I’ve spent most of this week troubleshooting an infuriatingly elusive problem with Windows Hooks. The use case is that the application I develop needs to monitor for certain third-party application window to appear and then install a windows hook to “do stuff”. What it does is unimportant here. The problem was, catching the target window and installing the hook worked fine, but then after some time the hook DLL was suddenly unloaded from all processes and the hook terminated. Here is a sample log from DebugView:
9.88870430 [1744] WinWrangler: CreateSharedMemory ok : C:\tools\TextPad 5\TextPad.exe [1a78]
9.88872910 [1744] WinWrangler: DLL_PROCESS_ATTACH : C:\tools\TextPad 5\TextPad.exe [1a78]
78.08577728 [1744] WinWrangler: DLL_PROCESS_DETACH : C:\tools\TextPad 5\TextPad.exe [1a78]
78.08594513 [1744] WinWrangler: CloseSharedMemory : C:\tools\TextPad 5\TextPad.exe [1a78]
First column is the event’s time in seconds. After a minute and a few seconds my hook was nuked from orbit by an unknown entity.
After scratching my head for a while I decided to pinpoint the code that actually called FreeLibrary on my DLL. It seemed that the OS itself was doing that. I started Api Monitor, set API filter to catch DLL loads/unloads and launched the test.
# TID Module API Return Error Duration
1 6200 USER32.dll LoadLibraryExW ( "C:\code\AppEmbedder\Release\WinWrangler.dll", NULL, LOAD_WITH_ALTERED_SEARCH_PATH ) 0x6f310000 0.0010937
2 6200 USER32.dll FreeLibrary ( 0x6f310000 ) TRUE 0.0003345
After clicking on the FreeLibrary call we can see the call stack:
# Module Address Offset Location
1 USER32.dll 0x76922ed9 0x22ed9 GetKeyboardLayoutList + 0x70
2 ntdll.dll 0x77b4011a 0x1011a KiUserCallbackDispatcher + 0x2e
3 USER32.dll 0x769260dc 0x260dc PeekMessageA + 0x168
4 TextPad.exe 0x004bcbd3 0xbcbd3
5 TextPad.exe 0x006279aa 0x2279aa
GetKeyboardLayoutList on the top? What the hell? Anyway, TextPad (my guinea pig for the test) was calling PeekMessage as seen on line 4. This is not helpful. I’ve spent some time with a kernel debugger (windbg) looking around but it was mostly a waste of time, so I won’t go into that here.
I tested my sample on a few different machines, all were experiencing this behavior. After that I started commenting my code out to see what’s the minimal piece producing the issue. Needless to say I was pretty shocked when DLL was still being unloaded with the whole hook procedure empty and “loader” code reduced mostly to a SetWindowsHookEx call. At the same time other programs using hooks worked just fine. Something was really wrong.
Finally after one more code simplification it stopped the hook from breaking. What interesting was there? Well, I was creating a timer that periodically searched for the target window, and if it found one, the hook was being set up. And this turned out to be the culprit: if you call SetWindowsHookEx from a thread and that thread is later terminated, your hook will be silently killed, even if hook procedure resides in a different DLL and the whole process still lives. And CreateTimerQueueTimer runs the timer function in a thread from a thread pool.
I knew that Windows GUI and multi-threading don’t mix well usually, but wasn’t expecting this. Nowhere in MSDN documentation on hooks is this behavior explained, or any requirement for the calling thread stated. I’ve modified my code so that the timer now sends message to the main thread that does the actual hook setup. And it works. You live and learn, I guess.
Edit: now that I looked at SetWindowsHookEx MSDN page again, this behavior is given in the last user comment. Still, it should be in the damn documentation where I expect it.
~ ~ ~
Posted by omeg in C/C++, code, GUI, rant, winapi, windows internals
