Spinning mirrors

I’ve spent most of this week troubleshooting an infuriatingly elusive problem with Windows Hooks. The use case is that the application I develop needs to monitor for certain third-party application window to appear and then install a windows hook to “do stuff”. What it does is unimportant here. The problem was, catching the target window and installing the hook worked fine, but then after some time the hook DLL was suddenly unloaded from all processes and the hook terminated. Here is a sample log from DebugView:

9.88870430	[1744] WinWrangler: CreateSharedMemory ok : C:\tools\TextPad 5\TextPad.exe [1a78]
9.88872910	[1744] WinWrangler: DLL_PROCESS_ATTACH : C:\tools\TextPad 5\TextPad.exe [1a78]
78.08577728	[1744] WinWrangler: DLL_PROCESS_DETACH : C:\tools\TextPad 5\TextPad.exe [1a78]
78.08594513	[1744] WinWrangler: CloseSharedMemory : C:\tools\TextPad 5\TextPad.exe [1a78]

First column is the event’s time in seconds. After a minute and a few seconds my hook was nuked from orbit by an unknown entity.

After scratching my head for a while I decided to pinpoint the code that actually called FreeLibrary on my DLL. It seemed that the OS itself was doing that. I started Api Monitor, set API filter to catch DLL loads/unloads and launched the test.

#	TID	Module	API	Return	Error	Duration
1	6200	USER32.dll	LoadLibraryExW ( "C:\code\AppEmbedder\Release\WinWrangler.dll", NULL, LOAD_WITH_ALTERED_SEARCH_PATH )	0x6f310000		0.0010937
2	6200	USER32.dll	FreeLibrary ( 0x6f310000 )	TRUE		0.0003345

After clicking on the FreeLibrary call we can see the call stack:

#	Module	Address	Offset	Location
1	USER32.dll	0x76922ed9	0x22ed9	GetKeyboardLayoutList + 0x70
2	ntdll.dll	0x77b4011a	0x1011a	KiUserCallbackDispatcher + 0x2e
3	USER32.dll	0x769260dc	0x260dc	PeekMessageA + 0x168
4	TextPad.exe	0x004bcbd3	0xbcbd3	
5	TextPad.exe	0x006279aa	0x2279aa	

GetKeyboardLayoutList on the top? What the hell? Anyway, TextPad (my guinea pig for the test) was calling PeekMessage as seen on line 4. This is not helpful. I’ve spent some time with a kernel debugger (windbg) looking around but it was mostly a waste of time, so I won’t go into that here.

I tested my sample on a few different machines, all were experiencing this behavior. After that I started commenting my code out to see what’s the minimal piece producing the issue. Needless to say I was pretty shocked when DLL was still being unloaded with the whole hook procedure empty and “loader” code reduced mostly to a SetWindowsHookEx call. At the same time other programs using hooks worked just fine. Something was really wrong.

Finally after one more code simplification it stopped the hook from breaking. What interesting was there? Well, I was creating a timer that periodically searched for the target window, and if it found one, the hook was being set up. And this turned out to be the culprit: if you call SetWindowsHookEx from a thread and that thread is later terminated, your hook will be silently killed, even if hook procedure resides in a different DLL and the whole process still lives. And CreateTimerQueueTimer runs the timer function in a thread from a thread pool.

I knew that Windows GUI and multi-threading don’t mix well usually, but wasn’t expecting this. Nowhere in MSDN documentation on hooks is this behavior explained, or any requirement for the calling thread stated. I’ve modified my code so that the timer now sends message to the main thread that does the actual hook setup. And it works. You live and learn, I guess.

Edit: now that I looked at SetWindowsHookEx MSDN page again, this behavior is given in the last user comment. Still, it should be in the damn documentation where I expect it.

~ by omeg on August 26, 2011.

C/C++, code, GUI, rant, winapi, windows internals