Crash in PeekMessage when hot reloading

Hello Everyone,

I have made a very simple hot reloading host program, that just loads the functions start() and frame() from a dll and runs a game loop:

image.png

I'm then running a thread which uses ReadDirectoryChangesW to listen for file writes and then trigger a reload which just reloads the dll and pulls the functions out again:

image.png

However, not all the time but occasionally after a reload I get a crash in PeekMessage and I can't see why this would happen.

image.png image.png

Calling start() and frame() is pretty much all the host program does, so all the window creation, events loop, opengl stuff is all done in the dll which is a bit different to how Handmade Hero does it. I'm assuming some resource in the windows api is tied to the dll and might get destroyed when the dll is unloaded. But it's strange that it only happens occasionally.

I'm not sure how to debug this, or even if this kind of thing is expected to work in the first place. I haven't found any info on this on the web so far. So if anyone had some advice or is more knowledgeable about this windows stuff I'd really appreciate it.

Thanks very much!

P.S. I'm on Windows 11 and using MinGW and gcc to compile.

On Windows the "window" (HWND) does not care about dll where it is created. Does not matter what dlls are loaded or unloaded.

Where is the code that does PeekMessage ? Inside dll that runs part of frame() function? In such case what is window pointer? where does it live?

What about other variables - seems you're using OpenGL? where are all the function pointers you load? are they in memory that's valid after dll reload? you're not keeping any pointers to functions inside dll that gets unloaded?

This is not a problem with crash - but you should not pass HWND handle to PeekMessage. Otherwise you'll be missing a lot of window messages. On Windows the WM messages are produced in thread message queue. Queue belongs to thread where it is processed, not to window. Passing window HWND handle to Peek/GetMessage makes it process only messages target to the window, and leave others unprocessed. Which is not good. There are bunch of messages that are "global" - and are not target to any specific window. Always process all of them, unless you have really really good reason to not to (which is very rare situation).


Edited by Mārtiņš Možeiko on

Thanks Martins,

Where is the code that does PeekMessage ? Inside dll that runs part of frame() function? In such case what is window pointer? where does it live?

Yeah PeekMessage gets called from my library function which is called in frame(): https://github.com/m4tthartley/core/blob/master/platform.h#L493

The window pointer is a structure in my library that contains the window handle etc. It's inside the state of the dll, which gets malloc()ed inside start() and then returned so the reloader can pass it to frame(). https://github.com/m4tthartley/reloader/blob/master/test.c#L11

What about other variables - seems you're using OpenGL? where are all the function pointers you load? are they in memory that's valid after dll reload? you're not keeping any pointers to functions inside dll that gets unloaded?

The program currently only uses the opengl immediate functions that don't need loading. The only opengl extension functions I load are the ones used during the initialisation. I don't have any other global variables in dll, everything is inside the state structure which is malloc()ed.

This is not a problem with crash - but you should not pass HWND handle to PeekMessage.

Ah ok, that makes sense, I'll take that out.

The code I've linked to is pretty rough and unfinished so please excuse the untidiness, :)

Thank you!


Replying to mmozeiko (#29629)

Are you not running out of space in this buffer? https://github.com/m4tthartley/reloader/blob/master/reloader.c#L161-L163 Because on every reload you're putting extra strings there: https://github.com/m4tthartley/core/blob/master/core.h#L544

Have you tried compiling with assertions enabled? It seems you have assert to check if you're not running out of space.
What about address sanitizer - clang and MSVC both support it on windows and should detect many different memory corruptions.


Edited by Mārtiņš Možeiko on

If I understand correctly: window creation and its window procedure callback are both in the DLL, but after reloading the function may have moved.

You could either move the callback outside the dll or update the address with SetWindowLongPtr (GWLP_WNDPROC).

Ah, yep. Good catch. That will do it. Cannot have pointers to functions in dll across reloads.

you're not keeping any pointers to functions inside dll that gets unloaded?

:)

CallWindowProc/DispatchMessageW on top of callstack in your screenshot is actually showing that - that it really wants to call window proc but fails.


Edited by Mārtiņš Možeiko on

Ah Damn! I completely forgot about the window callback function pointer!

Thank you very much both for your help!

you're not keeping any pointers to functions inside dll that gets unloaded?

:)

Sorry, it seems I may have lied 😂

Thanks again for your help!


Replying to mmozeiko (#29633)