Hey everyone!

I'm having a weird error when debugging the screen recording utility that Martins (mmozieko) wrote and I'm very curious as to what triggers it.

The error comes from the Address Sanitizer in MSVC and it looks like this:

==13576==ERROR: AddressSanitizer: attempting to call malloc_usable_size() for pointer which is not owned: 0x000005b00008

#0 0x1400534ed  (.\wcap.exe+0x1400534ed)
#1 0x7ffffcda16b6 (C:\WINDOWS\system32\IconCodecService.dll+0x1800016b6)
#2 0x7ff834783b37 (C:\WINDOWS\System32\USER32.dll+0x180013b37)
#3 0x7ff8347835f8  (C:\WINDOWS\System32\USER32.dll+0x1800135f8)
#4 0x7ff834782e59  (C:\WINDOWS\System32\USER32.dll+0x180012e59)
#5 0x7ff834785587  (C:\WINDOWS\System32\USER32.dll+0x180015587)
#6 0x7ff834781f4c  (C:\WINDOWS\System32\USER32.dll+0x180011f4c)
#7 0x1400017a3  (.\wcap.exe+0x1400017a3)
#8 0x1400746a1  (.\wcap.exe+0x1400746a1)
#9 0x1400745fd  (.\wcap.exe+0x1400745fd)
#10 0x1400744bd  (.\wcap.exe+0x1400744bd)
#11 0x14007471d  (.\wcap.exe+0x14007471d)
#12 0x7ff833d8269c  (C:\WINDOWS\System32\KERNEL32.DLL+0x18001269c)
#13 0x7ff83584a9f7  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18005a9f7)

Address 0x000005b00008 is a wild pointer.
SUMMARY: AddressSanitizer: bad-malloc_usable_size (.\wcap.exe+0x1400534ed) 

The error occurs after I step over line 1293 in the wcap.c file:

...
1293.	gIcon1 = LoadIconW(WindowClass.hInstance, MAKEINTRESOURCEW(1));
1294.	gIcon2 = LoadIconW(WindowClass.hInstance, MAKEINTRESOURCEW(2));
1295.	Assert(gIcon1 && gIcon2);
...

I want to mention that if I turn off the Address Sanitizer then everything works fine and I can keep on debugging.

So if any one of you guys, or even Martins himself, have encountered this odd behavior, I'd very appreciate it if you could shed some light on the matter <3

Right click addresses in call-stack and choose "Load Symbols" so VS downloads pdb file from Microsoft symbol server.

Then check the call stack if it has anything related to GlobalAlloc/GlobalFree/GlobalSize functions. If yes, then this is known issue with newer Windows versions & asan: https://learn.microsoft.com/en-us/cpp/sanitizers/asan-runtime?view=msvc-170#msvc-specific-addresssanitizer-runtime-options

The workaround it is to set ASAN_OPTIONS=windows_hook_legacy_allocators=false env variable when running .exe.

If I'm not mistaken - another alternative is to change /MTd to /MDd so .exe uses dynamic debug CRT runtime. Only issue is then you won't be able to run .exe outside of VS/vsvarsall environment, because it'll depend on debug asan dll files (which are not in PATH by default).

The workaround worked like a charm!
As always, thank you very much for the valuable info <3

Hey Martins. Sorry to bother you again!

I've stumbled upon another weird debugging error. Whenever I click on the "..." button to choose an output folder, the dialog window doesn't open and instead the program crashes :(

The program starts to display error messages after executing lines 440, 446 and throws an access violation exception after line 448:

...
439.	IFileDialog* Dialog;
440.	HR(CoCreateInstance(&CLSID_FileOpenDialog, NULL, CLSCTX_INPROC, &IID_IFileDialog, &Dialog));
441.
442.	WCHAR Text[MAX_PATH];
443.	GetDlgItemTextW(Window, ID_OUTPUT_FOLDER, Text, _countof(Text));
444.	
445.	IShellItem* Folder;
446.	if (SUCCEEDED(SHCreateItemFromParsingName(Text, NULL, &IID_IShellItem, &Folder)))
447.	{
448.		HR(IFileDialog_SetFolder(Dialog, Folder));
449.		IShellItem_Release(Folder);
450.	}
...

The error messages I get are:

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(1) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(2) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(3) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(4) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(5) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(6) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(7) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(8) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(9) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(10) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(11) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(12) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(13) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(14) tid(964) 80110474
The COM+ registry database detected a system error
Exception thrown at 00007FFBD175536C in windows.storage.dll: Microsoft C++ exception: [email protected]@@ at memory location 000000000014A1A0

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(15) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(16) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(17) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(18) tid(964) 80110474
The COM+ registry database detected a system error
Exception thrown at 00007FFBD175536C in windows.storage.dll: Microsoft C++ exception: [email protected]@@ at memory location 000000000014A1A0

Exception thrown at 0x0000000140009345: 0xC0000005: Access violation reading location 0x0000000000000000.

I might add that the above happens in both debug and release mode when running the program from the debugger. However, when I compile the program in release mode and run it outside of the debugger then everything runs fine.

Are there any more environment variables or compiler switches that I need to add?

Is this unmodified source? Or are there any changes? From error messages it almost sounds like there is some memory corruption happening.

Or it could be that Windows again is throwing C++ exceptions outside of C functions, but not sure... I have not seen this happening for IFileDialog.

It's indeed an unmodified source that I took directly from your GitHub project page. However, I did notice that you wrote there that code always assumes Windows 10 and I was running the program on a Windows 11 machine.

I don't know if it's actually supposed to make a difference in that particular piece of code but, out of curiosity, I decided to give it a go on a Windows 10 machine for comparison. The results were a bit different but the end result was the same access violation error as before:

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(1) tid(39d0)
8007007E The specified module could not be found.

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(2) tid(39d0) 
8007007E The specified module could not be found.

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(3) tid(39d0) 
8007007E The specified module could not be found.

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(4) tid(39d0) 
8007007E The specified module could not be found.

Exception thrown at 0x0000000140016A8B: 0xC0000005: Access violation reading location 0x0000000000000000.

Are you able to open that file dialog window during debugging?

Yes, I can. I am doing this on Win11. It shows a lot of garbage messages, but nothing crashes.

Make sure you have updated VS to latest version. They often fix things in address sanitizer to work better. Check also windows updates.

Or you can just disable it completely. It's not like you need it much if you're running unmodified builds.

If you want to debug this more, then check the call stack on crash - make sure symbols are downloaded and maybe there will be something obvious from function names in windows dll's.

That was the culprit!

We're using different debuggers... You're using Visual Studio and I'm using RemedyBG. In Visual Studio, everything indeed works perfectly fine!

But why does RemedyBG cause so much trouble in this specific case? Do you know of any feasible way to overcome this hurdle without crossing over to the dark side (Visual Studio)?

Then it is probably bug in Remedy. Supporting address sanitizer requires extra functionality in debugger. If that's not handled correctly it may look like a crash. I don't know much about RemedyBG, I don't really use it.

If this reproduces you can report it to x13pixels - we have #remedydbg channel on Handmade Network discord. Or here: https://github.com/x13pixels/remedybg-issues/issues

After much debugging, I think I finally managed to get everything under control without ditching Remedy!

Turned out that the ASAN_OPTIONS=windows_hook_legacy_allocators=false environment variable fixed one problem and, for some reason, caused another one :D

Luckily though, you provided me with two options for fixing my original problem which were: setting that environment variable in the debugger or changing the compiler switch /MTd to /MDd in the build script.

As soon as I tried the second option - both problems disappeared.

Thank you for everything <3<3<3