Hey everyone!

I'm having a weird error when debugging the screen recording utility that Martins (mmozieko) wrote and I'm very curious as to what triggers it.

The error comes from the Address Sanitizer in MSVC and it looks like this:

==13576==ERROR: AddressSanitizer: attempting to call malloc_usable_size() for pointer which is not owned: 0x000005b00008

#0 0x1400534ed  (.\wcap.exe+0x1400534ed)
#1 0x7ffffcda16b6 (C:\WINDOWS\system32\IconCodecService.dll+0x1800016b6)
#2 0x7ff834783b37 (C:\WINDOWS\System32\USER32.dll+0x180013b37)
#3 0x7ff8347835f8  (C:\WINDOWS\System32\USER32.dll+0x1800135f8)
#4 0x7ff834782e59  (C:\WINDOWS\System32\USER32.dll+0x180012e59)
#5 0x7ff834785587  (C:\WINDOWS\System32\USER32.dll+0x180015587)
#6 0x7ff834781f4c  (C:\WINDOWS\System32\USER32.dll+0x180011f4c)
#7 0x1400017a3  (.\wcap.exe+0x1400017a3)
#8 0x1400746a1  (.\wcap.exe+0x1400746a1)
#9 0x1400745fd  (.\wcap.exe+0x1400745fd)
#10 0x1400744bd  (.\wcap.exe+0x1400744bd)
#11 0x14007471d  (.\wcap.exe+0x14007471d)
#12 0x7ff833d8269c  (C:\WINDOWS\System32\KERNEL32.DLL+0x18001269c)
#13 0x7ff83584a9f7  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18005a9f7)

Address 0x000005b00008 is a wild pointer.
SUMMARY: AddressSanitizer: bad-malloc_usable_size (.\wcap.exe+0x1400534ed) 

The error occurs after I step over line 1293 in the wcap.c file:

...
1293.	gIcon1 = LoadIconW(WindowClass.hInstance, MAKEINTRESOURCEW(1));
1294.	gIcon2 = LoadIconW(WindowClass.hInstance, MAKEINTRESOURCEW(2));
1295.	Assert(gIcon1 && gIcon2);
...

I want to mention that if I turn off the Address Sanitizer then everything works fine and I can keep on debugging.

So if any one of you guys, or even Martins himself, have encountered this odd behavior, I'd very appreciate it if you could shed some light on the matter <3

Right click addresses in call-stack and choose "Load Symbols" so VS downloads pdb file from Microsoft symbol server.

Then check the call stack if it has anything related to GlobalAlloc/GlobalFree/GlobalSize functions. If yes, then this is known issue with newer Windows versions & asan: https://learn.microsoft.com/en-us/cpp/sanitizers/asan-runtime?view=msvc-170#msvc-specific-addresssanitizer-runtime-options

The workaround it is to set ASAN_OPTIONS=windows_hook_legacy_allocators=false env variable when running .exe.

If I'm not mistaken - another alternative is to change /MTd to /MDd so .exe uses dynamic debug CRT runtime. Only issue is then you won't be able to run .exe outside of VS/vsvarsall environment, because it'll depend on debug asan dll files (which are not in PATH by default).

The workaround worked like a charm!
As always, thank you very much for the valuable info <3

Hey Martins. Sorry to bother you again!

I've stumbled upon another weird debugging error. Whenever I click on the "..." button to choose an output folder, the dialog window doesn't open and instead the program crashes :(

The program starts to display error messages after executing lines 440, 446 and throws an access violation exception after line 448:

439.	IFileDialog* Dialog;
440.	HR(CoCreateInstance(&CLSID_FileOpenDialog, NULL, CLSCTX_INPROC, &IID_IFileDialog, &Dialog));
441.
442.	WCHAR Text[MAX_PATH];
443.	GetDlgItemTextW(Window, ID_OUTPUT_FOLDER, Text, _countof(Text));
444.	
445.	IShellItem* Folder;
446.	if (SUCCEEDED(SHCreateItemFromParsingName(Text, NULL, &IID_IShellItem, &Folder)))
447.	{
448.		HR(IFileDialog_SetFolder(Dialog, Folder));
449.		IShellItem_Release(Folder);
450.	}

The error messages I get are:

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(1) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(2) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(3) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(4) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(5) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(6) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(7) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(8) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(9) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(10) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(11) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(12) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(13) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(14) tid(964) 80110474
The COM+ registry database detected a system error
Exception thrown at 00007FFBD175536C in windows.storage.dll: Microsoft C++ exception: [email protected]@@ at memory location 000000000014A1A0

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(15) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(16) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(17) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(18) tid(964) 80110474
The COM+ registry database detected a system error
Exception thrown at 00007FFBD175536C in windows.storage.dll: Microsoft C++ exception: [email protected]@@ at memory location 000000000014A1A0

I might add that the above happens in both debug and release mode when running the program from the debugger. However, when I compile the program in release mode and run it outside of the debugger then everything runs fine.

Are there any more environment variables or compiler switches that I need to add?