Hey everyone!

I'm having a weird error when debugging the screen recording utility that Martins (mmozieko) wrote and I'm very curious as to what triggers it.

The error comes from the Address Sanitizer in MSVC and it looks like this:

==13576==ERROR: AddressSanitizer: attempting to call malloc_usable_size() for pointer which is not owned: 0x000005b00008

#0 0x1400534ed  (.\wcap.exe+0x1400534ed)
#1 0x7ffffcda16b6 (C:\WINDOWS\system32\IconCodecService.dll+0x1800016b6)
#2 0x7ff834783b37 (C:\WINDOWS\System32\USER32.dll+0x180013b37)
#3 0x7ff8347835f8  (C:\WINDOWS\System32\USER32.dll+0x1800135f8)
#4 0x7ff834782e59  (C:\WINDOWS\System32\USER32.dll+0x180012e59)
#5 0x7ff834785587  (C:\WINDOWS\System32\USER32.dll+0x180015587)
#6 0x7ff834781f4c  (C:\WINDOWS\System32\USER32.dll+0x180011f4c)
#7 0x1400017a3  (.\wcap.exe+0x1400017a3)
#8 0x1400746a1  (.\wcap.exe+0x1400746a1)
#9 0x1400745fd  (.\wcap.exe+0x1400745fd)
#10 0x1400744bd  (.\wcap.exe+0x1400744bd)
#11 0x14007471d  (.\wcap.exe+0x14007471d)
#12 0x7ff833d8269c  (C:\WINDOWS\System32\KERNEL32.DLL+0x18001269c)
#13 0x7ff83584a9f7  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18005a9f7)

Address 0x000005b00008 is a wild pointer.
SUMMARY: AddressSanitizer: bad-malloc_usable_size (.\wcap.exe+0x1400534ed) 

The error occurs after I step over line 1293 in the wcap.c file:

...
1293.	gIcon1 = LoadIconW(WindowClass.hInstance, MAKEINTRESOURCEW(1));
1294.	gIcon2 = LoadIconW(WindowClass.hInstance, MAKEINTRESOURCEW(2));
1295.	Assert(gIcon1 && gIcon2);
...

I want to mention that if I turn off the Address Sanitizer then everything works fine and I can keep on debugging.

So if any one of you guys, or even Martins himself, have encountered this odd behavior, I'd very appreciate it if you could shed some light on the matter <3

Right click addresses in call-stack and choose "Load Symbols" so VS downloads pdb file from Microsoft symbol server.

Then check the call stack if it has anything related to GlobalAlloc/GlobalFree/GlobalSize functions. If yes, then this is known issue with newer Windows versions & asan: https://learn.microsoft.com/en-us/cpp/sanitizers/asan-runtime?view=msvc-170#msvc-specific-addresssanitizer-runtime-options

The workaround it is to set ASAN_OPTIONS=windows_hook_legacy_allocators=false env variable when running .exe.

If I'm not mistaken - another alternative is to change /MTd to /MDd so .exe uses dynamic debug CRT runtime. Only issue is then you won't be able to run .exe outside of VS/vsvarsall environment, because it'll depend on debug asan dll files (which are not in PATH by default).

The workaround worked like a charm!
As always, thank you very much for the valuable info <3

Hey Martins. Sorry to bother you again!

I've stumbled upon another weird debugging error. Whenever I click on the "..." button to choose an output folder, the dialog window doesn't open and instead the program crashes :(

The program starts to display error messages after executing lines 440, 446 and throws an access violation exception after line 448:

...
439.	IFileDialog* Dialog;
440.	HR(CoCreateInstance(&CLSID_FileOpenDialog, NULL, CLSCTX_INPROC, &IID_IFileDialog, &Dialog));
441.
442.	WCHAR Text[MAX_PATH];
443.	GetDlgItemTextW(Window, ID_OUTPUT_FOLDER, Text, _countof(Text));
444.	
445.	IShellItem* Folder;
446.	if (SUCCEEDED(SHCreateItemFromParsingName(Text, NULL, &IID_IShellItem, &Folder)))
447.	{
448.		HR(IFileDialog_SetFolder(Dialog, Folder));
449.		IShellItem_Release(Folder);
450.	}
...

The error messages I get are:

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(1) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(2) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(3) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(4) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(5) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(6) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(7) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(8) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(9) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(10) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(11) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(12) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(13) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(14) tid(964) 80110474
The COM+ registry database detected a system error
Exception thrown at 00007FFBD175536C in windows.storage.dll: Microsoft C++ exception: .?AVResultException@wil@@ at memory location 000000000014A1A0

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(15) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(16) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(3174)\combase.dll!00007FFBD28CBB92: (caller: 00007FFBD28CBC92) ReturnHr(17) tid(964) 80110474
The COM+ registry database detected a system error

onecore\com\combase\catalog\catalog.cxx(1026)\combase.dll!00007FFBD28CBD16: (caller: 00007FFBD2866C80) ReturnHr(18) tid(964) 80110474
The COM+ registry database detected a system error
Exception thrown at 00007FFBD175536C in windows.storage.dll: Microsoft C++ exception: .?AVResultException@wil@@ at memory location 000000000014A1A0

Exception thrown at 0x0000000140009345: 0xC0000005: Access violation reading location 0x0000000000000000.

I might add that the above happens in both debug and release mode when running the program from the debugger. However, when I compile the program in release mode and run it outside of the debugger then everything runs fine.

Are there any more environment variables or compiler switches that I need to add?

Is this unmodified source? Or are there any changes? From error messages it almost sounds like there is some memory corruption happening.

Or it could be that Windows again is throwing C++ exceptions outside of C functions, but not sure... I have not seen this happening for IFileDialog.

It's indeed an unmodified source that I took directly from your GitHub project page. However, I did notice that you wrote there that code always assumes Windows 10 and I was running the program on a Windows 11 machine.

I don't know if it's actually supposed to make a difference in that particular piece of code but, out of curiosity, I decided to give it a go on a Windows 10 machine for comparison. The results were a bit different but the end result was the same access violation error as before:

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(1) tid(39d0)
8007007E The specified module could not be found.

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(2) tid(39d0) 
8007007E The specified module could not be found.

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(3) tid(39d0) 
8007007E The specified module could not be found.

onecore\com\combase\objact\dllcache.cxx(2127)\combase.dll!00007FFED780138E: (caller: 00007FFED77064CA) ReturnHr(4) tid(39d0) 
8007007E The specified module could not be found.

Exception thrown at 0x0000000140016A8B: 0xC0000005: Access violation reading location 0x0000000000000000.

Are you able to open that file dialog window during debugging?

Yes, I can. I am doing this on Win11. It shows a lot of garbage messages, but nothing crashes.

Make sure you have updated VS to latest version. They often fix things in address sanitizer to work better. Check also windows updates.

Or you can just disable it completely. It's not like you need it much if you're running unmodified builds.

If you want to debug this more, then check the call stack on crash - make sure symbols are downloaded and maybe there will be something obvious from function names in windows dll's.

That was the culprit!

We're using different debuggers... You're using Visual Studio and I'm using RemedyBG. In Visual Studio, everything indeed works perfectly fine!

But why does RemedyBG cause so much trouble in this specific case? Do you know of any feasible way to overcome this hurdle without crossing over to the dark side (Visual Studio)?

Then it is probably bug in Remedy. Supporting address sanitizer requires extra functionality in debugger. If that's not handled correctly it may look like a crash. I don't know much about RemedyBG, I don't really use it.

If this reproduces you can report it to x13pixels - we have #remedydbg channel on Handmade Network discord. Or here: https://github.com/x13pixels/remedybg-issues/issues

After much debugging, I think I finally managed to get everything under control without ditching Remedy!

Turned out that the ASAN_OPTIONS=windows_hook_legacy_allocators=false environment variable fixed one problem and, for some reason, caused another one :D

Luckily though, you provided me with two options for fixing my original problem which were: setting that environment variable in the debugger or changing the compiler switch /MTd to /MDd in the build script.

As soon as I tried the second option - both problems disappeared.

Thank you for everything <3<3<3

Hey Martins!

Quick question: How do I remove the yellow border that always appears when the recording starts?

I know there's a function that controls the appearance of the border: https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.graphicscapturesession.isborderrequired?view=winrt-22621

Which is very similar to the function that controls the visibility of the mouse cursor. However, when I add the two new functions to this struct:

struct IGraphicsCaptureSession2Vtbl {
	IInspectable_Parent(IGraphicsCaptureSession2);
	HRESULT(STDMETHODCALLTYPE* get_IsCursorCaptureEnabled)(IGraphicsCaptureSession2* this, char* value);
	HRESULT(STDMETHODCALLTYPE* put_IsCursorCaptureEnabled)(IGraphicsCaptureSession2* this, char value);
	HRESULT(STDMETHODCALLTYPE* get_IsBorderRequired)(IGraphicsCaptureSession2* this, char* value);
	HRESULT(STDMETHODCALLTYPE* put_IsBorderRequired)(IGraphicsCaptureSession2* this, char value);
};

Then, I call the necessary function here:

void Capture_Start(Capture* Capture, BOOL WithMouseCursor, BOOL WithBorder)
{
	IGraphicsCaptureSession* Session;
	HR(Capture->FramePool->vtbl->CreateCaptureSession(Capture->FramePool, Capture->Item, &Session));

	IGraphicsCaptureSession2* Session2;
	if (SUCCEEDED(Session->vtbl->QueryInterface(Session, &IID_IGraphicsCaptureSession2, (LPVOID*)&Session2)))
	{
		Session2->vtbl->put_IsCursorCaptureEnabled(Session2, (char)WithMouseCursor);
		Session2->vtbl->put_IsBorderRequired(Session2, (char)WithBorder);
		Session2->vtbl->Release(Session2);
	}

	HR(Capture->Item->vtbl->add_Closed(Capture->Item, &Capture->OnCloseHandler, &Capture->OnCloseToken));
	HR(Capture->FramePool->vtbl->add_FrameArrived(Capture->FramePool, &Capture->OnFrameHandler, &Capture->OnFrameToken));
	HR(Session->vtbl->StartCapture(Session));

	Capture->Session = Session;
}

And finally, I disable the feature by setting the parameter to false like so:

Capture_Start(&gCapture, gConfig.MouseCursor, FALSE);

The yellow border still appears. What am I doing wrong?

The Remarks section mentions what you need to do for this to work - you first need to request permissions to remove yellow border. I expect that is needed to be able to call put_IsBorderRequired with FALSE.

Also if I'm not mistaken border can be disabled only on Windows 11. Earlier versions won't support it, in case you're running Windows 10. That's why I did not add this feature to wcap. Nowadays Win11 is pretty common, so I'm probably fine adding this as option in config. We'll check it later.

That was quite an oversight on my part, thank you for putting me on the right track.

Now, is there a way to add the "graphicsCaptureWithoutBorder" capability to the already existing wcap.manifest file or do I need to create a new manifest file?

I'd expect that part of adding thing to manifest file does not matter for desktop application. That is UWP app thing. Code just need to call the API function.

I see...

So, basically, I need to create something like this:

struct IGraphicsCaptureAccessVtbl {
	IInspectableParent(IGraphicsCaptureAccess);
	HRESULT(STDMETHODCALLTYPE *RequestAccessAsync)(IGraphicsCaptureAccess *this, char value);
};

And then call RequestAccessAsync() with the second parameter being 0, which means access for borderless capture.

However, what do I pass for the first parameter? How do I initialize the IGraphicsCaptureAccess instance?