Handmade Network»Forums
34 posts / 1 project
windows: mmap file, reorder pages, multithreaded
Edited by bumbread on

I was experimenting with writing a PDB parser that would use mmaping to read the pages of MSF streams as if they are contiguous.

The thing about memory mapping on windows is that there doesn't seem to be a way to reserve a range of virtual memory pages and then fill that range up with multiple mmaps. Just reserving pages with VirtualAlloc still makes MapViewOfFileEx fail.

The way magic ring buffer does it is it reserves some memory to get a pointer to first free range of pages, then frees that memory and uses that pointer as the base for mmaps. Which is dicey, because if another thread does virtual alloc it might get those pages before they get mmaped. That's why the magic ring buffer does several attempts of this process and if all of them fail it gives up.

I tried using placeholder pages, but it seems like the only way you can use placeholders, the size of mmaped region should exactly match the size of the placeholder, which means I won't be able to map uncontiguous file pages contiguously in memory.

I was wondering whether anyone knows how that can be done and if so, how can it be done

Mārtiņš Možeiko
2559 posts / 2 projects
windows: mmap file, reorder pages, multithreaded
Edited by Mārtiņš Možeiko on

I tried using placeholder pages, but it seems like the only way you can use placeholders, the size of mmaped region should exactly match the size of the placeholder

VirtualFree with MEM_PRESERVE_PLACEHOLDER flag can split whole placeholder into separate parts - I'm 100% sure that allows to map individual pages from file in any order over placeholder regions, regardless of page order in file. That's what "magic ring buffer" example code in VirtualAlloc2 is doing.

34 posts / 1 project
windows: mmap file, reorder pages, multithreaded

That worked, thanks!

Luke Kasz
1 posts
windows: mmap file, reorder pages, multithreaded

Are you familiar with RawPDB (https://github.com/MolecularMatters/raw_pdb) ?

It uses memory mapped files in the examples (https://github.com/MolecularMatters/raw_pdb/blob/main/src/Examples/ExampleMemoryMappedFile.cpp) and the library itself provides low level raw access (hence the name :-)) to the different MSFs streams.

34 posts / 1 project
windows: mmap file, reorder pages, multithreaded
Replying to lukekasz (#26575)

RawPDB is what I used for reference, but I didn't read the whole codebase so I probably didn't notice where they would be mapping file