I have a lot of big binary files (usually 300-400 MB). The file format has changed slightly and I would like to convert old files to the new format.
What is the fastest method to do this? It needs to work for both Windows and Linux and preferably be in C++ or C. Should I use C FILE or C++ streams or something completely different?
Specific about the file format:
It consists of blocks of data. A block consists of
- a header,
- a body of data, and
- a footer.
Header: In the new format the header stays the same size, but one of the fields has changed slightly in definition, so the value does the same. The conversion depends on the footers of nearby blocks, in most cases prior blocks, but footer of the current block needs to be checked, so I need to read a bit ahead. If those footers aren't aren't valid in relation to the current block, I want to look ahead for valid footers.
Body: The size of the body is dependent on the header, but in virtually every case it has a fixed size close to 8 kB. It is unchanged in the new format, so it should just be copied.
Footer: The footer is variable size in the old format (at least four bytes) and fixed four bytes in the new. The definition is completely different. In the conversion the value may depend on nearby blocks.