I'm currently in the process of writing a serialization system for my game(C++), trouble is I don't want to keep myself locked down to a set struct/layout; dumping a struct to disk is simple enough but I don't want to have to rewrite my load/save code every time I change a type or add members, getting back any data that was "untouched" would be a nice bonus too. My current approach is to use some introspection data like what Casey put together back in episode 207(?), write that info disk along with struct data itself, and during loading compare what members are available during runtime vs. what was saved. I haven't actually rolled out with this so I have yet to see how this shakes out in practice, any alternatives? suggestions?
Note: goal is not small file sizes or load times, actual amount of data is very small, but it keeps changing and I expect it to change more as I keep working on my project.
Note: goal is not small file sizes or load times, actual amount of data is very small, but it keeps changing and I expect it to change more as I keep working on my project.
Hi and welcome.
In OpenTTD they appear to do something like that where they also keep a version number around in the savegame, where an older/newer save will be loaded with all understood fields upgraded to a new format if needed and missing/superfluous fields ignored. If some field is mandatory or can't be upgraded from a previous version, you could provide an error message.
Point being that it seems to work for them, but I can't imagine maintaining that code to be much fun.
You can certainly alleviate some of the problems with introspection, tagging struct members with version and other info (from version 25 to 33, mandatory/optional), with an auto-generated skeleton that loads the header, figures out which version it was and has struct definitions for each of the versions from the lowest version with a mandatory tag onward.
From there this auto-generated code would deserialise the data in the given version and you'd end up with structs for that version, and what you're left with is any munging left to do to upgrade a struct of one version to another version.
And even in that case you could add support to your meta-programming tool which marks up your structs with version info that additionally points to a macro or function which says how to munge an older version of that struct and upgrade it, so the load logic at least can be pretty much entirely generated.
On the other hand, that may just be overkill and a serialisation format like protocol buffers or one of its competitors might make more sense, at least until your data layout stabilises to a point where you can write a more optimised loader that doesn't care about older versions quite so much.
Good luck :)
In OpenTTD they appear to do something like that where they also keep a version number around in the savegame, where an older/newer save will be loaded with all understood fields upgraded to a new format if needed and missing/superfluous fields ignored. If some field is mandatory or can't be upgraded from a previous version, you could provide an error message.
Point being that it seems to work for them, but I can't imagine maintaining that code to be much fun.
You can certainly alleviate some of the problems with introspection, tagging struct members with version and other info (from version 25 to 33, mandatory/optional), with an auto-generated skeleton that loads the header, figures out which version it was and has struct definitions for each of the versions from the lowest version with a mandatory tag onward.
From there this auto-generated code would deserialise the data in the given version and you'd end up with structs for that version, and what you're left with is any munging left to do to upgrade a struct of one version to another version.
And even in that case you could add support to your meta-programming tool which marks up your structs with version info that additionally points to a macro or function which says how to munge an older version of that struct and upgrade it, so the load logic at least can be pretty much entirely generated.
On the other hand, that may just be overkill and a serialisation format like protocol buffers or one of its competitors might make more sense, at least until your data layout stabilises to a point where you can write a more optimised loader that doesn't care about older versions quite so much.
Good luck :)
Edited by Jeroen van Rijn
on
Have you considered the protocol buffer format? IT was designed for exactly the the issues you are trying to solve I think.
I have used it with C# code and it is quite nice. In fact you will sometimes end up with smaller payload than dumping a struct because it does some basic compression. I've never used any C libraries for it but there are some:
I think this is the most commonly used:
https://github.com/protobuf-c/protobuf-c
and then there is this lightweight one:
http://koti.kapsi.fi/~jpa/nanopb/
I have used it with C# code and it is quite nice. In fact you will sometimes end up with smaller payload than dumping a struct because it does some basic compression. I've never used any C libraries for it but there are some:
I think this is the most commonly used:
https://github.com/protobuf-c/protobuf-c
and then there is this lightweight one:
http://koti.kapsi.fi/~jpa/nanopb/
I had heard of protocol buffers but I was more interested in doing things myself, this is just a small personal project, so i'm just looking for general approaches really. And i'm obviously no expert, since I came in here asking about all this, but I had heard that protocol buffers were obnoxious to work with?
Quarteri'm unclear on what you mean... what debug API exactly?
You could try the thing I messed around with a while ago...
KelimionThanks! Been lurking a while actually, er'one seems pretty nice here :)
Hi and welcome.
Hey, sorry if I'm resurrecting this thread too late. I went with the approach you are thinking and it's working pretty well for me. It is pretty finicky to lock down, but hasn't given me any trouble since. I basically made it into an n^2 problem.
I guess the pseudocode would be
1. Get up to a type/name in the file
2. loop through the metadata looking for a match
if found, continue deserializing like normal
otherwise, eat forward in the file until you hit the next type/name.
Note that if you updated the metadata and are loading an outdated file, then that new field won't be filled at all. So you have to set the struct with default info before doing the deserialization.
I can share you my code if you'd like it for reference.
I guess the pseudocode would be
1. Get up to a type/name in the file
2. loop through the metadata looking for a match
if found, continue deserializing like normal
otherwise, eat forward in the file until you hit the next type/name.
Note that if you updated the metadata and are loading an outdated file, then that new field won't be filled at all. So you have to set the struct with default info before doing the deserialization.
I can share you my code if you'd like it for reference.