handmade.network » Forums » Approaches to Serialization
Floresy
Tom
21 posts

None

#8280 Approaches to Serialization
1 year, 2 months ago

I'm currently in the process of writing a serialization system for my game(C++), trouble is I don't want to keep myself locked down to a set struct/layout; dumping a struct to disk is simple enough but I don't want to have to rewrite my load/save code every time I change a type or add members, getting back any data that was "untouched" would be a nice bonus too. My current approach is to use some introspection data like what Casey put together back in episode 207(?), write that info disk along with struct data itself, and during loading compare what members are available during runtime vs. what was saved. I haven't actually rolled out with this so I have yet to see how this shakes out in practice, any alternatives? suggestions?

Note: goal is not small file sizes or load times, actual amount of data is very small, but it keeps changing and I expect it to change more as I keep working on my project.

0xdedededede.....
Kelimion
Jeroen van Rijn
240 posts
3 projects

A big ball of Wibbly-Wobbly, Timey-Wimey _stuff_

#8282 Approaches to Serialization
1 year, 2 months ago Edited by Jeroen van Rijn on Aug. 28, 2016, 10:01 a.m.

Hi and welcome.

In OpenTTD they appear to do something like that where they also keep a version number around in the savegame, where an older/newer save will be loaded with all understood fields upgraded to a new format if needed and missing/superfluous fields ignored. If some field is mandatory or can't be upgraded from a previous version, you could provide an error message.

Point being that it seems to work for them, but I can't imagine maintaining that code to be much fun.

You can certainly alleviate some of the problems with introspection, tagging struct members with version and other info (from version 25 to 33, mandatory/optional), with an auto-generated skeleton that loads the header, figures out which version it was and has struct definitions for each of the versions from the lowest version with a mandatory tag onward.

From there this auto-generated code would deserialise the data in the given version and you'd end up with structs for that version, and what you're left with is any munging left to do to upgrade a struct of one version to another version.

And even in that case you could add support to your meta-programming tool which marks up your structs with version info that additionally points to a macro or function which says how to munge an older version of that struct and upgrade it, so the load logic at least can be pretty much entirely generated.

On the other hand, that may just be overkill and a serialisation format like protocol buffers or one of its competitors might make more sense, at least until your data layout stabilises to a point where you can write a more optimised loader that doesn't care about older versions quite so much.

Good luck :)
MandleBro
Jack Mott
98 posts
1 project

Web Developer by day, game hobbyist by night. Fond of C and F#

#8283 Approaches to Serialization
1 year, 2 months ago

Have you considered the protocol buffer format? IT was designed for exactly the the issues you are trying to solve I think.

I have used it with C# code and it is quite nice. In fact you will sometimes end up with smaller payload than dumping a struct because it does some basic compression. I've never used any C libraries for it but there are some:

I think this is the most commonly used:
https://github.com/protobuf-c/protobuf-c

and then there is this lightweight one:
http://koti.kapsi.fi/~jpa/nanopb/

Quarter
stephen goglin
20 posts

aka Quartertron

#8289 Approaches to Serialization
1 year, 2 months ago

You could try the thing I messed around with a while ago...I loaded the .pdb for the currently executing program and used the debug api to parse it. Then used that to serialize all kinds of random data types within the program.

None
Floresy
Tom
21 posts

None

#8292 Approaches to Serialization
1 year, 2 months ago

I had heard of protocol buffers but I was more interested in doing things myself, this is just a small personal project, so i'm just looking for general approaches really. And i'm obviously no expert, since I came in here asking about all this, but I had heard that protocol buffers were obnoxious to work with?

Quarter
You could try the thing I messed around with a while ago...
i'm unclear on what you mean... what debug API exactly?

Kelimion
Hi and welcome.
Thanks! Been lurking a while actually, er'one seems pretty nice here :)

0xdedededede.....
Quarter
stephen goglin
20 posts

aka Quartertron

#8304 Approaches to Serialization
1 year, 2 months ago

Debug Interface Access SDK

or you can use the DbgHelp library, which is what cdbg is built on top of.

None
Yam
James Fulop
21 posts
1 project

None

#8981 Approaches to Serialization
1 year, 1 month ago

Hey, sorry if I'm resurrecting this thread too late. I went with the approach you are thinking and it's working pretty well for me. It is pretty finicky to lock down, but hasn't given me any trouble since. I basically made it into an n^2 problem.

I guess the pseudocode would be
1. Get up to a type/name in the file
2. loop through the metadata looking for a match
if found, continue deserializing like normal
otherwise, eat forward in the file until you hit the next type/name.

Note that if you updated the metadata and are loading an outdated file, then that new field won't be filled at all. So you have to set the struct with default info before doing the deserialization.

I can share you my code if you'd like it for reference.

None
Floresy
Tom
21 posts

None

#8998 Approaches to Serialization
1 year, 1 month ago

Sure, i'd love to see what you made if you're up for posting it. I actually never got around to really fleshing out the system though, it was kinda over-engineering for my given problem. But who knows who else will need the assistance in the future!

0xdedededede.....