Really awesome to see you're still working on this!
+1! This is a really useful (and difficult!) thing to work on.
We need a savegame format that is a tree, the ability to navigate nodes, return the number of child nodes, iterate through the nodes etc. There are two obvious choices: XML and JSON. I'm favoring JSON, because its easier to read - what do you think? I believe either can be platform independent.
Yes, definitely JSON since we already use it in other places, with some data that is currently stored in the form of Lua scripts being converted to JSON (we don't currently use XML anywhere).
The new savegame system will need some basic functions written for node navigation, and reading and writing the various data types (relatively painless - barring the "auto" type, but will cross that bridge when come to it).
Most of the low level (non-domain-specific) node navigation code and handling the raw value types (numbers, strings, arrays etc) should already be provided by the JSON parsing library that we use,
jsoncpp. We have a copy of the code for that library in the contrib/json/ directory of the repository... I don't think we've modified it at all (and I would recommend against modifying it without a really strong reason), though it may not be the latest version of the library. You'll probably need to write helper functions to serialise 3D vectors, colour values and perhaps some other basic value types, since JSON itself only supports string/number/boolean/null for basic values (plus array and "object" for structures).
I'm not sure what you mean by the "auto" type... are you referring to the
C++11 'auto' keyword(*)? That 'auto' is not itself a type, it tells the compiler to deduce the type automatically.
(* auto was a keyword in C++ prior to C++11, but it had a different meaning and was almost never used)
I've examined the code enough to implement a node-based structure of the existing save data - is this what you have in mind?
That's probably the right place to start. Long term, the major reason for changing the save format is to make it easier to improve the game without breaking save files, and doing that effectively may need some changes to the tree structure of the format, not just a switch to JSON.
The current format relies very strongly on having an exact match between the version of the save code that wrote the save file and the version of the code that is reading the file. For example, saving an extra data field for a particular object type will shift the offsets of all subsequent saved fields and objects, which breaks compatibility between different versions of the loading code. Switching to JSON will mean that changes of that kind can be made in a backward-compatible way (older saves will be missing the field but can use a default value). However, that level of compatibility is quite low level -- robust behavioural compatibility is harder to achieve and will probably have to be handled on a case-by-case basis as we continue to develop the game.
The structure of a samegame file will be essentially the same as the existing one, but written in a hierarchical tree structure (nodes containing data, nested inside other nodes etc). Each node type will have a name.
Yes, and probably. JSON doesn't have a built-in structure for named object types, but as you are designing the save format you can specify some convention to do this, like including a string 'type' field within each object. However, I think I might be misunderstanding what you mean by "Each node type will have a name".
Saving the number of objects (as int32) before saving the objects will no longer be required, because we can call the "get number of nodes function", and/or just iterate through the objects. The tree will be quite deep.
Yes, and yes.
Future savegame formats will have new node types. When loading an older file, the new version code should create a default/nominal value (e.g. when looking for a node "colour of crew hats" that did not exist in the old savegame file version).
Yes.
So it would seem that the remaining tasks before coding, are to learn JSON and pick a JSON parser.
Yes, you will need to learn JSON (it's
very simple). You don't need to pick a JSON parser -- as I mentioned above, we already use jsoncpp.
However, I do feel a bit out of my depth, what with work commitments and my other software project (a computer algebra system). I can't say when I'll get this task completed I'm afraid. I'll probably need a bit of help with some unfamilliar aspects of software development.
Don't worry -- we all have other commitments too, so we totally understand that it can take a long time to make progress on something like this. I'm happy to try to help answer any questions you have, although I may be very slow to reply sometimes due to my own time constraints. I'm sure others will be happy to help too.
Following on from the [existing] savegame process pseudocode (see 3 posts above) the savegame process for all the "body" types in Pioneer is shown below:
... full save tree description elided ...
I haven't read through the entire tree in detail yet, but a few things jump out at me:
- You have various 'Type of X' fields listed as [Int32]. These should use a string (the name of the type) rather than a number. We have some code already to deal with converting between enum values and strings, which you should be able to use for this purpose (and any other time that you want to save an enum value). Please ask if you have difficulty finding it or working out how to use it.
- Orientation matrices. I would probably store these as quaternions (4 numbers) instead of a matrix representation (9 numbers), purely because the quaternion is a more compact representation. This is a minor detail though.
- Why is the 'Properties' field of a 'Body' a string? It seems like that should be a JSON object (ie, key/value pairs).
- You have PhysRadius and ClipRadius stored for the Body type. These are currently saved, but I'm not sure that they should be saved. I think these values always come from somewhere else anyway (e.g., for a ModelBody, these values should come from the model data). There are probably other fields in the data that are like that -- values which actually come from some other static data (or from procedurally generated static data, like the PhysRadius and ClipRadius of procedurally generated planets). The values are copied into fields in the Body class so that they can be accessed quickly and uniformly by other code. Saving them makes it easy to recreate the object, but it's bad for robustness in the face of other changes: for example, the model data itself is not stored in the save game (and shouldn't be) -- if a ship design is changed then its ClipRadius or PhysRadius might change, but with the current system those changes might not be applied to ships that are loaded from save games, since the radius values are stored directly in the save. That could lead to weird behaviour, like ships vanishing or collisions being ignored because the clip/phys radius value for a particular object doesn't match the actual model that's being displayed. This is an example of the kind of backward compatibility problem that won't be solved simply by converting to JSON, but which requires more careful thought to avoid. Unfortunately, I suspect that there are a lot of potential pitfalls like this that will make old saves break in subtle ways.
Finally, as impaktor said, it would be a good idea to look at robn's json-savefile branch and perhaps use it as a starting point for this work.