Bobs scripting woes

[jpab]

Edit: Looks like robn already said all this while I was still writing my reply. I hope I haven't contradicted him.

First, the structure bases on the assumption that the indices of static bodies in a system is always the same. Is that assumption correct?

Yes, that's correct. Assuming by "index" you mean the value of the index attribute of the SystemBody type, which corresponds with the bodyIndex property of the SystemPath type.

Since the project I'm working on is a potential savefile bloater, it is important to me to save the stuff as efficiently as possible. I have the most efficient structure here, but I'm not sure if it's really going to work, and I have even less idea of what LUA will do with it exactly.

My advice would be: Worry about maintainability and forward compatibility, not savefile size. There will be lots of core changes that will affect the save file format, in particular we plan to switch to a JSON based format for flexibility (*), and that's not exactly a space-efficient format. If savefile size becomes a problem, the first thing we'll do (because it's basically trivial and doesn't require changes elsewhere in the code) is to pump the data through a general purpose compression library. After that, my guess is that speed of writing (generating) or loading (parsing) the save files is likely to be a more pressing problem than savefile size, and that will require more thought and wider changes.

Also, it seems unrealistic that any system has more than 256 static bodies, so a BYTE should be enough to store the body index.

No more than 256 bodies per system is probably valid right now, but I have no particular reason to believe it will be valid in the future. The internal SystemPath structure, which provides a unique identifier for any static body in the game, uses a 32-bit integer for the body index. That's probably overkill, but right now putting tight limits on things would just cause headaches for no useful gain. For example, what happens if we want to generate a large number of dwarf planets or major asteroids, or place large numbers of artificial satellites (comms. satellites, navigation markers, sensor pods, zero-g manufacturing plants, goods depots, etc etc) in orbit around heavily populated planets?

Having said all that, I don't believe it's my place to dictate how modders work, so I shall try to answer your other questions. As you correctly surmised, Lua doesn't pack things into the smallest possible data type. In fact, Lua only deals with a single numeric type (64-bit double precision floating point), and when we serialise Lua data into the save file, it's written as a decimal number, so it doesn't even remotely meet your criteria.

Your best bet if you want to pack data tightly is to do your own serialisation. Lua strings are just blobs (ie, byte arrays), and you can put whatever byte values you want into them. For efficiency you should be careful about how you build this data up: It's usually best to build a table of substrings and then use table.concat to join them into one. If you do it by having a single string variable that you keep appending small chunks to, then performance will suffer (there's an explanation of this in Programming in Lua). To make a string from an arbitrary series of byte values, use string.char. To extract the byte values from a string, use string.byte.

But: If you intend on submitting this work to be merged into the main game distribution (and I hope you will, because I would like us to have more missions and more content in general), then I would ask you to please make life easy for us and keep custom packing/serialisation code isolated, so that it's easy to remove if necessary.

(*) It's very embarrassing that updates to the game often make existing save files useless. We would like to stop that, but we need to be able to keep changing the content of the saves as features are added and changed. The point of switching to a JSON based format is to help make this possible.

[jpab]

One caveat here: Pioneer's Lua serialiser (used for savefiles) will treat a 0 byte in a string as end-of-string

This is now fixed (#2463), and should work correctly in the next available build.

John B

[TheBob]

Strings are stored as sequences of bytes, so you could do structure packing inside strings.

Of course! How didn't I think of that before?

One caveat here: Pioneer's Lua serialiser (used for savefiles) will treat a 0 byte in a string as end-of-string, so if you start packing structures into strings you might find you get corrupted savefiles.

Well, I won't use zeroes, then. Thanks for the heads-up (255 states is still enough for my purposes).

The other thing to remember is that most of the time you're trading space for time. You're going to be constantly packing and unpacking your data, which takes time.

That's pretty much a given. I was intending for unpacking the data on loading, really. I am not so much concerned about memory usage as about filesize. Loading the data from disk might take just as much time as unpacking it in memory... Depending on your disk, of course. But it's gonna be a while until everybody has SSDs.

Good luck :)

Thanks!

For example, what happens if we want to generate a large number of dwarf planets or major asteroids, or place large numbers of artificial satellites (comms. satellites, navigation markers, sensor pods, zero-g manufacturing plants, goods depots, etc etc) in orbit around heavily populated planets?

True, true... so, no compressing of the body index. The rest I control myself, so I shouldn't have troubles with it.

In fact, Lua only deals with a single numeric type (64-bit double precision floating point)

That's... utterly, terrifingly horrible. Yeah, it's a C++-programer speaking. Call me old-fashioned. But oh, the grief for the cycles that are lost!

then I would ask you to please make life easy for us and keep custom packing/serialisation code isolated, so that it's easy to remove if necessary

I will, don't worry. I hate functions of which I can't see the end and the beginning at the same time. Can't always get around them, but I don't like them...

Anyways, thanks a lot for the detailed answers! I'll probably take the middle road, saving the system path to the system as, well, a systempath, the indices of any bodies therein where discoveries were made as standard LUA-variables, and then my own stuff as strings. At least I'll have no redundant information stored...

[TheBob]

So, I learned a bit, and did some rewriting. Now I actually have something resembling classes in there. What I wonder, though, is, how can I shove them off to another file? It would greatly increase oversight, further compartementalise the code for future editing/fixing, and I could use them from other scripts as well...

Also, I have some uncertainties about "nested functions" (for lack of a better term) as for example timers. I understand that it's just a function that gets called after a certain time, but it can use variables that have been declared in the code block before, which does not get called periodically. For example I engage a timer onFrameChange, there's a variable defining the body the frame changed to, and the function declared in the timer can use it although it is declared local outside the scope of the timer function. So evidently that thing stays in memory... until when exactly? It should technically get destroyed when onFrameChange terminates, but obviously that doesn't happen or it would be invalid when the timer function is called the next time.

And rather more important, if I have a CallEvery running and function that initiated it gets called again, do I have two timers running (i.e. does the timer function get instanciated, or does it just run on with the new values)?

[TheBob]

Right, got my code ripped appart in several files and it's still working... I'm starting to actually get a feel for the whole structure. It would seem like a lot more is possible without sticking hands into the core than I thought.

I'm still majorly uncertain about how those timers behave, though...

[jpab]

So, I learned a bit, and did some rewriting. Now I actually have something resembling classes in there. What I wonder, though, is, how can I shove them off to another file? It would greatly increase oversight, further compartementalise the code for future editing/fixing, and I could use them from other scripts as well...

The import function can be used to link together multiple scripts. For example, data/ui/InfoView.lua imports the UI template for each tab from a separate script. Another example is Character.lua, which provides a class-like-object. Note that Character.lua defines the "class" as a table, and in the last line of the file it returns that table. The value returned from a script is the value that will be returned from import when you import that script. That value is cached the first time the script is imported (ie, everyone gets the same copy of the Character table). I'm sorry import is not documented yet -- we need to fix that.

Also, I have some uncertainties about "nested functions" (for lack of a better term) as for example timers. I understand that it's just a function that gets called after a certain time, but it can use variables that have been declared in the code block before, which does not get called periodically. For example I engage a timer onFrameChange, there's a variable defining the body the frame changed to, and the function declared in the timer can use it although it is declared local outside the scope of the timer function. So evidently that thing stays in memory... until when exactly? It should technically get destroyed when onFrameChange terminates, but obviously that doesn't happen or it would be invalid when the timer function is called the next time.

The variable is kept around until it's no longer accessible. In general, the actual game object it refers to might be destroyed if something in game causes that to happen (e.g., the body smashes into a planet and dies). There's an example in the main documentation for Timer.

Also, the term you're looking for instead of "nested functions" is closure (this may be useful to know if you need to search for more information).

And rather more important, if I have a CallEvery running and function that initiated it gets called again, do I have two timers running (i.e. does the timer function get instanciated, or does it just run on with the new values)?

Yes, if you call CallEvery again, then both callbacks will continue (with their own intervals). If your callback function returns a true value (ie, anything other than nil or false), then that timer will stop repeating. Perhaps we need CallEvery to return some separate timer object that you can use to cancel the callback externally. Although with closures it's possible to achieve this in Lua. For example:

Code: Select all

local Timer = import("Timer")
local Event = import("Event")

local CancellableCallEvery = function (interval, callback)
  local cancelled = false
  local function cancel_callback() cancelled = true; end
  local function callback_wrapper()
    if cancelled then return true; end
    return callback()
  end
  Timer:CallEvery(interval, callback_wrapper)
  return cancel_callback
end

local cancel

Event.Register("onShipUndocked", function (ship, station)
	if ship:IsPlayer() then
		cancel = CancellableCallEvery(3, function ()
			print('WARNING: BIG BLUE SPACE MEANIES DETECTED! DOCK IMMEDIATELY!')
		end)
	end
end)

Event.Register("onShipDocked", function (ship, station)
  if ship:IsPlayer() then cancel(); end
end)

[TheBob]

The variable is kept around until it's no longer accessible.

so, if I do for example something like this:

Code: Select all

local StartTimer = function()
      local countMe = 0
      Timer:CallEvery(10, function ()
          
                countMe = countMe + 1
                if countMe > 1000 then
                   return true
                end
      end)
end

countMe just stays in memory until the timer returns true? And even if I call StartTimer again, that will not influence countMe in the timer instance that got called earlier? Well, that's a relief... Makes it a lot easier to cancel them, really. I was afraid that if, in the above example, StartTimer got called again it would reset countMe even for the timer already running. So I can basically think of variables used in timers as static, did I understand that right?

That value is cached the first time the script is imported (ie, everyone gets the same copy of the Character table).

That's good to know. I was quite wondering whether the import order might affect things.

Thanks for the reply. I'm far enough now so that I can actually discover stuff... Now there's a dive into the GUI pending.

[jpab]

countMe just stays in memory until the timer returns true? And even if I call StartTimer again, that will not influence countMe in the timer instance that got called earlier?

Yes, that's correct.

John B

[TheBob]

Yay, I got saving, loading, UI... and exploding waypoints! Micheal Bay would be jealous :lol:

So I have pretty much everything together needed to find stuff... now I'll need to implement the exploitation too, I guess.

[TheBob]

So, LUA and deallocation... I don't seem to get the hang of this.

I have a class that stores the players discoveries. It has its own file which is in the libs folder, and it doesn't get instantiated, all references anywhere in the game point to the same object. There's two references to it, one in the module enabling the player to make discoveries, and one in the UI.

In the lib, I use the event onGameEnd to set Discoveries = nil. This doesn't work, because obviously it only sets this reference to the object to nil. Since there's two more references hanging around, the object doesn't get deallocated and is still accessed after loading a game. So I tried Discoveries = {}, same thing.

Of course I can just destroy all references, but should things get more complicated in the future this could become rather tedious (especially without a decent debugger and my tendency for forgetting things), so I wonder: Is there any function to force LUA to destroy the object even if it's still referenced at other places in the code? A google search hasn't really turned up anything...

Also, @FluffyFreak: I already wrote this a few days ago, but then there was server trouble and it would seem the backup ate my post.
You remember that faction stuff we talked about? Now that I know the Pioneer LUA-structure somewhat better, I see that it doesn't actually need core support for most things (some calls for starport behavior will be neccessary, but that isn't directly related). When I get to it, I can just shove the faction creation off into a lib and retrieve the faction list from there, without having to get it from the Pioneer core. Sorry for the work I've caused you.