Pioneer dev forum

Hi,

So after the post about Pioneer crashing despite not being out of memory I've decided to take a look at some things to do with the DDS loading etc.

Firstly having the "Compress Textures" option enabled within the options screen is a GOOD THING, however it comes with two variable negatives:

• It takes longer to load because it must compress the textures as it loads,
• As with all block based S3TC compression schemes it reduces quality and introduces artifacts.

That second point is worst on Linux, I actually disable it myself on my Linux machine because the compressor it uses is an unmitigated piece of shit that produces results so bad that I want to vomit... it's that bad.

However there is a way around the first problem, and of avoiding the worst of the issues for the second:
Generate the DDS files, which are compressed with dxt1 / dxt5, before loading.

This means you can use slow and higher-quality (not perfect but as good as you can get) compressors that are much better than those available on some platforms and drivers. It means that the whole game takes up more space on your Hard disk (~40MB), but LESS space on your GPU than the uncompressed versions.

I have submitted a PR for what I've currently done.
In that PR I have converted all of the PNG files in the models directory to DDS and then deleted the PNGs.
I had to fix a code issue to let patterns be DDS files and everything is compressed as DXT5 (I think) which means there might be room for some of the files being DXT1 and thus smaller.
All of those PNGs are not lost, they should be in pioneer-assets so I've submitted a PR for that too.

The optimisation used probably isn't the best too, I used the version of ImageMagick that I had installed and might go back and try a newer version with a slower and higher quality compressor, or use another tool entirely. This however was a bulk job that I didn't want to do by hand!

So what do people think to doing this?
What are you major objections?
If you have objections what can be done to fix them?

Cheers,

Andy

Wow. I just tried disabling "compress textures" and pioneer starts in seconds rather than several minutes (used to take 3-4 minutes), so now I can actually run it without having my laptop melt. I updated the wiki: http://pioneerwiki.com/wiki/Settings_Menu#Video_options some. Please feel free to elaborate/fill in/point out anything missing or errors on my part.

As for the suggested changes, I think most things that will improve performance is good, and an increase in usage of disk space seems like a reasonable trade off, but my knowledge in the field of graphics is non-existent.

I'm ok with this!

Do we need instructions on the wiki for how to save in these formats in common packages? Or do modelling/graphics people just know this?

That information would be good. At least I don't know how to save/convert to dds.

For this batch conversion I used ImageMagick and since I was doing it from the Git shell (in Windows 7) I used a command line like: To find ALL of the png files below the folder I was in "data/models/" and convert them to a dxt5 compressed DDS file.
In the end I did things a little bit differently but ImageMagick is mostly a command line tool and good for doing these bulk operations.

Doing files individually however you can use something like AMDs "The Compressonator" for which there is at least one guide.
It's got a GUI and you can see the results of your choices, generate mipmaps and do other wonderous things.

You can export DDS textures directly from GIMP with a plugin.

Yes, I already found that gimp plugin. :)
Which compression methods you recommend? Also should we generate MIP-maps?

Formats to use:

In Pioneer we support DXT1 and DXT5 formats, many places will give a long and detailed explanation for what this means but I will be brief:

• DXT1: 24-bit images, RGB no Alpha channel.
• DXT1a: 32-bit images, RGBA - results in 1 bit alpha DO NOT BOTHER.
• DXT5: 32-bit images, RGBA - best alpha quality, use this for patterns etc.

So: Got an alpha channel (like Patterns have)? Use DXT5 otherwise use DXT1.
Do not bother using the DXT1a or any other format, partly because Pioneer doesn't support anything but DXT1/DXT5, but also because you get worse quality with only a tiny saving in memory usage :(

Beware of forum threads that discuss the "performance" benefits of DXT1 vs DXT5 - if you're using any DXTn format on anything more powerful than a mobile phone then you've already gotten all the performance benefit you will be seeing.

MipMaps:
These are half-size images of your texture, they form what are called a "mip-chain" going from the full resolution image and halving in size until they are either 4x4 for DDS (DXT1/5) or 1x1 for uncompressed textures.
They are used to improve the quality of the texture on the 3D models when it is viewed at high angles or as it gets smaller on screen.
AMDs The Compressonator can generate the mip-chain for you if you ask it too, when I did the batch conversion it seems as though ImageMagick did the correct thing and generate mipmaps for all of the textures too, I don't know what the GIMP plugin will do.

Neat Trick: You can manually edit the sub-mips (the lower resolution copies) in a mip-chain to useful effect - make the full resolution image a distinct colour, then make all of the lower mip levels a different colour.
Then when you look at the model ingame you can see if the highest resolution is ever actually used.
If you make each level of the mips different colour you can even find out which mip is most commonly seen.

Reasons to use DDS (& DXT1/5):
PNG files look like they are smaller, they are only smaller on your harddrive. Once loaded onto the GPU they are between 4 and 6 times bigger than the same texture compressed using DXT1 or DXT5.
This has two drawbacks. The first is simply that they take up more room, OpenGL only has a finite amount of memory on the GPU to work with and once it's gone we get bugs saying that the game is crashing. The main problem here is OpenGL doesn't let us know how much memory we can use!!! This is for political, stupid and plain fucking wrong reasons that were decided by total morons who have never had to support software in the real world. AMD and nVidia do provide ways of getting this information, but since they're good graphics card makers they're not the ones that cause us grief, Intel who do cause us all of the grief don't provide us this information.
The second problem is not one we have to worry about very much but in theory if we started drawing a LOT more stuff, like trees/grass/rocks/animals/etc on planets, then we might start to use textures a lot more and that would eat up memory bandwidth between the GPU and ram. This is a bit of a worry with integrated GPUs (like the Intel ones) since they share the GPU bandwidth with a CPU, and it's a LOT (LOT LOT LOT) less than a dedicated GPU has available. In practical terms however I've not seen any evidence that this is a problem for Pioneer yet.

For us this is mainly about GPU memory saving, loading time reduction and controlling the final quality of the textures.

DXTn compression does introduce blocky artefacts into textures, so it does create quality issues of it's own, it is a lossy compression method.
This makes it unsuitable for most UI, character and some other uses however it's worst problems can be avoided with care.

Use it for 3D models like ships, stations, buildings.

I won't be removing support for PNG files ever, they're useful for development and sometimes, just very very rarely, they're higher quality is useful so use them exceedingly sparingly :)

Ask me anything else you want to know about it all.

Andy

nozmajner wrote:Yes, I already found that gimp plugin. :)
Which compression methods you recommend? Also should we generate MIP-maps?

I think... I just covered all of that :) took a while to type!!! :D

Thank you very much. I'll copy it to the wiki for further reference. :)

I merged this PR the other day so now all models should be using DDS textures, not the most optimal version for all of them but the safe DXT5 version at least which makes them 1/4 the size in memory and improves the loading time as I'm sure a few people have noticed :)