Interesting optimisation

[fromhell]

OA085 is 259mb after batch JPEGing all 24-bit TGA files (without the alphas) and encoding all WAV files as Ogg Vorbis, -q9.

Further, with the paks merged together into one overwriting whatever (mainly the patch doing this), it is 247mb.

This is down from 323mb.

Yeah i hacked together some batch file i meant to do for years, it uses imagemagick unfortunately. but at least it's nearly hands free. figuring out how to sniff a 24-bit TGA (as opposed to a 32-bit one) was tough for me to figure out



For future full releases it'll definitely be smaller than 259mb at least and i'll do the optimisation i should have done earlier. So far I haven't seen any awful glaring artifacts from the result.

[Udi]

Nice. Does the WAV -> Ogg conversion affect performance (or loading times)?

[fromhell]

Low-end performance I haven't yet tested yet.

[Gig]

I'm a bit unsure about:
- Possible speed decrease in using compressed sounds, on old machines... like mine!
- Possible problems caused by removing the alpha channel from some images?
- Real benefit in decreasing download size by 70MB? The game would still be very big for someone with a 56k connection... while for who is using adsl, 70 MB are few minutes less... no great change...

[Udi]

- Possible problems caused by removing the alpha channel from some images?

The alphas aren't removed, only those TGAs are converted to JPG which don't have alphas (the 24bit ones).

[Gig]

Oh, i thought that that "(without the alphas)" was meant that the operation would have removed the alpha channel. Instead it was talking about selecting only the images without the alpha channel.

Thus, ok, but the other doubts are still there..

[pulchr]

For future full releases it'll definitely be smaller than 259mb at least and i'll do the optimisation i should have done earlier.

what's the magical effect of the number 259?

[andrewj]

what's the magical effect of the number 259?

Re-read the first line of OP.

[sago007]

- Possible speed decrease in using compressed sounds, on old machines... like mine!

It might be good idea to have a minimum size the files should have before compressing them into vorbis. But again compressing can give performance for larger files. 

Uncompressing is fast, reading from harddisk is slow. Even on old machines there are good changes that the thing that slows down loading is the harddrive speed.

On newer machines this is even more extreme because harddrive read-speed has not grown as fast as other areas of technology.

[fromhell]

You might try to get faster loading times by using a USB stick to store the pk3 files, then using fs_cdpath to point to the stick'a OA folder. Use your fastest USB port for this.

[fromhell]

Right now to sooth the curiosity of PNG fanboys, i'm PNGing all the 32-bit TGAs (And there is a lot of those) and running optipng with -o9. So far, it's taken 1.6 HOURS and it's still not done! It hasn't even gotten to the textures/ folder yet!

I wouldn't use PNG in a release due to slow loading anyway.

UPDATE: Finished and packed - 239mb. Not a big decrease worth the long conversion time. Loading times still untested, but predictably it'll be too long. 84mb savings.

Compressing with LZMA brings it down to 214mb, loading times still untested but will probably be even longer especially with the memory requirements for just decompressing. 109mb savings.

[WaspKiller]

OA085 is 259mb after batch JPEGing all 24-bit TGA files (without the alphas) and encoding all WAV files as Ogg Vorbis, -q9...

I'm a bit unsure about:
...
- Real benefit in decreasing download size by 70MB? The game would still be very big for someone with a 56k connection... while for who is using adsl, 70 MB are few minutes less... no great change...

...i'm PNGing all the 32-bit TGAs... and running optipng with -o9...

UPDATE: Finished and packed - 239mb. Not a big decrease worth the long conversion time. Loading times still untested, but predictably it'll be too long. 84mb savings.

Compressing with LZMA brings it down to 214mb, loading times still untested but will probably be even longer especially with the memory requirements for just decompressing. 109mb savings.

Always great to hear about various progress but what's the concern regarding size?  At 323MB, OA is less than half the size of Urban Terror and still twice the game.  With over 30 times the number of active players as OA, I don't see UrT sufferring for lack of downloads owing to it's size (708MB for Windows version).

Also, why compress the Oggs and PNGs at level 9?  Isn't that the highest compression level?  I would image that you would obtain a more optimal trade-off of size vs loading times if you compressed at level 5 or 6.


**For those readers whose first language is not English, "still twice the game" means OA is two times better than UrT, mainly because UrT lacks all the great maps and mod game play that OA can leverage from Q3.

[fromhell]

Also, why compress the Oggs and PNGs at level 9?  

In oggenc, -q9 is a very high quality setting. In OptiPNG, -o9 means 9 more compression trials (lossless, the only tradeoff there is load performance)

what's the concern regarding size?

I don't like to upload large files. I don't really believe in 'bigger is better'. I'd HATE to hit the popular 500-800mb mark most games do for some reason. Nexuiz for example went to 600mb (from 400mb iirc) just for textures used by some few new maps. The original Nexuiz release was 150mb and isn't a lot different content-wise.


Is there a way to accurately log loading times in the engine?

[Gig]

A clarification: more than the loading times (when loading the map, weapons, etc), I was worrying about the more computational power needed to play compressed audio files. I don't want this could bring to a framerate reduction.


I'm not saying that the ogg sounds actually slow down fps, I'm just asking if they do and, in this case, to consider if it is worth to do the change or not.

For an example, with my 2001 PC I can see videos with some codecs (like xvid) with no problem, while with other codecs (like h264) I get about a frame every 5 seconds...

I remember that Q3 demos were used to benchmark computers for years. Can we run a demo benchmark test with the "standard" game and with the compressed sounds, to see if there is some change in the framerate? If a slowndown occurs, I suppose using an old machine may make it more clearly visible. I suppose also that we could do a similar test with the textures (tga, jpg, png)... and probably, loading the same demo could also be used as a test for the initial loading times, with a chronometer...

[RMF]

i like small filesizes, but running at more fps is always better ofcourse. The filesize doesn't improve a lot on the gameplay, fps does.

[Falkland]

The filesize doesn't improve a lot on the gameplay, fps does.

With fps independent gameplay or "cpu rounding error free" gameplay (  pmove_float ) YOU DON'T NEED TO HAVE A CONSTANT FPS VALUE NOR A HIGH VALUE NOR TO INCREASE IT

It's not hard to understand , isn't it ?

[Falkland]

OA085 is 259mb after batch JPEGing all 24-bit TGA files (without the alphas)

After JPEGing TGA files , we should find a way to use libjpeg-turbo as I suggested in another post : http://openarena.ws/board/index.php?topic=3785.msg32940#msg32940

http://fedoraproject.org/wiki/Features/libjpeg-turbo

libjpeg library has been replaced by libjpeg-turbo library which has same API/ABI but is at least twice faster on all primary architectures and about 25% faster on secondary architectures

libjpeg-turbo is an ASM/SIMD optimized JPEG lib used also in the VirtualGL project which is a software used to export GL accelerated visualization through network ( it's also used for remote data visualization at CERN )

[Gig]

The filesize doesn't improve a lot on the gameplay, fps does.

With fps independent gameplay or "cpu rounding error free" gameplay (  pmove_float ) YOU DON'T NEED TO HAVE A CONSTANT FPS VALUE NOR A HIGH VALUE NOR TO INCREASE IT

It's not hard to understand , isn't it ?

Framerate is not only related to game physics (that is fixed with fps independent gameplay option), but also to the way the game looks/feels. Try to play the same game at 30, 60 and 90 fps... you can notice the difference... I suppose there is no problem on a machine can can achieve 250 fps... but on a machine that can achieve 50-60 fps? Going down from 250 to 220 would not a problem, bug going down from 60 to 30 would be a problem (it's just an example: I don't know if and how much the ogg sounds and the jpg or png images slow down the framerate)...

[sago007]

A clarification: more than the loading times (when loading the map, weapons, etc), I was worrying about the more computational power needed to play compressed audio files. I don't want this could bring to a framerate reduction.

Except for music (sounds more than 1 minute long) and videos compressed data is decompressed during loading and converted to a format supported by the hardware. So loading is the only point of interest. The only data I know is being read during a game are models and by default they are not loaded until you watch the scoreboard.

EDIT:
Modern games with a lot of data do load and unload data during a game but that was not an option back in 1999.

[7]

With fps independent gameplay or "cpu rounding error free" gameplay (  pmove_float ) YOU DON'T NEED TO HAVE A CONSTANT FPS VALUE NOR A HIGH VALUE NOR TO INCREASE IT

It's not hard to understand , isn't it ?

Yes it is You don't need a very high FPS value anymore, but you do need a constant one, because mouse polling as well as packet sending happens on a frame basis. If your FPS varies a lot your mouse becomes inaccurate because it isn't polled at regular intervals, and if your FPS drops below the servers sv_fps for 2/sv_fps seconds or more the client fails to send a packet to the server for every snap, so you'll appear to be lagging to the other players on the server.

[Falkland]

Yes it is You don't need a very high FPS value anymore, but you do need a constant one, because mouse polling as well as packet sending happens on a frame basis. If your FPS varies a lot your mouse becomes inaccurate because it isn't polled at regular intervals, and if your FPS drops below the servers sv_fps for 2/sv_fps seconds or more the client fails to send a packet to the server for every snap, so you'll appear to be lagging to the other players on the server.

I should have though about a little bit more before press the post button , but anyway thank you for pointing that

[fromhell]

libjpeg-turbo is an ASM/SIMD optimized JPEG lib used also in the VirtualGL project which is a software used to export GL accelerated visualization through network ( it's also used for remote data visualization at CERN )

Interesting... could default that on if SSE is detected.

Also in speaking of JPEGs, Darkplaces (another engine) has a hack that loads alpha channels into JPGs by a seperate JPG file with an alpha prefix to its extension on load.  I imagine there'd be a way to have a similar hack in OA too (in renderer/tr_image.c somewhere, checking a .jpa file for each .jpg that loads)

A .jpa file would be a greyscale 256-color JPEG.

For testing i've packed a .jpg/.jpa version, it makes a 196MB PK3 file. That's the entire game, JPEGGED AND OGGED!  127MB SAVINGS!

[Gig]

An important thing to think about: how to preserve compatibility?

I suppose the engine should do something like:
If the program contains a link for "gasp.wav", it should search if "gasp.wav" exists, and if not, then search for "gasp.ogg".
If the map requires a texture "wall.tga", it should search if "wall.tga" exists, and if not, then search for "wall.png" and then for "wall.jpg".
Right?

We have to be sure that all the previous maps continue to work, and all the previous mods continue to work (is it possible to be sure that all the mods will follow the new behavior?). And what will happen when an higher pak file will contain the same file name, but with a different extension?

[fromhell]

If the program contains a link for "gasp.wav", it should search if "gasp.wav" exists, and if not, then search for "gasp.ogg".
If the map requires a texture "wall.tga", it should search if "wall.tga" exists, and if not, then search for "wall.png" and then for "wall.jpg".

It already does this for both, which is why I can get away with this crazy optimisation
it doesn't seek pcx files if missing though which can only be used explicitly.

[Falkland]

For testing i've packed a .jpg/.jpa version, it makes a 196MB PK3 file. That's the entire game, JPEGGED AND OGGED!  127MB SAVINGS!

Strictly about OGG , Xreal has introduced OGV/theora video support for demo conversion. Has anyone tried to port ogv/theora support to OA engine ?