OpenArena Message Boards

OpenArena Contributions => Development => Topic started by: GrosBedo on May 02, 2013, 02:17:54 PM



Title: Updated nice introductions to in-depth ioquake3 programming
Post by: GrosBedo on May 02, 2013, 02:17:54 PM
INTRODUCTION ARTICLES TO IN-DEPTH IOQUAKE3 PROGRAMMING
UPDATED ON 2013-06-16


For those who want to dive deeply in ioquake3 programming but lack the necessary overview understanding of the code, here are a few links (which are all working as of the date above) that may be useful to you.

The links are divided between logic/server programming and graphics/mapping programming, but keep in mind that both kind of areas intersect at some points and thus it may be useful to read a bit of the articles about the other areas of your speciality to at least get a vague idea of how everything works together.

I also try to post the official links whenever possible.

Please note that I tried to order the links by conciseness, which means by the rate of intrinsic content over content length (so this means I think you are more likely to learn a lot while spending less time reading if you choose to read the first links first).


Logic/Server Programming

  • How to build ioquake3 on any platform (http://wiki.ioquake3.org/Building_ioquake3)
  • Quagents's infos about their edits in ioquake3 code (http://www.cs.rochester.edu/trac/quagents/wiki/ioquake3): highly interesting read with infos you won't find anywhere else and simple step-by-step instructions to add your own content in ioquake3, (ie: how to add you own custom trap calls to access functions from inside a VM to outside, and inversely how to do a VM call to reach inside a VM from outside (http://www.cs.rochester.edu/trac/quagents/wiki/ServerProgrammingDealingWithVMs); and also how ioquake3 manages its I/O (http://www.cs.rochester.edu/trac/quagents/wiki/ServerProgrammingDealingWithIO); how to add your custom items without breaking compatibility (http://www.cs.rochester.edu/trac/quagents/wiki/ServerProgrammingItems))
  • The Quake3 Networking Model (http://trac.bookofhook.com/bookofhook/trac.cgi/wiki/Quake3Networking): very simple and abstract outline of how the networking works in ioquake3. Great way to get the big picture about the networking model.
  • Quake III Arena Game Structures (http://caia.swin.edu.au/reports/110209A/CAIA-TR-110209A.pdf): a great academic paper with schema to illustrate how the engine works
  • Laszlo Menczel's Structures and Typed Enums Used in the Quake III Arena Source Code (http://linradiant.intron-trans.hu/docs/q3a_structures.html): a great reference to quickly find the variables you need for your purpose. It's a great starting point to know where to look at in the engine, because if you know what variable you need to track, you can easily know what functions you need to modify with a quick search through the files.
  • Baseq3 source commentary (works about the same in ioquake3) (http://www.icculus.org/homepages/phaethon/q3mc/q3mc.html): this is a reference guide to get to know how the whole engine works (but it may be a bit too technical sometimes, so try to keep an abstract view of the engine in mind while reading this, or better you can read the game engine sourcecode at the same time).
  • Q3TA Menu System specifications (http://www.icculus.org/homepages/phaethon/q3tamenu/q3tamenu.html)
  • A short dev note from a third-party project (http://subversion.assembla.com/svn/ioquake3mocap/__NOTES.txt)
  • Highly interesting: there's no implementation of malloc in ioquake3 in order to enforce strict standards, but you can find one here: implementation of malloc for Quake 3 (http://www.icculus.org/homepages/phaethon/q3a/malloc/malloc.html)
  • Analyzing the AI Bot Library from the Quake 3 Source Code (http://aigamedev.com/open/article/quake3-engine/)
  • Looking at the Quake 3 Source -- Part 1 (http://element61.blogspot.com.au/2005/08/looking-at-quake-3-source-part-1.html)
  • Extracting position data from recorded DEMO files (http://openarena.ws/board/index.php?topic=4683.msg45659#msg45659)
  • Another thread to add your own custom trap function (http://www.quake3world.com/forum/viewtopic.php?p=597641#597641)
  • Quake 3 Network Protocol Format (http://www.tilion.org.uk/2011/11/quake-3-network-format/)

Also, the ioquake3 mailing list (http://ioquake3.org/discussion/) is a very good place to find interesting bits of informations (althrough you may not necessarily get an answer).

However, the best place to find how the engine work is probably by reading the headers *.h files, since ioquake3 is very cleanly coded, most of the interesting stuff and comments will be found in these files.


Graphics/Mapping Programming

  • Kekoa Proudfoot's Unofficial Quake 3 Map Specs (the reference guide to know how BSP format works) (http://www.mralligator.com/q3/)
  • Rendering Quake 3 Maps: how to render the basic geometry of a Quake 3 map using OpenGL. This is an unofficial followup on Unofficial Quake 3 Map Specs. (http://graphics.cs.brown.edu/games/quake/quake3.html)
  • Description of MD3 Format: MD3 is the format that describes all movable and animated 3D objects. (http://www.icculus.org/homepages/phaethon/q3a/formats/md3format.html) (beware the document was last updated in 2011, the old one can still be found here (http://www.icculus.org/homepages/phaethon/q3a/formats/md3-crapped.html))
  • Description of MD4 Format: a newer data format, appeared after MD3. (http://www.icculus.org/homepages/phaethon/q3a/formats/md4format.html)
  • Quake 3 BSP Collision Detection (http://www.nathanostgard.com/quake3-collision-detection/) (which is currently in the process of being rewritten because of mistakes and computational errors) and a french thread about the usage of the algorithm in the first version of this article (http://forum.games-creators.org/showthread.php?t=2314)
  • Michael Abrash's Ramblings in Realtime (http://www.bluesnews.com/abrash/contents.shtml): a great guide to dive deeply inside the 3D processing of Quake 1 (not Quake 3! But an interesting read nonetheless from one of the original programmers of Quake 1)


Interesting various links

  • QVMDisas Python (https://gist.github.com/lrq3000/5504503) (bit older but works) and QVMDisas Go (http://www.gamedeception.net/content/186) (latest version): very useful tool to disassemble QVMs. You can also cross-check the results with ollydbg manually. Also don't forget to make debug to get the full symbols.
  • An enhanced vote system with filters (eg: cannot vote an extreme fraglimit) (http://www.icculus.org/homepages/phaethon/q3a/vote/votebox.html)
  • MD3 import script into Blender (http://www.icculus.org/homepages/phaethon/q3a/md3import/md3import.html) (BEWARE: it is very OLD, may not work anymore or maybe there is a newer version out there!)
  • Q3ASM Turbo (may already be implemented in ioquake3) (http://www.icculus.org/homepages/phaethon/q3a/q3asm-turbo/q3asm-turbo.html)
  • Extended Math functions for VMs (http://www.icculus.org/homepages/phaethon/q3a/math/math.html)
  • Triseism, a standalone Q3VM Interpreter (http://www.icculus.org/triseism/triseism.html)
  • injx's Strafing Theory (http://www.funender.com/quake/articles/strafing_theory.html), Circle-Jump Theory (http://www.funender.com/quake/articles/cj_theory.html) and Wall Strafing Theory (http://www.funender.com/quake/articles/wall_strafing.html)
  • Editing already compiled QVM files (http://gamedev.stackexchange.com/questions/15107/how-can-i-edit-qvm-quake-virtual-machine-files): basically, use QVMDisas and then edit hexadecimally the qvm like any other software, just be careful to edit the correct offset (you will probably have to do some maths to find the aligned offset, because at runtime the offsets are different than when not running, but it's possible since I've done it this way to edit CPMA and GTV)
  • John Carmack Slashdot Archive (http://floodyberry.com/carmack/slashdot.html) and John Carmack Archive (http://floodyberry.com/carmack/)


Quick Quotes

Here are a few particularly interesting and concise quotes that I feel should be compiled in this post (extracted from some of the links above) along with some personal notes.

Quote
Quake 3 Filesystem

Quake 3 uses a private, isolated filesystem for all file I/O. This means that all data generated directly from the game - screenshots, log files, etc., will be stored in this filesystem. The "home directory" for this filesystem, where all loads and stores occur by default, is located at ~/.quagents3/quagents3. The '.' prefix on the directory indicates a hidden directory, which means you will not be able to see it from your standard GUI filesystem browser. Instead, use your terminal to access it.

For more information, see the "files.c" file located in code/qcommon.

Quote
IOQuake3 is organized into four major code modules:

- Game virtual code (the "code/game" folder, plus some code/qcommon).
- Client virtual code (the "code/cgame" folder, plus some code/qcommon).
- User Interface virtual code (the "code/ui" folder, plus some code/qcommon).
- Non-virtual code (everything else).

When Quake runs, the first three live within separate virtual machines. They have separate memory and typically cannot see each others functions. The fourth module binds everything together and provides code that is either too security-sensitive or too computationally intensive to run within the virtual machines.

Notes:
- The Non-virtual code "module" includes the server engine server-side (which manages core functionalities and networking synchronization across all clients, see code/server), or the renderers client-side (plus a lot of other stuffs like networking). This is basically what we call the engine.
- The VMs on the other side are most often used for game logic only (what we call gamecode) and is what mods modify in order to add more gameplay functionalities.
- This means that the VMs have full access on the game logic states (can access all variables in its scope, ie: of course CGAME, which is client-side, cannot access variables of other players from GAME which is server-side) but cannot directly access the real filesystem nor do critical operations, except by using trap calls (see below).
- Contrarywise, the engine can do critical operations and entirely manage the networking and launch/restart the VMs but it cannot access all the game logic state (scores, players health, damages and other game states are hardly accessible), except when using VM calls (see below).

Quote
Getting Out of the VM

Sometimes virtual code needs to call functions that exist outside the VM. These functions are called "trap functions". Best of all, you can add your own!

Quote
Getting Into the VM

Getting into the VM is very easy. Each module of VM code has a *_public.h file hosting a large *Export enum of function names. Add the function you want to be calling from outside the VM to that enum, go to *_main.c and add a case for that value and function into the switch in the vmMain(...) method. You can then call your function from outside the vm by using VM_Call(...). You should give VM_Call a legal vm (which can sometimes be a pain to get a hold of!) and the enum value you defined.

Quote
The Q3 networking model obviates the need to even have a discussion about UDP vs. TCP, unreliable vs. reliable, and out-of-order vs. in-order. It's all unreliable UDP delta compressed against the last known state.

Quote
Quake III game engine lacks a malloc() function (...) to enforce strict coding standards.

Quote
QVMs were never meant to be secure.

Quote
Debugging in the VM

Using gdb inside the VM is horrible because every so often you run into a bunch of inline assembly that gdb has trouble coping with. Try running the executable with "+set sv_pure 0 +set vm_game 0 +set vm_cgame 0 +set vm_ui 0" to force it to circumvent the VMs entirely and use dynamic library code. If you compile with debug flags ("make debug" will do that for you), it should be much easier to track your program. Additionally, you get nice familiar segfaults instead of strange VM_Free() errors.

Note: personally when I use valgrind to find memory leaks, I have to set vm_game 1 and not vm_game 0 nor 2 to make the debugging work. Here is an example commandline to use valgrind on OpenArena/ioquake3:

Code:
valgrind --tool=memcheck --leak-check=full --log-file=valgrind.txt -v --show-reachable=yes --track-origins=yes /home/user/openarena/openarena-0.8.8/oa_ded.x86  +set net_port 27960 +set dedicated 2 +set vm_game 1 +exec myconfig.cfg

Also don't forget to 'make debug' instead of just 'make' (which defaults to 'make release') to get a meaningful trace with the functions names.


Title: Re: Updated nice introductions to in-depth ioquake3 programming
Post by: grey matter on May 02, 2013, 04:45:48 PM
Quote
Quake 3 Filesystem

Quake 3 uses a private, isolated filesystem for all file I/O. This means that all data generated directly from the game - screenshots, log files, etc., will be stored in this filesystem. The "home directory" for this filesystem, where all loads and stores occur by default, is located at ~/.quagents3/quagents3. The '.' prefix on the directory indicates a hidden directory, which means you will not be able to see it from your standard GUI filesystem browser. Instead, use your terminal to access it.

For more information, see the "files.c" file located in code/qcommon.

"Isolated filesystem" makes this sound much more sophisticated than it is. The gamecode (cgame, ui, game) in QVM form can only write to files by using trap syscalls into the engine (assuming there are no bugs in the engine which allow direct filesystem access. Shared library game modules can just use the usual operating system calls to open files in the context of the engine). The engine then directs those (write-) calls onto files in fs_homepath/fs_game (e.g. cgame writes to "/foo/bar.log", which the engine actually stores in "/home/grey-matter/.q3a/baseq3/foo/bar.log"). Apart for some filtering regarding relative paths (which could break out of that folder) and writing to QVM/so, those files created by the engine are just regular files on your usual filesystem.
The engine itself also writes various files (qconsole, screenshots, videos etc.) into fs_homepath, because that's the fs_ dir which is supposed to be writable.


While we're at it, the ioquake3 code is available at https://github.com/ioquake/ioq3 and you should visit ircs://irc.freenode.net/ioquake3

P.S.: Did you already add those articles/links to the ioq3 wiki?


Title: Re: Updated nice introductions to in-depth ioquake3 programming
Post by: grey matter on May 02, 2013, 04:56:39 PM
huh, apparently i can't edit properly.. feel free to delete $this (former double post)


Title: Re: Updated nice introductions to in-depth ioquake3 programming
Post by: GrosBedo on May 04, 2013, 05:01:30 PM
Thank you for these precisions Grey Matter.

Yes indeed QVMs are not totally separated from the real filesystem, but I think the explanation given by Quagents is quite a good way to get an idea on how the engine manages its files inside a pseudo-virtual filesystem.

And I should add that it is good to keep in mind that at first the original choice was on JVM to make those VMs as John Carmack said, but they then chose to make their own implementation.

Quote from: John Carmack
Q3 also came fairly close to have a JVM instead of the QVM interpreter, but it didn't quite fit my needs.
Source link here. (http://floodyberry.com/carmack/slashdot.html#s20000806x021725)

This may help in thinking that QVMs are not VMs like today's virtual machines, but more like Java Virtual Machine. Then, the intent becomes clear that QVMs were not done to be secure and separate from the host machine, but rather to compile and execute code in a portable way.

P.S.: Did you already add those articles/links to the ioq3 wiki?

Not yet because I can't edit the wiki, because the wiki is bugging and can't send confirmation codes by mails currently. I've reported the problem and will soon be able to edit it.

However, I did not see a wiki page on ioq3 where they list tutorials except their own. Maybe I could not find the good one?


Title: Re: Updated nice introductions to in-depth ioquake3 programming
Post by: grey matter on May 05, 2013, 05:18:16 AM
P.S.: Did you already add those articles/links to the ioq3 wiki?
Not yet because I can't edit the wiki, because the wiki is bugging and can't send confirmation codes by mails currently. I've reported the problem and will soon be able to edit it.

However, I did not see a wiki page on ioq3 where they list tutorials except their own. Maybe I could not find the good one?

I must admit I don't have an account on the ioq3 wiki myself. The page which might be the best fit is Academic Articles (http://wiki.ioquake3.org/Academic_Articles), which despite its name lists a few non-academic websites or just specs.