Pages: [1]
  Print  
Author Topic: Updated nice introductions to in-depth ioquake3 programming  (Read 12663 times)
GrosBedo
Member


Cakes 20
Posts: 710


« on: May 02, 2013, 02:17:54 pm »

INTRODUCTION ARTICLES TO IN-DEPTH IOQUAKE3 PROGRAMMING
UPDATED ON 2013-06-16


For those who want to dive deeply in ioquake3 programming but lack the necessary overview understanding of the code, here are a few links (which are all working as of the date above) that may be useful to you.

The links are divided between logic/server programming and graphics/mapping programming, but keep in mind that both kind of areas intersect at some points and thus it may be useful to read a bit of the articles about the other areas of your speciality to at least get a vague idea of how everything works together.

I also try to post the official links whenever possible.

Please note that I tried to order the links by conciseness, which means by the rate of intrinsic content over content length (so this means I think you are more likely to learn a lot while spending less time reading if you choose to read the first links first).


Logic/Server Programming


Also, the ioquake3 mailing list is a very good place to find interesting bits of informations (althrough you may not necessarily get an answer).

However, the best place to find how the engine work is probably by reading the headers *.h files, since ioquake3 is very cleanly coded, most of the interesting stuff and comments will be found in these files.


Graphics/Mapping Programming



Interesting various links



Quick Quotes

Here are a few particularly interesting and concise quotes that I feel should be compiled in this post (extracted from some of the links above) along with some personal notes.

Quote
Quake 3 Filesystem

Quake 3 uses a private, isolated filesystem for all file I/O. This means that all data generated directly from the game - screenshots, log files, etc., will be stored in this filesystem. The "home directory" for this filesystem, where all loads and stores occur by default, is located at ~/.quagents3/quagents3. The '.' prefix on the directory indicates a hidden directory, which means you will not be able to see it from your standard GUI filesystem browser. Instead, use your terminal to access it.

For more information, see the "files.c" file located in code/qcommon.

Quote
IOQuake3 is organized into four major code modules:

- Game virtual code (the "code/game" folder, plus some code/qcommon).
- Client virtual code (the "code/cgame" folder, plus some code/qcommon).
- User Interface virtual code (the "code/ui" folder, plus some code/qcommon).
- Non-virtual code (everything else).

When Quake runs, the first three live within separate virtual machines. They have separate memory and typically cannot see each others functions. The fourth module binds everything together and provides code that is either too security-sensitive or too computationally intensive to run within the virtual machines.

Notes:
- The Non-virtual code "module" includes the server engine server-side (which manages core functionalities and networking synchronization across all clients, see code/server), or the renderers client-side (plus a lot of other stuffs like networking). This is basically what we call the engine.
- The VMs on the other side are most often used for game logic only (what we call gamecode) and is what mods modify in order to add more gameplay functionalities.
- This means that the VMs have full access on the game logic states (can access all variables in its scope, ie: of course CGAME, which is client-side, cannot access variables of other players from GAME which is server-side) but cannot directly access the real filesystem nor do critical operations, except by using trap calls (see below).
- Contrarywise, the engine can do critical operations and entirely manage the networking and launch/restart the VMs but it cannot access all the game logic state (scores, players health, damages and other game states are hardly accessible), except when using VM calls (see below).

Quote
Getting Out of the VM

Sometimes virtual code needs to call functions that exist outside the VM. These functions are called "trap functions". Best of all, you can add your own!

Quote
Getting Into the VM

Getting into the VM is very easy. Each module of VM code has a *_public.h file hosting a large *Export enum of function names. Add the function you want to be calling from outside the VM to that enum, go to *_main.c and add a case for that value and function into the switch in the vmMain(...) method. You can then call your function from outside the vm by using VM_Call(...). You should give VM_Call a legal vm (which can sometimes be a pain to get a hold of!) and the enum value you defined.

Quote
The Q3 networking model obviates the need to even have a discussion about UDP vs. TCP, unreliable vs. reliable, and out-of-order vs. in-order. It's all unreliable UDP delta compressed against the last known state.

Quote
Quake III game engine lacks a malloc() function (...) to enforce strict coding standards.

Quote
QVMs were never meant to be secure.

Quote
Debugging in the VM

Using gdb inside the VM is horrible because every so often you run into a bunch of inline assembly that gdb has trouble coping with. Try running the executable with "+set sv_pure 0 +set vm_game 0 +set vm_cgame 0 +set vm_ui 0" to force it to circumvent the VMs entirely and use dynamic library code. If you compile with debug flags ("make debug" will do that for you), it should be much easier to track your program. Additionally, you get nice familiar segfaults instead of strange VM_Free() errors.

Note: personally when I use valgrind to find memory leaks, I have to set vm_game 1 and not vm_game 0 nor 2 to make the debugging work. Here is an example commandline to use valgrind on OpenArena/ioquake3:

Code:
valgrind --tool=memcheck --leak-check=full --log-file=valgrind.txt -v --show-reachable=yes --track-origins=yes /home/user/openarena/openarena-0.8.8/oa_ded.x86  +set net_port 27960 +set dedicated 2 +set vm_game 1 +exec myconfig.cfg

Also don't forget to 'make debug' instead of just 'make' (which defaults to 'make release') to get a meaningful trace with the functions names.
« Last Edit: June 16, 2013, 01:11:47 am by GrosBedo » Logged
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #1 on: May 02, 2013, 04:45:48 pm »

Quote
Quake 3 Filesystem

Quake 3 uses a private, isolated filesystem for all file I/O. This means that all data generated directly from the game - screenshots, log files, etc., will be stored in this filesystem. The "home directory" for this filesystem, where all loads and stores occur by default, is located at ~/.quagents3/quagents3. The '.' prefix on the directory indicates a hidden directory, which means you will not be able to see it from your standard GUI filesystem browser. Instead, use your terminal to access it.

For more information, see the "files.c" file located in code/qcommon.

"Isolated filesystem" makes this sound much more sophisticated than it is. The gamecode (cgame, ui, game) in QVM form can only write to files by using trap syscalls into the engine (assuming there are no bugs in the engine which allow direct filesystem access. Shared library game modules can just use the usual operating system calls to open files in the context of the engine). The engine then directs those (write-) calls onto files in fs_homepath/fs_game (e.g. cgame writes to "/foo/bar.log", which the engine actually stores in "/home/grey-matter/.q3a/baseq3/foo/bar.log"). Apart for some filtering regarding relative paths (which could break out of that folder) and writing to QVM/so, those files created by the engine are just regular files on your usual filesystem.
The engine itself also writes various files (qconsole, screenshots, videos etc.) into fs_homepath, because that's the fs_ dir which is supposed to be writable.


While we're at it, the ioquake3 code is available at https://github.com/ioquake/ioq3 and you should visit ircs://irc.freenode.net/ioquake3

P.S.: Did you already add those articles/links to the ioq3 wiki?
« Last Edit: May 02, 2013, 04:57:30 pm by grey matter » Logged

This space is for rent.
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #2 on: May 02, 2013, 04:56:39 pm »

huh, apparently i can't edit properly.. feel free to delete $this (former double post)
Logged

This space is for rent.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #3 on: May 04, 2013, 05:01:30 pm »

Thank you for these precisions Grey Matter.

Yes indeed QVMs are not totally separated from the real filesystem, but I think the explanation given by Quagents is quite a good way to get an idea on how the engine manages its files inside a pseudo-virtual filesystem.

And I should add that it is good to keep in mind that at first the original choice was on JVM to make those VMs as John Carmack said, but they then chose to make their own implementation.

Quote from: John Carmack
Q3 also came fairly close to have a JVM instead of the QVM interpreter, but it didn't quite fit my needs.
Source link here.

This may help in thinking that QVMs are not VMs like today's virtual machines, but more like Java Virtual Machine. Then, the intent becomes clear that QVMs were not done to be secure and separate from the host machine, but rather to compile and execute code in a portable way.

P.S.: Did you already add those articles/links to the ioq3 wiki?

Not yet because I can't edit the wiki, because the wiki is bugging and can't send confirmation codes by mails currently. I've reported the problem and will soon be able to edit it.

However, I did not see a wiki page on ioq3 where they list tutorials except their own. Maybe I could not find the good one?
Logged
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #4 on: May 05, 2013, 05:18:16 am »

P.S.: Did you already add those articles/links to the ioq3 wiki?
Not yet because I can't edit the wiki, because the wiki is bugging and can't send confirmation codes by mails currently. I've reported the problem and will soon be able to edit it.

However, I did not see a wiki page on ioq3 where they list tutorials except their own. Maybe I could not find the good one?

I must admit I don't have an account on the ioq3 wiki myself. The page which might be the best fit is Academic Articles, which despite its name lists a few non-academic websites or just specs.
Logged

This space is for rent.
Pages: [1]
  Print  
 
Jump to: