OpenArena Server Crash 0.8.5 on Debian

[glabasnat]

Hi,
My server OpenArena cresh randomly

My OS is Debian Lenny 5.0.5
Only OpenArena is from unstable version
openarena-data (0.8.5-3)
openarena-server (0.8.5-4 and others)

Older version 0.8.1 was okay

Log and config:
http://game1.xline.cz/oa/log/

Tomas

[Gig]

That crashlog.txt file reports 403 Forbidden, I can't read it.

[glabasnat]

Sorry, is it log (crashlog.txt) OK

[Cacatoes]

Do these crash appear often ?

Seems unlikely, but do you add bots once the server is started ?

[glabasnat]

Yes,
If play boots or player, server crash in 30minutes - 2 hours

[Cacatoes]

Maybe there is no bot support in one of the maps you play.
If the crash always happen on the same maps, that would explain.

[glabasnat]

Server crashes independently of the map

Could be a old library of Debin Lenny? (That would be a problem)

[Cacatoes]

If it happens often, you may try to install the openarena.ws release (not the debian one) and see if it solves anything.
I run Debian too on most of my servers, your problem is not likely due to a library, more likely due to some exotic setup (if I learned my lessons well, your crash comes from the virtual machine of the game, not from the engine/system itself)
Do you use some custom pk3s in your server ? (like mods, maps/ressources...)

[glabasnat]

This is default (clean) install from Debian.

[glabasnat]

Test:
Debian Leny and OpenArena from unstable/testing repository - Server crash

Debian Testing and OpenArena from unstable/testing repository - Server OK crash

Where is error?

[Cacatoes]

And if you try with this+that ?

http://openarena.ws/download.php?view.2
http://openarena.ws/download.php?view.3

[glabasnat]

Server is OK on original file from http://openarena.ws/
Where is bug? (I prefer a package of Debian)

[Cacatoes]

Hard to know the reason of the bug.
Check last error messages every time, try to reproduce the crash (try to know its conditions : specific maps, specific settings with bots, specific game modes...).
It's not even sure official OA release runs better than the Debian one, maybe you just got luck with the first.

[sago007]

Debian has started to use compiled so files instead of qvms. The so-files was only meant for debugging and greatly increases risks of a crash.

[glabasnat]

Crash log often ends:
Kill: 2 2 26: Player_name killed Player_name by MOD_KAMIKAZE

[smcv]

Thanks for reporting this. My guess would be that it's a NULL pointer dereference somewhere in qagame, possibly related to the kamikaze - NULL pointer dereferences will crash the native-code version, but would just quietly do the wrong thing in the bytecode interpreter, as far as I can see.

If you can rebuild the Debian version of openarena with debug symbols (DEB_BUILD_OPTIONS="noopt nostrip") and get a backtrace from a crash with gdb, that would be very useful. I realise those instructions are rather too short to be useful, I'll try to provide more detailed instructions later.

I assume you're using some sort of wrapper script to run the server repeatedly and capture crashlog.txt whenever it dies? If so, please send me a copy, and hopefully I can modify it to pick up backtraces from gdb automatically.

I had to remove the bytecode from the PK3 files in Debian for licensing reasons, but the engine still has the bytecode interpreter/JIT, so if you replace the PK3 files installed by openarena-data with the ones available from here, the Debian engine build is still able to run the bytecode versions of the game logic (if both bytecode and native-code are available, it'll normally prefer to run bytecode, unless configured specially). It'd be very useful to get a crash backtrace, though - any crash in the native-code build indicates that there's a bug somewhere.

[smcv]

Here's a version built with debug symbols, with a script added to run it under gdb. Install the appropriate one of:

http://people.debian.org/~smcv/openarena-server_0.8.5-5~debug_i386.deb (32-bit)
http://people.debian.org/~smcv/openarena-server_0.8.5-5~debug_amd64.deb (64-bit)

From your crash log it looks as though you'll want the i386 version.

Instead of running openarena-server, if you run "/usr/share/games/openarena/openarena-server-gdb", it should add a backtrace to the crash log when it crashes. Please do that, try to crash it (adding a lot of players/bots might help to make it crash quicker), and send me the log.

To check that the crash logging is working, you can simulate a crash with:

pkill -ABRT openarena-server

Note that that command will "crash" every OpenArena server on your machine, if you're running more than one!

The log from simulating a crash like that won't be any help to me, but you can use it to check that the crash logging is actually working: if it is, you'll see a lot of gdb output at the end of the crash log, starting with something like this.

Program received signal SIGABRT, Aborted.
__strncpy_ssse3 () at ../sysdeps/x86_64/multiarch/strcpy.S:1107
1107   ../sysdeps/x86_64/multiarch/strcpy.S: No such file or directory.
   in ../sysdeps/x86_64/multiarch/strcpy.S

[smcv]

While trying this out with many bots, I got one server crash caused by a missing NULL check: client was NULL in the backtrace I saw. I can't see how it would happen in practice (both players and bots have a client structure) and couldn't reproduce it, so this may or may not the same bug glabasnat saw, but the change would be good to have regardless.

I'm doing an updated debug build with this fixed. When it finishes compiling and uploading, it'll be at:

http://people.debian.org/~smcv/openarena-server_0.8.5-5~debug2_i386.deb

Code:

commit 80f2967f82314bf10e975d1b00678b9b4327eae9
Author: Simon McVittie <smcv@debian.org>
Date:   2010-10-11 23:28:51 +0100

    G_Damage: check before dereferencing targ->client, which may be NULL

diff --git a/game/code/game/g_combat.c b/game/code/game/g_combat.c
index 22ce65a..7d1fba7 100644
--- a/game/code/game/g_combat.c
+++ b/game/code/game/g_combat.c
@@ -1043,7 +1043,7 @@ void G_Damage( gentity_t *targ, gentity_t *inflictor, gentity_t *attacker,
         
         //Sago: See if the client was sent flying
         //Check if damage is by somebody who is not a player!
-        if( (!attacker || attacker->s.eType != ET_PLAYER) && client->lastSentFlying>-1 && ( mod==MOD_FALLING || mod==MOD_LAVA || mod==MOD_SLIME || mod==MOD_TRIGGER_HURT || mod==MOD_SUICIDE) )  {
+        if( (!attacker || attacker->s.eType != ET_PLAYER) && client && client->lastSentFlying>-1 && ( mod==MOD_FALLING || mod==MOD_LAVA || mod==MOD_SLIME || mod==MOD_TRIGGER_HURT || mod==MOD_SUICIDE) )  {
             if( client->lastSentFlyingTime+5000<level.time) {
                 client->lastSentFlying = -1; //More than 5 seconds, not a kill!

[smcv]

Now uploaded. However, openarena-server-gdb turns out not to be quite right: you'll need to edit it and change where it says "/usr/games/openarena" to be "/usr/games/openarena-server". I'll fix that in the next build.

[Cacatoes]

Heh, your efforts are quite appreciated
I had something in mind, which is: try to find configurations which make the server crash (simple). Right now you may be aware if a map doesn't have some bot script, and the server tries to add bot, then it crashes (or at least it complains a lot. There are also a few issues about "spawnpoints" which if they do not exist in certain gametypes (starting a deathmatch on a CTF map) will crash the server. They're not related to the engine but to the QVM, I didn't take that initiative from now because I wasn't sure what developers were able to do against it, to fix it, nor if it was worth, I'm also ready to start back using the debian package if I know there is care for it and if it's maintained. Goal would be to get better stability for OpenArena servers.

[smcv]

They're not related to the engine but to the QVM, I didn't take that initiative from now because I wasn't sure what developers were able to do against it, to fix it

If you use native code for the game logic that'd normally be bytecode (the native-code version is qagamesomething.so or qagamesomething.dll, the bytecode version is qagame.qvm in the PK3 files), you're more likely to see hard crashes rather than things quietly going wrong... but if you can get a backtrace for those crashes, it's also easier to fix them :-)

We can't use the bytecode version in Debian/Ubuntu because there's no suitably licensed compiler for it, so we get more crashes, but more debuggability, as a side-effect. Hopefully that'll mean I send in a series of one-line patches and things get more reliable for everyone!

I've left an openarena-server running on a spare machine with 16 bots fragging each other at an accelerated g_speed, I'll see whether any of them manage to trigger another crash...

[glabasnat]

Good work,
I tested:
openarena-server_0.8.5-5~debug2_i386.deb
openarena-data (0.8.5-3)

Server is online 24 hours

Tomas

[smcv]

I can't see how it would happen in practice

It turns out you can trigger this by setting off the Kamikaze close to a non-player object (e.g. anything you can pick up), so I'm fairly confident that this is what caused glabasnat's crashes.

sago007 applied it in svn as r239, and it'll be fixed in Debian as version 0.8.5-5. Hopefully I can get a freeze exception to get this into Debian 6.

[Cacatoes]

I gave this debug package a try but I didn't find as many problems as I was expecting.
I might install it on other played servers, but it happens my servers are quite stable these days.
Only crash I was able to reproduce was because of missing spawn points.

Take this map:
http://download.tuxfamily.org/openarena/autodownload/baseoa/schism-b2.pk3

Start this map with g_gametype 4 (CTF), and join either blue or red team (or ask a bot to do so). It will crash with:

Code:

ClientUserinfoChanged: 0 n\^7Caca^2t^7oes\t\1\model\arachna/red\hmodel\arachna/red\g_redteam\\g_blueteam\\c1\7\c2\5\hc\100\w\0\l\0\tt\0\tl\1\id\144F10ADA851FC79008571529BE91EE4
Playerstore: Nothing to restore. Guid: 144F10ADA851FC79008571529BE91EE4
********************
ERROR: Couldn't find a spawn point
********************
----- Server Shutdown (Server crashed: Couldn't find a spawn point) -----
Sending heartbeat to dpmaster.deathmask.net
Sending heartbeat to dpmaster.deathmask.net
==== ShutdownGame ====
ShutdownGame:
------------------------------------------------------------
recursive error after: Couldn't find a spawn point
openarena-server: code/sys/sys_main.c:149: Sys_Exit: Assertion `ex == 0' failed.
Received signal 6, exiting...
----- Server Shutdown (Signal caught) -----
Sending heartbeat to dpmaster.deathmask.net
Sending heartbeat to dpmaster.deathmask.net
==== ShutdownGame ====
ShutdownGame:
]DOUBLE SIGNAL FAULT: Received signal 11, exiting...

This is only an issue if maps are not properly made, which shouldn't happen with standard OA maps. When used on a server in a regular way (in deathmatch mode), a simple callvote to change gametype would then crash the server.

This bug has been mentionned several times on the forum but is not listed on the DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/Bugs]Wiki as of now.

I also have a question about the OpenArena Debian package: would it be useful to have some default configuration for servers, and /etc/init.d scripts ? That would be another plus to encourage the use of Debian package over oa.ws ones when installing them on servers.

[sago007]

This is only an issue if maps are not properly made, which shouldn't happen with standard OA maps. When used on a server in a regular way (in deathmatch mode), a simple callvote to change gametype would then crash the server.

The problem is that the game cannot do anything about the problem so it tries to shutdown the game in order to let the script that started the server do its magic. Or in case of a listening server it can drop back to the menu to let the player pick another map.

There are not many options: The map has no spawnpoints, so running the game logic is not an option. nextmap might not be set, so the game cannot go to the next map. Restarting the map has no effect as the problem will still be there. The default config cannot be loaded directly because it is only known to the startup script and the problem might have been caused by it anyway.