The server-side demos thread, rebooted!

[GrosBedo]

Is the trap caught in SV_GameSystemCalls in sv_main.c?

Yes it is:

Code:

case G_DEMO_COMMAND:
                if ( sv.demoState == DS_RECORDING )
                {
                        if ( args[1] == -1 )
                                SV_DemoWriteServerCommand( VMA(2) );
                        else
                                SV_DemoWriteGameCommand( args[1], VMA(2) );
                }
                return 0;

I also tried to comment out the usage of G_DEMO_COMMAND:

Code:

void trap_DemoCommand( demoCommand_t cmd, const char *string ) {
//syscall( G_DEMO_COMMAND, cmd, string );
return;
}

Still produces the same error, so the problem really happens when calling trap_DemoCommand and not when syscalling G_DEMO_COMMAND. Weird.

[sago007]

Have you tried in g_public.h to define G_DEMO_COMMAND like this:

Code:

G_DEMO_COMMAND = 50,

[GrosBedo]

No still same error, even if I set G_DEMO_COMMAND = 47. But thank's a lot for the ideas.

However since when I comment out the G_DEMO_COMMAND call it still produces the error, I think the problem resides somewhere else in the declarations of the trap_DemoCommand.

Anyway, I think I'll do something else. Even if it would be better to first restore the original functionality of the patch before editing it, I think I'll skip directly to trying to move the gamelogic of the demorecording to the serverside. I'll see how it works to move every G_DEMO_COMMAND call directly to the server-side.

[GrosBedo]

Ok I directly hacked the recording of the game commands into the server, in sv_game.c:

Code:

intptr_t SV_GameSystemCalls( intptr_t *args ) {
switch( args[0] ) {
...
case G_SEND_SERVER_COMMAND:
if ( sv.demoState == DS_RECORDING )
{
if ( args[1] == -1 )
SV_DemoWriteServerCommand( VMA(2) );
else
SV_DemoWriteGameCommand( args[1], VMA(2) );
}
SV_GameSendServerCommand( args[1], VMA(2) );
return 0;
...

Which then gives me the following (with debug printings):

Code:

broadcast: print "Major^7 captured the RED flag!\n"
DebugGBOgameCommand: scores 4 0 2 36360 0 7 13 1 0 0 23 0 0 0 1 0 0 1 0 3 5 0 1 0 0 0 0 0 0 0 0 0 1 0 1 3 0 1 0 0 64 0 0 0 0 0 0 0 0 2 3 0 1 0 0 43 0 1 0 0 0 0 0 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 1  2 6 117 0 8 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 1  2 6 116 0 8 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 2 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 2 100 50 5 0
DebugGBOgameCommand: tinfo 1  2 6 115 0 8 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 2 100 50 5 0
DebugGBOgameCommand: chat "[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."

Seems encouraging! It's not yet being broadcasted but I think I know why (the demos still relies on g_demo_command in clients vm to process the commands, but if we redirect these commands directly into another SV_SendServerCommand it should work I think).

Do someone know what is tinfo? I never saw that before... Seems like infos about a player, but of what kind?

[grey matter]

tinfo is used for the team overlay, it includes your teammates' health, location etc (see TeamplayInfoMessage()).

[GrosBedo]

@Grey Matter: thank's for the info, I see... It seems that now all the previously missing data is now saved in the demo, just needs to replay them!

AH and another finding, I think I just understood the use of Cmd_SaveCmdContext() and Cmd_RestoreCmdContext() (added in the server-side demos patch).

I think it's used to save the current context of the system, before applying demos commands, and then restores the context so that normal commands can be processed (such as a real client connecting/disconnecting) without influence of the commands processed by demo.

An example:

Code:

case demo_gameCommand:
num = MSG_ReadByte(&msg);
Cmd_SaveCmdContext();
tmpmsg = MSG_ReadString(&msg);
Cmd_TokenizeString(tmpmsg);
VM_Call(gvm, GAME_DEMO_COMMAND, num);
Cmd_RestoreCmdContext();
break;

[GrosBedo]

Just one more question: what does exactly Cmd_TokenizeString(msg) ? Such as on an example line like this:

Code:

chat "[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."

Does it separate the elements like:

Code:

chat
"[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."

?

But then what would it do with a long line with several values such as:

Code:

tinfo 1  2 6 117 0 8 0

?

[grey matter]

It just parses a string into the argc and argv command buffers, which you can query with trap_Argc() and trap_Argv() in the gamecode and via Cmd_Argv() and Cmd_Argc() in the engine.

Quoted strings like in your example are treated in a special way (same goes for // and /**/ comments). Otherwise the string is just broken into pieces at spaces.

tinfo 1  2 6 117 0 8 0
should result in
argc 8
argv[0] tinfo
argv[1] 1
argv[2] 2
and so on.

chat "[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."
should result in
argc 2
argv[0] chat
argv[1] [Sarge^7] (Blue base): ^6UnnamedPlayer set up some D.

[GrosBedo]

@Grey Matter: thank's, that helps a lot!
-----
News: ok indeed now score and chat events are recorded, and I indeed made them replay using SV_SendServerCommand. Chat messages are shown well, but there's two small problems: chat messages are repeated as many times as there were players on the server at the time (because of the mechanism of G_Say() function: it sends the chat message to every client that should receive it) - which I'll fix by checking that the same say message wasn't already sent previously, and tell messages are shown too when they shouldn't (because they use the same command "chat" as normal say to all) - which I'm gonna fix by checking the primary color of the message (this is the only way I can think of because there's no other way to my knowledge to differenciate a sayall command and a tell command). Also, teamchats are recorded, but since they use a different command "tchat" I can easily filter them at recording. I'll do that when I'll fix the two other main problems described below.

Now my 2 main problems:
- scores are still freezed, flagcaptures are not updated (but I think it's related to the 2nd problem).
- democlient statuses are not updated (connection/disconnection/team switching aren't replayed). I think this is what also causes the scores freeze.

I'm now going to look towards fixing the second problem.

/EDIT: hypothesis: I think that the main reason that Armanieu done his gamecode patch is to workaround the problem of democlients management by giving them a special status of "demoClient = qtrue" and set them to the right team (or update them to the right team) with the G_DemoSetClient() function (which was called each time a client was set in the recording, either at ClientBegin or ClientUserinfoChanged). Then there are various checks (notably in the CalculateRanks() and cmd_FollowCycle_f() functions) that were modified to also consider democlients (eg: if ( level.clients[ clientnum ].pers.connected == CON_CONNECTED || level.clients[ clientnum ].pers.demoClient ) ...).

He probably done that because he could not find another way to simulate a client without producing networks errors.

But I _think_ it's possible to really simulate a client, proof is that by issuing a ClientBegin from the server's engine, the democlient are at least recognized and set to the right team, and we can even follow them (without any democlient check hacks). What I don't understand is why they are not updated, but I will find out.

----

Ah and VoIP is not recorded, nor GUID, nor IP. The only thing recorded are the configstrings and they seem to not contain any personal information, please tell me if I'm wrong so that I can fix it:

Code:

DebugGBOconfigString: n\UnnamedPlayer\t\2\model\sarge/default\hmodel\sarge/default\g_redteam\\g_blueteam\\c1\1\c2\5\hc\100\w\0\l\0\tt\0\tl\1

[GrosBedo]

New idea! I think I'll try to hack SV_ExecuteClientCommand(), it should be a lot easier to filter teamchat and tell chat, and it should help in switching players to teams.

[grey matter]

The only thing recorded are the configstrings and they seem to not contain any personal information, please tell me if I'm wrong so that I can fix it:

Code:

DebugGBOconfigString: n\UnnamedPlayer\t\2\model\sarge/default\hmodel\sarge/default\g_redteam\\g_blueteam\\c1\1\c2\5\hc\100\w\0\l\0\tt\0\tl\1

There's no personal information in the player configstrings/userinfo at all. Besides they are distributed to the connected clients anyways (try /configstrings while connected to a server). You can take a look at the end of ClientUserinfoChanged() in the gamecode to see how the player configstrings are generated.

[GrosBedo]

You can take a look at the end of ClientUserinfoChanged() in the gamecode to see how the player configstrings are generated.

Yes I've read it, I'm quite crawling the whole code lately :p

In fact, I've found where the personal informations are transmitted to the server: in the clientCommand userinfo. I will have to be careful with this one if I record client commands.

[GrosBedo]

Ok! Chat is now working (check...). Plus, tell, teamchat and userinfo are filtered (at recording time - so they are not saved at all). But my other problems are not solved!

2 problems left:

- Player status updates
- Captures/scores updates

Player statuses seem to be a bit more complicated than what I thought, but I'm getting more and more used to the engine. I think I can completely simulate a player, but first I'm going to see if there are other workaround this, but as a last resort, I may implement demoClient in a way such that they will be considered as real players.

Self-note about player statuses non-update:
- Frag score IS updated
- Flag hold IS updated and shown next to the player icon both in the HUD (when spectating the player) and on the scoreboard.
- No capture update
- No ping update
- No switch team update
- NEW: if a player connects, he CAN be spectated, but its player infos are not processed (no model, no name - defaults to UnnamedPlayer - and no health update - but if player was already connected, its health is updated!). However, ammo number is updated (?!).
- NEW: bots camera is not updated (stays on the same direction when spectated - for players it's ok!) - could not remember this bug before, maybe it appeared with my modifications? EDIT: tested with an older binary, bots' camera isn't moving...
- NEW: persistent score captures (player->persistant[PERS_CAPTURES]) is not saved in the demos nor updated with demos events. Probably the other persistent scores are also not updated nor saved.
- NEW: player status update is really a problem. Apart from the ammo and armor, the rest is NOT updated neither in scoreboard nor HUD (except for teaminfo HUD but that's because it's now handled). For example, health is NOT updated!

/EDIT: another self-note:

To summary the big picture, the gamecode rules all the game mechanisms. The engine (ideally) only runs server stuffs and general functions (such as basic network communication - but it's the gamecode that manages the clients and game statuses!).

In multiplayer, the client's gamecode communicate with the server gamecode, which communicate with the server's engine (using SV_GameSystemCalls() in sv_game.c) - the engine then processes the required stuff (without really knowing the exact content), then pass it back to the server's gamecode, which automatically calls the required network functions to broadcast the new game status to every client.

If I'm wrong please correct me

[GrosBedo]

Fix: teaminfos hud (and in fact all gamecommands) are now played back.

[GrosBedo]

Guys, do you know a way to access gamecode vars from the engine or is that just plainly impossible (without hacking the engine a lot)?

[GrosBedo]

Found a clue! About captures scores, I think the following strings aren't recorded nor played back:

Code:

#define CS_SCORES1 6
#define CS_SCORES2 7
#define CS_FLAGSTATUS 23 // string indicating flag status in CTF

This should fix the 3rd main problem on the 4 that are remaining.

[GrosBedo]

YES! The last clue was right, flag status (taken, dropped, at base) and capture score fixed!

Now one last main problem left: player status! But I think I'm on the right track

[GrosBedo]

Still working on flagstatus, because captures made previously to the demo recording start weren't reloaded (they were refreshed after another capture was made, but this made the capture score jump: eg: 0 -> 2). Now all configstrings initial state are recorded and restored at the beginning of the demo.

Yet a bug persists: the capture score isn't persistent. After a few seconds, a new CS_SCORES1 and CS_SCORES2 are sent with both values set at 0. This is the server's gamecode that is doing this. I think I don't properly update the configstrings, so they are not persistent.

I'll now try to fix this. Maybe it'll also fix some of the player status bugs.

[Gig]

GrosBedo... seeing that you are making an huge work to have serverside demos working... but are you doing all this "from scratch"? I thought the serverside demos systems already existed in some third-party unofficial executables (that you listed in the first posts), and that you had to import one of them into the official executables... but it looks like you are needed to re-do everything...

[GrosBedo]

@Gig: At first this project started as a simple port of the existing patchs, either for OA or for other games such as Tremfusion (Tremulous).

The problem is that, along the way, I noticed a lot of short-comings with these solutions:

- the OA patch by TheDoctor is simply an aggregation of all client-side demos (so if you have 12 players, you'll have 12 demos, just for one game! And you simply can't switch from one player's perspective to the other, you have to stop the demo and rewatch a whole other demo just to watch another player).

- the ioquake3 Alpha project implements a similar idea as TheDoctor, with some advantages (eg: demos are named meaningfully eg: oasago2-someplayername-date.dm_71, while TheDoctor's are named something like Z12ild83.dm_71 ...).

- the tremfusion server-side demos patch by Amanieu was the closest to what I aimed: full server-side demos, where you can switch the perspective directly from the demo, and with meaningful names.

Unfortunately, Amanieu has not polished his patch, and so there are some important missing data that were not recorded in the demo (eg: g_gametype, how the hell can you get playback a demo if you don't even get back the g_gametype that was used?), and a lot of issues that were not fixed (such as managing automatically sv_democlients and sv_fps), this is why Amanieu's patch is known to be buggy, but at least working (it's the unique full server-side demo patch for ioquake3 to my knowledge).

Also the patch wasn't entirely on the engine (binaries) but also on the gamecode, so that mean that to use its server-side demos you have to both patch the binaries AND the vm, which also mean that it's not compatible with mods. That limits a lot its usefulness and "universality".

Plus imagine telling to clients "to replay demos, you have to use a special QVM that I'm not maintaining anymore, so you have to use OA 0.8.1 with my QVM and binaries to replay the demos, then to play you put back your backed up QVM of OA x.x ...).

----

So my solution is at first based on Amanieu's patch, and we have to thank him a lot, his code is genius. But I aim to extend its idea and not only record a demo based on some parameters: demos recorded with my patch will be an almost total reproduction of the game when it was played.

That's why it takes me so much time: when Amanieu simply workaround the problem of managing democlients (clients that are replayed in a demo) by giving them a special status in the gamecode, I am currently totally recording the democlients data and simulate these players as if they were real, by the way tricking the engine into thinking it's real too.

So yes, all these additions are done from scratch, from crawling tons and tons of ioquake3 code and trying everything I can think of.

But now I'm near the conclusion! I still have some problems managing correctly the player's statuses, but almost all functionnalities are here and recording/playing demos is very straightforward, everything is automatically managed!

In the end, my solution should follow these criterias:
- completely managed by the engine - OK
- completely retrocompatible with any version of OA (just copy the binaries, you can run any gamecode version) - OK
- compatible with mods (for example - hopefully we may be able to record Generations Arena server-side demos!) - not tested!
- completely multiviewpoints demos: can follow and even freefly around the map while the demo is played! - OK
- privacy respectful: should not contain any personal data (such as ip nor guid) - OK
- OATV: could be used to retransmit matchs as an opensource and native implementation of GTV (you can launch a demo on your server and all clients can see it live!) - OK

It will also suffer of the same problems as normal demos (but that's not bad, normal demos are well recorded!):
- can't check the map version, the client will have to manage it to get the right map to replay the demo right
- can't check the mod: the client will have to load the mod by himself prior to replay the demo

What is also nice, is that this facility is greatly expandable, and I will try to comment it to my best to explain everything I can (now that I understand it!), so we can imagine that in the future, it could be the base foundation for all the demos facilities and it could even be enhanced to fix all of the above concerns (it CAN save the mod, and save an MD5 checksum of the map, the problem then is how to load them, because mods can't be hotswapped server-side).

----

So now I hope that you see why I'm putting so much effort. I don't want to leave this half finished now I'm near the ultimate goal

[GrosBedo]

Back to the matter, here are some debugs I've done:

Code:

clientid              connectvar                teamvar
DGBO TeamCvarSet: 0 connected=2 team=0
DGBO TeamCvarSet: 1 connected=2 team=0
DGBO TeamCvarSet: 2 connected=2 team=0
DGBO TeamCvarSet: 3 connected=2 team=0
DGBO TeamCvarSet: 4 connected=0 team=0
DGBO TeamCvarSet: 5 connected=0 team=0
DGBO TeamCvarSet: 6 connected=0 team=0

As you can see, the 4 first players are democlients, they are considered as CON_CONNECTED (the 2), but they are considered to be in the TEAM_UNKNOWN 0! This is very weird. And sometimes, it happens that they are switched to the right team, but it's very rare!

So I think it's the right track here

[Gig]

Wow!!! 
Anyway, about the hashes to find the right versions of pk3s (and thus, of maps and other files)... well, maybe you may make the demo store the list of used pk3 anyway, so when the player would in future play the demo, should play it correctly if it has got the required pk3s or more (e.g. playing a 0.8.1 demo with 0.8. not loading 0.8.5 and 0.8.8 pk3s). In case some pk3 will be missing on the player, of course the demo would not know where to download it... maybe it may try to play it anyway like it does now, but warning the user that some packs are missing so strange things may happen.
If working, maybe such coherency check may be extendend -optionally- to standard demos, who knows....

[GrosBedo]

There's a config string that stores every pk3 loaded by the server at the time, but when I save this special field (it's considered as reserved by the engine), the engine crashes when it tries to replay a demo. So I think it may not be a good idea for now.

To ensure coherency of demo we will have to create our very own functions because for now there's not really any coherency check (normal client-side demos included).

[grey matter]

Guys, do you know a way to access gamecode vars from the engine or is that just plainly impossible (without hacking the engine a lot)?

It's not impossible, but there's no sane reason to do this. Gamecode and thus offsets to structures change from mod to mod or even between versions of the same mod.
If you really want to go that way, peek around in some of the cheat forums or private clan forums for UrT, because runtime QVM modifications is the only real way for UrT server-side mods.

[..] (it CAN save the mod, and save an MD5 checksum of the map, the problem then is how to load them, because mods can't be hotswapped server-side).

You can easily restart the vfs. Hotswapping is not possible (and still a difficult topic in general. Read e.g. the kSplice whitepapers for some background info, though it'd be a little easier in ioq3). There's a /game_restart <modname> command in ioq3, so you don't even need to mod anything.

[GrosBedo]

You can easily restart the vfs. Hotswapping is not possible (and still a difficult topic in general. Read e.g. the kSplice whitepapers for some background info, though it'd be a little easier in ioq3). There's a /game_restart <modname> command in ioq3, so you don't even need to mod anything.

Mate, you've MADE MY DAY! I just tried it and it works!!! That's GREAT! This is a great command for all servers administrators, and I didn't even know its existence. Thank's a lot!

Gig, could you please add it to the wiki please? I don't know where this should belong, but it's important enough to be noted in the wiki.

----

About accessing gamecode cvars: ok Grey Matter, I don't want to do a dirty hack. As an alternative, is there a way to update the gamecode vars from the engine vars?

I ask this because I am now pretty sure that server engine's configstrings aren't synchronized with gamecode's configstrings, and this is the reason why there are weird stuff happening.

/EDIT: I really don't understand what's happening with configstrings so I will here detail further.

I will take the captures scores as an example.

My patch now sets correctly the configstrings at demo start and everytime there's an update. But every few moments (eg: just after a clientBegin), the CS_Scores get set back to 0 by an SV_GameSystemCalls G_SET_CONFIGSTRING (highlighted below by a "DGBO G_SET_CONFIGSTRING").

Code:

DebugGBOconfigString: 4 snaps 25 pmove_fixed 0
DebugGBOconfigString: 5
DebugGBOconfigString: 6 1
DebugGBOconfigString: 7 0
DebugGBOconfigString: 8
DebugGBOconfigString: 9
DebugGBOconfigString: 10
DebugGBOconfigString: 11
DebugGBOconfigString: 12
DebugGBOconfigString: 13
DebugGBOconfigString: 14
DebugGBOconfigString: 15
DebugGBOconfigString: 16
DebugGBOconfigString: 17
DebugGBOconfigString: 18
DebugGBOconfigString: 19
DebugGBOconfigString: 20 baseoa-1
DebugGBOconfigString: 21 29840
DebugGBOconfigString: 22
DebugGBOconfigString: 23 00
DebugGBOconfigString: 24
DebugGBOconfigString: 25
DebugGBOconfigString: 26
DebugGBOconfigString: 27 0010011011101111001101111000000000110000000000000000000000000
DebugGBOconfigString: 28
...
DebugGBOconfigString: 1021
DebugGBOconfigString: 1022
DebugGBOconfigString: 1023
Playing demo svdemos/demotest44.svdm_71.
DGBO G_SET_CONFIGSTRING: 6 0
DGBO G_SET_CONFIGSTRING: 7 0
DGBO G_CVAR_SET: g_humanplayers = 1
Item: 2 weapon_rocketlauncher
Item: 3 ammo_shells
Item: 1 ammo_rockets
Item: 1 weapon_rocketlauncher
Item: 1 ammo_rockets
Item: 3 ammo_cells
DebugGBOgameCommand: scores 4 1 0 39840 0 9 12 0 0 0 11 0 0 0 0 0 0 1 0 1 4 0 0 0 0 30 0 0 0 0 0 0 0 0 2 3 0 0 0 0 12 0 0 0 1 0 0 0 0 3 2 0 0 0 0 36 0 0 0 0 0 0 0 0
DGBO G_SET_CONFIGSTRING: 6 0
DGBO G_SET_CONFIGSTRING: 7 0

Note: configstring 6 is CS_SCORES1 and 7 is CS_SCORES2.

So I guess that the gamecode is using some trap somewhere to set back the engine configstrings, but this is weird, because I can't understand why it would set this value when the configstrings are stored in the engine.

So am I right? Configstrings are stored in the engine?