OpenArena Message Boards

OpenArena Contributions => Development => Topic started by: GrosBedo on January 28, 2012, 05:39:58 PM



Title: The server-side demos thread, rebooted!
Post by: GrosBedo on January 28, 2012, 05:39:58 PM
RELEASE AVAILABLE

The latest release can always be found here:
https://github.com/lrq3000/openarena_engine_serversidedemos

Also this patch is now ported to the experimental OA 0.8.8 + ioquake3 r2224 (any new update will be done exclusively here):
https://github.com/lrq3000/openarena_engine_serversidedemos/tree/latest

Unified diff patchs, precompiled binaries and a few other interesting patchs (rcon /tell, delay ...) can be found here:
https://github.com/lrq3000/openarena_engine_serversidedemos/downloads

This project is now considered stable. To check the latest news, please read the latest posts in this thread.

----
Hello guys,

I am currently trying to implement server-side demos. This thread is intended as a log of my progress and some of the questions that may arise from the different solutions that currently exists.

I know the devs are currently pretty busy with v0.8.8, and I don't intend to ask for help in code development.

----

My goal here is to provide a patch based on the latest OA binaries, found in this thread:
http://openarena.ws/board/index.php?topic=1933.0

I've focused on patching the binaries (instead of implementing that in a mod such as OAX), because I would like the feature to be compatible with any existing server and, if possible, mods. In summary: more portable, but less possible features. But since server-side demos is not something to be extended further than its minimum functionnality, I think that's ok (if the engine can generate and play server-side demos, that's enough for any use, and mods can still implement further features if needed).

Since I'm not really a specialist in hacking the ioquake3 engine, but I'm good at coding and algorithms, I chose to find currently available implementations in other games using the ioquake3 engine (or a modified version), and backport these changes to OA.

Now the serious matter:

-----

First, here is a previous thread about this subject:
http://openarena.ws/board/index.php?topic=3790.0

Second, here are 2 papers about the demos files specification, which could be very useful for future development/optimization:
http://skuller-vidnoe.narod.ru/
https://github.com/madprof/alpha-ioq3/wiki/Demos

Third, it seems there are only two different types of implementations:

- Server-side demos that record client-side demos for each client (everything is done server-side, but from a client point of view). In this case, each demo is saved in a different demo file per player. Demos should be playable with an unmodified client binary (even with ioquake3).
This type of the server-side demo is the simpler because it's basically a copy-paste of the client code for recording a demo, but ported to the server code. Not very optimized.

- Server-side demos that record full server-side demos by merging all the clients' streams into one file. Advantage is that this results in a single file, which can possibly be more space efficient, and when played it's just like you're a spectator in a game you can't join: you can switch the player you're spectating during the playback. Disadvantage is that it needs a special function to playback those files, clients must have a patched client binary. May also spawns problem with complex playback features (eg: oamme or q3mme rewind function), but not tested yet.

-----

Currently there are 4 implementations available that can easily be ported, and I will divide them by type:

**** Multi client-side demos ****

- TheDoctor (ScrewOA's server admin) patch for OpenArena. Can be found here:
http://openarena.ws/board/index.php?topic=3206.msg34546#msg34546
But this one is incomplete.

Complete patch can be found here:
http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110

Basic Usage: sv_autodemo 1, demos are stored in /homepath/baseoa/demos

- AlphaIOQ3: a fork of ioquake3 by some UrT guys and with collaboration of people from other communities such as Defrag. Pretty easy patch to port, since it's based on ioquake3. Project can be found here:
https://github.com/madprof/alpha-ioq3

Server-side patchs that interests us can be found here (in the order of patching):
https://github.com/madprof/alpha-ioq3/commit/45d78bda1adfd37a274a43bbf80dd3c9224fb779
https://github.com/madprof/alpha-ioq3/commit/94b77f3fb224df02371c952f6f1153c8abbfa71b
https://github.com/madprof/alpha-ioq3/commit/93ef087410157195b1c45b5e0145bfcc0401754e

Basic Usage: startserverdemo all, demos are stored in /homepath/baseoa/serverdemos
More infos here: https://github.com/madprof/alpha-ioq3/wiki

**** Fully server-side demos (merged in one file) ****

- Tremulous/Mercenarie's guild mod patch by Amarieu. This patch is harder to backport. Made in mid-2008. Here are the links:
https://dnc.tremforges.net/trac/ticket/127
http://patches.mercenariesguild.net/index.php?do=details&task_id=196
http://patches.mercenariesguild.net/index.php?getfile=804
http://code.google.com/p/fsm-trem/issues/detail?id=97

Newer resources with some fixes (2009):
http://code.google.com/p/fsm-trem/source/detail?r=178

Basic Usage: sv_autodemo 1, demos are stored in /homepath/baseoa/svdemos
More infos: http://tremulous.net/forum/index.php?topic=9278.0

IMPORTANT:
When recording:
sv_maxclients = whatever you want
sv_democlients = 0

When playing:
sv_maxclients = sv_maxclients of when you were recording + the max number of spectators
sv_democlients = sv_maxclients of when you were recording

Excerpt from a pm copy/pasted here:
http://tremulous.net/forum/index.php?topic=11170.msg167344#msg167344

- TremFusion patch by Amarieu. Made in the end of 2008. Theoretically the latest, but the 2009 commit in fsm-trem maybe implements newer fixes by other contributors (there's at least one). TremFusion patch can be found here:
http://patches.mercenariesguild.net/index.php?getfile=859
http://tremulous.net/forum/index.php?topic=9278.0
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296

But it seems that from reading the following post, the latest version is not the diff file but the direct patch to the qvm:
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296#p880

Note: the correct version is in Lakitu7's QVM, the version in p-g-qvm is buggy:
http://tremulous.net/forum/index.php?topic=9262.msg142914#msg142914

Another resource (don't know the version nor real release date, but it was done after the FSM patch at the very least):
http://www.assembla.com/code/arcade/git/nodes/src/server/sv_demo.c?rev=4b6ec00ac33ac847e825139609eb5a7d9ff584e5

Basic Usage: sv_autodemo 1, demos are stored in /homepath/baseoa/svdemos
More infos: http://tremulous.net/forum/index.php?topic=9278.0
http://www.tremfusion.net/wiki/Features_list#Server-side_demos

UPDATE: Also, the TremFusion project is officially dead (maybe it will revive someday?):
http://www.tremfusion.net/forum/viewtopic.php?f=11&t=350

This means that we can safely port the code to OA since there won't be any major changes in the future, and that it's considered pretty stable (at least, useable, we'll also have to test the ported code).

Update2:
Another patch by SlackersLinux (from Amanieu's code):
http://code.google.com/p/slackers-qvm/source/browse/branches/patches/02-SvDemo-Slackersqvm-1.1.patch?spec=svn99&r=99

- MVDSV for QuakeWorld: a specific engine written exclusively to support server-side demos, but is written for QuakeWorld:
http://qw-dev.net/projects/show/4


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 28, 2012, 05:52:28 PM
DEVLOG
--------

**** TheDoctor's patch for OA ****

Code comments:
Fixed the patch (missing header declarations, missing end of the patch) and applied it successfully.

The code is not very much optimized, it's basically a copy/paste of the client's record functions, and I think there may be some collisions and bugs, not to mention that it's not optimized. And I have to mention that the server recording functions did have the same names as clients functions (eg: CL_SaveRecord) and these had to be updated because the compiler would not compile the patch otherwise (eg: SV_SaveRecord).

Result: Demos are saved, but they are empty, containing only the headers and end of the demos specs. No data inside. Dunno if I did something wrong, maybe I'm missing more at the end of the patch than what I guessed. Or maybe I just didn't do the right config or tests for the demos to be recorded effectively, if what mentioned Falkland is true:

Quote from: Falkland
And I think it can be improved by adding a cvar for tuning parameters that activate autorecord : as now autorecord starts when a player has reached 0.9*fraglimit
excerpt from: http://openarena.ws/board/index.php?topic=3790.msg34583#msg34583

Then that may explains why I didn't get any output (I tried only with CTF, no fraglimit).

Conclusion: I think TheDoctor should be contacted about that to ask him to make a cleaner patch if his implementation is chosen.

Update: a complete version of the patch can be found here (thank's to Gig and r9k):
http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110

I'll update my port and see if that works.

----
Update: I've updated with the complete patch above and fixed some minor bugs, and it works. Demos are recorded and playable with a standard OA installation.

Here are the files:

Patch from engine v28: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/TheDoctor-serverside-demo_v0.4.patch
Full engine sources patched: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/openarena-engine-source-0.8.x-28-thedoctor-serversidedemopatch_v0.4.zip
Win32 binaries: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/TheDoctor-serverside-demo_engine-r28_v0.4-win32binaries.zip
An example demo produced: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/E0C277B1_oasago2_0001.dm_71

I won't do anymore work on this path. It fully works, and team chat messages as well as voip aren't recorded (and of course nor IP and GUID since it produces client-side demos).

/Update: I've taken another look over TheDoctor's patch, and it's quite interesting. Instead of hacking every engine's function that manages players and entities, it only hacks 3 functions: SV_WriteSnapshotToClient() and SV_UpdateServerCommandsToClient() and SV_SendClientSnapshot().

In clear terms: it stores and reproduces network snapshots packets instead of storing commands. This is quite an interesting approach, and it should be possible to modify its mechanism to allow for full server-side demos recordings (where you can freefly and spectate all players with a single demo), but I predict this would spawn some problems about how to manage these democlients and filtering of privacy data (because here this is raw data...). Also, there's no management of userinfo, only configstrings. Also, I think it's impossible to record bots this way (they have no network packet sent).

Maybe this path should be explored a bit to see if that's a better alternative than the modified Amanieu's patch.

**** AlphaIOQ3 server-side patchs ****

Did not implement it yet, but I don't think I will since it's pretty much straightforward to patch OA with these. Also, the feature is described as very experimental (but still it should be more mature than TheDoctor's patch), and also has no cvar to autorecord at map change (unlike TheDoctor's patch). But if everything else fail, I will try it.

Update: Maybe a merge between AIOQ3 and TheDoctor's patch (mainly to add the sv_autorecord function) would make a good implementation? Since they are very similar in both their codes and approachs, it should not be that difficult (I think).

Update2: a merge is unnecessary since these two patchs work pretty much the same, there are just some bits that are changing. Most notably, AIOQ3 does not offer a way to record automatically  demos (you have to issue a "startserverdemo all" everytime a player connects for the new player to be recorded...
So TheDoctor's patch is far superior to the AIOQ3 implementation in this respect. Anyway, TheDoctor's patch only records clients if they reach a score of 0.9*fraglimit, this should be fixed in the code in order to provide a real server-side demo recording.
Anyway, AIOQ3 provides a much cleaner hook inside sv_snapshot.c, maybe this should be ported back to TheDoctor's patch (but TheDoctor's way is much simpler, it's simply a mirroring of the normal messages with some added bits for the demo recording). However, the "force full frames every now and then" is very weird, this should probably be avoided, being a dirty hack that uses a constant that is not really well defined nor does explain why we should do that (so I guess here the authors did not really knew what they were doing here).

So in conclusion, not that interesting, except for the nice filenaming (but that's not hard), and clean sv_snapshot.c changes. But TheDoctor's patch should keep the preference for a server-side per client demo recording.

**** Tremulous FSM (Mercenarie's guild) patchs ****

Code comments:
The patch was not complete (some headers and functions were missing, mostly due because they are probably also used for other parts of Tremulous), but I could fix everything and compile it.

Result:
Demos seem to be recorded ok (size grow as time goes), but when trying to play the demo, an error is thrown:

Code:
\demo_play 20120128234413-ps37ctf
ERROR: Couldn't open svdemos/20120128234413-ps37ctf.svdm_71 for reading.

Don't know if this is because it can't find the file, or it really can't read the demo because it's corrupted. Going to investigate this.

Update: can't make it work. I tried to change the function to FS_FileExists, and still can't open it. But the demo is found if I try the /demo command, so there is a problem in the patch. Can't find where and why, arg. Nasty little bug.

Update2: it works, but I've not yet made the real fix, only a workaround (the fix is not hard, will do tomorrow). A stupid negative condition that checks a filename against an extensions blacklist...

Now the demo is found, in the right path (svdemos folder), loads the sv_demo, loads the map and the other files, but get stuck at "Awaiting Gamestate". Don't know if that's because my demo wasn't long enough (only recorded a few seconds for each, quick tests...) or if it's a bug of processing the playback because there are other missing functions (but it should have warned at compilation, so that's unlikely - or maybe an already existing function but missing some updated features that were not in the patch?).

I'll see later today.

Update3: I've ported some of the tremfusion changes (tremfusion v0.99r3, the latest version as of this writing), and it works better (no crash or hangup when loading the demo anymore), but now it seems that the demo can't load the map. Working on it.

Update4: ported qcommon/msg.c from tremfusion to openarena, now the demos are recorded with the right encoding and when played back, they load the right map and the other files. Unluckily, it gets stuck after the AAS loading, don't really know why. At least it gets stuck clientside (when using openarena_x86).

But when playing the demo serverside (when using oa_ded), it goes beyond: it loads the map (as done clientside), but it fully loads every other parameter, and the server becomes playable. But it's playable as a normal server, the demo doesn't get played back.

I noticed in the code that it's checking for some parameters, and if not the case, reload the map and wait with a command called "delay" (that doesn't exists in OA, only "wait" command exists), then retry to playback the demo. This is where I'm currently investigating.

So it seems that a sv_demo can be played back on a server, and dispatched to connected players. That would potentially be a great replacement for GTV if that's true!

Update5: Almost working: now it replays the demo, but the players movements aren't quite played. But at least the map is loaded and chat messages are displayed.

Quite some work has to be done before it will fully work, but at least there's some hope, and I'm grasping more and more of the engine and the sv_demo mechanism.

Update6: Even more working: loads the demo, the map, and plays it! At least, it replays most of the things: replays the events, replay the weapons, BUT it doesn't draw any entity such as players, flags or powerups yet XD But it WORKS! And indeed, it's possible to connect to a server which is replaying the demo. So it can be used as a GTV. GREAT! Also, I don't know if that's a bug or a feature, but it's possible to freefly in the game!

Update7: I ported the gamecode patch for replaying demos (this allows to prevent players from joining the game while the demo is replaying, and a few other adjustments like drawing with cp when a recorded player connects in the demo), but this doesn't solve the problem. Plus, even if I fixed most problems with the patch, we now get a "A reliable command cycled out" bug, which means that we didn't recorded some important frames at the beginning of the sv_demo, or at least the demo doesn't send them. But if you reconnect to the server replaying the demo, you can then watch the demo, but still no player shown, only actions and weapons and items.

So I got back to the patching of the engine, I leave the gamecode aside for the moment since it won't help to fix this problem, which is now obviously serverside (this is not a problem of rendering the players but of loading their config string).

I think I have a clue, it seems the server isn't recording the config strings. I'm looking for a way to fix it (because it is already implemented, just buggy). Hard and tedious, but it seems we finally see the light.

Update8: It WORKS! Not fully though, but the players are shown, along with their models. So my last clue was right after all! There are still a few more problems to work out, but the serverside code is almost complete. The last main problem is that players are not shown in the scoreboard, so that we can't spectate them, but we can freefly and actions are fully restituted. But I think that the client-side gamecode (that I already ported) should do the trick if it gets a bit fixed.

I have to pause this project for some weeks, so the code will be posted here as a proof of concept meanwhile for those adventurous enough to test it.

Update9: So here it is, the patch is up. Here are the files:

Amanieu's port of server-side demo patch for the engine v28: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_engine-r28_v0.4.7.6.patch
Full sources of engine v28 patched: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/openarena-engine-source-0.8.x-28-amanieu-sv-serversidedemo_v0.4.7.6.zip
Win32 binaries: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_engine-r28-win32-bin_v0.4.7.6.zip
Example of a demo produced (can be played back with demo_play with patched binaries): http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/demotest.svdm_71
Note: you should place this demo in baseoa/svdemos/ folder.
Note2: when replaying this demo, you should set sv_fps 25 (no any other value!) for the demo to be as fluent as when it was recorded (this will be fixed later).

Gamecode port (clientside) - Does NOT really work yet!
Gamecode patch over OAX B50: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_gamecode-B50_v0.3.patch
Full sources: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/openArenaExpansionB50-slackervm-serversidedemo-patched3.zip

What was done:
x port delay command
x re record demos with new msg.c
x ou porter de TremFusion patch de Amarieu en version compatible, par rapport a Tremulous (mais pas serverside demos?)
x autoset sv_democlients and sv_maxclients and save the old values (de toutes facon on restart la map, donc ca charge la config)
x singleplayer clientside demo_play awaiting gamestate bug when trying to replay a demo
x autoset bot_minplayers and fix bug value

Update10: Trying to patch the client-side gamecode. It's promising, now the bots and players are indeed shown in the scoreboard, and their scores (frags) is OK. The captures scores still aren't (was not implemented for Tremulous since there's no CTF gametype there, so we'll have to do it ourselves). And still can't spectate demo players, neither do say messages work, but at least it now prevents real players from joining in during the demo.

Update11: Ok it now WORKS! One can spectate players, freefly, see the full scoreboards (with ping, but it doesn't get updated after the first snapshot, but at least that gives an idea for demo spectators) and all actions are now handled by the server as if it was played right now (before many actions were performed but not loggued, wasn't very clean). Still the capture score is to be fixed and chat messages too, but at least now the patch is getting very functional!

Update12: I'm getting away from the original tremfsm and tremfusion patchs, I'm completely dropping the gamecode (which was in fact a server-side gamecode patch, not a client-side as I initially thought), and I'm trying to make a server-side demos functionnality completely handled by the engine. This would theoretically allow for more portability to other games, and also it would theoretically allow to use this feature for any mod.
I have implemented a few new functions and I'm extending the demos recording to a lot of other servers functions so as to accurately reproduce the events that happened in the real game. In the end it should be (almost) a full copy of the original game.
Chat messages are now fixed, also are the captures scores, along with a few other things.
Now the last main problem is the player status, should be fixed soon enough (hopefully).

TODO:
x chat messages are not recorded (or at least not displayed)
x client-side demo replaying bug: if while the demo is replaying, we leave to the menu, and try to launch another demo, it will crash (because values are latched but not updated)
x can't spectate (no democlients in the scoreboard nor teams) -> test with patched cgame -> NOT WORKING still same bug
Fake a real player by using something similar to: svs.clients[num].state = CS_ACTIVE;
x captures are not updated on the scoreboard (but they are announced ingame and in the console)
x really fix .sv_dmx extension detection (simply bypassed for now, see files.c)
x set sv_democlients as readonly with CVAR_ROM? -> tried and it works serverside, but for clientside replaying it crash
x save sv_fps in the demo and restore it in playback (and auto restore the normal value after the playback with savedSvFPS)
x save g_gametype
x IMPORTANT: clean up the patch to remove patchs that are maybe unnecessary (in the process). And add a lot of comments to clarify all this mess!
x set sv_democlients to 0 at startup (and always set to 0 when stoppingplayback? and 0 before recording? no more latched?) -> if sv_democlients is readonly, this fixes this problem
(for now it's set to 0 at first launch, but not at each subsequent startup)

FINAL UPDATE: a lot have been added and fixed, the patch is now pretty much stable. To see everything that was done, please check out the other pages of this thread, this is where you will find the rest of the discussion, because the previous updates in this post are up to about v0.4.7.

You can now download the stable release above on GitHub, and all new releases will be done there. There's also a full readme with all the changes that were done, and some of the planned features that may make it in someday (or not).

**** TremFusion patch ****

Not implemented yet, but I will, and we'll see later how it compares to the FSM patch (there are a great lot of things that change, patched files are not even the same at all).

I will port the patch from the Lakitu7's QVM since it seems to be the latest Amarieu has done from reading this post:
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296#p880

Update: The patch seems to be a lot more different than the FSM patch, and seems to apply only to Tremulous. Plus, it seems it's only an update (maybe of the FSM patch?), lots of functions declarations are missing.

So it seems that first we have to make the FSM patch work before going any further.

Update2: I have patched the FSM version with some of the tremfusion changes (every /server files are currently patched, but I didn't yet take a look at /qcommon nor /game files). It seems a complete version resides here:

http://www.assembla.com/code/arcade/git/nodes/src/server/sv_demo.c?rev=4b6ec00ac33ac847e825139609eb5a7d9ff584e5

Update3: the tremfusion patch can in fact be considered as an extension of the FSM patch: FSM patch being an engine patch, and tremfusion a gamecode patch (mainly for client-side replaying). See FSM devlog above.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 28, 2012, 05:56:12 PM
QUESTIONS
-----------

Now guys, if you had the courage to read all of what I've written above, I would like to know what you are thinking about these two questions:

1- What is the best type of server-side demo for you? Standard per-client demos or server-side demos?

2- Which patch is the newest, particularly between Tremulous FSM and TremFusion? (I will add more informations later so please leave this question aside for the moment).

PS: I'm creating multiple posts so that I get spaces to update them as I go along my tests.

PS2: I will provide unified patchs for everything I could implement (successfully or not), from the latest OA binaries (r28 from Sago's repository).


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 29, 2012, 01:13:52 AM
HELP: Do someone know if subfolders have to be specified somewhere, like the demos folder? I tried to crawl the code but could not find any declaration.

/EDIT: Ok I think I got it, testing...
/EDIT2: Ok got it, stupid condition that checks if a file is valid if it has the right extension. Viva comments (there are none that makes sense about that - thx ioq3 guys)!
Personal reminder: always avoid negative conditions, very awful to debug for someone else (or comment!).


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on January 29, 2012, 03:33:45 AM
1- What is the best type of server-side demo for you? Standard per-client demos or server-side demos?

2- Which patch is the newest, particularly between Tremulous FSM and TremFusion? (I will add more informations later so please leave this question aside for the moment).
1 - Server-side demoes that does not bind to a particular client. They should be clean enough to distribute: They should not contain rcon commands, admin commands or GUIDs.

2 - I have no idea.

There are a few things that I don't expect you to look at but if you stumbles across something it would be great:
An option to delag hitscan weapons in the server demo. Normally the server draws the shots when it receives them from the client because it does not know about them before. However during a replay the server could potentially know when the client hit the trigger and draw it then.

I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on January 29, 2012, 06:55:53 AM
GrosBedo, that "r9k" guy posted on the wiki DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/Talk:Manual/Demos#Server-side_demos]here (http://([b). He mentioned a link that you already wrote here (http://patches.mercenariesguild.net/index.php?do=details&task_id=196 server-side demos for Tremolous), and another one.... http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 29, 2012, 01:34:08 PM
1 - Server-side demoes that does not bind to a particular client. They should be clean enough to distribute: They should not contain rcon commands, admin commands or GUIDs.
From what I understand from the amarieu's code, it does not bind to any particular client, everything is fully done server-side with server-side packets.

About the cleaning of the data: I will see what I can do, but my main goal is to get something that works, I may not be able to implement this kind of filtering due to my lack of knowledge of the engine.

But I agree with this specification. These sort of data should be filtered.

There are a few things that I don't expect you to look at but if you stumbles across something it would be great:
An option to delag hitscan weapons in the server demo. Normally the server draws the shots when it receives them from the client because it does not know about them before. However during a replay the server could potentially know when the client hit the trigger and draw it then.

I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.

For the delag, I think that this should not be saved in the file but simulated in the playback (because anyway we can't simulate the delag for every player at once since this is a subjective, per-player shift). So this can be later added client-side as a toggleable cvar. ExcessivePlus implements a similar feature for their multi-view demo, so this is certainly possible.

If I can do it I will for sure, but I don't think I will be able.

GrosBedo, that "r9k" guy posted on the wiki DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/Talk:Manual/Demos#Server-side_demos]here (http://([b). He mentioned a link that you already wrote here (http://patches.mercenariesguild.net/index.php?do=details&task_id=196 server-side demos for Tremolous), and another one.... http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110

Thank's a lot Gig (and r9k)! Indeed, the patch provided in the rainbow's forum is complete, and indeed the end of the patch that I "guessed" was wrong, there are a lot of other modifications. This patch then should work, I'll update my port and see if that works better.


Title: Re: The server-side demos thread, rebooted!
Post by: PopeJo on January 30, 2012, 01:32:00 AM
**** TheDoctor's patch for OA ****

Quote from: Falkland
And I think it can be improved by adding a cvar for tuning parameters that activate autorecord : as now autorecord starts when a player has reached 0.9*fraglimit

the doctor !!?? falkland!!!??? they are still around?
havent seen any of them for a year or so. you are quoting old stuff, right?

if not, send them some greets   :D

/ontopic: and good luck with the server-side demos project  :)
can't really help you with your questions there..


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 30, 2012, 03:25:17 PM
I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.

Currently there's no facility for saving and reloading the same files, nor there seems to be for client-side demos. So I think that for now I won't be able to implement such a thing because I don't know the whole mechanism of file management and demos encoding, but if this feature gets implemented for client-side demos I may be able to port it to server-side demos.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on January 30, 2012, 03:33:29 PM
I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.

Are you talking about fs_referencedList (i.e. a list of used pk3s with names and checksums)? I don't quite see the advantage of this.
During playback you most often like to use different pk3s with brightskins, hd texture packs or replacement player models etc. Even if you had a list of pk3s being used, you'd still have to download them somehow if they're not on your pc already. Finally it would render OA demos incompatible to Q3/ioq3. If you're talking about complete server-side demos it would not matter that much, since those including all player data have their own custom file format anyways.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 31, 2012, 02:22:02 PM
For the sv_demo to work, I have to port some of other necessary commands such as /delay (which is similar to /wait but doesn't stop the engine from processing other commands meanwhile, and this is necessary for how sv_demo works, although I think this behaviour could be fixed without /delay but I'll leave it to someone else to do).

Interestingly, I stumbled upon other interesting extensions of the engine, such as a /random command to get a random integer, and other scripting extensions such as variables substitutions.

If I ever finish the server-side demo port, I may make a few other patchs for these scripting facilities if someone is interested. Also, these changes should (theoretically) work for mods since it's made in the binaries.

/EDIT: The TremFusion project is officially dead! Okay, this may not seem to be a very great news, but this means for us that we can safely port the TremFusion code because there won't expect major changes in the future.

/EDIT2: There are more and more sources that seems to point that indeed, a serverside demo from Amanieu's patch can be joined by players! So this could be used as a GTV server! We'll see if I can make it work! /EDIT: indeed, it seems it can be used as a replacement for GTV by loading a serverside demo on a server, and let players to connect in the server!


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on January 31, 2012, 04:58:38 PM
Sago, is there a command to queue another command to be executed later? I saw that there is a sort of queue implemented, how does it work? It can be either an ingame command or a C function, I can use both.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 01, 2012, 11:53:49 AM
Why do you need delayed command execution anyways?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 01, 2012, 04:21:16 PM
Why do you need delayed command execution anyways?

Because when you're playing a server-side demo, if the server do not meet the requirement, it executes a /devmap, and then a delay, and send again the server-side demo playback. So the delay needs to not stuck the whole engine, else it won't load the map before playing again the sv_demo, resulting in some weird errors.

That's why I think there can be other ways around, but for now I simply ported the delay command to make it work. When it will, we'll see how to enhance the code.

----

Back to the subject, every events really work, flags and weapons and items are shown, but characters are not displayed (neither bots nor players). This is weird: you see every shots and every action is in the log, but there's noone XD And it's possible to join in the game during the demo. This is also weird XD

I think I will now have to alter the client to make the server-side demos replaying totally functional, but it seems that server-side it fully works now.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 02, 2012, 11:20:57 AM
I've got a problem with bots, they're aren't properly managed in the demo (their actions are recorded but not their names and related player infos), as opposed to players which are managed properly.

I saw that OA added several changes to the behavior of bots.

Could someone (sago?) describe me succinctly those changes to help me understand how these work and why they were done?


Title: Re: The server-side demos thread, rebooted!
Post by: Graion Dilach on February 02, 2012, 11:48:52 AM
The changes were done by dmn_clown. You must understand that a huge part of the botcode wasn't released under the GPL2 (the entire content of the botfiles folder) and dmn_clown aimed for a more complexed botlogic than Mr Elusive for more challenge.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 02, 2012, 03:35:48 PM
Thank you Graion, I feared this answer, because dmn_clown is not active anymore in this project.

I'm mainly talking about the code in game/ and server/ folders, because it seems that not only the bot logic was changed but also the game logic and how bots are considered for the engine, so they are now kind of "special" in regards to the engine, which is not very good for server-side demos.

Well, I guess I'll have to find a way on my own.


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on February 02, 2012, 03:52:51 PM
I'm mainly talking about the code in game/ and server/ folders, because it seems that not only the bot logic was changed but also the game logic and how bots are considered for the engine, so they are now kind of "special" in regards to the engine, which is not very good for server-side demos.
I think id had a single programmer to create the bot code and he worked separately from the rest. The bot code seems hooked into the game in a very weird way. This is not new in OpenArena. The only difference I have made is that bots now look at the client struct to see what team they are on because sometimes that bot logic and server logic would get out of sync and the bot would attack teammates.


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 02, 2012, 04:04:46 PM
sometimes that bot logic and server logic would get out of sync and the bot would attack teammates.
I experienced this! Probably it is not yet totally fixed...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 02, 2012, 05:34:41 PM
@sago: Thank you for the clarification, but this is weird, because there are many bots functions that I can't find in the tremulous or tremfusion code that is also based on id tech 3:

in game/g_public.h:
Code:
BOTLIB_SETUP = 200, // ( void );
BOTLIB_SHUTDOWN, // ( void );
BOTLIB_LIBVAR_SET,
BOTLIB_LIBVAR_GET,
BOTLIB_PC_ADD_GLOBAL_DEFINE,
BOTLIB_START_FRAME,
BOTLIB_LOAD_MAP,
BOTLIB_UPDATENTITY,
BOTLIB_TEST,

BOTLIB_GET_SNAPSHOT_ENTITY, // ( int client, int ent );
BOTLIB_GET_CONSOLE_MESSAGE, // ( int client, char *message, int size );
BOTLIB_USER_COMMAND, // ( int client, usercmd_t *ucmd );

BOTLIB_AAS_ENABLE_ROUTING_AREA = 300,
BOTLIB_AAS_BBOX_AREAS,
BOTLIB_AAS_AREA_INFO,
BOTLIB_AAS_ENTITY_INFO,

BOTLIB_AAS_INITIALIZED,
BOTLIB_AAS_PRESENCE_TYPE_BOUNDING_BOX,
BOTLIB_AAS_TIME,

BOTLIB_AAS_POINT_AREA_NUM,
BOTLIB_AAS_TRACE_AREAS,

BOTLIB_AAS_POINT_CONTENTS,
BOTLIB_AAS_NEXT_BSP_ENTITY,
BOTLIB_AAS_VALUE_FOR_BSP_EPAIR_KEY,
BOTLIB_AAS_VECTOR_FOR_BSP_EPAIR_KEY,
BOTLIB_AAS_FLOAT_FOR_BSP_EPAIR_KEY,
BOTLIB_AAS_INT_FOR_BSP_EPAIR_KEY,

BOTLIB_AAS_AREA_REACHABILITY,

BOTLIB_AAS_AREA_TRAVEL_TIME_TO_GOAL_AREA,

BOTLIB_AAS_SWIMMING,
BOTLIB_AAS_PREDICT_CLIENT_MOVEMENT,

BOTLIB_EA_SAY = 400,
BOTLIB_EA_SAY_TEAM,
BOTLIB_EA_COMMAND,

BOTLIB_EA_ACTION,
BOTLIB_EA_GESTURE,
BOTLIB_EA_TALK,
BOTLIB_EA_ATTACK,
BOTLIB_EA_USE,
BOTLIB_EA_RESPAWN,
BOTLIB_EA_CROUCH,
BOTLIB_EA_MOVE_UP,
BOTLIB_EA_MOVE_DOWN,
BOTLIB_EA_MOVE_FORWARD,
BOTLIB_EA_MOVE_BACK,
BOTLIB_EA_MOVE_LEFT,
BOTLIB_EA_MOVE_RIGHT,

BOTLIB_EA_SELECT_WEAPON,
BOTLIB_EA_JUMP,
BOTLIB_EA_DELAYED_JUMP,
BOTLIB_EA_MOVE,
BOTLIB_EA_VIEW,

BOTLIB_EA_END_REGULAR,
BOTLIB_EA_GET_INPUT,
BOTLIB_EA_RESET_INPUT,


BOTLIB_AI_LOAD_CHARACTER = 500,
BOTLIB_AI_FREE_CHARACTER,
BOTLIB_AI_CHARACTERISTIC_FLOAT,
BOTLIB_AI_CHARACTERISTIC_BFLOAT,
BOTLIB_AI_CHARACTERISTIC_INTEGER,
BOTLIB_AI_CHARACTERISTIC_BINTEGER,
BOTLIB_AI_CHARACTERISTIC_STRING,

BOTLIB_AI_ALLOC_CHAT_STATE,
BOTLIB_AI_FREE_CHAT_STATE,
BOTLIB_AI_QUEUE_CONSOLE_MESSAGE,
BOTLIB_AI_REMOVE_CONSOLE_MESSAGE,
BOTLIB_AI_NEXT_CONSOLE_MESSAGE,
BOTLIB_AI_NUM_CONSOLE_MESSAGE,
BOTLIB_AI_INITIAL_CHAT,
BOTLIB_AI_REPLY_CHAT,
BOTLIB_AI_CHAT_LENGTH,
BOTLIB_AI_ENTER_CHAT,
BOTLIB_AI_STRING_CONTAINS,
BOTLIB_AI_FIND_MATCH,
BOTLIB_AI_MATCH_VARIABLE,
BOTLIB_AI_UNIFY_WHITE_SPACES,
BOTLIB_AI_REPLACE_SYNONYMS,
BOTLIB_AI_LOAD_CHAT_FILE,
BOTLIB_AI_SET_CHAT_GENDER,
BOTLIB_AI_SET_CHAT_NAME,

BOTLIB_AI_RESET_GOAL_STATE,
BOTLIB_AI_RESET_AVOID_GOALS,
BOTLIB_AI_PUSH_GOAL,
BOTLIB_AI_POP_GOAL,
BOTLIB_AI_EMPTY_GOAL_STACK,
BOTLIB_AI_DUMP_AVOID_GOALS,
BOTLIB_AI_DUMP_GOAL_STACK,
BOTLIB_AI_GOAL_NAME,
BOTLIB_AI_GET_TOP_GOAL,
BOTLIB_AI_GET_SECOND_GOAL,
BOTLIB_AI_CHOOSE_LTG_ITEM,
BOTLIB_AI_CHOOSE_NBG_ITEM,
BOTLIB_AI_TOUCHING_GOAL,
BOTLIB_AI_ITEM_GOAL_IN_VIS_BUT_NOT_VISIBLE,
BOTLIB_AI_GET_LEVEL_ITEM_GOAL,
BOTLIB_AI_AVOID_GOAL_TIME,
BOTLIB_AI_INIT_LEVEL_ITEMS,
BOTLIB_AI_UPDATE_ENTITY_ITEMS,
BOTLIB_AI_LOAD_ITEM_WEIGHTS,
BOTLIB_AI_FREE_ITEM_WEIGHTS,
BOTLIB_AI_SAVE_GOAL_FUZZY_LOGIC,
BOTLIB_AI_ALLOC_GOAL_STATE,
BOTLIB_AI_FREE_GOAL_STATE,

BOTLIB_AI_RESET_MOVE_STATE,
BOTLIB_AI_MOVE_TO_GOAL,
BOTLIB_AI_MOVE_IN_DIRECTION,
BOTLIB_AI_RESET_AVOID_REACH,
BOTLIB_AI_RESET_LAST_AVOID_REACH,
BOTLIB_AI_REACHABILITY_AREA,
BOTLIB_AI_MOVEMENT_VIEW_TARGET,
BOTLIB_AI_ALLOC_MOVE_STATE,
BOTLIB_AI_FREE_MOVE_STATE,
BOTLIB_AI_INIT_MOVE_STATE,

BOTLIB_AI_CHOOSE_BEST_FIGHT_WEAPON,
BOTLIB_AI_GET_WEAPON_INFO,
BOTLIB_AI_LOAD_WEAPON_WEIGHTS,
BOTLIB_AI_ALLOC_WEAPON_STATE,
BOTLIB_AI_FREE_WEAPON_STATE,
BOTLIB_AI_RESET_WEAPON_STATE,

BOTLIB_AI_GENETIC_PARENTS_AND_CHILD_SELECTION,
BOTLIB_AI_INTERBREED_GOAL_FUZZY_LOGIC,
BOTLIB_AI_MUTATE_GOAL_FUZZY_LOGIC,
BOTLIB_AI_GET_NEXT_CAMP_SPOT_GOAL,
BOTLIB_AI_GET_MAP_LOCATION_GOAL,
BOTLIB_AI_NUM_INITIAL_CHATS,
BOTLIB_AI_GET_CHAT_MESSAGE,
BOTLIB_AI_REMOVE_FROM_AVOID_GOALS,
BOTLIB_AI_PREDICT_VISIBLE_POSITION,

BOTLIB_AI_SET_AVOID_GOAL_TIME,
BOTLIB_AI_ADD_AVOID_SPOT,
BOTLIB_AAS_ALTERNATIVE_ROUTE_GOAL,
BOTLIB_AAS_PREDICT_ROUTE,
BOTLIB_AAS_POINT_REACHABILITY_AREA_INDEX,

BOTLIB_PC_LOAD_SOURCE,
BOTLIB_PC_FREE_SOURCE,
BOTLIB_PC_READ_TOKEN,
BOTLIB_PC_SOURCE_FILE_AND_LINE

} gameImport_t;

Code:
BOTAI_START_FRAME // ( int time );
} gameExport_t;

All these functions and variables do not exist in the trem code. But maybe trem is based on a previous version of id tech 3, so this may explain why.

Anyway, I think I now found the root of the problem of why the players are not shown. If at least I can properly draw players, we'll see later for bots.

sometimes that bot logic and server logic would get out of sync and the bot would attack teammates.
I experienced this! Probably it is not yet totally fixed...

For me, it happened a few times but only because I wasn't recognized as a player of the team (I was in TEAMFREE), so the bots, logically, considered me as an enemy. You can easily spot this bug when you see that all the bots get the same color (or the normal color if you didn't force model) whatever their team is (eg: for me all bots were red, even blue ones, and I could also hit them).

This is a very very rare bug (and I didn't modify the binaries when it happened, and it happened with all versions of OA I know of).


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 02, 2012, 05:46:05 PM
there are many bots functions that I can't find in the tremulous or tremfusion code that is also based on id tech 3
Quote from: Tremulous FAQ
Does Tremulous have bots?
No. Programming bots for a game as complex as Tremulous is not a trivial task. It is by no means impossible, but it is potentially extremely time consuming for questionable real world gain. Of course, given the open source nature of Tremulous, there is nothing stopping a third party implementing bots.
Maybe they've just removed those lines if they dropped botsupport? id Quake 1.32c still/already has these :)


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 03, 2012, 01:42:00 AM
For me, it happened a few times but only because I wasn't recognized as a player of the team...
Not sure about how it worked exactly when I noticed the problem. I was trying to test some variables that seem to -in theory- control console chats lasting (one seemed for team chat duration, in particular)... so I loaded oa_ctf4ish and added some bots, giving them team orders to make them answer via team chat. But they didn't reply (maybe later someone -maybe Graion or Chaoticsoldier?- told me that OpenArena bots chat does not include team chat yet!)... and they totally ignored my team orders (they did not follow me when I asked them to follow me, etc.) -so I was unable to really test those variables, maybe I could do some more tests when I have some time...-. More, IIRC, strange things happened like bots tending to all join the same team if I added them without specifying one, and attacking me when I was on their team. But time passed, and I should do more tests to be more detailed.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 04, 2012, 10:50:37 AM
UPDATE: It WORKS! Not fully though, still some bugs (eg: we can only freefly), but players infos and actions are fully restituted. So the server-side code is almost complete, only client-side gamecode will have to be fixed.

I'm going to clean a bit the code and try to fix some minor bugs, and then I'll post the patch + full sourcecode tonight.

I will be off for about 2 weeks, so I'll pause this project meanwhile, but the code will allow the most adventurous here to try it and see for themselves how it works.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 04, 2012, 12:31:10 PM
sv_maxclients and sv_democlients are now automatically set! Next: client-side demo replaying. Almost works, but one has to load the demo twice before it loads up. Shouldn't be too hard to fix :)

I've got a question to the devs around here: how to set a cvar as read-only? I would like to set sv_democlients to readonly since now it will be automatically set.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 04, 2012, 12:44:44 PM
Code:
sv_demoClients = Cvar_Get ("sv_demoClients", "", CVAR_ROM);


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 04, 2012, 01:21:29 PM
Thank you Grey Matter :)

But lol I thought this function was used to fetch the value of a cvar. So I've got one more question: could someone tell me how to fetch the value of a cvar by name?

/EDIT: found it:

Code:
Cvar_VariableIntegerValue()

I hope this is the right way to do it!

/EDIT2: ok I changed my mind about setting sv_democlients as read-only, because then, even the engine cannot change its value, and that's bad. I think I leave it as it is for now, but in the future this cvar could even be deleted and fully managed serverside (it's already managed though).


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 04, 2012, 01:57:36 PM
Well, it depends on where you are trying to query the cvar. The engine and gamecode have slightly different cvar structures and access methods.

If you have registered (i.e. bound to a cvar_t struct) your cvar with Cvar_Get() e.g. in sv_init.c, you can then just use cvar->value, cvar->integer or cvar->string to access it directly. You should however not modify these fields directly, rather use Cvar_Set().
If you did not register the cvar, you can use Cvar_VariableIntegerValue(), Cvar_VariableString(), Cvar_VariableValue() or Cvar_VariableStringBuffer().

If you have registered (vmCvar_t) your cvar with trap_Cvar_Register() e.g. in g_main.c, you can then use cvar->value, cvar->integer or cvar->string (and trap_Cvar_Set()) as well.
If you did not register the cvar, you can use trap_Cvar_VariableIntegerValue() or trap_Cvar_VariableStringBuffer().


Regarding your edit; the engine (and gamecode using it's trap_Cvar_Set()) can set any cvars using Cvar_Set(), even those with CVAR_ROM flag. Those cvars are inteded to be changed by the game logic, but not by the user.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 04, 2012, 03:03:02 PM
@grey matter: ok thank's a lot for the explanation.

Anyway about CVAR_ROM, I have to use Set_ValueLatched() (a new function that does the same as Set_Latched but with Set_Value preprocessing), and it didn't seem to work, so maybe CVAR_ROM + CVAR_LATCHED variables can't be modified even by the engine.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 04, 2012, 07:05:52 PM
Takes me a little longer than expected but I'm fixing most of the bugs. I'll post the sources and patch tomorrow.

/EDIT: do someone know how to force latched values to get set?

From sv_ccmds.c:

Code:
// force latched values to get set
Cvar_Get ("g_gametype", "0", CVAR_SERVERINFO | CVAR_USERINFO | CVAR_LATCH );

//Notice that we have done a restart
sv_dorestart->integer = 0;

But it doesn't work...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 05, 2012, 11:59:35 AM
Files are up! You can download all the ports I've done on the first thread.

I've posted the Amanieu's port and TheDoctor's fixed patch, along with win32 binaries (I don't have a linux machine at this moment sorry, but anyway you can compile them by yourself).

Also, I've ported the Smokin' Guns patch (which is itself ported from Urban Terror) to enable /rcon tell:

Rcon tell patch for engine r28: http://zsensible.free.fr/openarena/rconpatch/rcontellpatch-engine-r28.patch
Full source with engine r28 patched: http://zsensible.free.fr/openarena/rconpatch/openarena-engine-source-0.8.x-28-rcon-tell-patch.zip
Win32 binaries: http://zsensible.free.fr/openarena/rconpatch/rcontell-engine-r28-win32binaries.zip

-----

About server-side demos, it fully work server-side, so I don't think there will be much changes. Now the problem is that demo players are NOT considered as players (this was done on purpose), but thus they are not showing on the scoreboard and so there's no way to spectate them or update the scoreboard (captures are not update even if they are announced, see the provided demo example).

I am going to be off for about 2 weeks so the project will be in standby meanwhile.

Anyway, I ask for your ideas on how to fix this scoreboard problem.

I think we have 2 ways to fix that problem:

- Patch gamecode (clientside) to correctly manage democlients
- Patch server (engine) to simulate democlients as real players

It seems TremFusion and Amanieu chose the first solution, and they patched the gamecode to correctly manage democlients. I ported the code into OAX B50, but this produce a bug (see the patch and sources in the first post). I don't know why, but I could see that even without any patch, OAX B50 prevents the player from seeing the scoreboard or join the game, so it seems OAX B50 is a bit buggy, so maybe it was not a very good idea to try to patch this version.

About the second way, I also tried but players are just empty for the engine, so I think that simulating democlients as real players would imply a huge deal of changes in the engine, because the engine want to deal with network functions with real players, where for democlients there aren't any, everything is local.

I've done as far as I could with my low knowledge of the engine, now I need some help about what is possible currently, because I don't know at all how to manage those democlients.

Anyway, please test out my serverside patch and tell me if everything works for you. Please remember that after recording serverside demos with Amanieu's code, you also need a patched binary to replay them.

Commands:
/sv_autoDemo
/demo_play
/demo_record
/demo_stop


/EDIT: I've found a bug, you must set sv_fps 25 for the demo example I posted so that it's replayed as fluently as when it was recorded (it's easily fixable by saving the sv_fps value in the demo, I'll do it later). So when replaying a demo, sv_fps must be the same as when the demo was recorded. Else, players will be a bit jerky in their movements.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 16, 2012, 01:59:23 PM
I've just noticed that my patches aren't very clean, because my editor tends to automatically delete useless spaces and reformat the tabs (which is a good thing usually, but for a patch it's not).

I will post better patchs later.

/EDIT: client's gamecode patching works a bit, it's promising. See the devlog (in the 2nd post of the first page) for more details.


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on February 18, 2012, 06:09:34 AM
I've just remembered one thing.

If this fix would have been included in 0.8.8, it would have rendered Gig's benchmark demo as useless. :S


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 18, 2012, 06:56:12 AM
Which benchmark?


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on February 18, 2012, 07:31:19 AM
There's one in 0.8.8, in the demos folder inside of the pk3.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 18, 2012, 07:35:48 AM
What's the use of this benchmark (never heard of it before)? I don't understand why my patch would be a redundancy of Gig's work, it seems goals are entirely different.

/EDIT: ah ok I think you confused the threads. I guess you're here talking about the compatfix found in this thread:
http://openarena.ws/board/index.php?topic=4443.msg43061#msg43061

Did you try it? Does it work ok for you?


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on February 18, 2012, 08:19:18 AM
Gig did it way before the content freeze, last year.

http://openarena.ws/board/index.php?topic=4336.0
http://openarena.ws/board/index.php?topic=1945.msg41092#msg41092


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 18, 2012, 08:29:33 AM
Thank you for the links :)

Please excuse me if I sound stupid, but I still don't understand the correlation with server-side demos? What must be noted is that server-side demos are managed by an entirely different code than normal demos (at least for Amanieu's code), so that server-side demos can't really be used for benchmarking (I think).


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on February 18, 2012, 09:00:50 AM
Oh, then I might have been confused about the 0.8.X demo compatibility. -.-


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 18, 2012, 04:59:47 PM
Still working on the patchs, and I've got a big problem here.

The fact is, democlients get indeed put in the right team (at least they are SHOWN on the right team), but for the engine they are in fact considered as spectators! I know why they are spectators, it's in the code, all players get put to spectator at the beginning of the frame, and there's no code provided (at least not working as is) to put democlients into the right team:

The following code is an excerpt from game/g_main.c and this function is called everytime G_RunFrame is called:
Code:
void CheckDemo( void ) {
int i;

// Don't do anything if no change
if( g_demoState.integer == level.demoState ) {
return;
}
level.demoState = g_demoState.integer;

// empty teams and display a message
if( g_demoState.integer == DS_PLAYBACK ) {
trap_SendServerCommand( -1, "print \"A demo has been started on the server.\n\"" );
for( i = 0; i < level.maxclients; i++ ) {
if( level.clients[i].sess.sessionTeam != TEAM_SPECTATOR ) {
SetTeam( &g_entities[ i ], "spectator" );
}
}
}
}

As you can see, every player gets put into spectator at the beginning of the demo:
Code:
SetTeam( &g_entities[ i ], "spectator" );

But then, I don't understand why democlients (with a status of spectator) can be put in the right team (without having the status of being in the team)!

Do someone has any idea of why or how this can happen? Maybe it's in the config string (which is loaded)? But I didn't see any code that would make a player be drawn in a team without using SetTeam or .sessionTeam field... And I checked that there's no other function using SetTeam or .sessionTeam that would do this...

/EDIT: Sample of a bot config string:
Code:
n\Grism[b]\t\1\[/b]model\sarge/classic\hmodel\sarge/classic\c1\4\c2\5\hc\100\w\0\l\0\skill\ 4.00\tt\2\tl\0

So yes it seems it comes from the config string. The config string tells the engine that the player is in a team, but I expected the engine to automatically switch the player to the right team, when it doesn't. Weird.


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 18, 2012, 06:01:41 PM
Yes, the fix that would break compatibility with demo088-test1 (bundled with 088) is the one about 0.8.1 demo compatibility. My demo (intended to allow all players to have a common demo to use as a benchmark) was recorded before content freeze. In case a fix will disable 085 and current 088 demo compatibility, we should record a new demo to replace demo088-test1.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 18, 2012, 06:56:10 PM
@Gig: yes indeed, a new demo will have to be recorded with the compatibility patch. Sorry :/

@all About server-side demos, great news, IT WORKS! And the greatest news: NO DIRTY GAMECODE HACK! (ok I know gamecode hacking is better generally than engine hacking, but here I did it on purpose, because it should allow more interoperability).

Ok so now, spectating players WORKS, scoreboard WORKS, and even ping is shown (the first ping at connection, it never gets updated, but I don't think that's a big deal).

The bad news is that it still don't count the captures and don't show the chat, but replaying demos just WORKS!

I will continue to work on the code to fix the above stated issues, finish the todo list to polish the thing, and finally clean the patch from useless hacks.

@dev: I'm getting more and more the hang on the code, now I begin to understand how communication between clients and servers work, but could someone explain me a bit more about that (it will help me to enhance the patch)? I saw that configstrings commands and VM_Call are the two primary means of communication, how does that work exactly? And are there any other mean of communication?


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 19, 2012, 09:09:19 AM
You're requesting an abstract picture of the whole thing while there hardly exists any documentation :)

Here's a very rough picture; Engine and gamecode communicate via trap or system calls. You are interested in the engine part in SV_GameSystemCalls(), which is mapped as trap_Foo() in the gamecode. The gamecode part in vmMain() is exported by QVMs and shared libs, so the engine can call the predefined functions with VM_Call()

The engine thinks in fixed time slices (controlled by sv_fps), the main logic is in Com_Frame(). SV_Frame() just calls the gamecode with GAME_RUN_FRAME.
Other gamecode callbacks are clients connecting, disconnecting, the server shutting down/restarting, client commands, client "thinks", userinfo changes etc. The gamecode calls the engine for all the complex work (file access, collision detection, etc), especially when it comes to the botlib parts.

Apart from that, the engine and gamecode don't really communicate. The engine dictates a common struct for clients, everything else is left to the gamecode (take a look at e.g. SV_LocateGameData() and all the hardcoded structs in msg.c).

Ideally, the engine should not know and care about anything the gamecode does. There are some bad examples like the map_restart code, g_needpass in infoResponse etc.
What do you mean by "configstrings commands"?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 19, 2012, 10:12:11 AM
Thank's a lot Grey Matter for those infos! Yes I know there's hardly any documentation (I've already read pretty much everything that exists over the web about the engine, and also read the comments in the sourcecode), that's why I asked about it here :) Your explanation helps me a lot!

So, if I understand correctly what you wrote, this means that communication between client and server is basically the same as a communication between engine and gamecode? That's why if I do a VM_Call, it broadcasts the message accross all clients?

About configstrings, here are a few comments I found in the sourcecode.

In bg_public.h:
Code:
// config strings are a general means of communicating variable length strings
// from the server to all connected clients.

In g_public.h:
Code:
G_SET_CONFIGSTRING, // ( int num, const char *string );
// config strings hold all the index strings, and various other information
// that is reliably communicated to all clients
// All of the current configstrings are sent to clients when
// they connect, and changes are sent to all connected clients.
// All confgstrings are cleared at each level start.
Code:
//
// functions exported by the game subsystem
//
typedef enum {
GAME_INIT, // ( int levelTime, int randomSeed, int restart );
// init and shutdown will be called every single level
// The game should call G_GET_ENTITY_TOKEN to parse through all the
// entity configuration text and spawn gentities.

GAME_SHUTDOWN, // (void);

GAME_CLIENT_CONNECT, // ( int clientNum, qboolean firstTime, qboolean isBot );
// return NULL if the client is allowed to connect, otherwise return
// a text string with the reason for denial

GAME_CLIENT_BEGIN, // ( int clientNum );

GAME_CLIENT_USERINFO_CHANGED, // ( int clientNum );

GAME_CLIENT_DISCONNECT, // ( int clientNum );

GAME_CLIENT_COMMAND, // ( int clientNum );

GAME_CLIENT_THINK, // ( int clientNum );

GAME_RUN_FRAME, // ( int levelTime );

GAME_CONSOLE_COMMAND, // ( void );
// ConsoleCommand will be called when a command has been issued
// that is not recognized as a builtin function.
// The game can issue trap_argc() / trap_argv() commands to get the command
// and parameters.  Return qfalse if the game doesn't recognize it as a command.
...
} gameExport_t;

This seems to mean that a lot of the communication between the clients and the server relies on those configstrings.

Also, as a self notes, here are some others means of communication (in g_public.h):
Code:
//
// system traps provided by the main engine
//

G_ARGC, // ( void );
// ClientCommand and ServerCommand parameter access

G_ARGV, // ( int n, char *buffer, int bufferLength );

G_SEND_CONSOLE_COMMAND, // ( const char *text );
// add commands to the console as if they were typed in
// for map changing, etc

G_SEND_SERVER_COMMAND, // ( int clientNum, const char *fmt, ... );
// reliably sends a command string to be interpreted by the given
// client.  If clientNum is -1, it will be sent to all clients

G_GET_USERINFO, // ( int num, char *buffer, int bufferSize );
// userinfo strings are maintained by the server system, so they
// are persistant across level loads, while all other game visible
// data is completely reset

Code:
//
// functions exported by the game subsystem
//

GAME_CONSOLE_COMMAND, // ( void );
// ConsoleCommand will be called when a command has been issued
// that is not recognized as a builtin function.
// The game can issue trap_argc() / trap_argv() commands to get the command
// and parameters.  Return qfalse if the game doesn't recognize it as a command.

In server.h:

Code:
//
// sv_init.c
//
void SV_SetConfigstring( int index, const char *val );
void SV_GetConfigstring( int index, char *buffer, int bufferSize );
void SV_UpdateConfigstrings( client_t *client );

void SV_SetUserinfo( int index, const char *val );
void SV_GetUserinfo( int index, char *buffer, int bufferSize );

Code:
//
// sv_snapshot.c
//
void SV_UpdateServerCommandsToClient( client_t *client, msg_t *msg );
void SV_WriteFrameToClient (client_t *client, msg_t *msg);
void SV_SendMessageToClient( msg_t *msg, client_t *client );
void SV_SendClientMessages( void );
void SV_SendClientSnapshot( client_t *client );

In sv_client.c:
Code:
SV_UpdateUserinfo_f
SV_UserinfoChanged

/EDIT: do someone know what's the use of SV_WriteFrameToClient? I could not find a concrete use of it in the code... unnecessary command?


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 19, 2012, 11:05:22 AM
So, if I understand correctly what you wrote, this means that communication between client and server is basically the same as a communication between engine and gamecode? That's why if I do a VM_Call, it broadcasts the message accross all clients?
Ah, I thought you were refering to communication between engine and gamecode in one instance, e.g. a server.
Network communication works different, and to be honest I don't know much about it. There should be some documentation about it, back from the Quake 3 reverse engineering days. There are reliable commands, "normal" commands, out-of-the-box messages and so on. You should better ask this in the ioquake3 mailing list or IRC.
VM_Call() does no broadcasting, it's the generic way the engine calls functions in the gamecode (vmMain()). Do you mean something like SV_SendServerCommand(null, "foo") (a reliable command btw, also see CL_AddReliableCommand())?


This seems to mean that a lot of the communication between the clients and the server relies on those configstrings.
Configstrings are the means to communicate rather large text from server to all clients (where you don't want to use reliable commands for each client).
Engine configstrings contain information about cvars with CVAR_SYSTEMINFO and CS_SERVERINFO flag.
The gamecode maintains further configstrings for votes, items, models, sounds, players etc. The engine knowns nothing about them, it just provides the functionality to transmit them for the gamecode.

/EDIT: do someone know what's the use of SV_WriteFrameToClient? I could not find a concrete use of it in the code... unnecessary command?

SV_WriteFrameToClient() is indeed unused, only declared in headers. You can post it as a bug at ioquake3 if you wish :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 19, 2012, 02:24:32 PM
Thank you again Grey Matter for those infos, it enlightens me a lot.

-----

The following will contain notes about my current problem, as a note to myself in an attempt to see clearer into this, but if you are a developper and have an idea that I could try out feel free.

I have 3 current problems:
- Democlients do not get updated: if they stay in the same state as when they connected (except for frags and flagstatus that ARE updated, both on the player that holds it and on the scoreboard):
* The ping stays the same.
* If they switch team, the change is not reflected in the scoreboard, and they are not even considered to belong to the new team (an hint about this is that the team beacon do not get shown over their head - or the beacon of the other team, if they switched team during the demo recording).
* If a player or bot connects during the demo is recording, they are not shown in the scoreboard (their name is displayed in actions, you can see their player in the demo, but they can't be seen in the scoreboard - don't know if we still can spectate them?).

- Flag captures aren't displayed in the scoreboard (but they are announced both by text and by voice and sound).

- Chat messages aren't displayed (how can a player that is not updated can say anything?).

It must also be noted that if the demo started after some frags and flagcaptures were done, the frags are restored in the demo, but not the flagcaptures.

All the above leads me to think that the three problems have some correlations, and that they all points toward a scoreboard updating problem. But it seems the server do NOT really manage the scoreboard. So I'm very much puzzled. I can't even find anything about flag captures in servers files.


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on February 19, 2012, 02:43:36 PM
Scoreboard updates and chat messages are communicated through reliable messages. Scoreboard updates are not broadcaster to all players at once but calculated for every single client. I don't know why it is calculated for every single client because it is the same message being sent to all clients. The scoreboard updates are sent in g_cmds.c in "DeathmatchScoreboardMessage".

How do you determent what reliable messages the demo watcher should receive?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 09:14:26 AM
The scoreboard updates are sent in g_cmds.c in "DeathmatchScoreboardMessage".

Ah ok that's what I thought (but weird that the name is "Deathmatch" when it's used for any gametype). Does it also include flagcaptures?

And do you know how I can use it in a server/ file? (eg: from sv_main.c)? I don't really know if a SV_SendServerCommand could be used to issue such a command... I'll take a look on the mechanisms anyway.

How do you determent what reliable messages the demo watcher should receive?

I don't, in fact the server-side demo records all the entities placement and actions, and also server commands and players config strings at each iteration of SV_Frame in sv_main.c. But it seems that the problem is that it doesn't record everything, so that it doesn't record when there is a ClientBegin nor a ClientUserInfoChanged, and so probably other things are not recorded.

Then, when the demo is replayed, the server tries to replay all the events that happened in the demo, by restoring their context (there's a function for that):

Code:
typedef struct cmdContext_s
{
int argc;
char *argv[ MAX_STRING_TOKENS ]; // points into cmd.tokenized
char tokenized[ BIG_INFO_STRING + MAX_STRING_TOKENS ]; // will have 0 bytes inserted
char cmd[ BIG_INFO_STRING ]; // the original command we received (no token processing)
} cmdContext_t;

static cmdContext_t cmd;
static cmdContext_t savedCmd;

/*
============
Cmd_SaveCmdContext
============
*/
void Cmd_SaveCmdContext( void )
{
Com_Memcpy( &savedCmd, &cmd, sizeof( cmdContext_t ) );
}

/*
============
Cmd_RestoreCmdContext
============
*/
void Cmd_RestoreCmdContext( void )
{
Com_Memcpy( &cmd, &savedCmd, sizeof( cmdContext_t ) );
}

I don't really understand yet how this can save the context but it seems it does the job...

After, when replaying a demo, for each demo frame, it processes all saved events:

Code:
/*
====================
SV_DemoReadFrame

Play a frame from the demo file
====================
*/
void SV_DemoReadFrame(void)
{
(...)
// Parse the message
while (1)
{
cmd = MSG_ReadByte(&msg);
switch (cmd)
{
default:
Com_Error(ERR_DROP, "SV_DemoReadFrame: Illegible demo message\n");
return;
case demo_EOF:
MSG_Clear(&msg);
goto exit_loop;
case demo_endDemo:
SV_DemoStopPlayback();
return;
case demo_endFrame:
// Overwrite anything the game may have changed
for (i = 0; i < sv.num_entities; i++)
{
if (i >= sv_democlients->integer && i < MAX_CLIENTS)
continue;
*SV_GentityNum(i) = sv.demoEntities[i];
}
for (i = 0; i < sv_democlients->integer; i++)
*SV_GameClientNum(i) = sv.demoPlayerStates[i];
// Set the server time
sv.time = MSG_ReadLong(&msg);
return;
case demo_configString:
num = MSG_ReadBits(&msg, CLIENTNUM_BITS);
tmpmsg = MSG_ReadString(&msg);
SV_SetConfigstring(CS_PLAYERS + num, tmpmsg); //, qtrue
VM_Call( gvm, GAME_CLIENT_BEGIN, num );
break;
case demo_serverCommand:
Cmd_SaveCmdContext();
tmpmsg = MSG_ReadString(&msg);
Cmd_TokenizeString(tmpmsg);
SV_SendServerCommand(NULL, "%s \"%s\"", Cmd_Argv(0), Cmd_ArgsFrom(1));
Cmd_RestoreCmdContext();
break;
case demo_gameCommand:
num = MSG_ReadByte(&msg);
Cmd_SaveCmdContext();
Cmd_TokenizeString(MSG_ReadString(&msg));
VM_Call(gvm, GAME_DEMO_COMMAND, num);
Cmd_RestoreCmdContext();
break;
case demo_playerState:
num = MSG_ReadBits(&msg, CLIENTNUM_BITS);
player = SV_GameClientNum(num);
MSG_ReadDeltaPlayerstate(&msg, &sv.demoPlayerStates[num], player);
sv.demoPlayerStates[num] = *player;
break;
case demo_entityState:
while (1)
{
num = MSG_ReadBits(&msg, GENTITYNUM_BITS);
if (num == ENTITYNUM_NONE)
break;
entity = SV_GentityNum(num);
MSG_ReadDeltaEntity(&msg, &sv.demoEntities[num].s, &entity->s, num);
sv.demoEntities[num].s = entity->s;
}
break;
case demo_entityShared:
while (1)
{
num = MSG_ReadBits(&msg, GENTITYNUM_BITS);
if (num == ENTITYNUM_NONE)
break;
entity = SV_GentityNum(num);
MSG_ReadDeltaSharedEntity(&msg, &sv.demoEntities[num].r, &entity->r, num);

// Link/unlink the entity
if (entity->r.linked && (!sv.demoEntities[num].r.linked ||
    entity->r.linkcount != sv.demoEntities[num].r.linkcount))
SV_LinkEntity(entity);
else if (!entity->r.linked && sv.demoEntities[num].r.linked)
SV_UnlinkEntity(entity);

sv.demoEntities[num].r = entity->r;
if (num > sv.num_entities)
sv.num_entities = num;
}
break;
}
}
}
}

As you can see in the case demo_configString, I simply added:

Code:
VM_Call( gvm, GAME_CLIENT_BEGIN, num );

Which is for the moment only a hack to force clients to process the democlient (and it works well), but I think the real problem is that the demo doesn't record all the events, or that it can't replay all the events.

---------------------

AH OK! I think I've got it.

In fact, the demos also relies in a later revision (the tremfusion latest revision) on a modified QVM to also save events happening in the gamecode, which are then processed when replaying the demo with a GAME_DEMO_COMMAND:

Code:
case demo_gameCommand:
num = MSG_ReadByte(&msg);
Cmd_SaveCmdContext();
Cmd_TokenizeString(MSG_ReadString(&msg));
VM_Call(gvm, GAME_DEMO_COMMAND, num);
Cmd_RestoreCmdContext();
break;

Until now, I thought that the QVM modification was only used for client-side processing of demos, but it's also used to save server-side game commands!

This is nice, but I would like to avoid hacking the gamecode as much as possible and only rely on engine, because in the end I would like this feature to be available for any mod, no matter if they load their own QVM. I'm sure there must be another way to do it.

Ok I'll try to hack further and see if this is indeed the real problem.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 10:48:20 AM
Damn I need to add a trapped command but it keeps on crashing with the message:

Code:
********************
ERROR: Bad game system trap: 46
********************
----- Server Shutdown (Server crashed: Bad game system trap: 46) -----
==== ShutdownGame ====

I really can't understand why.

Here are the changes:
in g_main.c (this is where the trap_DemoCommand is called and is the only place, if it's commented out it does not crash anymore):
Code:
/*
============
G_DemoCommand

Store a demo command to a demo if we are recording
============
*/
void G_DemoCommand( demoCommand_t cmd, const char *string ) {
if( level.demoState == DS_RECORDING ) {
trap_DemoCommand( cmd, string );
}
}

in g_syscalls.asm:
Code:
equ trap_DemoCommand                -47

in g_syscalls.c:
Code:
void trap_DemoCommand( demoCommand_t cmd, const char *string ) {
syscall( G_DEMO_COMMAND, cmd, string );
return;
}

Of course the G_DEMO_COMMAND is also declared.

As you can see, everything is declared, and the code for this game system trap is 47, not 46! So I really don't understand why the error is not the right code number. Also, if I change the code to 49, the error is then 48! So it seems the error is always offset by -1, is this normal?


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 20, 2012, 11:05:25 AM
I can't give you an explaination right now, but the offset seems to be normal behaviour :)


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on February 20, 2012, 11:17:17 AM
Is the trap caught in SV_GameSystemCalls in sv_main.c?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 12:06:02 PM
Is the trap caught in SV_GameSystemCalls in sv_main.c?

Yes it is:
Code:
case G_DEMO_COMMAND:
                if ( sv.demoState == DS_RECORDING )
                {
                        if ( args[1] == -1 )
                                SV_DemoWriteServerCommand( VMA(2) );
                        else
                                SV_DemoWriteGameCommand( args[1], VMA(2) );
                }
                return 0;

I also tried to comment out the usage of G_DEMO_COMMAND:
Code:
void trap_DemoCommand( demoCommand_t cmd, const char *string ) {
//syscall( G_DEMO_COMMAND, cmd, string );
return;
}

Still produces the same error, so the problem really happens when calling trap_DemoCommand and not when syscalling G_DEMO_COMMAND. Weird.


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on February 20, 2012, 01:10:18 PM
Have you tried in g_public.h to define G_DEMO_COMMAND like this:

Code:
G_DEMO_COMMAND = 50,


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 05:11:06 PM
No still same error, even if I set G_DEMO_COMMAND = 47. But thank's a lot for the ideas.

However since when I comment out the G_DEMO_COMMAND call it still produces the error, I think the problem resides somewhere else in the declarations of the trap_DemoCommand.

Anyway, I think I'll do something else. Even if it would be better to first restore the original functionality of the patch before editing it, I think I'll skip directly to trying to move the gamelogic of the demorecording to the serverside. I'll see how it works to move every G_DEMO_COMMAND call directly to the server-side.



Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 06:15:06 PM
Ok I directly hacked the recording of the game commands into the server, in sv_game.c:

Code:
intptr_t SV_GameSystemCalls( intptr_t *args ) {
switch( args[0] ) {
...
case G_SEND_SERVER_COMMAND:
if ( sv.demoState == DS_RECORDING )
{
if ( args[1] == -1 )
SV_DemoWriteServerCommand( VMA(2) );
else
SV_DemoWriteGameCommand( args[1], VMA(2) );
}
SV_GameSendServerCommand( args[1], VMA(2) );
return 0;
...

Which then gives me the following (with debug printings):

Code:
broadcast: print "Major^7 captured the RED flag!\n"
DebugGBOgameCommand: scores 4 0 2 36360 0 7 13 1 0 0 23 0 0 0 1 0 0 1 0 3 5 0 1 0 0 0 0 0 0 0 0 0 1 0 1 3 0 1 0 0 64 0 0 0 0 0 0 0 0 2 3 0 1 0 0 43 0 1 0 0 0 0 0 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 1  2 6 117 0 8 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 1  2 6 116 0 8 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 1 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 2 100 50 5 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 2 100 50 5 0
DebugGBOgameCommand: tinfo 1  2 6 115 0 8 0
DebugGBOgameCommand: tinfo 3  0 1 88 10 5 0 1 1 90 0 8 0 3 2 100 50 5 0
DebugGBOgameCommand: chat "[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."

Seems encouraging! It's not yet being broadcasted but I think I know why (the demos still relies on g_demo_command in clients vm to process the commands, but if we redirect these commands directly into another SV_SendServerCommand it should work I think).

Do someone know what is tinfo? I never saw that before... Seems like infos about a player, but of what kind?


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 20, 2012, 06:21:10 PM
tinfo is used for the team overlay, it includes your teammates' health, location etc (see TeamplayInfoMessage()).


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 06:23:13 PM
@Grey Matter: thank's for the info, I see... It seems that now all the previously missing data is now saved in the demo, just needs to replay them!

AH and another finding, I think I just understood the use of Cmd_SaveCmdContext() and Cmd_RestoreCmdContext() (added in the server-side demos patch).

I think it's used to save the current context of the system, before applying demos commands, and then restores the context so that normal commands can be processed (such as a real client connecting/disconnecting) without influence of the commands processed by demo.

An example:
Code:
case demo_gameCommand:
num = MSG_ReadByte(&msg);
Cmd_SaveCmdContext();
tmpmsg = MSG_ReadString(&msg);
Cmd_TokenizeString(tmpmsg);
VM_Call(gvm, GAME_DEMO_COMMAND, num);
Cmd_RestoreCmdContext();
break;


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 20, 2012, 06:26:25 PM
Just one more question: what does exactly Cmd_TokenizeString(msg) ? Such as on an example line like this:

Code:
chat "[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."

Does it separate the elements like:

Code:
chat
"[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."

?

But then what would it do with a long line with several values such as:

Code:
tinfo 1  2 6 117 0 8 0

?


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 20, 2012, 06:48:12 PM
It just parses a string into the argc and argv command buffers, which you can query with trap_Argc() and trap_Argv() in the gamecode and via Cmd_Argv() and Cmd_Argc() in the engine.

Quoted strings like in your example are treated in a special way (same goes for // and /**/ comments). Otherwise the string is just broken into pieces at spaces.

tinfo 1  2 6 117 0 8 0
should result in
argc 8
argv[0] tinfo
argv[1] 1
argv[2] 2

and so on.

chat "[Sarge^7] (Blue base): ^6UnnamedPlayer set up some D."
should result in
argc 2
argv[0] chat
argv[1] [Sarge^7] (Blue base): ^6UnnamedPlayer set up some D.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 21, 2012, 08:52:49 AM
@Grey Matter: thank's, that helps a lot!
-----
News: ok indeed now score and chat events are recorded, and I indeed made them replay using SV_SendServerCommand. Chat messages are shown well, but there's two small problems: chat messages are repeated as many times as there were players on the server at the time (because of the mechanism of G_Say() function: it sends the chat message to every client that should receive it) - which I'll fix by checking that the same say message wasn't already sent previously, and tell messages are shown too when they shouldn't (because they use the same command "chat" as normal say to all) - which I'm gonna fix by checking the primary color of the message (this is the only way I can think of because there's no other way to my knowledge to differenciate a sayall command and a tell command). Also, teamchats are recorded, but since they use a different command "tchat" I can easily filter them at recording. I'll do that when I'll fix the two other main problems described below.

Now my 2 main problems:
- scores are still freezed, flagcaptures are not updated (but I think it's related to the 2nd problem).
- democlient statuses are not updated (connection/disconnection/team switching aren't replayed). I think this is what also causes the scores freeze.

I'm now going to look towards fixing the second problem.

/EDIT: hypothesis: I think that the main reason that Armanieu done his gamecode patch is to workaround the problem of democlients management by giving them a special status of "demoClient = qtrue" and set them to the right team (or update them to the right team) with the G_DemoSetClient() function (which was called each time a client was set in the recording, either at ClientBegin or ClientUserinfoChanged). Then there are various checks (notably in the CalculateRanks() and cmd_FollowCycle_f() functions) that were modified to also consider democlients (eg: if ( level.clients[ clientnum ].pers.connected == CON_CONNECTED || level.clients[ clientnum ].pers.demoClient ) ...).

He probably done that because he could not find another way to simulate a client without producing networks errors.

But I _think_ it's possible to really simulate a client, proof is that by issuing a ClientBegin from the server's engine, the democlient are at least recognized and set to the right team, and we can even follow them (without any democlient check hacks). What I don't understand is why they are not updated, but I will find out.

----

Ah and VoIP is not recorded, nor GUID, nor IP. The only thing recorded are the configstrings and they seem to not contain any personal information, please tell me if I'm wrong so that I can fix it:

Code:
DebugGBOconfigString: n\UnnamedPlayer\t\2\model\sarge/default\hmodel\sarge/default\g_redteam\\g_blueteam\\c1\1\c2\5\hc\100\w\0\l\0\tt\0\tl\1


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 21, 2012, 10:59:30 AM
New idea! I think I'll try to hack SV_ExecuteClientCommand(), it should be a lot easier to filter teamchat and tell chat, and it should help in switching players to teams.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 21, 2012, 11:46:05 AM
The only thing recorded are the configstrings and they seem to not contain any personal information, please tell me if I'm wrong so that I can fix it:

Code:
DebugGBOconfigString: n\UnnamedPlayer\t\2\model\sarge/default\hmodel\sarge/default\g_redteam\\g_blueteam\\c1\1\c2\5\hc\100\w\0\l\0\tt\0\tl\1

There's no personal information in the player configstrings/userinfo at all. Besides they are distributed to the connected clients anyways (try /configstrings while connected to a server). You can take a look at the end of ClientUserinfoChanged() in the gamecode to see how the player configstrings are generated.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 21, 2012, 12:18:32 PM
You can take a look at the end of ClientUserinfoChanged() in the gamecode to see how the player configstrings are generated.

Yes I've read it, I'm quite crawling the whole code lately :p

In fact, I've found where the personal informations are transmitted to the server: in the clientCommand userinfo. I will have to be careful with this one if I record client commands.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 21, 2012, 02:50:48 PM
Ok! Chat is now working (check...). Plus, tell, teamchat and userinfo are filtered (at recording time - so they are not saved at all). But my other problems are not solved!

2 problems left:

- Player status updates
- Captures/scores updates

Player statuses seem to be a bit more complicated than what I thought, but I'm getting more and more used to the engine. I think I can completely simulate a player, but first I'm going to see if there are other workaround this, but as a last resort, I may implement demoClient in a way such that they will be considered as real players.

Self-note about player statuses non-update:
- Frag score IS updated
- Flag hold IS updated and shown next to the player icon both in the HUD (when spectating the player) and on the scoreboard.
- No capture update
- No ping update
- No switch team update
- NEW: if a player connects, he CAN be spectated, but its player infos are not processed (no model, no name - defaults to UnnamedPlayer - and no health update - but if player was already connected, its health is updated!). However, ammo number is updated (?!).
- NEW: bots camera is not updated (stays on the same direction when spectated - for players it's ok!) - could not remember this bug before, maybe it appeared with my modifications? EDIT: tested with an older binary, bots' camera isn't moving...
- NEW: persistent score captures (player->persistant[PERS_CAPTURES]) is not saved in the demos nor updated with demos events. Probably the other persistent scores are also not updated nor saved.
- NEW: player status update is really a problem. Apart from the ammo and armor, the rest is NOT updated neither in scoreboard nor HUD (except for teaminfo HUD but that's because it's now handled). For example, health is NOT updated!

/EDIT: another self-note:

To summary the big picture, the gamecode rules all the game mechanisms. The engine (ideally) only runs server stuffs and general functions (such as basic network communication - but it's the gamecode that manages the clients and game statuses!).

In multiplayer, the client's gamecode communicate with the server gamecode, which communicate with the server's engine (using SV_GameSystemCalls() in sv_game.c) - the engine then processes the required stuff (without really knowing the exact content), then pass it back to the server's gamecode, which automatically calls the required network functions to broadcast the new game status to every client.

If I'm wrong please correct me :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 22, 2012, 05:32:59 PM
Fix: teaminfos hud (and in fact all gamecommands) are now played back.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 22, 2012, 06:08:02 PM
Guys, do you know a way to access gamecode vars from the engine or is that just plainly impossible (without hacking the engine a lot)?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 23, 2012, 08:57:06 AM
Found a clue! About captures scores, I think the following strings aren't recorded nor played back:

Code:
#define CS_SCORES1 6
#define CS_SCORES2 7
#define CS_FLAGSTATUS 23 // string indicating flag status in CTF

This should fix the 3rd main problem on the 4 that are remaining.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 23, 2012, 10:14:25 AM
YES! The last clue was right, flag status (taken, dropped, at base) and capture score fixed!

Now one last main problem left: player status! But I think I'm on the right track :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 23, 2012, 02:47:24 PM
Still working on flagstatus, because captures made previously to the demo recording start weren't reloaded (they were refreshed after another capture was made, but this made the capture score jump: eg: 0 -> 2). Now all configstrings initial state are recorded and restored at the beginning of the demo.

Yet a bug persists: the capture score isn't persistent. After a few seconds, a new CS_SCORES1 and CS_SCORES2 are sent with both values set at 0. This is the server's gamecode that is doing this. I think I don't properly update the configstrings, so they are not persistent.

I'll now try to fix this. Maybe it'll also fix some of the player status bugs.


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 23, 2012, 03:14:26 PM
GrosBedo... seeing that you are making an huge work to have serverside demos working... but are you doing all this "from scratch"? I thought the serverside demos systems already existed in some third-party unofficial executables (that you listed in the first posts), and that you had to import one of them into the official executables... but it looks like you are needed to re-do everything...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 23, 2012, 07:32:48 PM
@Gig: At first this project started as a simple port of the existing patchs, either for OA or for other games such as Tremfusion (Tremulous).

The problem is that, along the way, I noticed a lot of short-comings with these solutions:

- the OA patch by TheDoctor is simply an aggregation of all client-side demos (so if you have 12 players, you'll have 12 demos, just for one game! And you simply can't switch from one player's perspective to the other, you have to stop the demo and rewatch a whole other demo just to watch another player).

- the ioquake3 Alpha project implements a similar idea as TheDoctor, with some advantages (eg: demos are named meaningfully eg: oasago2-someplayername-date.dm_71, while TheDoctor's are named something like Z12ild83.dm_71 ...).

- the tremfusion server-side demos patch by Amanieu was the closest to what I aimed: full server-side demos, where you can switch the perspective directly from the demo, and with meaningful names.

Unfortunately, Amanieu has not polished his patch, and so there are some important missing data that were not recorded in the demo (eg: g_gametype, how the hell can you get playback a demo if you don't even get back the g_gametype that was used?), and a lot of issues that were not fixed (such as managing automatically sv_democlients and sv_fps), this is why Amanieu's patch is known to be buggy, but at least working (it's the unique full server-side demo patch for ioquake3 to my knowledge).

Also the patch wasn't entirely on the engine (binaries) but also on the gamecode, so that mean that to use its server-side demos you have to both patch the binaries AND the vm, which also mean that it's not compatible with mods. That limits a lot its usefulness and "universality".

Plus imagine telling to clients "to replay demos, you have to use a special QVM that I'm not maintaining anymore, so you have to use OA 0.8.1 with my QVM and binaries to replay the demos, then to play you put back your backed up QVM of OA x.x ...).

----

So my solution is at first based on Amanieu's patch, and we have to thank him a lot, his code is genius. But I aim to extend its idea and not only record a demo based on some parameters: demos recorded with my patch will be an almost total reproduction of the game when it was played.

That's why it takes me so much time: when Amanieu simply workaround the problem of managing democlients (clients that are replayed in a demo) by giving them a special status in the gamecode, I am currently totally recording the democlients data and simulate these players as if they were real, by the way tricking the engine into thinking it's real too.

So yes, all these additions are done from scratch, from crawling tons and tons of ioquake3 code and trying everything I can think of.

But now I'm near the conclusion! I still have some problems managing correctly the player's statuses, but almost all functionnalities are here and recording/playing demos is very straightforward, everything is automatically managed!

In the end, my solution should follow these criterias:
- completely managed by the engine - OK
- completely retrocompatible with any version of OA (just copy the binaries, you can run any gamecode version) - OK
- compatible with mods (for example - hopefully we may be able to record Generations Arena server-side demos!) - not tested!
- completely multiviewpoints demos: can follow and even freefly around the map while the demo is played! - OK
- privacy respectful: should not contain any personal data (such as ip nor guid) - OK
- OATV: could be used to retransmit matchs as an opensource and native implementation of GTV (you can launch a demo on your server and all clients can see it live!) - OK

It will also suffer of the same problems as normal demos (but that's not bad, normal demos are well recorded!):
- can't check the map version, the client will have to manage it to get the right map to replay the demo right
- can't check the mod: the client will have to load the mod by himself prior to replay the demo

What is also nice, is that this facility is greatly expandable, and I will try to comment it to my best to explain everything I can (now that I understand it!), so we can imagine that in the future, it could be the base foundation for all the demos facilities and it could even be enhanced to fix all of the above concerns (it CAN save the mod, and save an MD5 checksum of the map, the problem then is how to load them, because mods can't be hotswapped server-side).

----

So now I hope that you see why I'm putting so much effort. I don't want to leave this half finished now I'm near the ultimate goal ;)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 23, 2012, 07:36:42 PM
Back to the matter, here are some debugs I've done:

Code:
clientid              connectvar                teamvar
DGBO TeamCvarSet: 0 connected=2 team=0
DGBO TeamCvarSet: 1 connected=2 team=0
DGBO TeamCvarSet: 2 connected=2 team=0
DGBO TeamCvarSet: 3 connected=2 team=0
DGBO TeamCvarSet: 4 connected=0 team=0
DGBO TeamCvarSet: 5 connected=0 team=0
DGBO TeamCvarSet: 6 connected=0 team=0

As you can see, the 4 first players are democlients, they are considered as CON_CONNECTED (the 2), but they are considered to be in the TEAM_UNKNOWN 0! This is very weird. And sometimes, it happens that they are switched to the right team, but it's very rare!

So I think it's the right track here :)


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 24, 2012, 02:08:05 AM
Wow!!!  :)
Anyway, about the hashes to find the right versions of pk3s (and thus, of maps and other files)... well, maybe you may make the demo store the list of used pk3 anyway, so when the player would in future play the demo, should play it correctly if it has got the required pk3s or more (e.g. playing a 0.8.1 demo with 0.8.8, not loading 0.8.5 and 0.8.8 pk3s). In case some pk3 will be missing on the player, of course the demo would not know where to download it... maybe it may try to play it anyway like it does now, but warning the user that some packs are missing so strange things may happen.
If working, maybe such coherency check may be extendend -optionally- to standard demos, who knows....


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 24, 2012, 07:01:22 AM
There's a config string that stores every pk3 loaded by the server at the time, but when I save this special field (it's considered as reserved by the engine), the engine crashes when it tries to replay a demo. So I think it may not be a good idea for now.

To ensure coherency of demo we will have to create our very own functions because for now there's not really any coherency check (normal client-side demos included).


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 24, 2012, 08:23:09 AM
Guys, do you know a way to access gamecode vars from the engine or is that just plainly impossible (without hacking the engine a lot)?

It's not impossible, but there's no sane reason to do this. Gamecode and thus offsets to structures change from mod to mod or even between versions of the same mod.
If you really want to go that way, peek around in some of the cheat forums or private clan forums for UrT, because runtime QVM modifications is the only real way for UrT server-side mods.

[..] (it CAN save the mod, and save an MD5 checksum of the map, the problem then is how to load them, because mods can't be hotswapped server-side).

You can easily restart the vfs. Hotswapping is not possible (and still a difficult topic in general. Read e.g. the kSplice whitepapers for some background info, though it'd be a little easier in ioq3). There's a /game_restart <modname> command in ioq3, so you don't even need to mod anything.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 24, 2012, 08:42:04 AM
You can easily restart the vfs. Hotswapping is not possible (and still a difficult topic in general. Read e.g. the kSplice whitepapers for some background info, though it'd be a little easier in ioq3). There's a /game_restart <modname> command in ioq3, so you don't even need to mod anything.

Mate, you've MADE MY DAY! I just tried it and it works!!! That's GREAT! This is a great command for all servers administrators, and I didn't even know its existence. Thank's a lot!

Gig, could you please add it to the wiki please? I don't know where this should belong, but it's important enough to be noted in the wiki.

----

About accessing gamecode cvars: ok Grey Matter, I don't want to do a dirty hack. As an alternative, is there a way to update the gamecode vars from the engine vars?

I ask this because I am now pretty sure that server engine's configstrings aren't synchronized with gamecode's configstrings, and this is the reason why there are weird stuff happening.

/EDIT: I really don't understand what's happening with configstrings so I will here detail further.

I will take the captures scores as an example.

My patch now sets correctly the configstrings at demo start and everytime there's an update. But every few moments (eg: just after a clientBegin), the CS_Scores get set back to 0 by an SV_GameSystemCalls G_SET_CONFIGSTRING (highlighted below by a "DGBO G_SET_CONFIGSTRING").

Code:
DebugGBOconfigString: 4 snaps 25 pmove_fixed 0
DebugGBOconfigString: 5
DebugGBOconfigString: 6 1
DebugGBOconfigString: 7 0
DebugGBOconfigString: 8
DebugGBOconfigString: 9
DebugGBOconfigString: 10
DebugGBOconfigString: 11
DebugGBOconfigString: 12
DebugGBOconfigString: 13
DebugGBOconfigString: 14
DebugGBOconfigString: 15
DebugGBOconfigString: 16
DebugGBOconfigString: 17
DebugGBOconfigString: 18
DebugGBOconfigString: 19
DebugGBOconfigString: 20 baseoa-1
DebugGBOconfigString: 21 29840
DebugGBOconfigString: 22
DebugGBOconfigString: 23 00
DebugGBOconfigString: 24
DebugGBOconfigString: 25
DebugGBOconfigString: 26
DebugGBOconfigString: 27 0010011011101111001101111000000000110000000000000000000000000
DebugGBOconfigString: 28
...
DebugGBOconfigString: 1021
DebugGBOconfigString: 1022
DebugGBOconfigString: 1023
Playing demo svdemos/demotest44.svdm_71.
DGBO G_SET_CONFIGSTRING: 6 0
DGBO G_SET_CONFIGSTRING: 7 0
DGBO G_CVAR_SET: g_humanplayers = 1
Item: 2 weapon_rocketlauncher
Item: 3 ammo_shells
Item: 1 ammo_rockets
Item: 1 weapon_rocketlauncher
Item: 1 ammo_rockets
Item: 3 ammo_cells
DebugGBOgameCommand: scores 4 1 0 39840 0 9 12 0 0 0 11 0 0 0 0 0 0 1 0 1 4 0 0 0 0 30 0 0 0 0 0 0 0 0 2 3 0 0 0 0 12 0 0 0 1 0 0 0 0 3 2 0 0 0 0 36 0 0 0 0 0 0 0 0
DGBO G_SET_CONFIGSTRING: 6 0
DGBO G_SET_CONFIGSTRING: 7 0

Note: configstring 6 is CS_SCORES1 and 7 is CS_SCORES2.

So I guess that the gamecode is using some trap somewhere to set back the engine configstrings, but this is weird, because I can't understand why it would set this value when the configstrings are stored in the engine.

So am I right? Configstrings are stored in the engine?


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 24, 2012, 09:12:21 AM
I will try some tests with that game_restart <modname> command in the next days...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 24, 2012, 09:38:04 AM
I will try some tests with that game_restart <modname> command in the next days...

Ok, I can tell you from my tests that to be used effectively, one needs to issue an /exec <config.cfg> just after because the config is completely unloaded, or just to do tests you at least need to load a map /map oasago2 for example. But it really hotswap the gamemod!

I'll try to include the gamemod in the demos files :)

-------

Another example of the resetting of the CS_SCORES:
Code:
DebugGBOgameCommand: print "UnnamedPlayer^7 captured the BLUE flag!
"
broadcast: print "UnnamedPlayer^7 captured the BLUE flag!\n"
DebugGBOserverCommand: print "UnnamedPlayer^7 captured the BLUE flag!
"
broadcast: print "UnnamedPlayer^7 captured the BLUE flag!\n"
DebugGBOconfigString: 6 2
DebugGBOconfigString: 23 00
DebugGBOgameCommand: scores 4 2 0 39840 0 14 13 1 0 0 11 0 0 0 0 0 0 2 0 1 5 0 1 0 0 34 0 0 0 0 0 0 0 0 2 3 0 1 0 0 12 0 0 0 1 0 0 0 0 3 2 0 1 0 0 43 0 0 0 0 0 0 0 0
DGBO G_SET_CONFIGSTRING: 6 0
DGBO G_SET_CONFIGSTRING: 7 0

/EDIT: ok I've located where it comes from, I've made a mistake just before, I misplaced my debug print.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 24, 2012, 09:44:29 AM
Grey Matter, a precise question: since there's no nice way to access gamecode vars from engine, can it be done the other way around? accessing engine's vars from gamecode?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 24, 2012, 10:12:23 AM
@Grey Matter, I'm still interested in the previous question even if now I've fixed the problem because it came from a bug ;)

--------------

Capture scores FIXED and player status FIXED too !!!!!! All main problems are FIXED NOW!!!!

For those that are interested, it simply came from a buggy check that Amanieu implemented:

Code:
void SV_SetConfigstring (int index, const char *val) {
int i;
client_t *client;

if ( index < 0 || index >= MAX_CONFIGSTRINGS ) {
Com_Error (ERR_DROP, "SV_SetConfigstring: bad index %i\n", index);
}

// Don't allow the game to overwrite demo player configstrings
if ( sv.demoState == DS_PLAYBACK && index >= CS_PLAYERS && index < CS_PLAYERS + sv_democlients->integer ) {
return;
}
...

As you can see, when SV_SetConfigstring was called to set a client configstring, it was simply ignored. This was a safeguard against the engine resetting configstrings (which was happening for CS_SCORES1/2 because I didn't add CS_SCORES1/2 in this check).

BUT, the problem is that in the demo, it was SV_SetConfigstring that was called to set democlients configstrings! So it was rejected, even if it was legit!

I simply moved this check directly in the trap calls:

Code:
case G_SET_CONFIGSTRING:
// Don't allow the game to overwrite demo configstrings
if ( sv.demoState != DS_PLAYBACK ) {
SV_SetConfigstring( args[1], VMA(2) );
}

And now it works beautifully, because it both prevents the game from overwriting the configstrings with irrelevant data, and accepts legit configstring setting from the demo! (edit: this fix above is not completely correct: it fixes the problem stated above but other problems are spawned such as allowing normal clients configstrings to be overwritten, else real clients connecting during a playback won't have any infos loaded inside, this is fixed in my local repository by adding a few more conditions in the check).

-------------

Ok so from now on, my plan is to fix a few other things, try the server-side demos with other gametypes, record a few complete games (with capturelimit/timelimit/fraglimit reaching), and add a few more important data recording (such as SendConsoleCommand).

The completed patch should be released soon!


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 24, 2012, 10:50:38 AM
since there's no nice way to access gamecode vars from engine, can it be done the other way around? accessing engine's vars from gamecode?

No, not directly. You can "access" anything that the engine wraps via it's trap calls. Or store your data in the common player struct, user commands, cvars or configstrings.
Maybe you find some buffer overflow, but usually engine and gamecode are not supposed to access each other's memory directly. Gamecode should not be able to modify any of the engine's memory (possible security problem), the engine on the other hand controls the entire gamecode (at least in QVM compile and interpret) environment, but does not know that e.g. player struct + offset 0xDEADBEEF is splashMethodOfDeath. You can of course access that very location of memory if your engine hack is just targeted at that specific gamecode, but you rather shouldn't.

Just to make sure, you are talking about e.g. structs/fields like svs.nextHeartbeatTime, svEntity_t.areanum in engine and level.voteExecuteTime, gentity_t.gentity in gamecode?
You can easily access and modify c-vars in both directions.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 24, 2012, 10:53:45 AM
@Grey Matter: ok thank's a lot! And what about client_t, is it also shared?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 25, 2012, 08:20:22 PM
MEMENTO: Understood another thing about player's status management: clients configstrings are derived from their userinfo string.

Clients configstrings are constructed in the gamecode, in g_client.c in function ClientUserinfoChanged( int clientNum ):

Code:
void ClientUserinfoChanged( int clientNum ) {
...
// send over a subset of the userinfo keys so other clients can
// print scoreboards, display models, and play custom sounds
if (ent->r.svFlags & SVF_BOT)
{
s = va("n\\%s\\t\\%i\\model\\%s\\hmodel\\%s\\c1\\%s\\c2\\%s\\hc\\%i\\w\\%i\\l\\%i\\skill\\%s\\tt\\%d\\tl\\%d",
client->pers.netname, team, model, headModel, c1, c2,
client->pers.maxHealth, client->sess.wins, client->sess.losses,
Info_ValueForKey( userinfo, "skill" ), teamTask, teamLeader );
}
else
{
s = va("n\\%s\\t\\%i\\model\\%s\\hmodel\\%s\\g_redteam\\%s\\g_blueteam\\%s\\c1\\%s\\c2\\%s\\hc\\%i\\w\\%i\\l\\%i\\tt\\%d\\tl\\%d",
client->pers.netname, client->sess.sessionTeam, model, headModel, redTeam, blueTeam, c1, c2,
client->pers.maxHealth, client->sess.wins, client->sess.losses, teamTask, teamLeader);
}

trap_SetConfigstring( CS_PLAYERS+clientNum, s );

// this is not the userinfo, more like the configstring actually
G_LogPrintf( "ClientUserinfoChanged: %i %s\n", clientNum, s );

And that explains a lot of things! Actually, this explains why players statuses couldn't be updated in the scoreboard (even if now I could do it by a clever manipulation of configstrings), why configstrings would get reset if I didn't prevent the engine from rewriting them (see my post above about CS_SCORES1/2), and why this damn ClientUserinfoChanged wasn't logical when I was debugging (could not find where these configstrings came from, now I know! from userinfo!).

Now it makes no doubt as to the cause why Amanieu's patch was so buggy, it left a big part of the players management aside.

Also as a side-note, all client_t fields (client_t is, with player_t and gentity, the 3 main vars for players management) can be almost completely filled using a single userinfo string by using the function SV_UpdateUserinfo_f() which:
1- store the new userinfo string to the client.
2- call SV_UserinfoChanged to fill the other fields of client_t by deriving from userinfo string and some other functions call (eg: IP is directly fetched from networks functions).
3- issue a vm_call to update server gamecode and broadcast change to every clients (GAME_CLIENT_USERINFO_CHANGED).

Also, there are 2 main entries in the engine to set userinfo:
- G_SET_USERINFO trap in sv_game.c, which then calls SV_SetUserinfo() in sv_init.c (but this function only store the new userinfo, step 2 and 3 aren't processed - I suspect it's an error because name is copied manually when SV_UserinfoChanged could be used instead to fill all the blanks, name included). Not sure about this one.
- SV_UpdateUserinfo_f() in sv_client.c, which does all the steps outlined above.
- It seems there's a third one, at map_restart or when a client connects (because of client->csUpdated? in SV_UpdateConfigstrings() ?).


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on February 26, 2012, 05:51:52 AM
Gig, could you please add it to the wiki please? I don't know where this should belong, but it's important enough to be noted in the wiki.

Okay, I wrote about the "game_restart <modfolder>" command here:
(DO NOT LINK) h t t p s : / / openarena . wikia . com/wiki/Manual/Using_mods#How_to_launch_a_mod
(DO NOT LINK) h t t p s : / / openarena . wikia . com/wiki/Command_console#Some_commands
(DO NOT LINK) h t t p s : / / openarena . wikia . com/wiki/FAQ#When_I_launch_a_Mod.2C_the_sound_stops_working

Bye! :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 26, 2012, 01:54:40 PM
@Gig: thank's a lot, that's perfect!


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 26, 2012, 07:41:29 PM
News: userinfo is correctly updated and a few other things fixed. I also reordered the code a lot to aggregate as much changes as possible in the new sv_demo.c file (eg: all checks that were previously done in standard Q3 functions are now done directly in sv_demo.c functions), and I cleaned the code a lot. I will do some more cleaning by splitting the demo read frame function into several parts (because for now it's a BIG and messy switch, not really readable nor extendable). Also added a lot of comments, should ease a lot future modifications of this module.

I still have a few other things to fix, notably the health updating problem. Player's health is still not updated, I don't yet know if it's because the field is not properly recorded or not properly replayed.

/EDIT: forgot to say that userinfo is filtered (easy using Info_RemoveKey): cl_guid and ip are removed, cl_voip is set to 0 (anyway voip packets aren't recorded at all).


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 27, 2012, 09:42:09 AM
MEMENTO: about team management:

- teamcvarset only done at clientbegin (teamcvarset updates the cvars g_blueTeamClientNumbers and g_red... that contains the clientID of each player belonging to the team).
- broadcastteamchange done at clientconnect (broadcastteamchange() notify other clients about a team change for this client).
- clientuserinfochanged just trap_setconfigstring from userinfo and client fields (such as sessionTeam)
- setteam auto broadcastteamchange and clientuserinfochanged
- SV_DropClient calls ClientDisconnect which calls trap_SetConfigString to set an empty client configstring (since it will be blocked at replaying, we have to do it ourselves)
- ClientBegin only called when team change or at level load or a client connects (this is what is said in the comment)


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on February 27, 2012, 11:37:50 AM
- teamcvarset only done at clientbegin (teamcvarset updates the cvars g_blueTeamClientNumbers and g_red... that contains the clientID of each player belonging to the team).
These cvars are OpenArena specific. They are used to limit VOIP to your own team.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 27, 2012, 01:27:22 PM
@sago007: AH OK! That explains a lot of things! So i do not need to manage them in demos (anyway they will be since I now completely simulate demo clients management).

/EDIT: thinking about it, why is TeamCvarSet() only called in ClientBegin? There are other ways that a client can be put in a team without ClientBegin being summoned (eg: forceteam?). Putting it in the SetTeam() function seems more logical as it is the standard way for the engine to manage teams.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 08:19:56 AM
MEMENTO: gamecode management of clients statuses:

in cg_local.h
Code:
// each client has an associated clientInfo_t
// that contains media references necessary to present the
// client model and other color coded effects
// this is regenerated each time a client's configstring changes,
// usually as a result of a userinfo (name, model, etc) change


Title: Re: The server-side demos thread, rebooted!
Post by: sago007 on February 28, 2012, 09:28:03 AM
Putting it in the SetTeam() function seems more logical as it is the standard way for the engine to manage teams.
Yes, it would be more logical but SetTeam is not called if the player start on a team (after a map-change).

Quote
/*
===========
ClientBegin

called when a client has finished connecting, and is ready
to be placed into the level.  This will happen every level load,
and on transition between teams
, but doesn't happen on respawns
============
*/

ForceTeam is bugged if ClientBegin is not executed before the next respawn. It might be bugged. I know my shuffle team is bugged that is why it always execute a map_restart afterwards so that all necessary logic is executed.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 11:27:51 AM
@sago: ok thank's for the precision :)

----

News: fixed a few more stuffs, it becomes more and more stable. Also I could force clients to spectators and prevent them from joining in, all engine-side.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 28, 2012, 11:48:56 AM
While this all sounds nice, does it work with any mod? It seems like e.g. your configstring/userinfo fiddling is very mod (baseoa) specific.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 12:03:52 PM
While this all sounds nice, does it work with any mod? It seems like e.g. your configstring/userinfo fiddling is very mod (baseoa) specific.

I've quickly tried with ExcessivePlus mod and an older version of the patch and it seems it works nice. Now it should work even better. Anyway, I did not yet try extensively, I plan to do some more testing when everything I want to fix will be.

Anyway, normally everything should be alright with any mod, because I tried to use standard functions and mechanisms of the engine, and I don't interfer with the processing of configstrings and userinfo, I just store them as-is.

Normally, since mods can't edit the engine functions, everything should work alright, because the same mechanisms will be used. My patch's philosophy is: store as-is and restore as-is, with no or little processing involved.

An exception is my management of clientbegin and client team management, I had to do my own detection code (because this part of the clients management is fully gamecode-side), but this behaviour (of managing clients) should not change with any mod.

And on the subject, I'm now pretty satisfied how democlients are managed, everything seems pretty stable and faithful to the original.

Now onto the health bugfix!

/EDIT: also as a side-note, I think this patch should totally be compatible with ioquake3 because I believe the parts of the engine I'm touching and interacting are not modified at all by OA, so I'm basically working directly on the ioquake3 code.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 01:11:10 PM
Tested again with ExcessivePlus with the latest patch: everything I've implemented works pretty well, but there's a bug which makes the client reload the map every few seconds - but the demo still continues and replays everything faithfully!

I'm pretty sure this is caused by a small glitch, probably some consistency check that is specific to ExcessivePlus and which is not reproduced in the demo. I'll take a look later, but mods support for this functionnality should be ok in the end.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 04:00:22 PM
DEV HELP NEEDED: I don't know how the mechanism that manages health work. I guess that:
- Either clients send their health in their packets (so that the server's engine process them somewhere even if not knowing it),
- Either clients only send their actions and the server deduce their health (so health is computed serverside by the server's gamecode).

Can someone tell me which mechanism is true?

/EDIT: Ok anyway it seems gentities is shared between the gamecode and the engine, so the health variable should always be accessible. But I don't know how it will be updated to the clients... I guess the second way I described above is the right one, it seems logical and natural to avoid easy cheating. Is it right?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 06:13:29 PM
MEMENTO: OMG this one clarification was rather hard, not intended for the dyslexic!

sharedEntity_t = gentity_t != entityShared_t

gentity_t manages every aspect of a game entity, in the gamecode, concretely stored in level.gentities.
sharedEntity_t is the engine-side version and is concretely stored in sv.gentities
On the other side, entityShared_t is a field of gentity_t which, from what I understand, manages entities that are shared between several entities = relations between entities in the big picture.

Code:
// the server looks at a sharedEntity, which is the start of the game's gentity_t structure
typedef struct {
entityState_t s; // communicated by server to clients
entityShared_t r; // shared by both the server system and game
} sharedEntity_t;

In the server-side demos patch, demoEntities (sv.demoEntities) are the recorded version of sv.entities, but only sharedEntity_t and entityState_t (another field of gentity_t) are recorded, health field was not.

I'm not going to record all the fields (even if that would be the best solution ideally, but practically this would be a huge storage hog and some filtering would be needed, for example netchan and other networks fields shouldn't be stored nor replayed), but I'm going to do a quick fix for the health field and see how that works out.

But really, the sharedEntity_t vs entityShared_t made me laugh.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 06:53:50 PM
NEWS: I've found a HUGE storage, memory and CPU hog. entityShared and entityState was stored TWICE in the demo.

This fix should divide the storage space of demos and CPU usage by 2.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 28, 2012, 07:54:07 PM
PROBLEM: the sharedEntity_t is not exactly gentity_t: it seems it only contains entityShared_t and entityState_t. The problem is that I need to store and update the health variable. How can I do that if I can't even access that field? Do someone have a genius idea?

A side solution maybe: necessarily, clients have to send their actions, right? I could maybe recompute a whole simulation of the damages then, but is there a command that clients send with their actions? Can someone tell me a big picture of how clients damages on their target is communicated from client to server (I know that everything gets processed in g_damage, but where are those data updated and from where?).


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on February 29, 2012, 03:33:15 PM
Why not just memcpy the entire gentity_t? Take a look at trap_LocateGameData() to get its size from the engine.
gentity_t->health and STAT_HEALTH are gamecode and thus mod specific. The interesting damage code is in G_Damage() after the "// do the damage" comment.

What do you mean by "damage from client to server"? The client just sends a "shoot in this direction" command to the server, which then does the hit checks. Are you talking about things like EV_PAIN or PERS_HITS (from server to clients)?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 29, 2012, 06:23:22 PM
@grey matter: thank's a lot. I was indeed talking about data from clients to servers, so this anwers my question.

I didn't think about memcpy, maybe I could do it the other way (ie: move from demo constructed gentity_t to the gamecode's gentity_t). I will look towards this way.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 29, 2012, 09:57:10 PM
Ok this is confirmed, my problem really comes from gentity_t->health field. I debugged by changing in the gamecode's g_active.c:

Code:
ent->client->ps.stats[STAT_HEALTH] = ent->health; // FIXME: get rid of ent->health...
to:
Code:
ent->health = ent->client->ps.stats[STAT_HEALTH]; // FIXME: get rid of ent->health...

And now it works. But it's not a solution for me because it needs a gamecode edit.

What is interesting to note is that health is the only field that is not totally managed in stats[], and so is redundant. Proof is the FIXME message above.

Okay, I think memcpy will be the way to go.

/EDIT: AH also I think it wasn't remove to allow for special entities with health, such as obelisks or destructable items.


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on February 29, 2012, 10:06:17 PM
How many work is needed to get this done?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 29, 2012, 10:21:47 PM
How many work is needed to get this done?

If you mean the whole server-side demos functionnality, I can tell you now: a lot. And that probably explains why noone bothered to do it before.


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on February 29, 2012, 10:27:13 PM
Hmmm... basically it needs many months of work?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on February 29, 2012, 11:20:17 PM
No it already works. And pretty well. I'm just fixing one last thing and after I'll polish the first public release.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 01, 2012, 01:47:12 AM
OK, I give up on fixing the health bug engine-side. I tried to declare the entire gentity_t structure and make a getter function, but that didn't work. It seems sharedEntity_t can't be treated as a gentity_t, and I can see no other way. I'll just make the simple gamecode patch.

/EDIT: no in fact I can't even code it inside the gamecode in an elegant way because there's no client-side var that can tell if the server is replaying a demo or not. So first one should create such a variable (eg: g_demoPlayback, there's already a sv_demoState) in the server's gamecode.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 01, 2012, 06:13:10 AM
Polishing, polishing...

Good news about mods, the ExcessivePlus bug that kept restarting the map is from the sv patch, it seems it keeps restarting the map because some parameters don't fit in (it restarts the map to automatically adjust the needed values). This is a good news because it can be fixed, and it will.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 01, 2012, 02:35:30 PM
Still polishing, now gametype and mod are stored in the demo, loaded if available and automatically set back to their original values at the end of the demo playback.

Also I fixed the bot's camera playback. I was quite lucky on this one, I was looking for something very different. I just removed the bit SVF_BOT from the bots players, and it seems that then it forces the engine to not consider them as bots but as normal democlients and so their standard fields are accessed (viewangles?) rather than bot's special fields (such as "eyes"...).

And by the way I'm fixing a lot of other small things such as adding the sv_hostname in the demo filenames, along with a filename cleaner to avoid illegal characters, and so on...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 02, 2012, 07:23:59 PM
Still polishing, autorecording now works perfectly, and mods support also (only tested on ExcessivePlus, but it works great).


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 03, 2012, 04:14:50 PM
Polishing, polishing and polishing... Starting to look quite good and pretty stable, I'm quite pleased. Demos recorded on ExcessivePlus are perfectly replayed.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 03, 2012, 06:43:10 PM
Added some detailed infos in demos files, this should help in keeping an history, or to dig old demos (always hard to keep a track of when and where it was done), and could also help to debug bugs:

Code:
DEMO: Details of svdemos/demotest32.svdm_71 recorded 2012-03-04 01:37:26 timezone Europe de lӏuest (UTC timezone: 2012-03-04 00:37:26 W09) on server " ALocalTest": sv_fps: 25 start_time: 31040 clients: 12 fs_game: excessiveplus g_gametype: 4 map: oasago2 timelimit: 20 fraglimit: 0 capturelimit: 8


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 04, 2012, 10:04:37 AM
Polishing a lot, fixixng a lot, cleaning a lot. Every main functionality has now its very own function (instead of a big if or switch) along with detailed comments, and I hope this will also avoid bugs in the future (by avoiding having too many vars declared for a single function).


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 04, 2012, 05:08:07 PM
Apart Excessive Plus, it would be good to test it also with some other mod, for both OA (e.g. AfterShock) and Quake3 (e.g. BazookaQ3, AlternateFire... etc.)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 04, 2012, 07:30:51 PM
Apart Excessive Plus, it would be good to test it also with some other mod, for both OA (e.g. AfterShock) and Quake3 (e.g. BazookaQ3, AlternateFire... etc.)

Of course, I'm just taking E+ as the basis for my mod support development. If it works for E+, it should basically work for any other mod, I'll just do some adjustments if needed.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 04, 2012, 08:35:16 PM
PRE-RELEASE: I don't have the time to finish right now completely the patch, so here is a pre-release. This version is very close to the final release, it's pretty stable.

GBO server-side demos patch v0.8.10
http://zsensible.free.fr/openarena/gbopatch/openarena-engine-source-0.8.x-28-amanieu-sv-serversidedemo_v0.8.10-moved-functions-fixed.zip

There are already compiled win32 64bits binaries inside. For other platforms, you have to compile by yourself as I don't have a linux box close to me.

If you try it and feedback (which would make me very glad and help me a lot to fix issues!), please remember to always set sv_democlients to 0 prior to recording or replaying a demo (demo_play will automatically set sv_democlients but sometimes it does not reset it, particularly when there's a crash), and please remember that it's normal that the player's health is not updated in the HUD. These issues should be fixed in the final release.

Also, if you encounter a crash or a bug, please toggle "developer" cvar to 1, this will enable a lot of demo debugging infos.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 07, 2012, 10:48:13 AM
UPDATE: Here is another pre-release, with these changes:

- Fully automatic management of sv_democlients, which is now read-only.
- /timescale supported (weird behaviour for values < 0.1 to be fixed).
- /cl_freezeDemo supported

GBO server-side demos patch (sourcecode) v0.8.15:
http://zsensible.free.fr/openarena/gbopatch/openarena-engine-source-0.8.x-28-amanieu-sv-serversidedemo_v0.8.15-democlients-autoset-fixed.zip

As previously said, if you try it and encounter a bug, please set /developer 1, and attach the bugged demo.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 07, 2012, 03:11:18 PM
I just discovered two things with my latest patch:

- the end of the game is respected whatever the timelimit or capturelimit or fraglimit is (though I didn't test with a lower value than when the game ended, will have to test)
- enabling sv_autoDemo kicks players every map change because of a wrong guid? This is a major show-stopper, I'll have to fix that.


Title: Re: The server-side demos thread, rebooted!
Post by: hairball on March 07, 2012, 05:21:29 PM
Hey GrosBedo,

I merged your latest server side patch with the openarena_engine.git project.

https://github.com/undeadzy/openarena_engine_serverdemo

The only real change is that it uses FS_IsServerDemoExt().


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 09, 2012, 06:42:25 AM
GrosBedo, I'm sorry I haven't had the time to test the serverside demo thing yet. And I don't know when I will have some time.

Just guessing... when your server-side demo format will be quite "stable", maybe Bioxide may find a way to make such stuff integrate with OAMME?

PS: Does somebody know a download link for "GTV for OA"? I noticed the link DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/Manual/Demos#Additional_tools]here (http://([b) is broken, and I'd like to replace it with a working one.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on March 10, 2012, 06:22:23 AM
http://www.superbots.org/viewtopic.php?f=25&t=707 includes GTV for OA
--> http://www.multiupload.com/CPECWG6CXJ and http://superbots.free.fr/superbots/files/OpenArena-Server-PowerPack_v1.1.2.zip

I haven't actually tested it, but there are gtv executables within the download.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 10, 2012, 12:40:03 PM
@hairball: thank's a lot, the FS_IsServerDemoExt() was indeed a change to make (I commented it in the code)!

I am grateful for your port, but anyway this patch is not yet clean, and many core ioquake3 files have been edited when there's no need (this is because Tremulous used a weird old+edited version of ioquake3), and so I'd prefer to port it to the latest ioquake3 only when it will be cleaned.

Anyway I am currently creating a github repo, and I will upload all the history of my patch so that later people can take a look at any version to see better how this patch was contructed.

@Gig: I will reupload a newer version of GTV for OA later, but first I need to setup my permanent hosting (I don't want to host it again on a temporary one...). Anyway, GTV for OA is quite deprecated since my server-side demos patch is more reliable and can also be used as a GTV substitute, and maybe an opensource GTV-like will be done using the work outlined here (I will discuss that later on).

Also please don't excuse yourself for not testing my patch! Noone is forced to test my patch, and you're already doing a lot for the community even while you're pretty busy IRL ;)

Anyway I would really LIKE servers administrators to test this patch since they are the ones who will be using this feature!


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 10, 2012, 03:32:18 PM
Another pre-release, this one is much more stable and almost finalized. In fact, I won't really work on it very much anymore, I would like to add a few more features but they aren't necessary. Last thing I really want is to clean the code, but this won't change the features nor the stability.

All the previous bugs are fixed, mainly:
- no more are players disconnected when sv_autoDemo is set to 1
- health is now updated on HUD (this one was tough!)
- no more delay when replaying or stopping a demo (before, the code needed to use a delay, now you type demo_start and it starts the demo right away, no delay). Same when stopping a demo (and escaping a demo with escape key does not produce subsequent errors when replaying other demos afterwards).
- fully automatic management of sv_democlients and sv_maxclients vars. You can now safely just demo_play and demo_record without caring about these cvars.

This version is v0.9.3, but now anyway the very last version can always be found on github:

https://github.com/lrq3000/openarena_engine_serversidedemos

Also, there are win32 binaries for Windows users to easily test (sorry linux users, I don't have a linux box close to me!):
https://github.com/downloads/lrq3000/openarena_engine_serversidedemos/openarena-engine-serverdemo-v0.9.3-win32.zip

If you are a server administrator, please test the patch!


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 10, 2012, 03:35:04 PM
@admin: Arg, I can't edit my frontpage posts anymore! I didn't know there was a timelimit for edition in this forum.

Please can an admin edit the frontpage post to remove the version and replace the download link to the github's? (Also if you could change PRE-RELEASE to just RELEASE...). I would be very grateful!


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 10, 2012, 06:17:14 PM
Fromhell added this limit some months ago, due to some users that decided to abandon the community making *all* their previous posts empty (making such threads not easy to understand).

Anyway, I don't like this thing... e.g. the "front post" of the "0.8.x binaries" thread still contains the link to now-old version 22 executables, which had a major bug (mods do not work at all!).


Title: Re: The server-side demos thread, rebooted!
Post by: hairball on March 10, 2012, 06:42:24 PM
@hairball: thank's a lot, the FS_IsServerDemoExt() was indeed a change to make (I commented it in the code)!
Yeah I saw your comment. :)

I am grateful for your port, but anyway this patch is not yet clean, and many core ioquake3 files have been edited when there's no need (this is because Tremulous used a weird old+edited version of ioquake3), and so I'd prefer to port it to the latest ioquake3 only when it will be cleaned.
Was there any major cleanup needed?  The only thing I saw that stuck out was the ARRAY_LEN removal which 0.9.3 adds back.

By the way, I synced my copy of your patch with 0.9.3 in github.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 11, 2012, 04:52:20 AM
@Gig: ah ok, thank's for the info (I also witnessed this "presence deletion" behaviour...).

That's unfortunate, most visitors will only read the first post since this thread is pretty long, and it would be unfortunate that they use an old and buggy version... But I can't see an alternative solution for this "presence deletion" problem.

@hairball: oh man! You've already synced lol. This is nice but please don't waste your time syncing every version I release because I will port my patch to the latest oa binary synced with ioquake3 when I'm finished, it's just a  matter of time :)

And about the files that shouldn't be modified, for example all the game/*.* and most of qcommon/*.* shouldn't be modified, because they overwrite standard q3 functions with deprecated/weird tremfusion functions.


Title: Re: The server-side demos thread, rebooted!
Post by: Neon_Knight on March 11, 2012, 07:32:02 AM
@Gig: ah ok, thank's for the info (I also witnessed this "presence deletion" behaviour...).

That's unfortunate, most visitors will only read the first post since this thread is pretty long, and it would be unfortunate that they use an old and buggy version... But I can't see an alternative solution for this "presence deletion" problem.
Well, there's the option to move stuff to the Wiki.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 11, 2012, 07:34:20 AM
Indeed, good idea, but this is an original work, I mean it's not the standard OpenArena, so I'm not sure where it should reside in the wiki..


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 11, 2012, 04:52:29 PM
New version:

- Faults tolerance mode: when sv_demoTolerant is 1, demo_play will try to be more tolerant to errors and unknown markers. This should allow future versions of this patch to add more meta data (infos about the demo) or more demo events, but still be able to replay old demos. This means that any demo recorded with a patch versions >= v0.9.4.3 should always, forever be replayable from now on with any future version of this patch.

- limits (capturelimit, timelimit, fraglimit) are now automatically set and restored (this is not necessary to replay a demo, but this enables the voice announcement of SuddenDeath).

I have now implemented all the features I wanted, and fixed pretty much everything I had on my plate. I'm going to test a little further, and then I'll finalize this patch by cleaning the code and porting to the last OA+ioquake3 engine.


Title: Re: The server-side demos thread, rebooted!
Post by: hairball on March 11, 2012, 05:42:18 PM
I have now implemented all the features I wanted, and fixed pretty much everything I had on my plate. I'm going to test a little further, and then I'll finalize this patch by cleaning the code and porting to the last OA+ioquake3 engine.
Sounds good! :)  I deleted my copy of your server side demos to avoid needless duplication.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 11, 2012, 07:28:01 PM
Bad news: the patch crash for any other gametype that is not based on flags (CTF and Double Domination).

Here's a recap of my tests:

== Mod testing
- baseoa: works perfectly
- excessiveplus: works perfectly
- SuperHeroes Arena: works perfectly BUT suffers from the gametype bug (since there's no CTF gametype it seems?)
- Generations Arena: works ok but small problem in democlients teams management, it has to be refined

== Gametypes testing
- DM/TDM/Tourney: quit the demo when replayed after a few seconds. I don't know where this bug comes from...
- CTF: works great
- Double Domination: works great
- Overload: should work but for now it suffers from the warmup bug

So now my main concern is about the non flag based gametypes, it does not work quite good...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 11, 2012, 10:59:04 PM
Ok I've pinpointed a bit the problem, it is not related to the gametype at all, but to a problem in the democlients teams management. It's in fact a variant of an old problem I've dropped that made demos crash without any reason when bots were set to a team. The workaround was to avoid setting bots to teams, but now the problem also arise for players.

Anyway I'm pretty sure I'll be able to fix this bug now I've somewhat pinpointed it.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 13, 2012, 05:56:25 AM
Found the problem: it is not even related to the democlients team management, but to the warmup time:

in g_main.c CheckTournament():
Code:
// if the warmup time has counted down, restart
if ( level.time > level.warmupTime ) {
level.warmupTime += 10000;
trap_Cvar_Set( "g_restarted", "1" );
trap_SendConsoleCommand( EXEC_APPEND, "map_restart 0\n" );
level.restarted = qtrue;
return;
}

This is what causes the demo to stop.

Now I just have to find a way to cleanly fix this problem.

Meanwhile, a simple fix is to set an absurdly high g_warmup prior to replaying a demo, but I hope to come up with a better fix.

/EDIT: just set /g_doWarmup 0 and it fixes the problem. The demo playback function now does that automatically, and I'll also implement a check to avoid g_restarted to be set to 1 when a demo is playing, this should not happen no matter the reason when a demo plays.

/EDIT2: cannot safeguard againt level.restarted = qtrue; so I'll just drop the safeguarding.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 13, 2012, 08:26:02 AM
New version available v0.9.7.4, downloadable along with win32 binaries on my github repo (https://github.com/lrq3000/openarena_engine_serversidedemos).

All bugs fixed. I think (hope!) this release is now pretty much stable. I didn't have the time to retest all my demos but it should hopefully work smoothly from now on.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on March 13, 2012, 12:54:37 PM
I see you did not find a workaround for the gentity_t->health issue. This means that your patch might not compatible with any other mod than the one you used for compiling.
It's unlikely that mods add fields before the health field, but if they do its offset changes and your server code will blindly overwrite memory at the wrong locations. You could make the offset configurable via a cvar, but I doubt that admins would really bother to find the correct one, especially with old closed source mods.

Since you seem to have issues with map_restart, what happens if commands like that get executed during demo playback, say due to players calling a vote? (your fix using g_doWarmup is mod specific as well. Mods might execute commands like this on other occasions).

tell, say_team, chat and tchat are also mod specific. free, red, blue, spectator , skill, timelimit, fraglimit and capturelimit also depend on the gamecode, though those are unlikely to change.
forceteam is a game command, which might not even be present in some mods.

While most of those are unlikely to be modified in mods, the size of this list suggests that your code is quite likely to break.
The patch for Tremulous modified gamecode, so it didn't need to care about mod interna. Older server-side demos had the same problems with mod support like your patch has.

I can't be of any help here, I just wanted to spell this out. I would not want to be the one maintaining such a monster, where each and everything can go wrong :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 13, 2012, 02:08:39 PM
@grey matter: firstly, thank's for your input, they are very insightful.

About gentity_t->health, I DID find a workaround! :D And the workaround is based on the very definition of gentity_t, there's no direct access to the memory location, so even if mods change the order of the fields, the demo will still be recorded/replayed fine because it just does a cast of sharedEntity_t -> gentity_t.

About user input (such as callvote), they are not (yet) managed, but they can easily be filtered out by cancelling them when they reach the server. I will take a look at that feature. But anyway I'm not going to filter servers commands, the admin should be free to do whatever he wants to, even during a demo playback.

About chatting commands, I don't think so, every ioquake3 based game I saw used them (Smokin'Guns, Tremulous, ioquake3, OA).

About teams management, indeed that's ioquake3 specific, standalone games that modifies the team names will have to adapt a bit the code, but this way was the most effective I could find, because there's no function accessible server-side that can convert an int to a team (there is one in the gamecode, but inaccessible, I just copied it in my team management function, and added comments so that people that would port the patch would easily know what to do here).

About forceteam, it's implemented server-side in the binaries, so if my patch is ported, this command should be ported too, and it should work whatever the mod or standalone game.

That said, my primary goal is to make a patch that is compatible with OA, standard ioquake3 and any mod running on those engines, but not necessarily standalone games such as Tremulous. For these games, the patch should also work but with a few tweaks (such as the stage management for Tremulous, it was in the original Amanieu's patch but removed here because very specific to Tremulous).

For the moment, it seems it works with most mods (ExcessivePlus, Aftershock, Superheroes Arena, Generations Arena).

I will take again a look at the team management to see if I can do something more dynamical. But apart from that, I think that everything else is pretty much generic.

-----------

Anyway I'm going to try to make from scratch an alternative version of this functionnality, based on TheDoctor's patch, which should be a lot more generic and clean and short! But I'm not sure if my idea is gonna work, we'll see. At least this one patch already works.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 13, 2012, 05:15:36 PM
Ah also about team management, I forgot to mention that there's a weird beneficial side effect: in fact, whatever the team the client is set to, the gamecode will automatically affect the client to the right team.

So in fact I just can replace the command with "team o" (or in fact any string) and it should always work. So as it is, even if the team are changed, it WILL work.

/EDIT: about forceteam, you were right, it's dependent on the gamecode. But anyway, this should not hamper demo functionnalities, it is only used to force players into the spectator team prior to a demo playing, and I can't think of any other way to do it without using this command. Anyway, this was necessary before because there was a delay before the demo began, but now that's not the case anymore so it should not be necessary anymore.

/EDIT2: about the fact that we can do a "team o" with any team value and it works, it's because the gamecode will then automatically pick a team using the function PickTeam, so in fact it does NOT set to the right team, but a random team based on the availble slots. But the nice thing is that it does a CLIENTBEGIN, so that's why it shows the democlients correctly in their right team.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 14, 2012, 08:10:29 AM
Working on interoperability:

- changed forceteam into a clientcommand team, this should work on any ioquake3 based game.

- teams management isn't necessary for the patch to work, because we are recording clientcommands, so when a client change team it's completely mod-independent, the command is just replayed. Before it wasn't the case (in the original patch, these commands were not recorded), so now the team management is only useful for the initial team when the demo starts recording (because then there's no clientcommand, so the demo must still set the initial team). I'll try to move the team management from configstrings to userinfo since the userinfo string stores the full team name string, not just the int. I did that in the past but it didn't work, but maybe now it will.

But it works great even if the initial team is not set, it's just that spectators can then also be followed, which is not what we want.

Also it is to be noted that a simple CLIENT_BEGIN vmcall along with a correctly set client configstring allows the democlients to be shown in the right team, even if sessionTeam is not set. But since I want this patch to fully simulate the game state, the only way to set sessionTeam is to use the "team" clientcommand.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 14, 2012, 11:04:53 AM
BUG SPOTTED!

There is a bug in the recording of the userinfo string in the latest versions, I mistakenly changed strcpy to Q_strncpyz, which truncates the userinfo string.

I will soon post an updated version.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on March 14, 2012, 11:11:54 AM
There is a bug in the recording of the userinfo string in the latest versions, I mistakenly changed strcpy to Q_strncpyz, which truncates the userinfo string.

It should only do this when your target buffer is too small (hence the n in its name). Double check that your buffer is large enough, because plain strcpy might overflow it (and that's a security problem).


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 14, 2012, 11:36:33 AM
That's how I did it with Q_strncpyz:

Code:
static char fuserinfodata[MAX_STRING_CHARS];
static char *fuserinfo = fuserinfodata;

Q_strncpyz(fuserinfo, userinfo, sizeof(fuserinfo));

Did I do something wrong? I'm not really a C guru ;)

Anyway are there really any risk of overflowing if my var and the game var are both limited by the same constant (here MAX_STRING_CHAR)?


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on March 14, 2012, 12:12:23 PM
You can not use sizeof(fuserinfo) here. The compiler only knows that it's a char *, it could point to any arbitrary sized struct.
You'll have to use Q_strncpyz(fuserinfodata, userinfo, sizeof(fuserinfodata)); (or directly use the MAX_STRING_CHARS size as an argument, which is not quite as good). Why the additional pointer anyways?

It does not matter whether the define *should be the same in both modules, you must always use sizeof(targetbuffer) if you want to be safe.
Quote from: The ten commandments for C Programmers
Thou shalt check the array bound off all strings (indeed, all arrays), for surely where thou typest "foo" someone someday shall type "supercalifragilisticexpialidocious".


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 14, 2012, 04:56:30 PM
Thank's a lot, I didn't know sizeof should only point to the buffer, not the pointer to the buffer (yeah I'm a newb in C ;) ).

I've fixed all the issues and used Q_strncpyz. I'll later convert the other ones.

Also, should I free() every malloc'ed char at the end of my functions?


Title: Re: The server-side demos thread, rebooted!
Post by: hairball on March 14, 2012, 07:25:33 PM
Thank's a lot, I didn't know sizeof should only point to the buffer, not the pointer to the buffer (yeah I'm a newb in C ;) ).
You should review the code again and read up on C.  I looked at the log for 5 seconds and saw this:

Code: ("wrong")
qboolean SV_CheckClientCommand( client_t *client, const char *cmd )
{
    char *userinfo = ""; // init with an empty string so that the compiler doesn't shout errors

    if ( !strncmp(cmd, "userinfo", 8) ) { // If that's a userinfo command, we directly handle that with a specialized function
       strncpy(userinfo, cmd+9, strlen(cmd)-9); // trimming out the "userinfo " substring (because we only need the userinfo string)

There are multiple problems with that code. :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 15, 2012, 04:43:03 AM
@hairball: Ah indeed I didn't convert stncpy to Q_strncpyz, but what would be the problem here apart from that?

/EDIT: ah indeed the strlen(cmd)-9 was wrong because one shouldn't use the size of the input string. I've fixed that. Anyway I can assure you that there are similar mistakes in the ioquake3 engine since I've pretty much copied it.


Title: Re: The server-side demos thread, rebooted!
Post by: grey matter on March 15, 2012, 12:49:59 PM
The problem is, that your char *userinfo is just a pointer, no buffer. Initializing it like this means the compiler will store the "" string somewhere (just large enough to store the \0, maybe even in the program's read-only section), then let userinfo point to it.
You now copy arbitrarily sized data to wherever userinfo points to, not into a properly allocated buffer. Like hairball suggested, get some book about C. Pointers can be tricky and dangerous.
You pointed one problem yourself, another problem is that you blindly use cmd+9, while the string that it points to might just be "userinfo", so you'd start to copy from some random memory that's after the string.

If there are indeed portions of ioquake3 code that do this, please file a bug report about them.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 15, 2012, 04:01:24 PM
Yes indeed I also fixed that while checking the code.

Could you please tell me if that one is ok (because it's a char[size]), or should I here too do a malloc?

Code:
static char savedFsGameVal[MAX_QPATH] = "";
static char *savedFsGame = savedFsGameVal;

Thank you for your advice but I really don't have the time to read an entire book on C, and this is the only project I have for which I must use the C language. Moreover, even if I admit that I'm not enough experienced with C as I would like, I'm a very experienced programmer in general, and I do know how the memory is managed and how pointers work generally, just not the syntax and habits specific to the C language (but I'm very willing to learn if you have a good resource about it!).

Anyway I will do more checking now as you are suggesting and follow these good habits. For example, about the userinfo, I thought about that when I did cmd+9, but I assumed that there was no possible reason that the engine would issue a userinfo without at least a string after because I can remember the engine checks that the userinfo is not empty before issuing it. Anyway I will add a strlen check, this should fix this for good.

About ioquake3, if I ever stumble accross these unfixed things in the latest version of the engine, I will report them (maybe they were fixed already).


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 18, 2012, 10:13:39 PM
Code updated, fixed, and, most importantly, cleaned. I also added a readme with the changelog since about v0.5 and commands usage.


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 22, 2012, 04:44:21 PM
Hi! Where are the latest win32 binaries? I finally had some minutes to try the serverdemos thing, but I cannot find the binaries...  :(


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 23, 2012, 05:45:59 AM
Hi! Where are the latest win32 binaries? I finally had some minutes to try the serverdemos thing, but I cannot find the binaries...  :(

Sorry I forgot to upload the binaries. They are now available, and this is the final version for OpenArena v0.8.8 (until I get feedback!).

You can get them here:
https://github.com/lrq3000/openarena_engine_serversidedemos/downloads


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 23, 2012, 06:17:22 AM
Thank you. But I dont know if I will be able to do any test before sunday...


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 23, 2012, 11:27:33 AM
@Gig: no problem, don't be sorry, I am glad that at least someone plan to try this patch :p

NEWS: Finalized the code, and ported to OA+ioquake3 r2224. Unified diff patchs are provided. This is the same code for OA 0.8.8 and the hybrid OA 0.8.8 + ioquake3.

You can find the OA 0.8.8 server-side demos project here (branch "master"):
https://github.com/lrq3000/openarena_engine_serversidedemos

And the OA 0.8.8+ioquake3 r2224 server-side demos project here (branch "latest"):
https://github.com/lrq3000/openarena_engine_serversidedemos/tree/latest

Unified patchs for both branches and win32 binaries and a few demo examples can be found here:
https://github.com/lrq3000/openarena_engine_serversidedemos/downloads

I consider this project finished. I may update it if I get some feedback, but I won't promise any precise date.

As a side note, I have tested a bit the storage efficiency, it seems server-side demos produced by this patch are about nbplayers times the size of a standard demo. In my test, you can expect 125 KB/min per player. For a real game of 10 players (and no spectators) and 20 min timelimit, if we extrapolate from this observation, we should expect 125*10*20 = 25 MB for one demo. However, real tests on production servers should be performed in order to get a better estimate of the storage consumption.

My warm thank's go to all the people who participated in this thread and tried to help me in one way or another, especially sago and grey matter who both were very patient with me.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 23, 2012, 02:25:58 PM
Opening to the future

I have taken a look at a possible alternative way of making such a patch, based on TheDoctor's approach.

At first, I thought that the approach was different and based on flux. But in fact, it's the exact same approach as Amanieu's, the difference being that TheDoctor's patch rely on client-side facilities to replay demos (so it NEEDS a client to replay a demo), while Amanieu's code didn't: it has its very own replaying facility server-side.

But this approach is still interesting.

More details follow.

----

TheDoctor's patch simply dumps the data packets sent to clients.

Then, the client can replay the demo by using the standard demo facility (because this is how standard demos are made: from network packets dumps).

From cl_parse.c, CL_ParseServerMessage() function:

Code:
//
// parse the message
//
while ( 1 ) {
if ( msg->readcount > msg->cursize ) {
Com_Error (ERR_DROP,"CL_ParseServerMessage: read past end of server message");
break;
}

cmd = MSG_ReadByte( msg );

if (cmd == svc_EOF) {
SHOWNET( msg, "END OF MESSAGE" );
break;
}

if ( cl_shownet->integer >= 2 ) {
if ( (cmd < 0) || (!svc_strings[cmd]) ) {
Com_Printf( "%3i:BAD CMD %i\n", msg->readcount-1, cmd );
} else {
SHOWNET( msg, svc_strings[cmd] );
}
}

// other commands
switch ( cmd ) {
default:
Com_Error (ERR_DROP,"CL_ParseServerMessage: Illegible server message");
break;
case svc_nop:
break;
case svc_serverCommand:
CL_ParseCommandString( msg );
break;
case svc_gamestate:
CL_ParseGamestate( msg );
break;
case svc_snapshot:
CL_ParseSnapshot( msg );
break;
case svc_download:
CL_ParseDownload( msg );
break;
case svc_voip:
#ifdef USE_VOIP
CL_ParseVoip( msg );
#endif
break;
}
}

This is the code that parse the packets and replay a standard demo.

On the other hand, Amanieu's approach was to dump the events, not the data packets. This is more insightful in our case, because this is what allows to replay them.

If we only dump the data packets, there is absolutely no facility provided server-side to replay them (because they are designed to be an update specific to the target client), so to fix that we would either need to include client/ files to access the functions, or copy/mimic them, which is neither satisfying.

But if we dump events, we have a broad range of functions provided in the server/ files to play these events. Hence the approach of Amanieu.

-----

All that is to say that it's currently not possible to produce multi-view server-side demos with TheDoctor's approach since there's no replaying facility and data packets are designed per client. Everything is replayed client-side, so there's no way for example to allow for freeflying spectators since when you play the demo you are in the "eye of the beholder".

However, I think there's some ways to tweak this approach and come to an "hybrid" approach that would overcome these limitations:

- make a replay facility server-side, by feeding real clients some fake packets containing the messages of a demo. This can easily be done by just reading a message from a demo and using SV_SendMessageToClient(client, &msg). What is harder is to prevent the server from sending conflicting data to clients (because the server will continue to send them the valid gamestate), without crashing the clients nor the server.

- instead of recording every client's flux separately, we could save them all in the same file and add an event "End of frame" which would allow to replay these packets at the right time (similarly to Amanieu's approach, see my patch).

- hook clientCommand and when the client send a "follow" or "follownext" command, the server would send them the flux of the democlient associated with the number. But it seems that FollowCycle (when you click your mouse button) is not catched this way, so it would be needed to edit the gamecode at least to allow the hooking of FollowCycle.

- to allow freeflying, spectators that are known to be freeflying right now would be sent just the entities states and player states. A gamecode hook would also be needed here, because I don't know of a way to know the state of a spectator (and who he is following) completely server-side, but from the gamecode it's easily available.


As you can see, this implementation would greatly outdo my current implementation in terms of performance, because everything would be computed client-side and not anymore server-side (even if my implementation is already pretty inexpensive on CPU since there's nearly no gamecode event, everything is simulated, no actions processing involved). However, it would not be possible to use it with mods (unless workarounds are found for the above problems).

Anyway I do think this approach could be very interesting for a GTV replacement, since it would allow to support a big crowd of players watching the same demo flux.

----

As a memo, here's the big picture of standard demo replaying client-side:
CL_SetCGameTime -> CL_ReadDemoMessage() -> CL_ParseServerMessage() parse the servers messages -> various parsing functions...

And how TheDoctor's patch write demo messages:
SV_SendClientSnapshot() -> SV_UpdateServerCommandsToClient() and SV_WriteSnapshotToClient() -> back to sendclientsnapshot -> SV_SendMessageToClient()

----

In summary, both patchs are very similar, except for where the demo events are processed: client-side for TheDoctor's, server-side for Amanieu's. And this is all where the problem resides in making a fully server-side demo but processed client-side: how to allow freefly and viewpoint change when everything is processed client-side?


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 26, 2012, 04:14:18 PM
I've finally been able to try 0.9.9.5 win32 binaries.
A quick try, but I have to tell you GREAT! A cake for you!  :)  :)  :)

A little thing I noticed:
Quote
sv_demoState : show the current demo state (0: none, 1: waiting to play a demo, 2: demo playback, 3: waiting to stop a demo, 4: demo recording)
... what is value 3 exactly? Why I saw it as 3 instead of 0 when I was not playing a serverside demo?

A question:
demo_play, demo_sercord, demo_stop... what about a "sv" in the name of those commands, to better distinguinsh them from standard demos commands?

PS: Why are there two .dll in the zip file?

PPS: Everyone around here should try that!  :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 27, 2012, 03:21:56 AM
Thank you Gig, Im quite happy it ran well for you too :D

Quote
sv_demoState : show the current demo state (0: none, 1: waiting to play a demo, 2: demo playback, 3: waiting to stop a demo, 4: demo recording)
... what is value 3 exactly? Why I saw it as 3 instead of 0 when I was not playing a serverside demo?

This comes from qcommon/qshared.h :
Code:
typedef enum {
DS_NONE,

DS_WAITINGPLAYBACK, // demo will play after map_restart)
DS_PLAYBACK, // a demo is playing
DS_WAITINGSTOP, // demo is stopped but we must move clients over their normal slots

DS_RECORDING, // a demo is being recorded

DS_NUM_DEMO_STATES
} demoState_t;

In fact when replaying a serverside demo, real clients are moved by a certain offset, so that democlients can be assigned to their original slot (eg: if you were at slot 2 when recording the demo, when replaying this democlient will be at slot 2 too). This is necessary to make the simulation happening.

At the end of demo playback, the map is restarted to delete the democlients slots and move back the real clients, but to do this, we must have a way to know we are stopping the demo. Before, Amanieu's original patch used the "delay" command (a special command similar to "wait" but that doesn't freeze the game while it waits, it worked but was more unreliable because you couldn't know the exact time the map_restard took, and when clientside with no map_restard, it didn't work). Here I simply chose to add another demoState. In practice, you should never be able to see it because it should be really quick (just the time to restard the map).

So it is not normal that your demoState was stuck at 3, but I guess this is because you were playing your demo clientside, which does not happen to do a map_restart at the end of playback (it simply exits from the demo to the game main menu). This is not important and will not produce any bug, but thank's for pointing that, Ill fix that.

A question:
demo_play, demo_sercord, demo_stop... what about a "sv" in the name of those commands, to better distinguinsh them from standard demos commands?

There are no command in OA or ioquake with a prefix, only cvars do have prefix, so I just followed the spec. Also, demo_play and demo_stop do work clientside too, so prefixing sv would be a bit misleading, but for demo_record thats true. Dunno but this has to be given some thoughts. Maybe server_demo_* then?

PS: Why are there two .dll in the zip file?

These contain the OA renderers because of the new renderer modularity facility of the latest ioquake3. This is not related to my patch, undeadzy's patch also has them (see the OA 0.8.8 sources thread).


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 27, 2012, 06:12:50 AM
Just a (stupid?) idea... What about using a different name for these demos, e.g. "full demos" instead of "serverside demos"... (still to better divide their commands from classic demos)?
E.g. fulldemo_play, fulldemo_stop, fulldemo_record?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 27, 2012, 06:17:45 AM
I like the idea, I think Ill do that, thank's Gig :)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 27, 2012, 07:25:30 AM
Since my computer crashed yesterday (along with losing all my data, don't trust Windows 7 profile folders nor chkdsk), I won't be able to update the code in a while.
Meanwhile, I've updated the wiki page about demos.


Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 27, 2012, 04:00:12 PM
Hi! I'm not sure about this part:
(DO NOT LINK) h t t p s : / / openarena . wikia . com/wiki/Manual/Demos#About_privacy_and_server-side_demos
I feel I never actually heard real in-game OpenArena VoIP... but from what I read in ioquake3 documentation at the time... VoIP packets are recorded in standard demos... why should serverside demos be different? The same, I'm not sure, but probably standard demos record team chat... is it okay to change it for serverside demos?




Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 27, 2012, 04:27:00 PM
Yes standard demos have absolutely no filtering (thus it records say, teamsay and tell), and it explicitely records VoIP packets. The difference is the intention of usage: a client recording a standard demo records it for himself, and if he release it publicly, he implicitly consent to make public all the data contained inside the demo.

On the other hand, a server-side demo is recorded by a server administrator, and he cant know when releasing the demos publicly if all the clients consent to make these private data public. So the best thing to do is to avoid the recording of the sensible data.

Also, server-side demos potentially contain more sensible datas than standard demos, such as IP and GUID (I cant check the code right now but I think I can remember standard demos do not record these datas). Also the server has access to ALL IPs and GUIDs, so disclosing them could be very dangerous for clients (eg: hacking...).

PS: OA in-game VoIP works, I have tested it ;)


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 28, 2012, 02:05:34 PM
Ive given another thought about the alternative hybrid approach, and I think it's possible to overcome some of the limitations:

1)
- make a replay facility server-side, by feeding real clients some fake packets containing the messages of a demo. This can easily be done by just reading a message from a demo and using SV_SendMessageToClient(client, &msg). What is harder is to prevent the server from sending conflicting data to clients (because the server will continue to send them the valid gamestate), without crashing the clients nor the server.

Easy to be done: at replaying, we just have to use SV_SendMessageToClient(client, &msg). Both are already recorded in TheDoctor's patch.

And to avoid the interference of the engine, we simple impose a condition in SV_SendClientSnapshot() to call SV_SendMessageToClient() only if we are not replaying a demo (because if were replaying a demo, we will directly call SV_SendMessageToClient() in our demo loop).

2)
- instead of recording every client's flux separately, we could save them all in the same file and add an event "End of frame" which would allow to replay these packets at the right time (similarly to Amanieu's approach, see my patch).

As I said, TheDoctor's patch already records the messages, you then just have at each end of frame (at the end of SV_Frame(), see my current implementation) to record an "_endFrame" marker. This way, all messages will just be recorded when they are made by the engine, and we just add a marker so that at replaying we can know that we are changing frame.

To replay, we make a simple loop to process and send all demo messages (for the current frame) until we hit the _endFrame marker, then we exit the demo playback loop until next SV_Frame() iteration.

3)
- hook clientCommand and when the client send a "follow" or "follownext" command, the server would send them the flux of the democlient associated with the number. But it seems that FollowCycle (when you click your mouse button) is not catched this way, so it would be needed to edit the gamecode at least to allow the hooking of FollowCycle.

This one is tricky but should be possible and fairly easy to implement if done right.

Cmd_FollowCycle_f is in fact triggered in two way: either with follownext or followprev clientCommands, or either by just clicking the mouse button (which can be seen in the usercmd_t packets).

Code:
// game commands

  { "follow", CMD_NOTEAM, Cmd_Follow_f },
  { "follownext", CMD_NOTEAM, Cmd_FollowCycle_f },
  { "followprev", CMD_NOTEAM, Cmd_FollowCycle_f },

Code:
/*
=================
SpectatorThink
=================
*/
void SpectatorThink( gentity_t *ent, usercmd_t *ucmd ) {
...
client->oldbuttons = client->buttons;
client->buttons = ucmd->buttons;

 if ( ( client->buttons & BUTTON_ATTACK ) && ! ( client->oldbuttons & BUTTON_ATTACK ) ) {
Cmd_FollowCycle_f( ent );
}
...
}

This is a standard behaviour for all ioquake3 based game, so we can safely assume that clicking the button when spectator will always result in changing the viewpoint.

The usercmd_t packets are also available from the engine, and also we can access playerState_t->pm_flags.

Here's the idea is to basically to reproduce the behaviour of SpectatorThink() and Cmd_FollowCycle_f() in the engine instead of the gamecode, by just detecting when BUTTON_ATTACK is pressed while the client is a spectator (and it should always be in a demo playback, so this is optional).

But there will be some challenges:

- SpectatorThink(): detecting if the client want to change the viewpoint and maintain an array that says for all spectators who they are watching. This should be done at the beginning of any frame prior to sending demo messages to assign all spectators to the right democlient in case of a change.

- Cmd_FollowCycle_f(): to find the next democlient in the list, to achieve that we would need to maintain an array of the democlients indexes at any time. A non-empty configstring could be used to check that the democlient was active?
Note that the list would be needed anyway for "follow" to work, because we must make sure that we have a democlients that can be watched with the specified id.


4)
- to allow freeflying, spectators that are known to be freeflying right now would be sent just the entities states and player states. A gamecode hook would also be needed here, because I don't know of a way to know the state of a spectator (and who he is following) completely server-side, but from the gamecode it's easily available.

This one is HARD. I don't know even if it's possible to achieve it. Maybe it will be impossible, or just too easy to believe (because there's a chance that after step 3 is done, freeflying may be totally managed by the gamecode without having to change anything in the engine).

I explain: the problem here is not to allow for freeflying, as I wrote we just have to send the entities delta messages (update messages), and the client will just draw them and will be able to freefly around. The problem is to detect when the spectator wants to freefly.

To detail further, let me show you the code that allows to switch from following to freeflying (which is a call to StopFollowing()):
Code:
/*
=================
SpectatorThink
=================
*/
void SpectatorThink( gentity_t *ent, usercmd_t *ucmd ) {
...

if ( ( client->buttons & BUTTON_USE_HOLDABLE ) && ! ( client->oldbuttons & BUTTON_USE_HOLDABLE ) ) {
if ( ( g_gametype.integer == GT_ELIMINATION || g_gametype.integer == GT_CTF_ELIMINATION) &&
                g_elimination_lockspectator.integer>1 &&
                ent->client->sess.sessionTeam != TEAM_SPECTATOR ) {
                    return;
                }
StopFollowing(ent);
}
...
}

At first, you could think "well, why can't we use the same trick as in step 3 and detect when BUTTON_USE_HOLDABLE is pressed?"

Answer: because this is not a native behaviour of ioquake3. Ioquake3 does not even have any way to go to freefly once you've followed someone, except from joining the Spectator team again. So this behaviour was introduced in OpenArena.
Also, mods can further modify this behaviour. For example, ExcessivePlus does not go from Following directly to Freefly, but from Following -> Third-person view (Track Cam) -> TV View -> Freeflying.

Anyway, we could still implement this feature this way, by detecting if BUTTON_USE_HOLDABLE is pressed and then just sending him the entities states (and stopping to send him the democlient messages he was previously following), but this is not satisfying if you want to be faithful to the original way mods have made their spectating work. But this is without any doubt the easiest way to do it.

Now let's take a look on another way to do that. Let's see the code of StopFollowing():
Code:
/*
=================
StopFollowing

If the client being followed leaves the game, or you just want to drop
to free floating spectator mode
=================
*/
void StopFollowing( gentity_t *ent ) {
ent->client->ps.persistant[ PERS_TEAM ] = TEAM_SPECTATOR;
ent->client->sess.sessionTeam = TEAM_SPECTATOR;
ent->client->sess.spectatorState = SPECTATOR_FREE;
ent->client->ps.pm_flags &= ~PMF_FOLLOW;
ent->r.svFlags &= ~SVF_BOT;
ent->client->ps.clientNum = ent - g_entities;
}

As you can see, there's a playerState_t pm_flags PMF_FOLLOW that gets removed. Also the ps.clientNum gets changed too.

So a good way to do this would maybe be by checking the state of the pm_flags PMF_FOLLOW, but this would require that the gamecode would set it, which is not certain since the democlients do not exist on the server with this implementation. So the PMF_FOLLOW flag may not even be set.
Maybe we can then set PMF_FOLLOW flag by ourselves at demo playback in our functions made at step 3, and then the gamecode would be now able to call StopFollowing()? This is something to be explored...

--------------------------

So I now think this implementation could be pretty easily done. However, it may not be possible to get a satisfyingly working implementation of the Freeflying feature, but it would be posible to allow for follow switching (multiview) without any problem for sure.

I really do think this path should be explored. I currently have no time to dedicate to make a prototype of this implementation, but if someone out there is interested, I can give a hand and lend some of my knowledge.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on June 28, 2012, 10:26:16 AM
New server-side demo patch for Urban Terror 4.x, based on AlphaIoquake3 patch (cited in the first post of this thread) with a few enhancements:

http://www.urbanterror.info/forums/topic/28657-server-side-demo-recording/

Base patch: https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/b28aaea62ae1543828b71aaf6ff94afdd9463014

Incremental fixes:
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/b0d0f70110096e476a1f1836ddeda557a9e96bc5
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/384d356487d120c37ce4fbff1e1ed38f7d1dea83
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/669c5b2d2c6adb956b6415b3b11dfbde13e505a7
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/0d7185175aafc798eb12ef33916916035f182ae6
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/863f36d115f18bc64c87cbd6b1c1b9eafac3ca68
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/b49fa4f63a31a9e781ca3f5547162c9956db1104
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/90cf8a4616e5b17240b3668aa120802715f2102e
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/24512a775f7a177a79081d68b8c89cfb80ca3638
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/d0bf31ded108b6894ab61a8edd7e1f2c022614a6
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/2ac2562006ce2193170e0a25708f312b9ecf803d
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/7a43630bfa72d649d99c4883e26fbbddffd68f8e
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/0494025b74800cd7f99228bcc7708e16795f1b85

Note: it seems to suffer from the same advantages and issues as the original patch: demos produced are standard (no need for client to patch to read the demos), but the admin has to issue a command to record any subsequent player joining after the demo has started recording. There is a workaround by using an external bot (B3):
http://www.urbanterror.info/forums/topic/28665-urt-serverside-demo-recording-bigbrotherbot-plugin/


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 11, 2013, 09:46:26 AM
The patch was updated, now it works with OA 0.8.8 and even the very latest ioquake3 (on github).

I also have rebased the patch, so that now the commits are clearly at the top of the commits graph (thus it is now very easy to separate the ioquake3 changes, then on top the openarena_engine changes, and then at the very top the serverside demos commits).

I also have fixed the memory leaks, and the patch is currently being thoroughly used on the OAC servers, with no memory leaks problem anymore.

Old server-side demos are still compatible with the new release on github (and will normally always be thank's to sv_demoTolerant facility - but here you don't even need that, the latest changes are totally compatible).

However, there is one last problem: dynamic movers (like the hidden door on Kaos2) produce a crash when playing back the demo.

I will try to fix that last problem later, or just try to make the alternative patch I've been talking about in this thread (replaying the net messages instead of entities positions).


Title: Re: The server-side demos thread, rebooted!
Post by: Danfun64 on December 05, 2015, 04:37:19 PM
Sorry for bumping, but I have no idea how to rebase the serverside engine with the official engine.git. The automatic rebasing failed, and the instructions for manually rebasing confuse me. The fact that I am not a programmer doesn't help matters. Any ideas?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on December 06, 2015, 05:33:10 AM
Sorry for bumping, but I have no idea how to rebase the serverside engine with the official engine.git. The automatic rebasing failed, and the instructions for manually rebasing confuse me. The fact that I am not a programmer doesn't help matters. Any ideas?

Sorry Dan, I don't have the time to rebase and update the git history, and rebasing is indeed quite tricky (I'm not even sure I'll be able to do it by myself, I'll probably have to spend several days to get it done correctly).

However, if you just want to compile a binary for the latest official engine.git, there is probably an easier way, you'll loose the git history, but it will work: you can manually copy the bits of codes that were added into the engine by looking at the diff file, that I reuploaded here:

https://github.com/lrq3000/openarena_engine_serversidedemos/releases/tag/v0.9.9.5

Indeed, the server-side demos facility has been modularized as much as possible, to almost all the logic is managed by the files sv_demo.c (https://github.com/lrq3000/openarena_engine_serversidedemos/blob/v0.9.9.5/code/server/sv_demo.c) and sv_demo_ext.c (https://github.com/lrq3000/openarena_engine_serversidedemos/blob/v0.9.9.5/code/server/sv_demo_ext.c), so first you can just copy those two files right away onto the official engine.git. Then, what remains is to copy all the bits and codes that were added into the other files inside the server/ folder to implement various callbacks and variables definitions so that sv_demo.c and sv_demo_ext.c are called correctly by the engine. To do that, just look in the diff file, and copy the code into the corresponding files (the context will show you where to place those bits of codes, sometimes the placement is important, sometimes not, so I advise you to try to do the placement right).

This should not be too difficult to do. If you achieve that, please don't forget to post a link to your git repo, this may interest other developpers :)


Title: Re: The server-side demos thread, rebooted!
Post by: Danfun64 on December 06, 2015, 07:00:23 PM
There seems to be several patch files that I could use. I don't know the difference between "ioquake3-r2224-delaypatch.zip", "rcontellpatch-engine-r28.zip", "serverside-demo-event-entity-patch_v0.9.9.4.patch.zip", and "serverside-demo-patch-oa.ioquake3-r2224_v0.9.9.5.patch.zip".

What do you recommend?

edit: Maybe it's ignorance, but I find it odd that the serverside patch doesn't properly record the first person view of each player. At least the patch is open source, which Excessive Plus isn't.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on December 09, 2015, 03:56:52 AM
@Danfun64: Use either "serverside-demo-event-entity-patch_v0.9.9.4.patch.zip" or "serverside-demo-patch-oa.ioquake3-r2224_v0.9.9.5.patch.zip", they are the same patch but against different versions of the OA engine. The other patchs are for other functionalities that should now be included in the latest OA engine since a long time.

What do you mean by "the patch doesn't properly record the first person view of each player"? Because the patch does that, and more: it records the whole gamestate of every players, items and games data, and replays them when you replay the demo. There are only two missing features compared to E+ demos: it doesn't fully record movable objects such as platforms (due to a limitation of the accessible fields in the engine, this begs for a fix but I'm not sure how), and it doesn't replay the delag (which is possible to do, E+ does it).


Title: Re: The server-side demos thread, rebooted!
Post by: Danfun64 on December 10, 2015, 10:25:24 PM
How do I apply the patch? Do I have to rename the engine source directory?


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on December 12, 2015, 06:25:07 AM
This is simply a diff unified patch, you can apply it using the standard Linux tools (you need to apply on the whole engine source directory):

http://stackoverflow.com/a/9982282/1121352

However, since you want to apply it on a different version of the engine than the one the patch has been compiled against, I don't think it will work (maybe there are tools to do that? You may try to google search), so you will probably have to manually copy/paste the code from the diff patch to the correct places in the engine source code.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on December 12, 2015, 06:31:40 AM
And if you are on Windows and you need a diff file viewer, you can install TortoiseGit, it comes with TortoiseGitUDiff.


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 04, 2017, 11:24:20 AM
REJOICE! BIG NEWS! (this is not a clickbait ;) ): server-side demos for OpenArena/ioquake3 are now a reality.

1. All (big) issues have been fixed in my server-side demos patch: movers are correctly replayed, and the patch has been tested for years on my servers (and is being used for a new E+ competition (https://www.excessiveplus.net/forums/thread/hit-tournament-2017)), it worked without any issue. This leads to the release of the first stable release v1.0.0, you can grab it here: https://github.com/lrq3000/openarena_engine_serversidedemos/releases

2. Another developper implemented the snapshot-replay approach, an alternative approach to my patch. This implements something more akin to a multiview demo than a real freeflying server side demo, but it's also completely server side and mod-agnostic! This is like TheDoctor's patch, but it saves all demos in one file instead of one demo per player, and it has some more functionalities. To grab the diff patch, go here: http://edawn-mod.org/forum/viewtopic.php?f=5&t=7 (mirror here (https://github.com/lrq3000/openarena_engine_serversidedemos/releases/download/v1.0.0/q3e-multiview-patch-edawn.zip)) and the commands/cvars are here: http://edawn-mod.org/binaries/quake3e-mv.txt

3. My patch is also available for the latest ioquake3 (https://github.com/lrq3000/ioq3/releases) and has been submitted as a pull request (https://github.com/ioquake/ioq3/pull/265).

So that's pretty much the end of this quite long story, with a happy ending :) These patchs could be used to recreate a GTV replacement, a new adventure!



Title: Re: The server-side demos thread, rebooted!
Post by: Gig on March 06, 2017, 03:30:55 AM
Ohh!!! Quite interesting!!!  :)
I was wondering what happened with the development of this stuff...

Let's also say that more infos (including usage infos and "todo") are available here:
https://github.com/lrq3000/openarena_engine_serversidedemos

About including it in OA3, I don't know if Fromhell is going to sync anything more from ioquake3 repo. So it may be necessary to manually pull it to OA3 engine repo... maybe after getting some feedback from ioquake3 guys to make some fixes if necessary. ;)



Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 07, 2017, 04:13:19 PM
Thanks Gig, nice to see you :D

No I don't expect this to be merged in OA3, since it's really only useful to competitive server administrators (so only a very few subset of administrators, which are themselves a very few subset of players), but at least this solution now exists and admins can compile the patch themselves :)

But it should be possible to patch OA3 quite easily, I saw that OA3 is now almost completely synced to the latest ioq3, and the latest patch is now also made for ioq3:

https://github.com/lrq3000/ioq3/tree/server-side-demo


Title: Re: The server-side demos thread, rebooted!
Post by: fromhell on March 07, 2017, 07:53:09 PM
FYI it's not synced to the latest because that SDL2 implementation commit regresses a bunch of things on Windows and most of the changes in the past 2 years is in regards to the GL2 renderer we don't use nor do plan to use.  so for that reason it should patch to the OA3 tree fine


Title: Re: The server-side demos thread, rebooted!
Post by: GrosBedo on March 10, 2017, 05:03:04 AM
Thank you fromhell for the info, this explains why I could easily port the patch to the latest ioq3 then, and so it should also apply to OA3 easily :)

BTW I just discovered that ET-Legacy is using the patch since years, and they fixed a few things. Might be interesting to merge in these fixes, I'll have a look:

* https://dev.etlegacy.com/issues/278
* https://dev.etlegacy.com/projects/etlegacy/wiki/Changelog#Server-7

PS: you're right not to use GL2 renderer, it's awfully slow and incompatible with most hardwares (mine included, I had to disable it just to be able to launch the game).