DEVLOG
--------
**** TheDoctor's patch for OA ****
Code comments:
Fixed the patch (missing header declarations, missing end of the patch) and applied it successfully.
The code is not very much optimized, it's basically a copy/paste of the client's record functions, and I think there may be some collisions and bugs, not to mention that it's not optimized. And I have to mention that the server recording functions did have the same names as clients functions (eg: CL_SaveRecord) and these had to be updated because the compiler would not compile the patch otherwise (eg: SV_SaveRecord).
Result: Demos are saved, but they are empty, containing only the headers and end of the demos specs. No data inside. Dunno if I did something wrong, maybe I'm missing more at the end of the patch than what I guessed. Or maybe I just didn't do the right config or tests for the demos to be recorded effectively, if what mentioned Falkland is true:
And I think it can be improved by adding a cvar for tuning parameters that activate autorecord : as now autorecord starts when a player has reached 0.9*fraglimit
excerpt from:
http://openarena.ws/board/index.php?topic=3790.msg34583#msg34583Then that may explains why I didn't get any output (I tried only with CTF, no fraglimit).
Conclusion: I think TheDoctor should be contacted about that to ask him to make a cleaner patch if his implementation is chosen.
Update: a complete version of the patch can be found here (thank's to Gig and r9k):
http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110I'll update my port and see if that works.
----
Update: I've updated with the complete patch above and fixed some minor bugs, and it works. Demos are recorded and playable with a standard OA installation.
Here are the files:
Patch from engine v28:
http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/TheDoctor-serverside-demo_v0.4.patchFull engine sources patched:
http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/openarena-engine-source-0.8.x-28-thedoctor-serversidedemopatch_v0.4.zipWin32 binaries:
http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/TheDoctor-serverside-demo_engine-r28_v0.4-win32binaries.zipAn example demo produced:
http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/E0C277B1_oasago2_0001.dm_71I won't do anymore work on this path. It fully works, and team chat messages as well as voip aren't recorded (and of course nor IP and GUID since it produces client-side demos).
/Update: I've taken another look over TheDoctor's patch, and it's quite interesting. Instead of hacking every engine's function that manages players and entities, it only hacks 3 functions: SV_WriteSnapshotToClient() and SV_UpdateServerCommandsToClient() and SV_SendClientSnapshot().
In clear terms: it stores and reproduces network snapshots packets instead of storing commands. This is quite an interesting approach, and it should be possible to modify its mechanism to allow for full server-side demos recordings (where you can freefly and spectate all players with a single demo), but I predict this would spawn some problems about how to manage these democlients and filtering of privacy data (because here this is raw data...). Also, there's no management of userinfo, only configstrings. Also, I think it's impossible to record bots this way (they have no network packet sent).
Maybe this path should be explored a bit to see if that's a better alternative than the modified Amanieu's patch.
**** AlphaIOQ3 server-side patchs ****
Did not implement it yet, but I don't think I will since it's pretty much straightforward to patch OA with these. Also, the feature is described as very experimental (but still it should be more mature than TheDoctor's patch), and also has no cvar to autorecord at map change (unlike TheDoctor's patch). But if everything else fail, I will try it.
Update: Maybe a merge between AIOQ3 and TheDoctor's patch (mainly to add the sv_autorecord function) would make a good implementation? Since they are very similar in both their codes and approachs, it should not be that difficult (I think).
Update2: a merge is unnecessary since these two patchs work pretty much the same, there are just some bits that are changing. Most notably, AIOQ3 does not offer a way to record automatically demos (you have to issue a "startserverdemo all" everytime a player connects for the new player to be recorded...
So TheDoctor's patch is far superior to the AIOQ3 implementation in this respect. Anyway, TheDoctor's patch only records clients if they reach a score of 0.9*fraglimit, this should be fixed in the code in order to provide a real server-side demo recording.
Anyway, AIOQ3 provides a much cleaner hook inside sv_snapshot.c, maybe this should be ported back to TheDoctor's patch (but TheDoctor's way is much simpler, it's simply a mirroring of the normal messages with some added bits for the demo recording). However, the "force full frames every now and then" is very weird, this should probably be avoided, being a dirty hack that uses a constant that is not really well defined nor does explain why we should do that (so I guess here the authors did not really knew what they were doing here).
So in conclusion, not that interesting, except for the nice filenaming (but that's not hard), and clean sv_snapshot.c changes. But TheDoctor's patch should keep the preference for a server-side per client demo recording.
**** Tremulous FSM (Mercenarie's guild) patchs ****
Code comments:
The patch was not complete (some headers and functions were missing, mostly due because they are probably also used for other parts of Tremulous), but I could fix everything and compile it.
Result:
Demos seem to be recorded ok (size grow as time goes), but when trying to play the demo, an error is thrown:
\demo_play 20120128234413-ps37ctf
ERROR: Couldn't open svdemos/20120128234413-ps37ctf.svdm_71 for reading.
Don't know if this is because it can't find the file, or it really can't read the demo because it's corrupted. Going to investigate this.
Update: can't make it work. I tried to change the function to FS_FileExists, and still can't open it. But the demo is found if I try the /demo command, so there is a problem in the patch. Can't find where and why, arg. Nasty little bug.
Update2: it works, but I've not yet made the real fix, only a workaround (the fix is not hard, will do tomorrow). A stupid negative condition that checks a filename against an extensions blacklist...
Now the demo is found, in the right path (svdemos folder), loads the sv_demo, loads the map and the other files, but get stuck at "Awaiting Gamestate". Don't know if that's because my demo wasn't long enough (only recorded a few seconds for each, quick tests...) or if it's a bug of processing the playback because there are other missing functions (but it should have warned at compilation, so that's unlikely - or maybe an already existing function but missing some updated features that were not in the patch?).
I'll see later today.
Update3: I've ported some of the tremfusion changes (tremfusion v0.99r3, the latest version as of this writing), and it works better (no crash or hangup when loading the demo anymore), but now it seems that the demo can't load the map. Working on it.
Update4: ported qcommon/msg.c from tremfusion to openarena, now the demos are recorded with the right encoding and when played back, they load the right map and the other files. Unluckily, it gets stuck after the AAS loading, don't really know why. At least it gets stuck clientside (when using openarena_x86).
But when playing the demo serverside (when using oa_ded), it goes beyond: it loads the map (as done clientside), but it fully loads every other parameter, and the server becomes playable. But it's playable as a normal server, the demo doesn't get played back.
I noticed in the code that it's checking for some parameters, and if not the case, reload the map and wait with a command called "delay" (that doesn't exists in OA, only "wait" command exists), then retry to playback the demo. This is where I'm currently investigating.
So it seems that a sv_demo can be played back on a server, and dispatched to connected players. That would potentially be a great replacement for GTV if that's true!
Update5: Almost working: now it replays the demo, but the players movements aren't quite played. But at least the map is loaded and chat messages are displayed.
Quite some work has to be done before it will fully work, but at least there's some hope, and I'm grasping more and more of the engine and the sv_demo mechanism.
Update6: Even more working: loads the demo, the map, and plays it! At least, it replays most of the things: replays the events, replay the weapons, BUT it doesn't draw any entity such as players, flags or powerups yet XD But it WORKS! And indeed, it's possible to connect to a server which is replaying the demo. So it can be used as a GTV. GREAT! Also, I don't know if that's a bug or a feature, but it's possible to freefly in the game!
Update7: I ported the gamecode patch for replaying demos (this allows to prevent players from joining the game while the demo is replaying, and a few other adjustments like drawing with cp when a recorded player connects in the demo), but this doesn't solve the problem. Plus, even if I fixed most problems with the patch, we now get a "A reliable command cycled out" bug, which means that we didn't recorded some important frames at the beginning of the sv_demo, or at least the demo doesn't send them. But if you reconnect to the server replaying the demo, you can then watch the demo, but still no player shown, only actions and weapons and items.
So I got back to the patching of the engine, I leave the gamecode aside for the moment since it won't help to fix this problem, which is now obviously serverside (this is not a problem of rendering the players but of loading their config string).
I think I have a clue, it seems the server isn't recording the config strings. I'm looking for a way to fix it (because it is already implemented, just buggy). Hard and tedious, but it seems we finally see the light.
Update8: It WORKS! Not fully though, but the players are shown, along with their models. So my last clue was right after all! There are still a few more problems to work out, but the serverside code is almost complete. The last main problem is that players are not shown in the scoreboard, so that we can't spectate them, but we can freefly and actions are fully restituted. But I think that the client-side gamecode (that I already ported) should do the trick if it gets a bit fixed.
I have to pause this project for some weeks, so the code will be posted here as a proof of concept meanwhile for those adventurous enough to test it.
Update9: So here it is, the patch is up. Here are the files:
Amanieu's port of server-side demo patch for the engine v28:
http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_engine-r28_v0.4.7.6.patchFull sources of engine v28 patched:
http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/openarena-engine-source-0.8.x-28-amanieu-sv-serversidedemo_v0.4.7.6.zipWin32 binaries:
http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_engine-r28-win32-bin_v0.4.7.6.zipExample of a demo produced (can be played back with demo_play with patched binaries):
http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/demotest.svdm_71Note: you should place this demo in baseoa/svdemos/ folder.
Note2: when replaying this demo, you should set sv_fps 25 (no any other value!) for the demo to be as fluent as when it was recorded (this will be fixed later).
Gamecode port (clientside) - Does NOT really work yet!
Gamecode patch over OAX B50:
http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_gamecode-B50_v0.3.patchFull sources:
http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/openArenaExpansionB50-slackervm-serversidedemo-patched3.zipWhat was done:
x port delay command
x re record demos with new msg.c
x ou porter de TremFusion patch de Amarieu en version compatible, par rapport a Tremulous (mais pas serverside demos?)
x autoset sv_democlients and sv_maxclients and save the old values (de toutes facon on restart la map, donc ca charge la config)
x singleplayer clientside demo_play awaiting gamestate bug when trying to replay a demo
x autoset bot_minplayers and fix bug value
Update10: Trying to patch the client-side gamecode. It's promising, now the bots and players are indeed shown in the scoreboard, and their scores (frags) is OK. The captures scores still aren't (was not implemented for Tremulous since there's no CTF gametype there, so we'll have to do it ourselves). And still can't spectate demo players, neither do say messages work, but at least it now prevents real players from joining in during the demo.
Update11: Ok it now WORKS! One can spectate players, freefly, see the full scoreboards (with ping, but it doesn't get updated after the first snapshot, but at least that gives an idea for demo spectators) and all actions are now handled by the server as if it was played right now (before many actions were performed but not loggued, wasn't very clean). Still the capture score is to be fixed and chat messages too, but at least now the patch is getting very functional!
Update12: I'm getting away from the original tremfsm and tremfusion patchs, I'm completely dropping the gamecode (which was in fact a server-side gamecode patch, not a client-side as I initially thought), and I'm trying to make a server-side demos functionnality completely handled by the engine. This would theoretically allow for more portability to other games, and also it would theoretically allow to use this feature for any mod.
I have implemented a few new functions and I'm extending the demos recording to a lot of other servers functions so as to accurately reproduce the events that happened in the real game. In the end it should be (almost) a full copy of the original game.
Chat messages are now fixed, also are the captures scores, along with a few other things.
Now the last main problem is the player status, should be fixed soon enough (hopefully).
TODO:
x chat messages are not recorded (or at least not displayed)
x client-side demo replaying bug: if while the demo is replaying, we leave to the menu, and try to launch another demo, it will crash (because values are latched but not updated)
x can't spectate (no democlients in the scoreboard nor teams) -> test with patched cgame -> NOT WORKING still same bug
Fake a real player by using something similar to: svs.clients[num].state = CS_ACTIVE;
x captures are not updated on the scoreboard (but they are announced ingame and in the console)
x really fix .sv_dmx extension detection (simply bypassed for now, see files.c)
x set sv_democlients as readonly with CVAR_ROM? -> tried and it works serverside, but for clientside replaying it crash
x save sv_fps in the demo and restore it in playback (and auto restore the normal value after the playback with savedSvFPS)
x save g_gametype
x IMPORTANT: clean up the patch to remove patchs that are maybe unnecessary (in the process). And add a lot of comments to clarify all this mess!
x set sv_democlients to 0 at startup (and always set to 0 when stoppingplayback? and 0 before recording? no more latched?) -> if sv_democlients is readonly, this fixes this problem
(for now it's set to 0 at first launch, but not at each subsequent startup)
FINAL UPDATE: a lot have been added and fixed, the patch is now pretty much stable. To see everything that was done, please check out the other pages of this thread, this is where you will find the rest of the discussion, because the previous updates in this post are up to about v0.4.7.
You can now download the stable release above on GitHub, and all new releases will be done there. There's also a full readme with all the changes that were done, and some of the planned features that may make it in someday (or not).
**** TremFusion patch ****
Not implemented yet, but I will, and we'll see later how it compares to the FSM patch (there are a great lot of things that change, patched files are not even the same at all).
I will port the patch from the Lakitu7's QVM since it seems to be the latest Amarieu has done from reading this post:
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296#p880Update: The patch seems to be a lot more different than the FSM patch, and seems to apply only to Tremulous. Plus, it seems it's only an update (maybe of the FSM patch?), lots of functions declarations are missing.
So it seems that first we have to make the FSM patch work before going any further.
Update2: I have patched the FSM version with some of the tremfusion changes (every /server files are currently patched, but I didn't yet take a look at /qcommon nor /game files). It seems a complete version resides here:
http://www.assembla.com/code/arcade/git/nodes/src/server/sv_demo.c?rev=4b6ec00ac33ac847e825139609eb5a7d9ff584e5Update3: the tremfusion patch can in fact be considered as an extension of the FSM patch: FSM patch being an engine patch, and tremfusion a gamecode patch (mainly for client-side replaying). See FSM devlog above.