Pages: [1] 2 3 ... 7
  Print  
Author Topic: The server-side demos thread, rebooted!  (Read 77653 times)
GrosBedo
Member


Cakes 20
Posts: 710


« on: January 28, 2012, 05:39:58 pm »

RELEASE AVAILABLE

The latest release can always be found here:
https://github.com/lrq3000/openarena_engine_serversidedemos

Also this patch is now ported to the experimental OA 0.8.8 + ioquake3 r2224 (any new update will be done exclusively here):
https://github.com/lrq3000/openarena_engine_serversidedemos/tree/latest

Unified diff patchs, precompiled binaries and a few other interesting patchs (rcon /tell, delay ...) can be found here:
https://github.com/lrq3000/openarena_engine_serversidedemos/downloads

This project is now considered stable. To check the latest news, please read the latest posts in this thread.

----
Hello guys,

I am currently trying to implement server-side demos. This thread is intended as a log of my progress and some of the questions that may arise from the different solutions that currently exists.

I know the devs are currently pretty busy with v0.8.alien and I don't intend to ask for help in code development.

----

My goal here is to provide a patch based on the latest OA binaries, found in this thread:
http://openarena.ws/board/index.php?topic=1933.0

I've focused on patching the binaries (instead of implementing that in a mod such as OAX), because I would like the feature to be compatible with any existing server and, if possible, mods. In summary: more portable, but less possible features. But since server-side demos is not something to be extended further than its minimum functionnality, I think that's ok (if the engine can generate and play server-side demos, that's enough for any use, and mods can still implement further features if needed).

Since I'm not really a specialist in hacking the ioquake3 engine, but I'm good at coding and algorithms, I chose to find currently available implementations in other games using the ioquake3 engine (or a modified version), and backport these changes to OA.

Now the serious matter:

-----

First, here is a previous thread about this subject:
http://openarena.ws/board/index.php?topic=3790.0

Second, here are 2 papers about the demos files specification, which could be very useful for future development/optimization:
http://skuller-vidnoe.narod.ru/
https://github.com/madprof/alpha-ioq3/wiki/Demos

Third, it seems there are only two different types of implementations:

- Server-side demos that record client-side demos for each client (everything is done server-side, but from a client point of view). In this case, each demo is saved in a different demo file per player. Demos should be playable with an unmodified client binary (even with ioquake3).
This type of the server-side demo is the simpler because it's basically a copy-paste of the client code for recording a demo, but ported to the server code. Not very optimized.

- Server-side demos that record full server-side demos by merging all the clients' streams into one file. Advantage is that this results in a single file, which can possibly be more space efficient, and when played it's just like you're a spectator in a game you can't join: you can switch the player you're spectating during the playback. Disadvantage is that it needs a special function to playback those files, clients must have a patched client binary. May also spawns problem with complex playback features (eg: oamme or q3mme rewind function), but not tested yet.

-----

Currently there are 4 implementations available that can easily be ported, and I will divide them by type:

**** Multi client-side demos ****

- TheDoctor (ScrewOA's server admin) patch for OpenArena. Can be found here:
http://openarena.ws/board/index.php?topic=3206.msg34546#msg34546
But this one is incomplete.

Complete patch can be found here:
http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110

Basic Usage: sv_autodemo 1, demos are stored in /homepath/baseoa/demos

- AlphaIOQ3: a fork of ioquake3 by some UrT guys and with collaboration of people from other communities such as Defrag. Pretty easy patch to port, since it's based on ioquake3. Project can be found here:
https://github.com/madprof/alpha-ioq3

Server-side patchs that interests us can be found here (in the order of patching):
https://github.com/madprof/alpha-ioq3/commit/45d78bda1adfd37a274a43bbf80dd3c9224fb779
https://github.com/madprof/alpha-ioq3/commit/94b77f3fb224df02371c952f6f1153c8abbfa71b
https://github.com/madprof/alpha-ioq3/commit/93ef087410157195b1c45b5e0145bfcc0401754e

Basic Usage: startserverdemo all, demos are stored in /homepath/baseoa/serverdemos
More infos here: https://github.com/madprof/alpha-ioq3/wiki

**** Fully server-side demos (merged in one file) ****

- Tremulous/Mercenarie's guild mod patch by Amarieu. This patch is harder to backport. Made in mid-2008. Here are the links:
https://dnc.tremforges.net/trac/ticket/127
http://patches.mercenariesguild.net/index.php?do=details&task_id=196
http://patches.mercenariesguild.net/index.php?getfile=804
http://code.google.com/p/fsm-trem/issues/detail?id=97

Newer resources with some fixes (2009):
http://code.google.com/p/fsm-trem/source/detail?r=178

Basic Usage: sv_autodemo 1, demos are stored in /homepath/baseoa/svdemos
More infos: http://tremulous.net/forum/index.php?topic=9278.0

IMPORTANT:
When recording:
sv_maxclients = whatever you want
sv_democlients = 0

When playing:
sv_maxclients = sv_maxclients of when you were recording + the max number of spectators
sv_democlients = sv_maxclients of when you were recording

Excerpt from a pm copy/pasted here:
http://tremulous.net/forum/index.php?topic=11170.msg167344#msg167344

- TremFusion patch by Amarieu. Made in the end of 2008. Theoretically the latest, but the 2009 commit in fsm-trem maybe implements newer fixes by other contributors (there's at least one). TremFusion patch can be found here:
http://patches.mercenariesguild.net/index.php?getfile=859
http://tremulous.net/forum/index.php?topic=9278.0
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296

But it seems that from reading the following post, the latest version is not the diff file but the direct patch to the qvm:
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296#p880

Note: the correct version is in Lakitu7's QVM, the version in p-g-qvm is buggy:
http://tremulous.net/forum/index.php?topic=9262.msg142914#msg142914

Another resource (don't know the version nor real release date, but it was done after the FSM patch at the very least):
http://www.assembla.com/code/arcade/git/nodes/src/server/sv_demo.c?rev=4b6ec00ac33ac847e825139609eb5a7d9ff584e5

Basic Usage: sv_autodemo 1, demos are stored in /homepath/baseoa/svdemos
More infos: http://tremulous.net/forum/index.php?topic=9278.0
http://www.tremfusion.net/wiki/Features_list#Server-side_demos

UPDATE: Also, the TremFusion project is officially dead (maybe it will revive someday?):
http://www.tremfusion.net/forum/viewtopic.php?f=11&t=350

This means that we can safely port the code to OA since there won't be any major changes in the future, and that it's considered pretty stable (at least, useable, we'll also have to test the ported code).

Update2:
Another patch by SlackersLinux (from Amanieu's code):
http://code.google.com/p/slackers-qvm/source/browse/branches/patches/02-SvDemo-Slackersqvm-1.1.patch?spec=svn99&r=99

- MVDSV for QuakeWorld: a specific engine written exclusively to support server-side demos, but is written for QuakeWorld:
http://qw-dev.net/projects/show/4
« Last Edit: March 23, 2012, 02:47:05 pm by GrosBedo » Logged
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #1 on: January 28, 2012, 05:52:28 pm »

DEVLOG
--------

**** TheDoctor's patch for OA ****

Code comments:
Fixed the patch (missing header declarations, missing end of the patch) and applied it successfully.

The code is not very much optimized, it's basically a copy/paste of the client's record functions, and I think there may be some collisions and bugs, not to mention that it's not optimized. And I have to mention that the server recording functions did have the same names as clients functions (eg: CL_SaveRecord) and these had to be updated because the compiler would not compile the patch otherwise (eg: SV_SaveRecord).

Result: Demos are saved, but they are empty, containing only the headers and end of the demos specs. No data inside. Dunno if I did something wrong, maybe I'm missing more at the end of the patch than what I guessed. Or maybe I just didn't do the right config or tests for the demos to be recorded effectively, if what mentioned Falkland is true:

Quote from: Falkland
And I think it can be improved by adding a cvar for tuning parameters that activate autorecord : as now autorecord starts when a player has reached 0.9*fraglimit
excerpt from: http://openarena.ws/board/index.php?topic=3790.msg34583#msg34583

Then that may explains why I didn't get any output (I tried only with CTF, no fraglimit).

Conclusion: I think TheDoctor should be contacted about that to ask him to make a cleaner patch if his implementation is chosen.

Update: a complete version of the patch can be found here (thank's to Gig and r9k):
http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110

I'll update my port and see if that works.

----
Update: I've updated with the complete patch above and fixed some minor bugs, and it works. Demos are recorded and playable with a standard OA installation.

Here are the files:

Patch from engine v28: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/TheDoctor-serverside-demo_v0.4.patch
Full engine sources patched: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/openarena-engine-source-0.8.x-28-thedoctor-serversidedemopatch_v0.4.zip
Win32 binaries: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/TheDoctor-serverside-demo_engine-r28_v0.4-win32binaries.zip
An example demo produced: http://zsensible.free.fr/openarena/TheDoctor%20serverside%20demo/E0C277B1_oasago2_0001.dm_71

I won't do anymore work on this path. It fully works, and team chat messages as well as voip aren't recorded (and of course nor IP and GUID since it produces client-side demos).

/Update: I've taken another look over TheDoctor's patch, and it's quite interesting. Instead of hacking every engine's function that manages players and entities, it only hacks 3 functions: SV_WriteSnapshotToClient() and SV_UpdateServerCommandsToClient() and SV_SendClientSnapshot().

In clear terms: it stores and reproduces network snapshots packets instead of storing commands. This is quite an interesting approach, and it should be possible to modify its mechanism to allow for full server-side demos recordings (where you can freefly and spectate all players with a single demo), but I predict this would spawn some problems about how to manage these democlients and filtering of privacy data (because here this is raw data...). Also, there's no management of userinfo, only configstrings. Also, I think it's impossible to record bots this way (they have no network packet sent).

Maybe this path should be explored a bit to see if that's a better alternative than the modified Amanieu's patch.

**** AlphaIOQ3 server-side patchs ****

Did not implement it yet, but I don't think I will since it's pretty much straightforward to patch OA with these. Also, the feature is described as very experimental (but still it should be more mature than TheDoctor's patch), and also has no cvar to autorecord at map change (unlike TheDoctor's patch). But if everything else fail, I will try it.

Update: Maybe a merge between AIOQ3 and TheDoctor's patch (mainly to add the sv_autorecord function) would make a good implementation? Since they are very similar in both their codes and approachs, it should not be that difficult (I think).

Update2: a merge is unnecessary since these two patchs work pretty much the same, there are just some bits that are changing. Most notably, AIOQ3 does not offer a way to record automatically  demos (you have to issue a "startserverdemo all" everytime a player connects for the new player to be recorded...
So TheDoctor's patch is far superior to the AIOQ3 implementation in this respect. Anyway, TheDoctor's patch only records clients if they reach a score of 0.9*fraglimit, this should be fixed in the code in order to provide a real server-side demo recording.
Anyway, AIOQ3 provides a much cleaner hook inside sv_snapshot.c, maybe this should be ported back to TheDoctor's patch (but TheDoctor's way is much simpler, it's simply a mirroring of the normal messages with some added bits for the demo recording). However, the "force full frames every now and then" is very weird, this should probably be avoided, being a dirty hack that uses a constant that is not really well defined nor does explain why we should do that (so I guess here the authors did not really knew what they were doing here).

So in conclusion, not that interesting, except for the nice filenaming (but that's not hard), and clean sv_snapshot.c changes. But TheDoctor's patch should keep the preference for a server-side per client demo recording.

**** Tremulous FSM (Mercenarie's guild) patchs ****

Code comments:
The patch was not complete (some headers and functions were missing, mostly due because they are probably also used for other parts of Tremulous), but I could fix everything and compile it.

Result:
Demos seem to be recorded ok (size grow as time goes), but when trying to play the demo, an error is thrown:

Code:
\demo_play 20120128234413-ps37ctf
ERROR: Couldn't open svdemos/20120128234413-ps37ctf.svdm_71 for reading.

Don't know if this is because it can't find the file, or it really can't read the demo because it's corrupted. Going to investigate this.

Update: can't make it work. I tried to change the function to FS_FileExists, and still can't open it. But the demo is found if I try the /demo command, so there is a problem in the patch. Can't find where and why, arg. Nasty little bug.

Update2: it works, but I've not yet made the real fix, only a workaround (the fix is not hard, will do tomorrow). A stupid negative condition that checks a filename against an extensions blacklist...

Now the demo is found, in the right path (svdemos folder), loads the sv_demo, loads the map and the other files, but get stuck at "Awaiting Gamestate". Don't know if that's because my demo wasn't long enough (only recorded a few seconds for each, quick tests...) or if it's a bug of processing the playback because there are other missing functions (but it should have warned at compilation, so that's unlikely - or maybe an already existing function but missing some updated features that were not in the patch?).

I'll see later today.

Update3: I've ported some of the tremfusion changes (tremfusion v0.99r3, the latest version as of this writing), and it works better (no crash or hangup when loading the demo anymore), but now it seems that the demo can't load the map. Working on it.

Update4: ported qcommon/msg.c from tremfusion to openarena, now the demos are recorded with the right encoding and when played back, they load the right map and the other files. Unluckily, it gets stuck after the AAS loading, don't really know why. At least it gets stuck clientside (when using openarena_x86).

But when playing the demo serverside (when using oa_ded), it goes beyond: it loads the map (as done clientside), but it fully loads every other parameter, and the server becomes playable. But it's playable as a normal server, the demo doesn't get played back.

I noticed in the code that it's checking for some parameters, and if not the case, reload the map and wait with a command called "delay" (that doesn't exists in OA, only "wait" command exists), then retry to playback the demo. This is where I'm currently investigating.

So it seems that a sv_demo can be played back on a server, and dispatched to connected players. That would potentially be a great replacement for GTV if that's true!

Update5: Almost working: now it replays the demo, but the players movements aren't quite played. But at least the map is loaded and chat messages are displayed.

Quite some work has to be done before it will fully work, but at least there's some hope, and I'm grasping more and more of the engine and the sv_demo mechanism.

Update6: Even more working: loads the demo, the map, and plays it! At least, it replays most of the things: replays the events, replay the weapons, BUT it doesn't draw any entity such as players, flags or powerups yet XD But it WORKS! And indeed, it's possible to connect to a server which is replaying the demo. So it can be used as a GTV. GREAT! Also, I don't know if that's a bug or a feature, but it's possible to freefly in the game!

Update7: I ported the gamecode patch for replaying demos (this allows to prevent players from joining the game while the demo is replaying, and a few other adjustments like drawing with cp when a recorded player connects in the demo), but this doesn't solve the problem. Plus, even if I fixed most problems with the patch, we now get a "A reliable command cycled out" bug, which means that we didn't recorded some important frames at the beginning of the sv_demo, or at least the demo doesn't send them. But if you reconnect to the server replaying the demo, you can then watch the demo, but still no player shown, only actions and weapons and items.

So I got back to the patching of the engine, I leave the gamecode aside for the moment since it won't help to fix this problem, which is now obviously serverside (this is not a problem of rendering the players but of loading their config string).

I think I have a clue, it seems the server isn't recording the config strings. I'm looking for a way to fix it (because it is already implemented, just buggy). Hard and tedious, but it seems we finally see the light.

Update8: It WORKS! Not fully though, but the players are shown, along with their models. So my last clue was right after all! There are still a few more problems to work out, but the serverside code is almost complete. The last main problem is that players are not shown in the scoreboard, so that we can't spectate them, but we can freefly and actions are fully restituted. But I think that the client-side gamecode (that I already ported) should do the trick if it gets a bit fixed.

I have to pause this project for some weeks, so the code will be posted here as a proof of concept meanwhile for those adventurous enough to test it.

Update9: So here it is, the patch is up. Here are the files:

Amanieu's port of server-side demo patch for the engine v28: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_engine-r28_v0.4.7.6.patch
Full sources of engine v28 patched: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/openarena-engine-source-0.8.x-28-amanieu-sv-serversidedemo_v0.4.7.6.zip
Win32 binaries: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_engine-r28-win32-bin_v0.4.7.6.zip
Example of a demo produced (can be played back with demo_play with patched binaries): http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/demotest.svdm_71
Note: you should place this demo in baseoa/svdemos/ folder.
Note2: when replaying this demo, you should set sv_fps 25 (no any other value!) for the demo to be as fluent as when it was recorded (this will be fixed later).

Gamecode port (clientside) - Does NOT really work yet!
Gamecode patch over OAX B50: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/full-serverside-demo_gamecode-B50_v0.3.patch
Full sources: http://zsensible.free.fr/openarena/Amanieu%20full%20serverside%20demo/openArenaExpansionB50-slackervm-serversidedemo-patched3.zip

What was done:
x port delay command
x re record demos with new msg.c
x ou porter de TremFusion patch de Amarieu en version compatible, par rapport a Tremulous (mais pas serverside demos?)
x autoset sv_democlients and sv_maxclients and save the old values (de toutes facon on restart la map, donc ca charge la config)
x singleplayer clientside demo_play awaiting gamestate bug when trying to replay a demo
x autoset bot_minplayers and fix bug value

Update10: Trying to patch the client-side gamecode. It's promising, now the bots and players are indeed shown in the scoreboard, and their scores (frags) is OK. The captures scores still aren't (was not implemented for Tremulous since there's no CTF gametype there, so we'll have to do it ourselves). And still can't spectate demo players, neither do say messages work, but at least it now prevents real players from joining in during the demo.

Update11: Ok it now WORKS! One can spectate players, freefly, see the full scoreboards (with ping, but it doesn't get updated after the first snapshot, but at least that gives an idea for demo spectators) and all actions are now handled by the server as if it was played right now (before many actions were performed but not loggued, wasn't very clean). Still the capture score is to be fixed and chat messages too, but at least now the patch is getting very functional!

Update12: I'm getting away from the original tremfsm and tremfusion patchs, I'm completely dropping the gamecode (which was in fact a server-side gamecode patch, not a client-side as I initially thought), and I'm trying to make a server-side demos functionnality completely handled by the engine. This would theoretically allow for more portability to other games, and also it would theoretically allow to use this feature for any mod.
I have implemented a few new functions and I'm extending the demos recording to a lot of other servers functions so as to accurately reproduce the events that happened in the real game. In the end it should be (almost) a full copy of the original game.
Chat messages are now fixed, also are the captures scores, along with a few other things.
Now the last main problem is the player status, should be fixed soon enough (hopefully).

TODO:
x chat messages are not recorded (or at least not displayed)
x client-side demo replaying bug: if while the demo is replaying, we leave to the menu, and try to launch another demo, it will crash (because values are latched but not updated)
x can't spectate (no democlients in the scoreboard nor teams) -> test with patched cgame -> NOT WORKING still same bug
Fake a real player by using something similar to: svs.clients[num].state = CS_ACTIVE;
x captures are not updated on the scoreboard (but they are announced ingame and in the console)
x really fix .sv_dmx extension detection (simply bypassed for now, see files.c)
x set sv_democlients as readonly with CVAR_ROM? -> tried and it works serverside, but for clientside replaying it crash
x save sv_fps in the demo and restore it in playback (and auto restore the normal value after the playback with savedSvFPS)
x save g_gametype
x IMPORTANT: clean up the patch to remove patchs that are maybe unnecessary (in the process). And add a lot of comments to clarify all this mess!
x set sv_democlients to 0 at startup (and always set to 0 when stoppingplayback? and 0 before recording? no more latched?) -> if sv_democlients is readonly, this fixes this problem
(for now it's set to 0 at first launch, but not at each subsequent startup)

FINAL UPDATE: a lot have been added and fixed, the patch is now pretty much stable. To see everything that was done, please check out the other pages of this thread, this is where you will find the rest of the discussion, because the previous updates in this post are up to about v0.4.7.

You can now download the stable release above on GitHub, and all new releases will be done there. There's also a full readme with all the changes that were done, and some of the planned features that may make it in someday (or not).

**** TremFusion patch ****

Not implemented yet, but I will, and we'll see later how it compares to the FSM patch (there are a great lot of things that change, patched files are not even the same at all).

I will port the patch from the Lakitu7's QVM since it seems to be the latest Amarieu has done from reading this post:
http://www.tremfusion.net/forum/viewtopic.php?f=3&t=296#p880

Update: The patch seems to be a lot more different than the FSM patch, and seems to apply only to Tremulous. Plus, it seems it's only an update (maybe of the FSM patch?), lots of functions declarations are missing.

So it seems that first we have to make the FSM patch work before going any further.

Update2: I have patched the FSM version with some of the tremfusion changes (every /server files are currently patched, but I didn't yet take a look at /qcommon nor /game files). It seems a complete version resides here:

http://www.assembla.com/code/arcade/git/nodes/src/server/sv_demo.c?rev=4b6ec00ac33ac847e825139609eb5a7d9ff584e5

Update3: the tremfusion patch can in fact be considered as an extension of the FSM patch: FSM patch being an engine patch, and tremfusion a gamecode patch (mainly for client-side replaying). See FSM devlog above.
« Last Edit: March 28, 2012, 08:16:03 pm by GrosBedo » Logged
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #2 on: January 28, 2012, 05:56:12 pm »

QUESTIONS
-----------

Now guys, if you had the courage to read all of what I've written above, I would like to know what you are thinking about these two questions:

1- What is the best type of server-side demo for you? Standard per-client demos or server-side demos?

2- Which patch is the newest, particularly between Tremulous FSM and TremFusion? (I will add more informations later so please leave this question aside for the moment).

PS: I'm creating multiple posts so that I get spaces to update them as I go along my tests.

PS2: I will provide unified patchs for everything I could implement (successfully or not), from the latest OA binaries (r28 from Sago's repository).
Logged
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #3 on: January 29, 2012, 01:13:52 am »

HELP: Do someone know if subfolders have to be specified somewhere, like the demos folder? I tried to crawl the code but could not find any declaration.

/EDIT: Ok I think I got it, testing...
/EDIT2: Ok got it, stupid condition that checks if a file is valid if it has the right extension. Viva comments (there are none that makes sense about that - thx ioq3 guys)!
Personal reminder: always avoid negative conditions, very awful to debug for someone else (or comment!).
« Last Edit: January 29, 2012, 01:40:14 am by GrosBedo » Logged
sago007
Posts a lot
*

Cakes 62
Posts: 1652


Open Arena Developer


WWW
« Reply #4 on: January 29, 2012, 03:33:45 am »

1- What is the best type of server-side demo for you? Standard per-client demos or server-side demos?

2- Which patch is the newest, particularly between Tremulous FSM and TremFusion? (I will add more informations later so please leave this question aside for the moment).
1 - Server-side demoes that does not bind to a particular client. They should be clean enough to distribute: They should not contain rcon commands, admin commands or GUIDs.

2 - I have no idea.

There are a few things that I don't expect you to look at but if you stumbles across something it would be great:
An option to delag hitscan weapons in the server demo. Normally the server draws the shots when it receives them from the client because it does not know about them before. However during a replay the server could potentially know when the client hit the trigger and draw it then.

I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.
Logged

There are nothing offending in my posts.
Gig
In the year 3000
***

Cakes 48
Posts: 4281


WWW
« Reply #5 on: January 29, 2012, 06:55:53 am »

GrosBedo, that "r9k" guy posted on the wiki DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/Talk:Manual/Demos#Server-side_demos]here. He mentioned a link that you already wrote here (http://patches.mercenariesguild.net/index.php?do=details&task_id=196 server-side demos for Tremolous), and another one.... http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&p=14110#14110
Logged

I never want to be aggressive, offensive or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #6 on: January 29, 2012, 01:34:08 pm »

1 - Server-side demoes that does not bind to a particular client. They should be clean enough to distribute: They should not contain rcon commands, admin commands or GUIDs.
From what I understand from the amarieu's code, it does not bind to any particular client, everything is fully done server-side with server-side packets.

About the cleaning of the data: I will see what I can do, but my main goal is to get something that works, I may not be able to implement this kind of filtering due to my lack of knowledge of the engine.

But I agree with this specification. These sort of data should be filtered.

There are a few things that I don't expect you to look at but if you stumbles across something it would be great:
An option to delag hitscan weapons in the server demo. Normally the server draws the shots when it receives them from the client because it does not know about them before. However during a replay the server could potentially know when the client hit the trigger and draw it then.

I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.

For the delag, I think that this should not be saved in the file but simulated in the playback (because anyway we can't simulate the delag for every player at once since this is a subjective, per-player shift). So this can be later added client-side as a toggleable cvar. ExcessivePlus implements a similar feature for their multi-view demo, so this is certainly possible.

If I can do it I will for sure, but I don't think I will be able.


Thank's a lot Gig (and r9k)! Indeed, the patch provided in the rainbow's forum is complete, and indeed the end of the patch that I "guessed" was wrong, there are a lot of other modifications. This patch then should work, I'll update my port and see if that works better.
Logged
PopeJo
Lesser Nub


Cakes 12
Posts: 105



WWW
« Reply #7 on: January 30, 2012, 01:32:00 am »

**** TheDoctor's patch for OA ****

Quote from: Falkland
And I think it can be improved by adding a cvar for tuning parameters that activate autorecord : as now autorecord starts when a player has reached 0.9*fraglimit

the doctor !!?? falkland!!!??? they are still around?
havent seen any of them for a year or so. you are quoting old stuff, right?

if not, send them some greets   Cheesy

/ontopic: and good luck with the server-side demos project  Smiley
can't really help you with your questions there..
Logged

GrosBedo
Member


Cakes 20
Posts: 710


« Reply #8 on: January 30, 2012, 03:25:17 pm »

I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.

Currently there's no facility for saving and reloading the same files, nor there seems to be for client-side demos. So I think that for now I won't be able to implement such a thing because I don't know the whole mechanism of file management and demos encoding, but if this feature gets implemented for client-side demos I may be able to port it to server-side demos.
Logged
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #9 on: January 30, 2012, 03:33:29 pm »

I would like the demo files to remember the files that was loaded while they where recorded, so the same files can be loaded while replaying.

Are you talking about fs_referencedList (i.e. a list of used pk3s with names and checksums)? I don't quite see the advantage of this.
During playback you most often like to use different pk3s with brightskins, hd texture packs or replacement player models etc. Even if you had a list of pk3s being used, you'd still have to download them somehow if they're not on your pc already. Finally it would render OA demos incompatible to Q3/ioq3. If you're talking about complete server-side demos it would not matter that much, since those including all player data have their own custom file format anyways.
Logged

This space is for rent.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #10 on: January 31, 2012, 02:22:02 pm »

For the sv_demo to work, I have to port some of other necessary commands such as /delay (which is similar to /wait but doesn't stop the engine from processing other commands meanwhile, and this is necessary for how sv_demo works, although I think this behaviour could be fixed without /delay but I'll leave it to someone else to do).

Interestingly, I stumbled upon other interesting extensions of the engine, such as a /random command to get a random integer, and other scripting extensions such as variables substitutions.

If I ever finish the server-side demo port, I may make a few other patchs for these scripting facilities if someone is interested. Also, these changes should (theoretically) work for mods since it's made in the binaries.

/EDIT: The TremFusion project is officially dead! Okay, this may not seem to be a very great news, but this means for us that we can safely port the TremFusion code because there won't expect major changes in the future.

/EDIT2: There are more and more sources that seems to point that indeed, a serverside demo from Amanieu's patch can be joined by players! So this could be used as a GTV server! We'll see if I can make it work! /EDIT: indeed, it seems it can be used as a replacement for GTV by loading a serverside demo on a server, and let players to connect in the server!
« Last Edit: January 31, 2012, 05:50:57 pm by GrosBedo » Logged
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #11 on: January 31, 2012, 04:58:38 pm »

Sago, is there a command to queue another command to be executed later? I saw that there is a sort of queue implemented, how does it work? It can be either an ingame command or a C function, I can use both.
Logged
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #12 on: February 01, 2012, 11:53:49 am »

Why do you need delayed command execution anyways?
Logged

This space is for rent.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #13 on: February 01, 2012, 04:21:16 pm »

Why do you need delayed command execution anyways?

Because when you're playing a server-side demo, if the server do not meet the requirement, it executes a /devmap, and then a delay, and send again the server-side demo playback. So the delay needs to not stuck the whole engine, else it won't load the map before playing again the sv_demo, resulting in some weird errors.

That's why I think there can be other ways around, but for now I simply ported the delay command to make it work. When it will, we'll see how to enhance the code.

----

Back to the subject, every events really work, flags and weapons and items are shown, but characters are not displayed (neither bots nor players). This is weird: you see every shots and every action is in the log, but there's noone XD And it's possible to join in the game during the demo. This is also weird XD

I think I will now have to alter the client to make the server-side demos replaying totally functional, but it seems that server-side it fully works now.
Logged
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #14 on: February 02, 2012, 11:20:57 am »

I've got a problem with bots, they're aren't properly managed in the demo (their actions are recorded but not their names and related player infos), as opposed to players which are managed properly.

I saw that OA added several changes to the behavior of bots.

Could someone (sago?) describe me succinctly those changes to help me understand how these work and why they were done?
Logged
Graion Dilach
Member


Cakes 12
Posts: 403



« Reply #15 on: February 02, 2012, 11:48:52 am »

The changes were done by dmn_clown. You must understand that a huge part of the botcode wasn't released under the GPL2 (the entire content of the botfiles folder) and dmn_clown aimed for a more complexed botlogic than Mr Elusive for more challenge.
Logged

One shall remind what have he left behind... to actually realize that it's still cool.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #16 on: February 02, 2012, 03:35:48 pm »

Thank you Graion, I feared this answer, because dmn_clown is not active anymore in this project.

I'm mainly talking about the code in game/ and server/ folders, because it seems that not only the bot logic was changed but also the game logic and how bots are considered for the engine, so they are now kind of "special" in regards to the engine, which is not very good for server-side demos.

Well, I guess I'll have to find a way on my own.
Logged
sago007
Posts a lot
*

Cakes 62
Posts: 1652


Open Arena Developer


WWW
« Reply #17 on: February 02, 2012, 03:52:51 pm »

I'm mainly talking about the code in game/ and server/ folders, because it seems that not only the bot logic was changed but also the game logic and how bots are considered for the engine, so they are now kind of "special" in regards to the engine, which is not very good for server-side demos.
I think id had a single programmer to create the bot code and he worked separately from the rest. The bot code seems hooked into the game in a very weird way. This is not new in OpenArena. The only difference I have made is that bots now look at the client struct to see what team they are on because sometimes that bot logic and server logic would get out of sync and the bot would attack teammates.
Logged

There are nothing offending in my posts.
Gig
In the year 3000
***

Cakes 48
Posts: 4281


WWW
« Reply #18 on: February 02, 2012, 04:04:46 pm »

sometimes that bot logic and server logic would get out of sync and the bot would attack teammates.
I experienced this! Probably it is not yet totally fixed...
« Last Edit: February 02, 2012, 04:22:50 pm by Gig » Logged

I never want to be aggressive, offensive or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #19 on: February 02, 2012, 05:34:41 pm »

@sago: Thank you for the clarification, but this is weird, because there are many bots functions that I can't find in the tremulous or tremfusion code that is also based on id tech 3:

in game/g_public.h:
Code:
BOTLIB_SETUP = 200, // ( void );
BOTLIB_SHUTDOWN, // ( void );
BOTLIB_LIBVAR_SET,
BOTLIB_LIBVAR_GET,
BOTLIB_PC_ADD_GLOBAL_DEFINE,
BOTLIB_START_FRAME,
BOTLIB_LOAD_MAP,
BOTLIB_UPDATENTITY,
BOTLIB_TEST,

BOTLIB_GET_SNAPSHOT_ENTITY, // ( int client, int ent );
BOTLIB_GET_CONSOLE_MESSAGE, // ( int client, char *message, int size );
BOTLIB_USER_COMMAND, // ( int client, usercmd_t *ucmd );

BOTLIB_AAS_ENABLE_ROUTING_AREA = 300,
BOTLIB_AAS_BBOX_AREAS,
BOTLIB_AAS_AREA_INFO,
BOTLIB_AAS_ENTITY_INFO,

BOTLIB_AAS_INITIALIZED,
BOTLIB_AAS_PRESENCE_TYPE_BOUNDING_BOX,
BOTLIB_AAS_TIME,

BOTLIB_AAS_POINT_AREA_NUM,
BOTLIB_AAS_TRACE_AREAS,

BOTLIB_AAS_POINT_CONTENTS,
BOTLIB_AAS_NEXT_BSP_ENTITY,
BOTLIB_AAS_VALUE_FOR_BSP_EPAIR_KEY,
BOTLIB_AAS_VECTOR_FOR_BSP_EPAIR_KEY,
BOTLIB_AAS_FLOAT_FOR_BSP_EPAIR_KEY,
BOTLIB_AAS_INT_FOR_BSP_EPAIR_KEY,

BOTLIB_AAS_AREA_REACHABILITY,

BOTLIB_AAS_AREA_TRAVEL_TIME_TO_GOAL_AREA,

BOTLIB_AAS_SWIMMING,
BOTLIB_AAS_PREDICT_CLIENT_MOVEMENT,

BOTLIB_EA_SAY = 400,
BOTLIB_EA_SAY_TEAM,
BOTLIB_EA_COMMAND,

BOTLIB_EA_ACTION,
BOTLIB_EA_GESTURE,
BOTLIB_EA_TALK,
BOTLIB_EA_ATTACK,
BOTLIB_EA_USE,
BOTLIB_EA_RESPAWN,
BOTLIB_EA_CROUCH,
BOTLIB_EA_MOVE_UP,
BOTLIB_EA_MOVE_DOWN,
BOTLIB_EA_MOVE_FORWARD,
BOTLIB_EA_MOVE_BACK,
BOTLIB_EA_MOVE_LEFT,
BOTLIB_EA_MOVE_RIGHT,

BOTLIB_EA_SELECT_WEAPON,
BOTLIB_EA_JUMP,
BOTLIB_EA_DELAYED_JUMP,
BOTLIB_EA_MOVE,
BOTLIB_EA_VIEW,

BOTLIB_EA_END_REGULAR,
BOTLIB_EA_GET_INPUT,
BOTLIB_EA_RESET_INPUT,


BOTLIB_AI_LOAD_CHARACTER = 500,
BOTLIB_AI_FREE_CHARACTER,
BOTLIB_AI_CHARACTERISTIC_FLOAT,
BOTLIB_AI_CHARACTERISTIC_BFLOAT,
BOTLIB_AI_CHARACTERISTIC_INTEGER,
BOTLIB_AI_CHARACTERISTIC_BINTEGER,
BOTLIB_AI_CHARACTERISTIC_STRING,

BOTLIB_AI_ALLOC_CHAT_STATE,
BOTLIB_AI_FREE_CHAT_STATE,
BOTLIB_AI_QUEUE_CONSOLE_MESSAGE,
BOTLIB_AI_REMOVE_CONSOLE_MESSAGE,
BOTLIB_AI_NEXT_CONSOLE_MESSAGE,
BOTLIB_AI_NUM_CONSOLE_MESSAGE,
BOTLIB_AI_INITIAL_CHAT,
BOTLIB_AI_REPLY_CHAT,
BOTLIB_AI_CHAT_LENGTH,
BOTLIB_AI_ENTER_CHAT,
BOTLIB_AI_STRING_CONTAINS,
BOTLIB_AI_FIND_MATCH,
BOTLIB_AI_MATCH_VARIABLE,
BOTLIB_AI_UNIFY_WHITE_SPACES,
BOTLIB_AI_REPLACE_SYNONYMS,
BOTLIB_AI_LOAD_CHAT_FILE,
BOTLIB_AI_SET_CHAT_GENDER,
BOTLIB_AI_SET_CHAT_NAME,

BOTLIB_AI_RESET_GOAL_STATE,
BOTLIB_AI_RESET_AVOID_GOALS,
BOTLIB_AI_PUSH_GOAL,
BOTLIB_AI_POP_GOAL,
BOTLIB_AI_EMPTY_GOAL_STACK,
BOTLIB_AI_DUMP_AVOID_GOALS,
BOTLIB_AI_DUMP_GOAL_STACK,
BOTLIB_AI_GOAL_NAME,
BOTLIB_AI_GET_TOP_GOAL,
BOTLIB_AI_GET_SECOND_GOAL,
BOTLIB_AI_CHOOSE_LTG_ITEM,
BOTLIB_AI_CHOOSE_NBG_ITEM,
BOTLIB_AI_TOUCHING_GOAL,
BOTLIB_AI_ITEM_GOAL_IN_VIS_BUT_NOT_VISIBLE,
BOTLIB_AI_GET_LEVEL_ITEM_GOAL,
BOTLIB_AI_AVOID_GOAL_TIME,
BOTLIB_AI_INIT_LEVEL_ITEMS,
BOTLIB_AI_UPDATE_ENTITY_ITEMS,
BOTLIB_AI_LOAD_ITEM_WEIGHTS,
BOTLIB_AI_FREE_ITEM_WEIGHTS,
BOTLIB_AI_SAVE_GOAL_FUZZY_LOGIC,
BOTLIB_AI_ALLOC_GOAL_STATE,
BOTLIB_AI_FREE_GOAL_STATE,

BOTLIB_AI_RESET_MOVE_STATE,
BOTLIB_AI_MOVE_TO_GOAL,
BOTLIB_AI_MOVE_IN_DIRECTION,
BOTLIB_AI_RESET_AVOID_REACH,
BOTLIB_AI_RESET_LAST_AVOID_REACH,
BOTLIB_AI_REACHABILITY_AREA,
BOTLIB_AI_MOVEMENT_VIEW_TARGET,
BOTLIB_AI_ALLOC_MOVE_STATE,
BOTLIB_AI_FREE_MOVE_STATE,
BOTLIB_AI_INIT_MOVE_STATE,

BOTLIB_AI_CHOOSE_BEST_FIGHT_WEAPON,
BOTLIB_AI_GET_WEAPON_INFO,
BOTLIB_AI_LOAD_WEAPON_WEIGHTS,
BOTLIB_AI_ALLOC_WEAPON_STATE,
BOTLIB_AI_FREE_WEAPON_STATE,
BOTLIB_AI_RESET_WEAPON_STATE,

BOTLIB_AI_GENETIC_PARENTS_AND_CHILD_SELECTION,
BOTLIB_AI_INTERBREED_GOAL_FUZZY_LOGIC,
BOTLIB_AI_MUTATE_GOAL_FUZZY_LOGIC,
BOTLIB_AI_GET_NEXT_CAMP_SPOT_GOAL,
BOTLIB_AI_GET_MAP_LOCATION_GOAL,
BOTLIB_AI_NUM_INITIAL_CHATS,
BOTLIB_AI_GET_CHAT_MESSAGE,
BOTLIB_AI_REMOVE_FROM_AVOID_GOALS,
BOTLIB_AI_PREDICT_VISIBLE_POSITION,

BOTLIB_AI_SET_AVOID_GOAL_TIME,
BOTLIB_AI_ADD_AVOID_SPOT,
BOTLIB_AAS_ALTERNATIVE_ROUTE_GOAL,
BOTLIB_AAS_PREDICT_ROUTE,
BOTLIB_AAS_POINT_REACHABILITY_AREA_INDEX,

BOTLIB_PC_LOAD_SOURCE,
BOTLIB_PC_FREE_SOURCE,
BOTLIB_PC_READ_TOKEN,
BOTLIB_PC_SOURCE_FILE_AND_LINE

} gameImport_t;

Code:
BOTAI_START_FRAME // ( int time );
} gameExport_t;

All these functions and variables do not exist in the trem code. But maybe trem is based on a previous version of id tech 3, so this may explain why.

Anyway, I think I now found the root of the problem of why the players are not shown. If at least I can properly draw players, we'll see later for bots.

sometimes that bot logic and server logic would get out of sync and the bot would attack teammates.
I experienced this! Probably it is not yet totally fixed...

For me, it happened a few times but only because I wasn't recognized as a player of the team (I was in TEAMFREE), so the bots, logically, considered me as an enemy. You can easily spot this bug when you see that all the bots get the same color (or the normal color if you didn't force model) whatever their team is (eg: for me all bots were red, even blue ones, and I could also hit them).

This is a very very rare bug (and I didn't modify the binaries when it happened, and it happened with all versions of OA I know of).
Logged
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #20 on: February 02, 2012, 05:46:05 pm »

there are many bots functions that I can't find in the tremulous or tremfusion code that is also based on id tech 3
Quote from: Tremulous FAQ
Does Tremulous have bots?
No. Programming bots for a game as complex as Tremulous is not a trivial task. It is by no means impossible, but it is potentially extremely time consuming for questionable real world gain. Of course, given the open source nature of Tremulous, there is nothing stopping a third party implementing bots.
Maybe they've just removed those lines if they dropped botsupport? id Quake 1.32c still/already has these Smiley
Logged

This space is for rent.
Gig
In the year 3000
***

Cakes 48
Posts: 4281


WWW
« Reply #21 on: February 03, 2012, 01:42:00 am »

For me, it happened a few times but only because I wasn't recognized as a player of the team...
Not sure about how it worked exactly when I noticed the problem. I was trying to test some variables that seem to -in theory- control console chats lasting (one seemed for team chat duration, in particular)... so I loaded oa_ctf4ish and added some bots, giving them team orders to make them answer via team chat. But they didn't reply (maybe later someone -maybe Graion or Chaoticsoldier?- told me that OpenArena bots chat does not include team chat yet!)... and they totally ignored my team orders (they did not follow me when I asked them to follow me, etc.) -so I was unable to really test those variables, maybe I could do some more tests when I have some time...-. More, IIRC, strange things happened like bots tending to all join the same team if I added them without specifying one, and attacking me when I was on their team. But time passed, and I should do more tests to be more detailed.
Logged

I never want to be aggressive, offensive or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #22 on: February 04, 2012, 10:50:37 am »

UPDATE: It WORKS! Not fully though, still some bugs (eg: we can only freefly), but players infos and actions are fully restituted. So the server-side code is almost complete, only client-side gamecode will have to be fixed.

I'm going to clean a bit the code and try to fix some minor bugs, and then I'll post the patch + full sourcecode tonight.

I will be off for about 2 weeks, so I'll pause this project meanwhile, but the code will allow the most adventurous here to try it and see for themselves how it works.
Logged
GrosBedo
Member


Cakes 20
Posts: 710


« Reply #23 on: February 04, 2012, 12:31:10 pm »

sv_maxclients and sv_democlients are now automatically set! Next: client-side demo replaying. Almost works, but one has to load the demo twice before it loads up. Shouldn't be too hard to fix Smiley

I've got a question to the devs around here: how to set a cvar as read-only? I would like to set sv_democlients to readonly since now it will be automatically set.
Logged
grey matter
Member


Cakes 8
Posts: 381

>9k


« Reply #24 on: February 04, 2012, 12:44:44 pm »

Code:
sv_demoClients = Cvar_Get ("sv_demoClients", "", CVAR_ROM);
Logged

This space is for rent.
Pages: [1] 2 3 ... 7
  Print  
 
Jump to: