Pages: 1 ... 5 6 [7]
  Print  
Author Topic: The server-side demos thread, rebooted!  (Read 8927 times)
Gig
Millenial poster
**

Cakes 39
Posts: 2297


WWW
« Reply #150 on: March 23, 2012, 06:17:22 am »

Thank you. But I dont know if I will be able to do any test before sunday...
Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #151 on: March 23, 2012, 11:27:33 am »

@Gig: no problem, don't be sorry, I am glad that at least someone plan to try this patch :p

NEWS: Finalized the code, and ported to OA+ioquake3 r2224. Unified diff patchs are provided. This is the same code for OA 0.8.8 and the hybrid OA 0.8.8 + ioquake3.

You can find the OA 0.8.8 server-side demos project here (branch "master"):
https://github.com/lrq3000/openarena_engine_serversidedemos

And the OA 0.8.8+ioquake3 r2224 server-side demos project here (branch "latest"):
https://github.com/lrq3000/openarena_engine_serversidedemos/tree/latest

Unified patchs for both branches and win32 binaries and a few demo examples can be found here:
https://github.com/lrq3000/openarena_engine_serversidedemos/downloads

I consider this project finished. I may update it if I get some feedback, but I won't promise any precise date.

As a side note, I have tested a bit the storage efficiency, it seems server-side demos produced by this patch are about nbplayers times the size of a standard demo. In my test, you can expect 125 KB/min per player. For a real game of 10 players (and no spectators) and 20 min timelimit, if we extrapolate from this observation, we should expect 125*10*20 = 25 MB for one demo. However, real tests on production servers should be performed in order to get a better estimate of the storage consumption.

My warm thank's go to all the people who participated in this thread and tried to help me in one way or another, especially sago and grey matter who both were very patient with me.
« Last Edit: March 24, 2012, 03:22:59 am by GrosBedo » Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #152 on: March 23, 2012, 02:25:58 pm »

Opening to the future

I have taken a look at a possible alternative way of making such a patch, based on TheDoctor's approach.

At first, I thought that the approach was different and based on flux. But in fact, it's the exact same approach as Amanieu's, the difference being that TheDoctor's patch rely on client-side facilities to replay demos (so it NEEDS a client to replay a demo), while Amanieu's code didn't: it has its very own replaying facility server-side.

But this approach is still interesting.

More details follow.

----

TheDoctor's patch simply dumps the data packets sent to clients.

Then, the client can replay the demo by using the standard demo facility (because this is how standard demos are made: from network packets dumps).

From cl_parse.c, CL_ParseServerMessage() function:

Code:
//
// parse the message
//
while ( 1 ) {
if ( msg->readcount > msg->cursize ) {
Com_Error (ERR_DROP,"CL_ParseServerMessage: read past end of server message");
break;
}

cmd = MSG_ReadByte( msg );

if (cmd == svc_EOF) {
SHOWNET( msg, "END OF MESSAGE" );
break;
}

if ( cl_shownet->integer >= 2 ) {
if ( (cmd < 0) || (!svc_strings[cmd]) ) {
Com_Printf( "%3i:BAD CMD %i\n", msg->readcount-1, cmd );
} else {
SHOWNET( msg, svc_strings[cmd] );
}
}

// other commands
switch ( cmd ) {
default:
Com_Error (ERR_DROP,"CL_ParseServerMessage: Illegible server message");
break;
case svc_nop:
break;
case svc_serverCommand:
CL_ParseCommandString( msg );
break;
case svc_gamestate:
CL_ParseGamestate( msg );
break;
case svc_snapshot:
CL_ParseSnapshot( msg );
break;
case svc_download:
CL_ParseDownload( msg );
break;
case svc_voip:
#ifdef USE_VOIP
CL_ParseVoip( msg );
#endif
break;
}
}

This is the code that parse the packets and replay a standard demo.

On the other hand, Amanieu's approach was to dump the events, not the data packets. This is more insightful in our case, because this is what allows to replay them.

If we only dump the data packets, there is absolutely no facility provided server-side to replay them (because they are designed to be an update specific to the target client), so to fix that we would either need to include client/ files to access the functions, or copy/mimic them, which is neither satisfying.

But if we dump events, we have a broad range of functions provided in the server/ files to play these events. Hence the approach of Amanieu.

-----

All that is to say that it's currently not possible to produce multi-view server-side demos with TheDoctor's approach since there's no replaying facility and data packets are designed per client. Everything is replayed client-side, so there's no way for example to allow for freeflying spectators since when you play the demo you are in the "eye of the beholder".

However, I think there's some ways to tweak this approach and come to an "hybrid" approach that would overcome these limitations:

- make a replay facility server-side, by feeding real clients some fake packets containing the messages of a demo. This can easily be done by just reading a message from a demo and using SV_SendMessageToClient(client, &msg). What is harder is to prevent the server from sending conflicting data to clients (because the server will continue to send them the valid gamestate), without crashing the clients nor the server.

- instead of recording every client's flux separately, we could save them all in the same file and add an event "End of frame" which would allow to replay these packets at the right time (similarly to Amanieu's approach, see my patch).

- hook clientCommand and when the client send a "follow" or "follownext" command, the server would send them the flux of the democlient associated with the number. But it seems that FollowCycle (when you click your mouse button) is not catched this way, so it would be needed to edit the gamecode at least to allow the hooking of FollowCycle.

- to allow freeflying, spectators that are known to be freeflying right now would be sent just the entities states and player states. A gamecode hook would also be needed here, because I don't know of a way to know the state of a spectator (and who he is following) completely server-side, but from the gamecode it's easily available.


As you can see, this implementation would greatly outdo my current implementation in terms of performance, because everything would be computed client-side and not anymore server-side (even if my implementation is already pretty inexpensive on CPU since there's nearly no gamecode event, everything is simulated, no actions processing involved). However, it would not be possible to use it with mods (unless workarounds are found for the above problems).

Anyway I do think this approach could be very interesting for a GTV replacement, since it would allow to support a big crowd of players watching the same demo flux.

----

As a memo, here's the big picture of standard demo replaying client-side:
CL_SetCGameTime -> CL_ReadDemoMessage() -> CL_ParseServerMessage() parse the servers messages -> various parsing functions...

And how TheDoctor's patch write demo messages:
SV_SendClientSnapshot() -> SV_UpdateServerCommandsToClient() and SV_WriteSnapshotToClient() -> back to sendclientsnapshot -> SV_SendMessageToClient()

----

In summary, both patchs are very similar, except for where the demo events are processed: client-side for TheDoctor's, server-side for Amanieu's. And this is all where the problem resides in making a fully server-side demo but processed client-side: how to allow freefly and viewpoint change when everything is processed client-side?
« Last Edit: March 27, 2012, 07:23:49 am by GrosBedo » Logged
Gig
Millenial poster
**

Cakes 39
Posts: 2297


WWW
« Reply #153 on: March 26, 2012, 04:14:18 pm »

I've finally been able to try 0.9.9.5 win32 binaries.
A quick try, but I have to tell you GREAT! A cake for you!  Smiley  Smiley  Smiley

A little thing I noticed:
Quote
sv_demoState : show the current demo state (0: none, 1: waiting to play a demo, 2: demo playback, 3: waiting to stop a demo, 4: demo recording)
... what is value 3 exactly? Why I saw it as 3 instead of 0 when I was not playing a serverside demo?

A question:
demo_play, demo_sercord, demo_stop... what about a "sv" in the name of those commands, to better distinguinsh them from standard demos commands?

PS: Why are there two .dll in the zip file?

PPS: Everyone around here should try that!  Smiley
Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #154 on: March 27, 2012, 03:21:56 am »

Thank you Gig, Im quite happy it ran well for you too Cheesy

Quote
sv_demoState : show the current demo state (0: none, 1: waiting to play a demo, 2: demo playback, 3: waiting to stop a demo, 4: demo recording)
... what is value 3 exactly? Why I saw it as 3 instead of 0 when I was not playing a serverside demo?

This comes from qcommon/qshared.h :
Code:
typedef enum {
DS_NONE,

DS_WAITINGPLAYBACK, // demo will play after map_restart)
DS_PLAYBACK, // a demo is playing
DS_WAITINGSTOP, // demo is stopped but we must move clients over their normal slots

DS_RECORDING, // a demo is being recorded

DS_NUM_DEMO_STATES
} demoState_t;

In fact when replaying a serverside demo, real clients are moved by a certain offset, so that democlients can be assigned to their original slot (eg: if you were at slot 2 when recording the demo, when replaying this democlient will be at slot 2 too). This is necessary to make the simulation happening.

At the end of demo playback, the map is restarted to delete the democlients slots and move back the real clients, but to do this, we must have a way to know we are stopping the demo. Before, Amanieu's original patch used the "delay" command (a special command similar to "wait" but that doesn't freeze the game while it waits, it worked but was more unreliable because you couldn't know the exact time the map_restard took, and when clientside with no map_restard, it didn't work). Here I simply chose to add another demoState. In practice, you should never be able to see it because it should be really quick (just the time to restard the map).

So it is not normal that your demoState was stuck at 3, but I guess this is because you were playing your demo clientside, which does not happen to do a map_restart at the end of playback (it simply exits from the demo to the game main menu). This is not important and will not produce any bug, but thank's for pointing that, Ill fix that.

A question:
demo_play, demo_sercord, demo_stop... what about a "sv" in the name of those commands, to better distinguinsh them from standard demos commands?

There are no command in OA or ioquake with a prefix, only cvars do have prefix, so I just followed the spec. Also, demo_play and demo_stop do work clientside too, so prefixing sv would be a bit misleading, but for demo_record thats true. Dunno but this has to be given some thoughts. Maybe server_demo_* then?

PS: Why are there two .dll in the zip file?

These contain the OA renderers because of the new renderer modularity facility of the latest ioquake3. This is not related to my patch, undeadzy's patch also has them (see the OA 0.8.8 sources thread).
Logged
Gig
Millenial poster
**

Cakes 39
Posts: 2297


WWW
« Reply #155 on: March 27, 2012, 06:12:50 am »

Just a (stupid?) idea... What about using a different name for these demos, e.g. "full demos" instead of "serverside demos"... (still to better divide their commands from classic demos)?
E.g. fulldemo_play, fulldemo_stop, fulldemo_record?
« Last Edit: March 27, 2012, 07:45:19 am by Gig » Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #156 on: March 27, 2012, 06:17:45 am »

I like the idea, I think Ill do that, thank's Gig Smiley
Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #157 on: March 27, 2012, 07:25:30 am »

Since my computer crashed yesterday (along with losing all my data, don't trust Windows 7 profile folders nor chkdsk), I won't be able to update the code in a while.
Meanwhile, I've updated the wiki page about demos.
Logged
Gig
Millenial poster
**

Cakes 39
Posts: 2297


WWW
« Reply #158 on: March 27, 2012, 04:00:12 pm »

Hi! I'm not sure about this part:
http://openarena.wikia.com/wiki/Manual/Demos#About_privacy_and_server-side_demos
I feel I never actually heard real in-game OpenArena VoIP... but from what I read in ioquake3 documentation at the time... VoIP packets are recorded in standard demos... why should serverside demos be different? The same, I'm not sure, but probably standard demos record team chat... is it okay to change it for serverside demos?


Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #159 on: March 27, 2012, 04:27:00 pm »

Yes standard demos have absolutely no filtering (thus it records say, teamsay and tell), and it explicitely records VoIP packets. The difference is the intention of usage: a client recording a standard demo records it for himself, and if he release it publicly, he implicitly consent to make public all the data contained inside the demo.

On the other hand, a server-side demo is recorded by a server administrator, and he cant know when releasing the demos publicly if all the clients consent to make these private data public. So the best thing to do is to avoid the recording of the sensible data.

Also, server-side demos potentially contain more sensible datas than standard demos, such as IP and GUID (I cant check the code right now but I think I can remember standard demos do not record these datas). Also the server has access to ALL IPs and GUIDs, so disclosing them could be very dangerous for clients (eg: hacking...).

PS: OA in-game VoIP works, I have tested it Wink
Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #160 on: March 28, 2012, 02:05:34 pm »

Ive given another thought about the alternative hybrid approach, and I think it's possible to overcome some of the limitations:

1)
- make a replay facility server-side, by feeding real clients some fake packets containing the messages of a demo. This can easily be done by just reading a message from a demo and using SV_SendMessageToClient(client, &msg). What is harder is to prevent the server from sending conflicting data to clients (because the server will continue to send them the valid gamestate), without crashing the clients nor the server.

Easy to be done: at replaying, we just have to use SV_SendMessageToClient(client, &msg). Both are already recorded in TheDoctor's patch.

And to avoid the interference of the engine, we simple impose a condition in SV_SendClientSnapshot() to call SV_SendMessageToClient() only if we are not replaying a demo (because if were replaying a demo, we will directly call SV_SendMessageToClient() in our demo loop).

2)
- instead of recording every client's flux separately, we could save them all in the same file and add an event "End of frame" which would allow to replay these packets at the right time (similarly to Amanieu's approach, see my patch).

As I said, TheDoctor's patch already records the messages, you then just have at each end of frame (at the end of SV_Frame(), see my current implementation) to record an "_endFrame" marker. This way, all messages will just be recorded when they are made by the engine, and we just add a marker so that at replaying we can know that we are changing frame.

To replay, we make a simple loop to process and send all demo messages (for the current frame) until we hit the _endFrame marker, then we exit the demo playback loop until next SV_Frame() iteration.

3)
- hook clientCommand and when the client send a "follow" or "follownext" command, the server would send them the flux of the democlient associated with the number. But it seems that FollowCycle (when you click your mouse button) is not catched this way, so it would be needed to edit the gamecode at least to allow the hooking of FollowCycle.

This one is tricky but should be possible and fairly easy to implement if done right.

Cmd_FollowCycle_f is in fact triggered in two way: either with follownext or followprev clientCommands, or either by just clicking the mouse button (which can be seen in the usercmd_t packets).

Code:
// game commands

  { "follow", CMD_NOTEAM, Cmd_Follow_f },
  { "follownext", CMD_NOTEAM, Cmd_FollowCycle_f },
  { "followprev", CMD_NOTEAM, Cmd_FollowCycle_f },

Code:
/*
=================
SpectatorThink
=================
*/
void SpectatorThink( gentity_t *ent, usercmd_t *ucmd ) {
...
client->oldbuttons = client->buttons;
client->buttons = ucmd->buttons;

 if ( ( client->buttons & BUTTON_ATTACK ) && ! ( client->oldbuttons & BUTTON_ATTACK ) ) {
Cmd_FollowCycle_f( ent );
}
...
}

This is a standard behaviour for all ioquake3 based game, so we can safely assume that clicking the button when spectator will always result in changing the viewpoint.

The usercmd_t packets are also available from the engine, and also we can access playerState_t->pm_flags.

Here's the idea is to basically to reproduce the behaviour of SpectatorThink() and Cmd_FollowCycle_f() in the engine instead of the gamecode, by just detecting when BUTTON_ATTACK is pressed while the client is a spectator (and it should always be in a demo playback, so this is optional).

But there will be some challenges:

- SpectatorThink(): detecting if the client want to change the viewpoint and maintain an array that says for all spectators who they are watching. This should be done at the beginning of any frame prior to sending demo messages to assign all spectators to the right democlient in case of a change.

- Cmd_FollowCycle_f(): to find the next democlient in the list, to achieve that we would need to maintain an array of the democlients indexes at any time. A non-empty configstring could be used to check that the democlient was active?
Note that the list would be needed anyway for "follow" to work, because we must make sure that we have a democlients that can be watched with the specified id.


4)
- to allow freeflying, spectators that are known to be freeflying right now would be sent just the entities states and player states. A gamecode hook would also be needed here, because I don't know of a way to know the state of a spectator (and who he is following) completely server-side, but from the gamecode it's easily available.

This one is HARD. I don't know even if it's possible to achieve it. Maybe it will be impossible, or just too easy to believe (because there's a chance that after step 3 is done, freeflying may be totally managed by the gamecode without having to change anything in the engine).

I explain: the problem here is not to allow for freeflying, as I wrote we just have to send the entities delta messages (update messages), and the client will just draw them and will be able to freefly around. The problem is to detect when the spectator wants to freefly.

To detail further, let me show you the code that allows to switch from following to freeflying (which is a call to StopFollowing()):
Code:
/*
=================
SpectatorThink
=================
*/
void SpectatorThink( gentity_t *ent, usercmd_t *ucmd ) {
...

if ( ( client->buttons & BUTTON_USE_HOLDABLE ) && ! ( client->oldbuttons & BUTTON_USE_HOLDABLE ) ) {
if ( ( g_gametype.integer == GT_ELIMINATION || g_gametype.integer == GT_CTF_ELIMINATION) &&
                g_elimination_lockspectator.integer>1 &&
                ent->client->sess.sessionTeam != TEAM_SPECTATOR ) {
                    return;
                }
StopFollowing(ent);
}
...
}

At first, you could think "well, why can't we use the same trick as in step 3 and detect when BUTTON_USE_HOLDABLE is pressed?"

Answer: because this is not a native behaviour of ioquake3. Ioquake3 does not even have any way to go to freefly once you've followed someone, except from joining the Spectator team again. So this behaviour was introduced in OpenArena.
Also, mods can further modify this behaviour. For example, ExcessivePlus does not go from Following directly to Freefly, but from Following -> Third-person view (Track Cam) -> TV View -> Freeflying.

Anyway, we could still implement this feature this way, by detecting if BUTTON_USE_HOLDABLE is pressed and then just sending him the entities states (and stopping to send him the democlient messages he was previously following), but this is not satisfying if you want to be faithful to the original way mods have made their spectating work. But this is without any doubt the easiest way to do it.

Now let's take a look on another way to do that. Let's see the code of StopFollowing():
Code:
/*
=================
StopFollowing

If the client being followed leaves the game, or you just want to drop
to free floating spectator mode
=================
*/
void StopFollowing( gentity_t *ent ) {
ent->client->ps.persistant[ PERS_TEAM ] = TEAM_SPECTATOR;
ent->client->sess.sessionTeam = TEAM_SPECTATOR;
ent->client->sess.spectatorState = SPECTATOR_FREE;
ent->client->ps.pm_flags &= ~PMF_FOLLOW;
ent->r.svFlags &= ~SVF_BOT;
ent->client->ps.clientNum = ent - g_entities;
}

As you can see, there's a playerState_t pm_flags PMF_FOLLOW that gets removed. Also the ps.clientNum gets changed too.

So a good way to do this would maybe be by checking the state of the pm_flags PMF_FOLLOW, but this would require that the gamecode would set it, which is not certain since the democlients do not exist on the server with this implementation. So the PMF_FOLLOW flag may not even be set.
Maybe we can then set PMF_FOLLOW flag by ourselves at demo playback in our functions made at step 3, and then the gamecode would be now able to call StopFollowing()? This is something to be explored...

--------------------------

So I now think this implementation could be pretty easily done. However, it may not be possible to get a satisfyingly working implementation of the Freeflying feature, but it would be posible to allow for follow switching (multiview) without any problem for sure.

I really do think this path should be explored. I currently have no time to dedicate to make a prototype of this implementation, but if someone out there is interested, I can give a hand and lend some of my knowledge.
Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #161 on: June 28, 2012, 10:26:16 am »

New server-side demo patch for Urban Terror 4.x, based on AlphaIoquake3 patch (cited in the first post of this thread) with a few enhancements:

http://www.urbanterror.info/forums/topic/28657-server-side-demo-recording/

Base patch: https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/b28aaea62ae1543828b71aaf6ff94afdd9463014

Incremental fixes:
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/b0d0f70110096e476a1f1836ddeda557a9e96bc5
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/384d356487d120c37ce4fbff1e1ed38f7d1dea83
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/669c5b2d2c6adb956b6415b3b11dfbde13e505a7
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/0d7185175aafc798eb12ef33916916035f182ae6
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/863f36d115f18bc64c87cbd6b1c1b9eafac3ca68
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/b49fa4f63a31a9e781ca3f5547162c9956db1104
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/90cf8a4616e5b17240b3668aa120802715f2102e
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/24512a775f7a177a79081d68b8c89cfb80ca3638
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/d0bf31ded108b6894ab61a8edd7e1f2c022614a6
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/2ac2562006ce2193170e0a25708f312b9ecf803d
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/7a43630bfa72d649d99c4883e26fbbddffd68f8e
https://github.com/Barbatos/ioq3-for-UrbanTerror-4/commit/0494025b74800cd7f99228bcc7708e16795f1b85

Note: it seems to suffer from the same advantages and issues as the original patch: demos produced are standard (no need for client to patch to read the demos), but the admin has to issue a command to record any subsequent player joining after the demo has started recording. There is a workaround by using an external bot (B3):
http://www.urbanterror.info/forums/topic/28665-urt-serverside-demo-recording-bigbrotherbot-plugin/
Logged
GrosBedo
Member


Cakes 14
Posts: 674


« Reply #162 on: March 11, 2013, 09:46:26 am »

The patch was updated, now it works with OA 0.8.8 and even the very latest ioquake3 (on github).

I also have rebased the patch, so that now the commits are clearly at the top of the commits graph (thus it is now very easy to separate the ioquake3 changes, then on top the openarena_engine changes, and then at the very top the serverside demos commits).

I also have fixed the memory leaks, and the patch is currently being thoroughly used on the OAC servers, with no memory leaks problem anymore.

Old server-side demos are still compatible with the new release on github (and will normally always be thank's to sv_demoTolerant facility - but here you don't even need that, the latest changes are totally compatible).

However, there is one last problem: dynamic movers (like the hidden door on Kaos2) produce a crash when playing back the demo.

I will try to fix that last problem later, or just try to make the alternative patch I've been talking about in this thread (replaying the net messages instead of entities positions).
Logged
Pages: 1 ... 5 6 [7]
  Print  
 
Jump to: