Automatic Downloading rebooted!

[GrosBedo]

Hello there,

Ok yet another subject I revive, but I'm feeling quite in a mood of contributing to the OA project now that I'm getting used more and more to the engine and gamecode

This topic is close to my heart because I think it's really one of the key feature that is currently quite broken in the current OA and ioquake3 implementations. Of course, it's needless to say that if this feature ever gets implemented, it can only be for a later version of OA.

But first, let's take a look at what would be the benefits of a better automatic downloading (or third-party extensions management):

- Less confusion and frustration for new players, who are currently disconnected from many servers just because they can't download the brightskin, or a pack of textures/audio addons.

- More control/security for clients: depending on the implementation (trust system/ergonomic dialog box) they will be able to choose which third-party packages they want and which they don't want. Currently, the only way to do this is to turn off autodownload, and activate it only when one can't join a server. But still, it's impossible to know which exact files will be downloaded (unless you manually look in the console, before each file is downloaded), nor stop prevent one specific file to download.

- Better management for servers admins: there is currently a way to manage which files will be required to download for clients, and which will be optional (by placing them in homepath/baseoa folder instead of root/baseoa folder, but then they won't be automatically downloaded, only used if the client already has them). There should be a better way, clearer way to manage that, and give the possibility to admins to make optional content downloadable if the client wants it.

- More diversified servers and gametypes: currently, admins are very reluctant to mods, because they fear that clients won't be able to connect (and apart from AfterShock admins, noone want only elite players to be able to connect to your server just because new players don't know the existence of the autodownload option).
(eg: more players for Generations Arena wink)

Now the problems we are faced:

- Problem of trust and security: third-party downloads can break the game, and even open a breach into your OS (it's not that hard to modify a QVM to make such an exploit and take the control of your computer).

- Problem of ergonomy: the client should be notified about each file that will be downloaded and the risks associated.

- Problem of management: admins can't really specify optional downloads, and when downloads are required, the admin must be pretty careful to not break the original install of the clients (easy fix is to create another homepath folder and launch the game with +set fs_game "yourfolder", but generally it's not applied).

------------------------------

Taking into account what has been discussed in this previous thread:
http://openarena.ws/board/index.php?topic=3793.0

I propose these solutions for each of the above stated problems:

- Ergonomy, security and management: use a similar dialog box that IoURT show before downloading a content, with a warning message and a list of files that will be downloaded. Also, this dialog box could contain a list of optional files to download, and the user could select/deselect them (the selection is not necessary for required files since they will be checked if server has sv_pure - but of course if server has sv_pure 0 it should also allow for selection/deselection of these items).

- Security: when servers offers third-party downloads, but use the baseoa default mod, instead of saving and loading files from homepath/baseoa, they could all be redirected to a homepath/baseoa-thirdparty folder, which will only be loaded when connecting to an internet server requiring those files. This solution has the double benefits that then it would be impossible to break the basic installation of OpenArena (since it will only load normal files placed in root/baseoa and homepath/baseoa), and we coders won't have to modify a lot of the code to create this behaviour (just add a new folder search if files are missing when connecting).


- Management: similarly to the first 2 solutions, to make a download optional, an admin could simply place it in a -optional folder. Eg: if gamemod is baseoa, folder would be baseoa-optional. If excessiveplus, it would be excessiveplus-optional. Plus this would have the advantage for admins to be able to have different optional and required downloads depending on the gamemod/folder (because one can create a CTF folder - that's what is currently applied to super servers).

----------------------------------

So, what do you think guys? Are these solutions viable? Interesting? A waste of time to implement? Or maybe you have some other solutions to propose!

[grey matter]

- Problem of trust and security: third-party downloads can break the game, and even open a breach into your OS (it's not that hard to modify a QVM to make such an exploit and take the control of your computer).

Actually, QVMs should not be able to do this. If you happen to have a working PoC then submit it to the ioquake3 team to fix. QVMs should only be able to read and write files from your game installation (homepath, basepath) but nowhere else. They can not launch external programs and can not run arbitrary commands.
You can do a few nasty things by setting the vm_* cvars and then autodownloading native libs for your platform. This circumvents all of the QVM security considerations, but that's due to their design.

Please don't support the "ioUrT" name., it's just plain wrong. They've ripped the ioquake3 sourcecode and call it io*. "io" here is for icculus.org, who has nothing to do with their fork. Other games using ioquake3 are not called io* either, think of World of Padman, Smokin' Guns, Tremulous, Turtle Arena etc.


While a confirmation dialog gives control to the players, I don't think it'll actually work. Just assume I want to join a server and the game keeps asking me whether I want to download file 1, file 2, file 3, .. , file 20. I'll sure as hell just click "yes" or even "yes, always", because all I care for is playing on that server. Besides, what is this zzz-proskins2.pk3 anyways?
It does nothing for security that way either. As a player I expect the game to be safe, no matter what servers I join. There's no such thing as a web-of-trust between players and server admins regarding their pk3 files. You'd need to have a public list of verified files with checksums and descriptions, which is just overkill.

After all, even with such a system in place it would still require admins to setup their servers properly. I've seen so many misconfigured public servers that I simply assume that they won't be able to do so (in general).

[sago007]

I have been working a little on the side on a cache system based on secure checksums, to see if I could get a prototype working.

Basically:

The system requires assumes a pure-server.
Client reads all pk3-files known with there md5 sum. (Caching of md5 sums might be a good idea)
Client connects.
The server calcualtes md5 checksums of all its files pk3-files in use and generates a list with: checksums and filesize.
The client checks if it has all the files based on md5 sum. If it has it loads those files and connects.
Otherwize it generates a list of files it needs. Because the server send all files required the client knows just how many MB it needs to download.
The client downloads the files to the cahce-folder. The filenames are MD5SUM.pk3.
After download is completed the client reconnects and play.
If anything goes wrong: Quit and the next time the game starts up all files in the cache-folder are never loaded until needed.
By redirecting all downloaded files to the cache folder the main game will never be affected by malfunctioning downloads.

The disadvantage is that the files the client actually want to use offline are hard to recognize. A management program with known files might help with that.
Generally I have not yet given much thought to prevent the cache from growing very large but it can be a problem on some systems.

My focus has been elsewhere for some time but I think it could work.

[Neon_Knight]

Can I also ask for an option in the Game Options menu to enable/ask/disable AD as well?

(By the point 0.9.0/OA3 is released, we should be using the Missionpack UI, but if not, such option should be available in the menu)

[WaspKiller]

...This topic is close to my heart because I think it's really one of the key feature that is currently quite broken in the current OA and ioquake3 implementations...

GrosBedo, I am in complete agreement.  As you know, I started the thread - http://openarena.ws/board/index.php?topic=3793.0 - that you referred to, so this has been and continues to be a sore point for me.

I am less interested in the precise mechanics of the implementation than I am in the OA Devs just resolving the problem.  To me this trumps the creation of addition player models, art and all that other fluff and should have been implement in 0.8.8.

Sago wrote 20 months ago,

...The current system is broken and needs a real fix.

yet here we are just talking the same stuff over and over.

It's ridiculous that I cannot setup a regular (baseoa) server that uses custom maps and have players connect to it.  And it's hilarious how the naysayers sit in their glass towers and wonder why its difficult to attract casual players to a graphically challenged game... perhaps, among other things, if these players had a better variety of maps to experience.

If I understand Grey Matter correctly, his suggestion is to do nothing because any solution would be unsafe.  Well, everyday I get up I face a plethora of unsafe situations.  Just using a web browser is inherently dangerous because it has proven to be one of the easier vectors in which to exploit a system.  But, does that mean we should stop using them?

Autodownloads are enabled by default in Quake III Arena and Team Arena.  OA chooses differently because of security concerns.  There is nothing wrong with that decision but it's just dumb to be modeled after a game down to the point that we can leverage most of its assets (maps, bot skins, mods, etc.) and then not have a way to alert casual/n00b players of why they have been prevented from getting these assets other than via a stupid message that says "Can't load maps/mapname.pk3."

The players who are old enough to play this game are old enough to decide whether they want the custom maps or sound pack or mod, etc.  They just need a screen which explains what is going on.

Similarly, when a player using OA 0.8.1 connects to a 0.8.5 server and the map rotation includes a map like hydronex2 which is not included in 0.8.1, the player should not receive a cryptic disconnect message.  There should be a clear explanation of what happened and how to get the map manually or via autodownload.

[Neon_Knight]

It's not only for security reasons, but also for choice reasons. (Not speaking about anyone in this thread, but I find quite ironic that the most fervent FS crusaders defenders usually attack one of the fundamental freedoms, the freedom of choice)

Players shouldn't be obligated to download stuff from servers.

[WaspKiller]

...Players shouldn't be obligated to download stuff from servers.

I concur.  Moreover, anyone who feels differently is an idiot.  But that is not in the issue of either GrosBedo's or my Forum Threads.

In the days when Q3 and its Mods ruled and shortly after that, a large percentage of FPS players understood how to enable and disable autodownloads in id tech 3 engine based games.  But, nowadays, the average casual player has no familiarity with that system.  Additionally, many new players to OA especially on North American servers, are from Latin America where English is not their First language.  I watch these players trying to re-connect, sometimes over and over, without success and I know they do not understand why they cannot connect.  The error message is so cryptic that even English only speakers do not understand what happened.

What GrosBedo, I and others (including Sago) want is a method to alert players not familiar with OA of why they cannot connect to a server and to give them an easy on-screen way to allow the downloads if they chose to do so.

I (and most people) do not want OA to enable autodownloads by default.  The only stumbling block that I can discern is that many people differ in how best to implement this system.  But given the amount of discussion on the subject the Devs ought to be able to implement something basic and then revisit and refine the solution in future iterations.

[Neon_Knight]

The thing also should be what should be obligatory and what optional.
For me, models, skins and audiopacks should be optional. Maps, ofc, should be obligatory.

[grey matter]

If I understand Grey Matter correctly, his suggestion is to do nothing because any solution would be unsafe.

No, this is not what I wanted to say. I was replying to GrosBedo who said that QVMs downloaded via autodownload can "take the control of your computer". As far as I'm aware there is no such possibility (in ioquake3). There have indeed been classical buffer overflows and similar exploit vectors, but these had nothing to do with pk3 autodownload.
I don't think that enabled autodownloads would impose any security risk beyond what's already present in the engine.

There are however other reasons why I don't think enabling autodownload by default is a good idea. Like Neon_Knight said, it's about freedom of choice. I as a user may not want to download additional pk3s. A rather lame reason is that I'm on a limited traffic connection. A more obvious reason would be that I'm only interested in free (as in GPL, PD) content. There's currently no way to know the license of a pk3 files before downloading. This'd require the kind of system I mentioned before; a public list with known checksums for pk3 files as well as their license (if it is known at all).

I don't think that a simple "Yes/No" button will solve this. Like I've already said, I assume that any player confronted with such a dialogue when connecting to a server will just click "yes" or even "yes, always" (if there's such a button). This renders the whole dialogue useless and the time could have been spent for just changing the cl_allowDownload default, which is a bad idea as stated above.

As a result I don't think that the OA developers will solve this problems. From what I see, this requires quite a sophisticated system, involving many code changes, which in turn requires knowledge, time to write and even more time to test and document. If you do think otherwise, just go ahead and implement it, I won't stop you. It's just that I don't think it'll succeed in the end and the time could be spent on other, smaller things (which are more likely to succeed).

You could/should try the easiest way; display a simple dialogue to enable or disable cl_allowDownload upon first launching OA (like the one for the network settings). You'd then include a (few) lines about the autodownload system right in this very menu or even link to a page in the OA wiki.
This is neither sophisticated nor elegant, but does not require much work and is thus more likely to cause a change. Even if players just click "yes", it means they had/have their choice and can then connect happily.

P.S.: While exact error messages are always helpful, they're not that easy to create for such problems as missing pk3s. There could be just a missing sound (luckily we have silence.wav), missing textures or a problem that prevents the gamecode from being loaded. To display a meaningful error message to the player would require quite some work for the developers. This is further complicated that a pk3 file can contain any kind of content and thus trigger all different kind of errors when missing.

[GrosBedo]

First of all, I don't think OA devs will implement this feature. I will. As far as I can do it, I will try. OA devs have a lot of other stuffs in their roadmap, so that's totally understandable that they did not implement this feature yet. From what I understand, their primary concern is to make a fully working game with a complete set of content. Enhancing the gameplay's feature is not on the roadmap (yet), but the community can then contribute. And after all, who can better enhance the gameplay than the community itself?

Secondly, I must point out that this feature is of even more importance since for future versions of OA it is considered to put most of the current maps into a tribute pack (which will never ever be played if this feature is not enhanced).

Thirdly, I would like to summarize concisely the ideas I propose:

- a client's dialog box summarizing the list of the packages to be downloaded, splitted into two categories: required and optional, and with the possibility for the user to choose what packages to download, or just refuse to connect.

- 1 new folder where to put optional contents for admins (will be shown in the optional list for clients).

- 1 new folder to separate third-party content from the baseoa folder (for clients, so that baseoa is never touched by downloaded pk3 files). This would allow players to keep a pretty clean install. See Sago's proposition of a cached pk3 as an alternative solution.

Now onto replies:

I have been working a little on the side on a cache system based on secure checksums, to see if I could get a prototype working.

Good news But I've got some questions:

- How the client's can really trust what the server is sending? I mean, isn't there already a CRC checksum? Of course an MD5 checksum is a lot better, but still the server can be misguiding the client by sending corrupting pk3?

- About the cache pk3, will it still be loaded when the client is playing in single player, or in other servers, or would only some parts of the content will be loaded when required by the server?

Actually, QVMs should not be able to do this. If you happen to have a working PoC then submit it to the ioquake3 team to fix. QVMs should only be able to read and write files from your game installation (homepath, basepath) but nowhere else. They can not launch external programs and can not run arbitrary commands.
You can do a few nasty things by setting the vm_* cvars and then autodownloading native libs for your platform. This circumvents all of the QVM security considerations, but that's due to their design.

Nice to know, I wasn't sure about the read/write protection. Anyway, (I didn't write it) but I'm aware that QVM security has been enhanced a lot in ioquake3, but as it was clearly stated on the ioquake3 mailing list, QVM has never been designed for security, so even if it's enhanced, I still expect to see some security concerns, because the initial design is flawed.

But, I can be mistaken, but anyway since I have limited knowledge on this issue, I prefer to be more cautious and consider this as a possible issue.

Please don't support the "ioUrT" name., it's just plain wrong. They've ripped the ioquake3 sourcecode and call it io*. "io" here is for icculus.org, who has nothing to do with their fork. Other games using ioquake3 are not called io* either, think of World of Padman, Smokin' Guns, Tremulous, Turtle Arena etc.

You're right but that's the name. How should I call it then?

While a confirmation dialog gives control to the players, I don't think it'll actually work. Just assume I want to join a server and the game keeps asking me whether I want to download file 1, file 2, file 3, .. , file 20. I'll sure as hell just click "yes" or even "yes, always", because all I care for is playing on that server. Besides, what is this zzz-proskins2.pk3 anyways?

I did not say that it would ask one dialog per file, I think I stated that it would be a dialog box with a full list of Required and then Optional files that will be downloaded, from which list players can just choose what they want or not.

I think the freedom of choice can't be better applied than that.

After all, even with such a system in place it would still require admins to setup their servers properly. I've seen so many misconfigured public servers that I simply assume that they won't be able to do so (in general).

Indeed but this is another problem. First, these features aren't exist. If they did, I'm pretty sure quite a few servers (the most played) would take benefit. At least, I will.

The thing also should be what should be obligatory and what optional.
For me, models, skins and audiopacks should be optional. Maps, ofc, should be obligatory.

That's also what I propose and I think it's a very important feature. Nowadays it already works, but it's kind of tricky and not very flexible (only one folder for all optional files for all gamemods).

I am less interested in the precise mechanics of the implementation than I am in the OA Devs just resolving the problem.  To me this trumps the creation of addition player models, art and all that other fluff and should have been implement in 0.8.8.

I know what you mean WaspKiller and you know we share the same vision about the subject, but I don't concur to what you are saying here. The devs aren't the only one to implement features, the community also can, and for the moment noone did anything, so don't unnecessarily blame the devs, the community is to be blamed too. I know you aren't really part of this community, so I'm not blaming you, I know what you are doing for E+ and OA, but please also consider the responsibility of the community for the advancement of the game (since it's an opensource game, the community is a whole part of it).

Can I also ask for an option in the Game Options menu to enable/ask/disable AD as well?

Yes there should be such an option. If impossible with the current UI, it would be splitted on two separate options (Enable autodownload Yes/No - if Yes another option: Always accept autodownload?).

There are however other reasons why I don't think enabling autodownload by default is a good idea. (...)
A more obvious reason would be that I'm only interested in free (as in GPL, PD) content. There's currently no way to know the license of a pk3 files before downloading. This'd require the kind of system I mentioned before; a public list with known checksums for pk3 files as well as their license (if it is known at all).

No a list is not needed here. We could also add to the system a code to read from a txt file that would list all the files required or optional to download along with a description and their licence, but I think that for now it's unnecessary (maybe later?), because players are free to download the file anyway and check the licence for themselves, or just look over the web since there will be a list of the files before downloading.

Really I think that the GPL idea is a bit far-fetched here.

P.S.: While exact error messages are always helpful, they're not that easy to create for such problems as missing pk3s. There could be just a missing sound (luckily we have silence.wav), missing textures or a problem that prevents the gamecode from being loaded. To display a meaningful error message to the player would require quite some work for the developers. This is further complicated that a pk3 file can contain any kind of content and thus trigger all different kind of errors when missing.

Meaningful error messages for any type of error would be great, but practically impossible to implement.

Here this is not the issue: what I describe is a way to handle a precise type of error, caused by missing packages that can be downloaded from the server which the player is trying to connect.

[GrosBedo]

I've rethought about the practical implementation for additional contents, and I think that instead of creating an "mod-additional" folder for each mod (which would be quite messy), one could also just make a specification file with a syntax similar to voteconf.txt:

Code:

somecontent.pk3 {
  required:true
}
someadditionalcontent.pk3 {
  required:false
}

Of course, by default, all pk3 not specified would be required as usual. Implementing this would also allow for future extension over the processing of pk3, like possible specifying loading order or exclusion list, etc...

Practically speaking, in the code, at server spawning (when the game references all the pk3), it would look inside this specification file (if it exists), and load accordingly either in the usual referencedPaks var, or in a new additionalreferencedPaks.

Then, these two vars could be both sent to the client when requesting a connection to the server, and the client will just have to load in a new GUI form the list of referencedPaks and additionalreferencedPaks in two separate lists.

All this would theoretically not require any big change of the code (except the new GUI form, but it can be made in a separate file of course), and thus should be pretty clean, while being quite extendable.

----

That said, I put this project on a standby for now because I have no time currently to dedicate to it. But hopefully I'll implement it someday.