Restructure the auto download system

[sago007]

There is something I have thought of for some time and I might as well get it out for a discussion.

The current Auto download system has some problems, not least:
It might download files that breaks the base game.
It downloads maps that might contain textures: while mapping those textures are impossible to recognize so you might make one of your maps depend on it.

What we want:
Ability to download new maps without getting a new version of OpenArena
Ability to download mods to get a different gameplay. Mods should not need special work before working.
Ability to distinguish downloaded maps from original ones.

My idea:
Introduce a new extension in addition to pk3: ".oamap".
.oamap-files should always have priority under pk3-files, so you cannot overload the original files with them. Only map-relevant files are read: bsp, png, pcx, aas, arena, shader, ogg and the like (shader is a bit dangerous, because it can affect baseoa files that does not have a shader. I have not completly though of how to handle this).
ONLY .oamap-files can be downloaded to baseoa and missionpack folders!
pk3 files can be downloaded to other folders such as "myserver".

This would protect the base game while allowing maps and mods to be downloaded.
Because *Radiant does not read oamap-files only official textures will appear while designing maps, preventing the java-trap.

This would require some server admins to do a little extra:
Custom maps would need to be renamed to use the oamap extension.
If a server would like to use other that maps (like models) these should be placed in a different folder such as "SERVERNAME" so that they wont affect baseoa.
Mods that has there own folder should just work.

Problems I see so far (aside from technical ones):
Different
Patches such as 0.8.5 must be downloaded once for each server (unless manually installed), because the servers cannot place it in baseoa.
Work renaming files.
Must be done in a larger upgrade (not patch).
The different folders makes the server list stop showing gametype... there must be a technical solution to this.



So far this is just something to think about... there might be many reasons why work on this should never start or extra problems it will create. There might exist a better or simpler solution.

[Falkland]

I've suggested a better download system in some of my previous post , in particular about Q4 autodownload which prompts for files to be downloaded by client side ( a description here : http://openarena.ws/board/index.php?topic=3291.msg27298#msg27298 ) and about dfengine download system which has the explicit /download command and some controls over the file to be downloaded  eg a control based on MIME/type to allow only pk3 zip files format and a control over the name of the file ( http://openarena.ws/board/index.php?topic=3363.msg27738#msg27738 and also on RN forum : http://rainbow.furver.se/index.php?name=Forums&file=viewtopic&t=1558 )

EDIT : and the system has also an exception that treats the case of name collision :

[...] Name collision with an existing pk3 file will result in a failure and
be left to the user to sort out.

They've improved the system in their latest version.

                                HTTP DOWNLOAD

DFEngine has built-in support for HTTP downloads based on libcurl. It adds a
/download command that allows requesting map packs from a web server. Such a
feature has been made possible thanks to the efforts involved in the
http://q3a.ath.cx/ online repository which covers an extensive number of maps.
An alternative repository can be set up with the dl_source cvar.

  download [mapname] (&)
      download command - appending & to the command makes the download run in
      the background

  cl_mapAutodownload [0,1]
      whether to download missing maps automatically when the client connects to
      a server

  curl_version
      version string of the curl library built into the engine

  dl_showmotd [0,1]
      whether to show the download server motd in the console

  dl_showprogress [0,2]
      whether to show the download progress in the console (1 writes new lines
      continuously, 2 writes in place)

  dl_source [URI]
      URI from which to request maps - token %m inside the string is replaced
      with the requested mapname before issuing the HTTP request

  dl_usebaseq3 [0,1]
      whether to issue the downloaded packs in baseq3 (1) or in the current
      gamedir (0)

  dl_verbose [0,2]
      verbose modes for debugging purposes

[fromhell]

I liked UT2K4/Darkplaces's cache folder idea.

[Falkland]

I liked UT2K4/Darkplaces's cache folder idea.

Uhm ... how does it work ? I've never played any of those games.

[fromhell]

From what I recall in Darkplaces: You autodownload, it goes into a subfolder called cache/, gets slapped with a checksum in its name somehow, and loads only when package checksum matches, if not, tries to download the correct package from the server. The cache folder is never automatically loaded.

UT2004 did the same, but it completely mangles up the filename with a more elaborate sum, and has days of expiry.

[Falkland]

From what I recall in Darkplaces: You autodownload, it goes into a subfolder called cache/, gets slapped with a checksum in its name somehow, and loads only when package checksum matches, if not, tries to download the correct package from the server. The cache folder is never automatically loaded.

UT2004 did the same, but it completely mangles up the filename with a more elaborate sum, and has days of expiry.

They sound interesting ... both systems prevent an eventual overwrite of existing files and preserve games installations.

[sago007]

I have also thought about the cached folder, mainly because it will allow files to be reused between servers as long as the checksum matches.

In Unreal Tournament it was annoying that you could not use autodownloaded maps (there did exist tools though). The pk3 packages sorta solves that problem by grouping relevant files together (unlike all the files in Unreal Tournament, one for the map, some for textures, some for sound).

The cache does not solve the problem that any pk3-files I manually installed would pollute NetRadiant.

As a good design principle I would never ask the user what to download either the user gets used to just clicking yes or they have absolutely no idea what files to accept.

[davidd]

why dont you download the map in a separate different directory what gets read with lower priority.

perhaps mirror the directory tree structure in .openarena/downloaded/

Code:

.openarena/downloaded/baseoa/mapname.pk3

perhaps even with some download history so that one can remove the last few breaking .pk3
with mapping you could perhaps disable the download directory from getting listed.

I dont see any benefit in getting prompted after I have enabled downloading.
It would be nice if somehow a total amount of megabytes to be downloaded could be shown instead of on a per file basis. After a few pk3s at slow speed I disconnect.

[andrewj]

why dont you download the map in a separate different directory what gets read with lower priority.

I was about to suggest the same thing -- use folders instead of a different file extension.

[davidd]

why dont you download the map in a separate different directory what gets read with lower priority.

I was about to suggest the same thing -- use folders instead of a different file extension.

Ok thx, let me emphasise it a bit extra,

I would like to see a file "downloadhistory.txt" that lists the downloaded files, which time they were downloaded and what the remote filesize was at that time (perhaps also list the server that provided the download)

[RMF]

i like davids ideas about total filesize and a downloadlog, i'm sometimes annoyed too by having to download many pk3s without no clear endtime. And with the dllog you can actually fix any changes pk3s made to OA without having to remember each dled file or go through the whole folder.