pyqscore

[kernel panic]

pyqscore is a python script that reads OA logs and outputs an HTML file with numbers.







It has options to change how entries are sorted, eliminate players from output, set maximum number of entries and a couple of them more. More information in the README file. I expect it to contain some bugs that passed unnoticed given the limited number of log files I have. Download it here:

http://www.4shared.com/file/LvpOcmg4/pyqscore.html

[Gig]

I just linked it here:
(DO NOT LINK) h t t p s : / / openarena . wikia . com/wiki/Servers#Server_managers

If you will release a new version, please update the link there, too.  

PS: In your example screenshots, why does your name appear more times, even if written in the same way? Were there different colors, that are not shown in the table? But I see that some colors appear there...

[kernel panic]

Hi, thanks!

Good question about the names, I wondered that myself the first time I saw it. The example screenshot at the top was taken by processing my 'games.log', i.e. the log OpenArena writes when you play against bots. Unless I deleted something at some point, that log contains everything I did since I started playing, and so there are some variations of my nick there. Those two that look identical actually are not; one of them is "^4kernel panic", whereas the other is "^4 kernel panic" (notice the space). It doesn't bother me particularly, but it all has to do with how I parse the coloured codes, which could be improved a lot...

[Gig]

Maybe you could add a column (even at the end of the table) that shows the full "source code" of the names...

[Cacatoes]

Maybe you could use the GUID to identify the player.
I suppose you're aware Sago made some stats tool, and aftershock did one too.

[kernel panic]

The thing is that, tecnically, pyqscore is doing what it should do. After all, both nicks are different. The problem, of course, is that they are different only because at some point many months ago I accidentally hit a space when entering my nick. I don't think it's a real issue, and probably it doesn't deserve a special case. A different situation would be if pyqscore were actually used by somebody and this turned out to be a problem.

What would need some attention is the case of repeated nicks. As all my logs are from 0.8.1 I wasn't aware of GUIDs--apparently this is logged since 0.8.5. This would be a partial solution, but not definite, because identical nicks would still show up in the final data as identical nicks. When writing this tool I decided that I wouldn't complicate matters trying to handle unsolvable problems (unsolvable this far down the ladder). Unfortunately, idiots impersonating other players are one of these, and server admins are the ones that should deal with them.

But thanks for the suggestion, I'd probably rely on GUIDs for something if I knew they existed. Then again, there are like 2 persons using pyqscore (me and my mate), and none of us are parsing logs of public servers filled with a lot of players, so...

As for sago's tool and AfterShock, yes, I was aware of them. I don't like the approach of oastat (I think that's the name), it reminds me of vsp, and I'm not even sure there is a frontend for it. AfterShock's solution--coding the whole thing in the mod--is the only right one. The rest, including pyqscore, are broken  : )

[sago007]

Using version 0.9 (19-04-2011) and Python 2.6.6 I get a couple of errors, depending on the log file I try to parse:

Code:

./pyqscore.py ~/.openarena/baseoa/games.log 

No cache file found. Processing the entire log file.

31469 new lines read.

Traceback (most recent call last):
  File "./pyqscore.py", line 201, in <module>
    server.gtype  = int(game.gametype)
ValueError: invalid literal for int() with base 10: 'k'

Code:

./pyqscore.py ~/.openarena/oaxB48/games.log 

No cache file found. Processing the entire log file.

1050 new lines read.

Traceback (most recent call last):
  File "./pyqscore.py", line 799, in <module>
    1.*    R[i]['frags']  / R[i]['deaths'],               \
ZeroDivisionError: float division

I actually believe that parsing the logs are the right way to do it. That way the games only responsibility it to write the all events to the log. The game does not do that at the moment but that is because the logging is flawed not the idea. OpenArena have added some extra logs (like CTF:) but even more is needed.

My needs as a user might be very different. For my own personal logs I would prefer a program like pyqscore to quickly parse my personal logs while on my server I would prefer to get the stats into a database, so I could make more advanced queries. 

[kernel panic]

Hi, sago

I was expecting some stuff to break when people tried some logs. Does this happen with all the logs you tried? Would you mind sending me these problematic logs somehow? I've introduced several changes in the script but I haven't uploaded anything yet. I would like to have a look.

Regarding the "right" way to deal with this, it's true that I'm biased against log parsing because of the flaws present in idtech3's implementation. Still, I don't see the advantage of doing it outside the game. Actually, my personal needs seem pretty similar to yours. I only wrote this to have some more or less nice looking stats for my games in some private server. A database solution is overkill for this problem. But even if I were running a public server with lots of players joining every month, I don't think I'd need more complex information. Sure, you can come up with pretty exotic queries, but those aren't really useful, are they? The most obvious thing pyqscore can't do at the moment that would be remotely useful is per map and item stats (the basis for the latter are actually in place). Other than that, I can't think of anything I would like to know out of my games...

[WaspKiller]

pyqscore is a python script that reads OA logs and outputs an HTML file with numbers...

Nice work KP.

...As all my logs are from 0.8.1 I wasn't aware of GUIDs--apparently this is logged since 0.8.5. This would be a partial solution, but not definite, because identical nicks would still show up in the final data as identical nicks...

Yes, very true.

...I'd probably rely on GUIDs for something if I knew they existed. Then again, there are like 2 persons using pyqscore (me and my mate), and none of us are parsing logs of public servers filled with a lot of players, so...

But you posted the program for others to use so it would be nice to include GUID support.  Anyway, do continue to improve it, more options are always ideal for Server Admins.



Shameless plug:  Our solution in E+ which is compatible with a variety of games and mods - http://vsp.goquake.com/

[kernel panic]

Thanks!

But you posted the program for others to use so it would be nice to include GUID support.
Anyway, do continue to improve it, more options are always ideal for Server Admins.

The thing is that I don't see what fundamental problem GUIDs solve. I can only think of
two use cases (but I probably lack imagination and real world experience in dealing with
large server logs): a) identical nicks; b) gather statistics _per player_, as opposed to
_per nick_ (like your vspxp does I think). The first problem is not fully solved, and in
any case I don't think log massaging is the right approach to solve it, but I appreciate
that you may prefer trying than doing nothing. The second is not a problem per se, but a
question of how you want to deal with player data. Personally, I don't mind players changing
their nicks and counting their statistics separately (actually, I look at it as almost a
privacy issue).

Shameless plug:  Our solution in E+ which is compatible with a variety of games and
mods - http://vsp.goquake.com/

Very nice, I din't know about this one. I tried vsp some time ago and found that it didn't
suit my needs. Something like pyqscore (or qscore) is much more manageable for private
servers (or for regular users wishing to parse their games.log files and check how bad they
fare against nightmare bots :-) But apart from its complexity, which renders it unusable for
all but the most determined players, what I didn't like of vsp was the closed license and
the ridiculous decision of its developer to obfuscate the code. I wonder whether this didn't
give you any headaches when you created vspxp.

[Akts^Itachi]

hi kp,
nice job !

i'm using your script, i'll tweak-it soon to fit my site, ty for your effort.

there's a point i'll talk to you about, regarding the programming.
running your script on my server's log and having no player that used handicap, i've got an exception (line 297)

so i think it will be a good ideea to handle the exception with a try/except in a future release

thanks again

[kernel panic]

You are very kind, and you are very welcome!

Not that you need me to say it, but do introduce as many changes as you want, hopefully it will suit your needs. Your post forced me to do some quick stuff that was needed to release a new version containing changes I'd done weeks ago. So if you improve the script (and decide to release it), it would be best for both of us if you worked on the new version. It contains some bug fixes (I believe Sago's problems should be gone), some tiny improvements, and some minor changes not noticeable by the user. A list of changes is now included within the zip file.

You are absolutely right about the handicap issue. Since all my logs contain a handicap value, I did not realize this did not apply universally. I invite you to make the change yourself (I can't test this with my logs) and let me know if you obtain the expected behavior.

You can get the latest version here:

http://www.4shared.com/file/HKCDYEKJ/pyqscore_091.html

Cheers

[Apo]

Hi

First of all i wanted to thank Kernel Panic for making pyqscore. I'm using your script on a daily basis at http://linuxiuvat.de/stats/oa/daily.htm. It's a very useful and lightwight tool to analyse OpenArena logs and for creating a simple static statistics page. It definitely wins out over these PHP+MySQL solutions.

I've discovered some minor issues with your script which doesn't check very carefully about some special characters. I also use Tidy to clean up some minor html errors.

But one thing people should change in your script are these two lines at line 544 in your name_colour function.

limit = len(nick) -1
if nick.rfind('^') > -1 and nick.rfind('^') < limit:

Otherwise pyqscore would exit with an out of bound error if the ^ symbol is the last character in the player name. Besides from that it's perfect. 

[kernel panic]

Hey, hi, Apo.

I'm very happy to see somebody using the script in a public server--and with some customisations, nice one!

Good catch about the nicks. I didn't think that through very well, did I?

By the way, do you think the server frag count is being reported correctly?

Cheers

[Apo]

Hello

sorry for the late reply. I didn't check the "notify me" box. Hopefully i will be notified from now on.

Well, there are some minor issues but in general pyqscore just works and it is very useful. 

I think the player names are a special case. It would be ideal if every player name could be stripped of special characters like <>. For example one player called himself <html5> thus one table cell was rendered incorrectly. At the moment i solve this issue by adding the name to the ignore list.

Yes, i think the server frag count is not being reported correctly because four bots play on the server. I didn't want pyqscore to count those frags so i put all bot names on the ignore list. I guess that's the reason why the frag count differs.

Cheers

[RMF]

Downloadlink renewal? It says the publisher can still access it.

If it's under a megabyte or 5, I can host it for lifetime if you want. Also if it's larger, but then the download takes forever (my speed goes up in a few months when we get fiber though).

[Apo]

You can download my slightly modified version of pyqscore here.

http://dl.linuxiuvat.de/oa/pyqscore.tar.gz

Feel free to mirror it or to use it under the GPL-2 license.

The result looks like this:

http://linuxiuvat.de/stats/oa/daily.htm

[capodecima]

is some stats program for oa friendly for windows ?

[Gig]

Hi!
I've just updated the download link in the wiki (DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/Servers#Server_statistics]here).

Maybe the next time, update version infos, too.  

[Apo]

Thanks Gig for updating the links!

I think a version bump wasn't necessary and the changes were relatively small. Besides I'm not the original author. However I can recommend pyqscore to every OpenArena server admin who would like to keep it simple. Parsing the log files with pyqscore instead of installing a full blown php and mysql server is sufficient in my opinion.

@capodecima

You can install Python for Windows and run the script on your pc. Pyqscore is indeed very simple to use.

[capodecima]

so bad there isnt any readme so i dont know how the program use :/

[*Ruthless*]

Gig,

I have to say the same I don't know how to use this program either. I have python fully installed, is there something else needed?

[Gig]

Sorry, I don't know what to tell you about usage of the program.

Maybe you may try to send PMs to Kernel Panic and/or Apo, asking them to reply here.

[*Ruthless*]

Thanks, I messaged Apo, I dont know how active he is with his inbox on here. I know Panic probably doesn't check about this most likely.

[Gig]

Well, also if Panic does not check the forum directly, and his email address is hidden, maybe he may get the "email notification" when he receives a PM...