Does the engine even build anymore?

[stigmha]

Hi

I have spent quite some time attempting to build the engine on various modern platforms with different results, and I would truly appreciate any help.

Background

I started off by being lazy and doing things the wrong way; skimming through the Wiki and by downloading a copy of the game engine source and the QVM's source. I quickly found the Visual Studio project in /misc/msvc/ioq3.sln and opened it in Visual Studio 2012 on a Windows 8 machine. It converted the project to VS12 format and I eventually managed to get the engine to build and run (I later got it to work in VS13 too). I made a few modifications to the engine with positive results, but problems arise once I started working on the source of the VMs in the VS12 project. I managed to compile them in to dlls and made the engine load them on launch, but I experienced multiple bugs. One being that every bot renamed to "scoreboard" and became a spectator immediately after connecting to the game. I delved into the code, found the source of the problem and decided to compare this file with its equivalent in the QVMs project; they were different. I realized that the QVM code in the engine's Visual Studio project are terrible outdated, and that the VS project (.sln file) is leftovers from ioquake3. I will suggest to remove it from the source of OA, as it is merely useless and leads to confusion.

I decided to do it the right way from now on and started to read your forums in detail.

Windows 7/8

I followed all your instructions on the Coding section of the DO NOT LINK[/b]) h t t p s : / / openarena . wikia . com/wiki/DeveloperFAQ#Coding]DeveloperFAQ; Installed MinGW with the specified packages, modified the cross-make-mingw.sh, and entered the engine's root directory in a "MinGW Shell" instance. I then ran "sh cross-make-mingw.sh". It compiled successfully, but the linker (LD) gives the following errors:

Code:

LD build/release-mingw32-x86/openarena.x86.exe
code/libs/win32/libcurl.a(strequal.o):strequal.c:(.text+0x14): undefined reference to 'strcasecmp'
code/libs/win32/libcurl.a(strequal.o):strequal.c:(.text+0x48): undefined reference to 'strncasecmp'
collect2.exe: error: ld returned 1 exit status

I spent almost a day Googling, reading and attempting to fix this issue with no luck. The only references I found to the functions strcasecmp and strncasecmp in the engine's source are the prototypes defined in /code/libcurl/curl/stdcheaders.h. So I disassembled libcurl.a and found a call to _strcasecmp in the function _curl_strequal and _strncasecmp in _curl_strnequal, none which are implemented in the libcurl.a itself. I have currently no clue on how to fix this, anyone?

OSX Mavericks (10.9.1)

I had already installed the latest version of XCode with the command line tools installed. There is a XCode project file available in the project's source but it also looks like some ioquake3 leftovers that haven't been modified to be compliant with OpenArena, so I ignored it. I yet again followed the Wiki and ran "make". Result:

Code:

error: unknown target CPU 'prescott'

I Googled it, found this in the Makefile: "OPTIMIZEVM += -march=prescott -mfpmath=sse" and changed prescott to native: "OPTIMIZEVM += -march=native -mfpmath=sse". So I ran make again and everything seemed to work until:

Code:

code/qcommon/vm_x86.c:86:23: error: initializer element is not a compile-time constant
static  int             ftolPtr = (int)qftol0F7F;

I played a bit in the code and Googled the error, but without any success. I honestly didn't gave it much effort as I was quite sick of these issues, I wanted to try Linux next. Any OSX gurus out there who know a quick fix?

Ubuntu 13.10

Yet again back to your wiki, and I made sure all dependencies in your list were installed before I tried to build it. I did find it a bit strange that your Wiki states libsdl2-dev when I knew OA uses SDL 1.2 and not SDL 2, but I decided to try anyways. "make" resulted in duplicate errors as code/SDL12/include/SDL_config_minimal.h declares typedef unsigned int size_t which is already declared in one of the includes. My quick fix was to comment out this line, but then I got compilation errors on unknown type size_t ! So I decided to include the file stddef.h which worked and the compilation went well. The linker failed however badly complaining about the SDL calls which it couldn't find. I then installed SDL 1.2 ("sudo apt-get install libsdl1.2-dev"), tried again and it worked! I also reverted my changes to the SDL_config_minimal.h file and it worked. You should definitely fix the Wiki to say "libsdl1.2-dev" instead of "libsdl2-dev"!

--------------

Summary: any help on Windows and OSX is highly appreciated. I don't currently care for Solaris, but it's probably not a walk in the park there either. Thank you for your time.

[Neon_Knight]

I have sent a PM to sago007, who knows about the code. He may help you.

[grey matter]

On a sidenote, does the current ioquake3 source code build without problems on Windows with VisualStudio? Afaik the VS project files are not really being used.
All the other platforms should work fine with ioq3 as well. If you can confirm this, it might be easier to use ioq3 to fix oa's crufty codebase.

P.S.: ioq3 has a SDL2 branch.

[Gig]

For building engine from OSX, you may send a PM to Jackoverfull, linking this thread. He created OSX executables.
But I don't know if he's also used to compiling QVM.

[stigmha]

I have sent a PM to sago007, who knows about the code. He may help you.

Thanks

On a sidenote, does the current ioquake3 source code build without problems on Windows with VisualStudio? Afaik the VS project files are not really being used.

Good idea, I cloned it and opened it in VisualStudio 2013 which converts it into a VS13 project. It did not build "out of the box" as the project contained old references to their renderer, which they now has separated into two subdirectories; renderergl1 and renderergl2. I honestly think the names says it all, but I replaced the references to the old renderer directory to renderergl2. Compilation went fine, but the linker is unhappy:

Code:

Error	261	error LNK2026: module unsafe for SAFESEH image.	C:\~\ioq3\misc\msvc10\ftola.obj	quake3
Error	262	error LNK2026: module unsafe for SAFESEH image.	C:\~\ioq3\misc\msvc10\snapvector.obj	quake3
Error	264	error LNK2005: _Com_Error already defined in common.obj	C:\~\ioq3\misc\msvc10\tr_subs.obj	quake3
Error	265	error LNK2005: _Com_Printf already defined in common.obj	C:\~\ioq3\misc\msvc10\tr_subs.obj	quake3
Error	267	error LNK1281: Unable to generate SAFESEH image.	C:\~\ioq3\build\quake3_debug\ioquake3.x86.exe quake3

I checked LNK2026 at MSDN and it means: "/SAFESEH was specified, but a module was not compatible with the safe exception handling feature. If you want to use this module with /SAFESEH, then you will need to recompile the module using the Visual C++ .NET 2003 (or later) compiler.". Not so weird as it is programmed in C with the project set to use the C compiler (/TC flag), and not the Visual C++ compiler. I disabled /SAFESEH by opening the quake3 project's Property Pages then expanded Configuration Properties->Linker->Advanced and changed "Image Has Safe Exception Handlers" to "No (/SAFESEH:NO)". I rebuilt it and that specific error was gone. LNK2005 remained, but I also got tons of linking errors from the rendering system  . So I included the files in renderecommon to the project. It gave me the same errors. I removed all the renderer files from the project, and tried the renderegl1 instead combined with the renderecommon files. It fixed the rendering linker errors. LNK2005 next.

Both Com_Error's and Com_Printf's prototypes are defined in qshared.h, while the implementations are defined the following places:

• cg_main.c l.441 (client)
• g_main.c l.540 (server)
• ui_atoms.c l.33 (user interface)
• tr_subs.c l.38 (renderer)
• common.c l.260 (engine)

The line numbers points to the Com_Error functions with the Com_Printf's right above. I commented out the one in the renderer as it leads to a duplicate in the engine. Clean, rebuild, and 318 warnings later; success! I then tried to run it setting "Command Arguments" to "+set fs_basepath "C:\Program Files (x86)\openarena-0.8.8\"". I also made sure to change the Linker settings to display the console, so I could see the program output ("Property Pages"->"Configuration Properties"->"Linker"->"System": set "SubSystem" to "Console (/SUBSYSTEM:CONSOLE)"). I got the following pop-up:

Code:

---------------------------
Error
---------------------------
"pak0.pk3" is missing. Please copy it from your legitimate Q3 CDROM. Point Release files are missing. Please re-install the 1.32 point release. Also check that your ioq3 executable is in the correct place and that every file in the "baseq3" directory is present and readable. Copy console log to clipboard?
---------------------------
Ja   Nei   
---------------------------

Console says "0 files in pk3 files". I obviously have to change the "baseoa" directorie's name to "baseq3". New attempt, "computer says no":

Code:

**************************************************
WARNING: baseq3/pak0.pk3 is present but its checksum (3235491247)
is not correct. Please re-copy pak0.pk3 from your
legitimate Q3 CDROM.
**************************************************

I patched this by commenting out the function calls to FS_CheckPak0() at line 3901 and 3936 in files.c. Game launched and worked well with OpenArena! But I am not sure whether I'll continue with ioquake3 as the engine since it worked just as well With Visual Studio using OA's engine. I haven't tested it on any other platforms yet, but I honestly want the OA MinGW procedure to work as it is the "proper" OA way of doing it.

For building engine from OSX, you may send a PM to Jackoverfull, linking this thread. He created OSX executables.

Thanks, PM has been sent. I might experiment using ioquake3's Makefiles to see if I can get it to work in OA as well.

But I don't know if he's also used to compiling QVM.

I haven't really focused on compiling the QVMs yet, but I have successfully managed to compile the QVMs on both Windows and Linux using the provided scripts in the QVM projects. OSX complains: Unknown target CPU 'prescott', which I guess I can fix the same way as with the engine. But my first priority is to get the engine to compile as the QVMs are portable.

FYI: Linux seems to compile the QVMs as dynamic libraries (*.so) rather than QVMs (*.qvm).

[Neon_Knight]

I thought the QVMs could be compiled with the batch script Sago has included into that SDK.

[stigmha]

Compiling the QVMs isn't a big issue. The scripts provided in the QVM download works perfectly well on Windows and Linux. It's the engine that won't build the "proper" way on Windows and OSX.

[jackoverfull]

Here I am…

No, never compiled QVMs in my life…And I haven't been doing this in a long time, actually…Nor ever used Mavericks yet.

Right about the XCode project, that's probably pathetically outdated and I never used it. What I can recommend is trying to build ioq3, then, once it works, see what's different in OA and try to work around the differences.

[fromhell]

Most OA's changes should be mostly platform independent though... particularly the renderer gets the most changes for cosmetic things (specular, envmap, flares and bloom) and additional compatibility fallbacks - and then there's videoflags which enforces visual balance.

[jackoverfull]

"Should be"…But I always had to tinker a bit to get the sources compile on OS X in the past. 

[fromhell]

Most of my new compatibility fallback changes aren't even relevant for the Mac since no Apple OSX-powered computer ever sold with a video card far too crap for id Tech 3.  Every OS X machine sold is technically ideal.  

[sago007]

Based on the error message I imagine that a new version of libcurl.a would solve the problem.

[stigmha]

Thanks for your input so far, I'll try to figure something out and I will get back to you. Here's my first attempt of a naive make on OSX Mavericks with the pure ioquake3 source;

Code:

code/tools/lcc/src/dag.c:252:27: error: expected expression
                        assert(flab == 0),
                                         ^
code/tools/lcc/src/dag.c:254:9: error: expected expression
                      else if (flab) {
                      ^
code/tools/lcc/src/dag.c:533:34: error: expected expression
                                                assert(k < LONG_MAX),
                                                                    ^
code/tools/lcc/src/dag.c:560:25: error: expected expression
                        assert(p->count == 0),

make clean and ./make-macosx.sh x86. Same errors plus:

Code:

code/tools/lcc/src/enode.c:107:13: error: expected expression
                assert(t3),
                          ^
code/tools/lcc/src/enode.c:111:2: error: expected expression
        else {

x86_64 gave me the same results. Running ./make-macosx-app.sh release requires the binaries that failed, and the final ./make-macosx-ub.sh is to build for other platforms.

NB: it is not my intention to spam you with info, I do it to document and track the progress.

[jackoverfull]

First of all, forget about the sh script for a start, begin with a standard make as if you were on linux. The shell script automatizes a few things like building a .app bundle, optimizing and doing universal binaries but also adds complexity. I _always_ had to edit it to make it work eventually. Start getting the basic thing to build, then you can work into fixing the script.


Then, I'd look into code/tools/lcc/src/dag.c:252:27 and see what goes wrong there…


Attaching the make-macosx-ub.sh edited for oa 8.8. I sent it at the time but I have no idea if it eventually went into the sources.

[grey matter]

https://github.com/ioquake/ioq3/pull/32 as well as https://github.com/ioquake/ioq3/pull/31 might be of interest for OA devs (there might be a ticket on https://bugzilla.icculus.org/ as well).

[andrewj]

Firstly, you don't need 'lcc' to make an engine binary -- it is the tool for compiling qvms.

Secondly, I guess that platform assert() macro is messing up how the LCC code uses it.  You can either make the LCC code better -- instead of "if (xxx) assert(yyy), zzz;" then have "if (xxx) { assert(yyy); zzz; }" -- which may be painful if that idiom is used a lot.

Or else stick something like the following in c.h (after #include <assert.h>) :

Code:

#undef assert
#define assert(x)  do {} while  (0)

[Hitchhiker]

I dont know if this helps? regarding the first error in the original post (undefined reference to `strcasecmp')
On Tremulous site they say this:
"I always remove the build directory, but that didn't help. I finally solved the problem switching to gcc v4.6.3, everything compiled right!
So, there must be some linking bug in gcc 4.8.1 with curl."

[stigmha]

Attaching the make-macosx-ub.sh edited for oa 8.8. I sent it at the time but I have no idea if it eventually went into the sources.

Thanks, might be handy! 

https://github.com/ioquake/ioq3/pull/32 as well as https://github.com/ioquake/ioq3/pull/31 might be of interest for OA devs (there might be a ticket on https://bugzilla.icculus.org/ as well).

Thanks for the input, but I haven't really had any problems related to that.

Andrewj: my inital goal was to be able to compile and build both the engine and the QVMs without modifying anything. More on that later in this post.

Hitchiker: good catch, this is probably the reason. I haven't tried it yet as I have been busy.

-------------------------------------------

The primary reason why I made this post was because I intentionally wanted to build both the engine and QVMs without any modifications what-so-ever and by following your official guidelines of building the game. So I was originally against modifying the code and project files, and only looking for tweaks for the build tools. My intention is to use the game for my Master's Thesis and I hoped that some of my later changes perhaps could be implemented in the game. Merging my changes with the code base will of course be more tricky if I've done lots of modifications to the build procedure and changes in the code only to make it build in the first place. I've selected to stay away from the SVN repo as my work requires a stable version of the game for the base.

I eventually decided not to spend any more time on this as I learned that source code available from your source downloads page is a mess. Firstly, splitting the QVM code from the engine makes it really difficult and cumbersome to debug the QVMs together with the engine. Especially when this project primarily focuses on building them as *.qvm files and not dynamic link libraries (dll). To make it even more confusing, the engine source project also contains the QVM source code, but it's really outdated and buggy - but you are at least able to debug them as dlls. Both projects also contains several unused leftovers from ioquake3, and there's very little documentation of what to use and what to not use.

This led me to the conclusion that I want one clean project/solution with both the latest version of the QVM sources and the engine without the confusing leftovers. I started from scratch with an empty Visual Studio project and manually merged the latest engine source with the source of the latest QVMs. I am now able to build the latest QVMs in Visual Studio as DLLs and to live debug them. I have also included the scripts/tools to build the QVMs as native *.qvm files. It is currently Windows and Visual Studio 2013 only. Support for MinGW, OSX and Linux should be implemented soon.

Please read the README in my Git repo for full details and changelog: https://bitbucket.org/shalvorsen/openarena-0.8.8-stigmha/.

I have probably broken several of your development principles by doing this, but at least I have a single stable and clean repo of the sources that works for my purposes. You can do whatever you want with it in terms of cloning and adapting modifications from it. I appreciate feedback/comments and any input and even help - except for with my thesis related work ofc, but engine maintenace and build procedures aren't really a part of that.

Thank you for your time.

[jackoverfull]

This could be very interesting actually, leading to some cleaning i the code, perhaps.


BTW
The script was attached to my previous post…

[grey matter]

My intention is to use the game for my Master's Thesis and I hoped that some of my later changes perhaps could be implemented in the game.

Is this due to the included GPLv2 assets? Regarding a maintained code base I'd rather go with ioquake3 itself, which also has a larger developer/maintainer count.
As of now you can also use the ioquake3 engine with OpenArena's game assets (i.e. game code as QVM from OA's pk3s) to play on regular OA servers.

Use the file manager to replace the QVMs in the vm subdirectory of the pk3 file with your new qvms.

I get that this is your research project, but modifying pk3s will prevent you from joining any pure online server (which is most of them).

Please try contributing your patches to ioquake3, which is the upstream not only for OA but quite a few other games.

re:libspeex. Speex is deprecated in favour of Opus.
re:Makefile. The current one supports building the game logic as QVM/so, which includes building lcc/q3asm for the QVM part. Not sure what you'd like to change about that?

It is the first part of my thesis work on the engine and currently allows a client to bypass the main menu and jump straight into a game

I didn't look at the related code, but you can do this currently via +exec, +map, +connect etc.

re:COPYING. lcc is not licensed under GPL.

[stigmha]

This could be very interesting actually, leading to some cleaning i the code, perhaps.


BTW
The script was attached to my previous post…

Yeah, that was my intention. I have downloaded your script  .

Is this due to the included GPLv2 assets?

Yes, at least partly... I've actually written a paper on open source games for computer scientific research, and Open Arena turned out to be one of the best candidates. It was the first part of my thesis so I will be able to defend my choice at the end. The paper was accepted for and presented at LSDVE 2013 in August, but the publishers, Springer, have been really slow so it isn't officially published yet - I'll let you know when.

Regarding a maintained code base I'd rather go with ioquake3 itself, which also has a larger developer/maintainer count.
As of now you can also use the ioquake3 engine with OpenArena's game assets (i.e. game code as QVM from OA's pk3s) to play on regular OA servers.

...

Please try contributing your patches to ioquake3, which is the upstream not only for OA but quite a few other games.

Are these your thoughts or how you're actually intending to maintain the engine for future releases? I will upgrade the engine of my solution to the latest stable version of ioquake3 if it is the case. I'll stick to the OA version otherwise as I'd like to contribute to you, not ioquake3.

I get that this is your research project, but modifying pk3s will prevent you from joining any pure online server (which is most of them).

Most of my planned modifications are engine features only, but I will modify the QVMs if I have to. I am then totally aware of that most servers will reject my client, but it will most likely changes explicitly related to specific parts of the thesis research - something that might not benefit OA.

re:libspeex. Speex is deprecated in favour of Opus.
re:Makefile. The current one supports building the game logic as QVM/so, which includes building lcc/q3asm for the QVM part. Not sure what you'd like to change about that?

Cool, I should change Speex to Opus in the README.

The reason why I don't like the Makefile is that it currently only works in Linux (at least for me) and that there are lots of ioquake3 leftover garbage which OA doesn't use, making the Makefile really huge and cumbersome to interpret. I also consider the MinGW, OSX, and Linux build tools (gcc, etc.) to be so similar now that I believe a much shorter and simpler approach can be done in the Makefile to achieve easy and user friendly cross-platform support. This is just my current opinion though.

I didn't look at the related code, but you can do this currently via +exec, +map, +connect etc.

re:COPYING. lcc is not licensed under GPL.

Nice, I haven't gotten all up-to-speed with all the commands yet. The commands you mention will work for similar functionality to what I have now, but I will later/soon implement virtual clients that are spawned based upon my commands. That's specialized functionality I do not want to mix with the already existing commands. Thanks for the input on LCC, I will add the proper licensing shortly... I'll soon have to go to bed.

[grey matter]

Are these your thoughts or how you're actually intending to maintain the engine for future releases? I will upgrade the engine of my solution to the latest stable version of ioquake3 if it is the case. I'll stick to the OA version otherwise as I'd like to contribute to you, not ioquake3.

I'm neither an OA dev nor an official ioq3 dev (more of a contributor and complainer ). What I tried to say is to provide general fixes you make for your (currently) Windows centric fork to the ioq3 crowd, as they're not really targeting Windows anyways (only cross-builds via mingw, that's why the VS files which are coming from ioq3 we're most likely broken to begin with). So any contribution for getting the VS Windows build up-to-speed would be highly appreciated, even more so if you find related linking/compiling issues.
I can understand if it's too much trouble trying to compare the outdated OA code-base to the ioq3 one to check whether those fixes are still needed at all.

Cool, I should change Speex to Opus in the README.

See https://github.com/ioquake/ioq3/pull/25 (by one of the official devs) and the note on http://www.speex.org/ itself.

The reason why I don't like the Makefile is that it currently only works in Linux (at least for me) and that there are lots of ioquake3 leftover garbage which OA doesn't use, making the Makefile really huge and cumbersome to interpret. I also consider the MinGW, OSX, and Linux build tools (gcc, etc.) to be so similar now that I believe a much shorter and simpler approach can be done in the Makefile to achieve easy and user friendly cross-platform support. This is just my current opinion though.

I was talking about the ioq3 Makefile, which should work on OSX as well (possibly using the make-macosx*.sh wrapper scripts) and which does support MinGW iirc.

The commands you mention will work for similar functionality to what I have now, but I will later/soon implement virtual clients that are spawned based upon my commands. That's specialized functionality I do not want to mix with the already existing commands.

Are those "virtual clients" different to the regular in-game bots? You can spawn those already using /addbot.

[grey matter]

The lonely ioq3 wiki has http://wiki.ioquake3.org/Building_ioquake3 http://wiki.ioquake3.org/Console_Reference
and http://wiki.ioquake3.org/Academic_Articles (note the bot section).

[stigmha]

Ok, thanks, I'll see what I can do to get the engine up to speed with the ioquake3 engine. I'll take a look at the Opus and Speex related resources you provided at the next opportunity. I will also eventually implement a Makefile for other platforms, probably basing it on the ioquake3 Makefile as you suggest. Thanks for the links to the ioquake3 resources, will investigate soon.

The problem with /addbot and the current bot setup is that they are all running in the server code, and not on the client. The entire purpose of virtual clients in the terms of research is to be able to simulate houndres or even thousands of virtual players in order to monitor the server's performance. The bots may not run on the server as it will affect it's processing capabilities. I will indeed use the already available bots, but I need to move them over to the client code, so I can simulate players. Yes, this can in worst case be used for cheating as they will in theory be able to connect to any server, but it is not my intention to include this in the actual game. It is entirely for the case of science ;P. I will create a separate branch for it.

This is my last post today as it's nearly midnight here, I truly appreciate all the feedback and interest so far. It is truly both valuable and motivating to hear from you! Fridays are currently reserved for writing the actual thesis and I seldom have time during weekends, but you'll hear from me soon enough.

[grey matter]

Ok, thanks, I'll see what I can do to get the engine up to speed with the ioquake3 engine. I'll take a look at the Opus and Speex related resources you provided at the next opportunity. I will also eventually implement a Makefile for other platforms, probably basing it on the ioquake3 Makefile as you suggest. Thanks for the links to the ioquake3 resources, will investigate soon.

I didn't mean to suggest you try fixing those things anytime soon. Speex/Opus will most likely be solved by the ioq3 devs themselves. I'd also leave the Makefile as is, given that your current VS solution seems to work and research projects tend to take a lot longer than estimated

The problem with /addbot and the current bot setup is that they are all running in the server code, and not on the client. The entire purpose of virtual clients in the terms of research is to be able to simulate houndres or even thousands of virtual players in order to monitor the server's performance. The bots may not run on the server as it will affect it's processing capabilities. I will indeed use the already available bots, but I need to move them over to the client code, so I can simulate players.

Ah, I see where this is going to. You can either be lazy and just provide RMI for the botlib plus the existing connection/slot reservation functions or go the hard way and truly implement a client bot, which'll require some (realttime) computer vision plus the usual wallhack/aimbot logic.
Be aware that the engine has a harcoded MAX_CLIENTS limit, which is at 64, but even that doesn't work entirely (meaning there are some related bugs which do not properly obey this limit). In id Q3 one ran into problems with the old maximum of 32 clients already.