Gulpe.de is down, the ioquake 3 DDOS UDP Bug further still exists

[schorsch]

I'm so sorry,

but there was a lot of ddos UDP attacks with ip spoofing over Gulpe.de (Gulpe.de was the relay server) to other servers. Another server admin informed me about the ddos attack is coming from Gulpe.de i cant believe that, then i can see with tcpdump there is coming in and going out a lot of UDP ddos attacks with ip-spoofing, the Port is openarena. The OS is Debian Jessie, but the Bug is in ioquake3, but i can't run further the openarena server, when it attacks other servers with ddos by this Bug. 

The game is great, I am very sad that this bug there are still 2015 January further continues...  :-(

Bye bye, it was a great time with this Game, had had a lot of fun with it. 

If the bug is solved, i can start a new server on Gulpe.de

best regards
schorsch

[Gig]

Do you know something more about the problem?
Is this ioquake3 bug?
https://bugzilla.icculus.org/describecomponents.cgi?product=ioquake3

(To list more ioquake3 bugs, see https://bugzilla.icculus.org/describecomponents.cgi?product=ioquake3)

[schorsch]

Hello,

i think you mistake me and the Problem, the Bug is not a component, the bug is inside in ioquake3.

It is a old public known bug from quake3 engine over years, i am really sad about this problem. Possible follow of such problem
can really will be expensive for a server administrator, claims for damages, and other scary things it will bring.

I hate this stupid script kiddies, they use an really very old bug and destroy the gaming fun of a lot of people.
They are not actual heroes, they are stupid and not creative thinking for other people, still egoists.

Bye, bye, i hope you can solve the bug.

best regards
schorsch

[grey matter]

i think you mistake me and the Problem, the Bug is not a component, the bug is inside in ioquake3.

Those are "components" in the icculus.org bugtracker for ioquake3. You can just search for bugs without specifying a component.

Bye, bye, i hope you can solve the bug.

From what I know, the related bugs (getchallenge, getstatus, rcon are all rate-limited) are already fixed in ioq3 (e.g. #5954, #5678 but there are related commits without tickets as well).

[Gig]

i think you mistake me and the Problem, the Bug is not a component, the bug is inside in ioquake3.

Oops, I pasted the wrong link (I pasted the second one also at the place of the first one!), sorry.
This is the page I wanted to link: https://bugzilla.icculus.org/show_bug.cgi?id=5954 (gray matter also linked it in the meanwhile)

[schorsch]

So i will show you,

how the attack on gulpe.de has worked. This "Bug" is in the UDP Protocol from ioquake3, there is a get gamestatus request from the client with 5kb and the Server answer is like 500kb size sent. This and the UDP Protocol is perfect for ip spoofing, so you can not see who the ip packets sends.

Here is a example to look how old is this "Bug":

klick to understand..

This works on Openarena Server also.

here look :-(

there a lot more people with the same ddos attacks, i think in tcp you cant so easy make ip spoofing like in UDP, that is the old problem.

best regards
schorsch

[fromhell]

Would a server-only update work out for this?

[grey matter]

Would a server-only update work out for this?

The only thing that might need updating are the OA server binaries, i.e. openarena-ded. Might be possible to just use (nightly) ioq3 builds to run a dedicated OA server as well.

[schorsch]

hi,

all that is not working, but the server will ever answer on udp ip spoofing pakets. The only thing what can help is away from udp and go to tcp protocol this is a lot harder for ip spoofing. Because the routers and many other network hardware can better and secure handle that protocol then udp. This only can solve this old problem.

mfg
schorsch

[fromhell]

Might be possible to just use (nightly) ioq3 builds to run a dedicated OA server as well.

I haven't looked into this but ioq3 binaries should still work with the videoflags and human counting stuff right?

[Gig]

all that is not working, but the server will ever answer on udp ip spoofing pakets. The only thing what can help is away from udp and go to tcp protocol this is a lot harder for ip spoofing. Because the routers and many other network hardware can better and secure handle that protocol then udp. This only can solve this old problem.

Uhm... even if it's not easy/possible to detect IP spoofing on UDP... isn't it possible to detect excessive queries for that feature? I mean "If I get more than X requests for these packets in Y time from Z IP, then it's probably a DDOS amplification try: stop responding to that IP to make the attack useless.". Is this the way the ioquake3 patch was meant to work?

[grey matter]

Might be possible to just use (nightly) ioq3 builds to run a dedicated OA server as well.

I haven't looked into this but ioq3 binaries should still work with the videoflags and human counting stuff right?

There's code to expose g_humanplayers. I don't know about videoflags.

all that is not working, but the server will ever answer on udp ip spoofing pakets. The only thing what can help is away from udp and go to tcp protocol this is a lot harder for ip spoofing. Because the routers and many other network hardware can better and secure handle that protocol then udp. This only can solve this old problem.

Uhm... even if it's not easy/possible to detect IP spoofing on UDP... isn't it possible to detect excessive queries for that feature? I mean "If I get more than X requests for these packets in Y time from Z IP, then it's probably a DDOS amplification try: stop responding to that IP to make the attack useless.". Is this the way the ioquake3 patch was meant to work?

There's no way to fix the Quake 3 protocol in a backwards-compatible way. The overhead of TCP is too high and I doubt that anyone is going to write a port anyways.
Regarding your UDP DoS problem; ISPs should do ingress filtering. Rate-limiting per IP is what the ioq3 code is doing, and you could achieve the same effect with nftables or your favourite firewall.

[sago007]

all that is not working, but the server will ever answer on udp ip spoofing pakets. The only thing what can help is away from udp and go to tcp protocol this is a lot harder for ip spoofing. Because the routers and many other network hardware can better and secure handle that protocol then udp. This only can solve this old problem.

You don't need TCP but you do need to perform a light handshake. You can perform a handshake over UDP too.

Instead of currently:
Client sends 5 kb
Server responds with 500 kb

it would be:
Client sends a 2 kb getpassword request
Sever reponds with 1 kb (including a key generated from a nonce+client IP)
Client sends a 5 kb getgamestatus request with the key
Server reponds with 500 kb if the provided key matches the client IP

It would be very hard for an attacker to get the key for a different IP as long as the nonce is random enough and hash that forms the unique key is significantly safe.

Compared to other cases the unique key doesn't even need to be that extreme. As long as the attacker needs to send more data to the server than the server sends to the victim it would not cause an amplification. Even if the attacker can limit her to 1000 keys and just send them all, she would still need to send 5000 kb to create a 500 kb result. A little case should be taken here as the attacker might be able to see if the result was correct by timing the server (if sending the correct key can cause lag, the attacker knows she has the right key then the server starts to lag).

[kealper]

I had a similar issue around the same time as schorsch on the Raspberry Pi server, however, my server wasn't actively participating in the attacks. I believe the latest versions of OA (085, 088) have the patch that adds rate limiting to out-of-band requests like getstatus, getinfo, etc. Looking at a tcpdump of the traffic, most of the attempted reflection was using the source port 2980 with a 14-byte getstatus request which makes mitigating it through firewall filtering fairly easy. There were some other times more recently that I had some problems with getting huge amounts of getstatus requests (5k-10k a second) that didn't match the firewall rule. That forced me to move the server off the default port, 27960, as the number of requests the server was getting was just too much for the poor little Pi to handle when the spoofed requests would burst that high. Changing off the default port and dropping all traffic destined to the default port seems to have done the trick, as it's been about a week since I've done that and I haven't had any non-OA traffic to my server that I've seen yet.

My server version has rate limiting and only responds to ~10 requests per-second so it's not really vulnerable to attack, but servers running older OA versions such as 081 do not have this same rate limiting on things like getinfo and getstatus, and this means they're very vulnerable to a reflection attack unless the admins of those server still running 081 have put external firewall rules in-place to prevent it.

The version string of my server, since it may be relevant: ioq3 1.36+svn2287-1/Debian linux-arm Jul 3 2012

EDIT: Just tried with the stock dedicated server that ships in 088, it has the same rate limit as the ARM build I run on the Raspberry Pi server, so it is not vulnerable. I haven't tested 085, but I did confirm that 081 is vulnerable.

So the easiest fix would be to update your servers to 088. It won't totally stop your server from responding to the bogus requests, but it will greatly reduce the amount of traffic your server is sending if you find yourself being used as a reflector in an attack. Without the rate limiting, your server could be using the entire connection's upload speed for reflecting an attack, with the rate limiting it will use somewhere around 5-10KB/s, or about the same as having one more player on the server.

[Suicizer]

So the easiest fix would be to update your servers to 088. It won't totally stop your server from responding to the bogus requests, but it will greatly reduce the amount of traffic your server is sending if you find yourself being used as a reflector in an attack. Without the rate limiting, your server could be using the entire connection's upload speed for reflecting an attack, with the rate limiting it will use somewhere around 5-10KB/s, or about the same as having one more player on the server.

Wait wut?
Why would you still run some ancient 0.8.1 server in the first place when clients use 0.8.8?

[kealper]

Wait wut?
Why would you still run some ancient 0.8.1 server in the first place when clients use 0.8.8?

The 088 client will run on 081 servers, so there's still servers that either haven't updated because they don't feel like it, or because they don't like some of the changes made between 081 and 088.

[Neon_Knight]

Well, not really our fault. 0.8.8 has been around for two years. And we had like four years between 0.8.1 and 0.8. with plenty devtest versions in between.

Servers should update to 0.8. as 0.8.1 is officially unsupported.

[kealper]

And we had like four years between 0.8.1 and 0.8.8

Oh man I feel old now...

[Suicizer]

Well, not really our fault. 0.8.8 has been around for two years. And we had like four years between 0.8.1 and 0.8. with plenty devtest versions in between.

Servers should update to 0.8. as 0.8.1 is officially unsupported.

So this topic is just a troll actually.

[CASE SOLVED]

[Neon_Knight]

I don't think it's a troll. But we aren't going to solve problems of 7-year outdated servers when we had two released patches. (WHICH EVEN HAD THOSE ISSUES FIXED!)

[Suicizer]

I don't think it's a troll. But we aren't going to solve problems of 7-year outdated servers when we had two released patches. (WHICH EVEN HAD THOSE ISSUES FIXED!)

In any case, the topic could be modified as being solved.

[Gig]

So this topic is just a troll actually.

Wait Suicizer... don't you think you may risk to offend someone?

Schorsch, could you please tell us which binaries your server was using?

Can you bring up your server using offiicial 0.8.8 binaries, and check if you still find those problems happening?