Pages: [1] 2
  Print  
Author Topic: My server is getting bot striked...  (Read 9430 times)
MilanFIN
Nub


Cakes 1
Posts: 41


WWW
« on: December 08, 2011, 11:37:40 am »

Hi.
My server has 2 ports open on the firewall as needed, and there came not so much unwanted tries to other ports...
But 1 month ago started the problems, some bot found the open default port of oa and the computer started to send massive amounts of stuff outbound.

2 weeks ago I changed the port, and it worked ok for 1 week, then same thing happened... I had re-installed all stuff after the first strike.

week ago, I changed the port again and after 2 days the bot found the port again Rest In PEACE!, now I cant keep the server on...

The bot haven't found the master server port any time, only the port that clients use to connect on the server...

Is there anything I could do to be able to keep the server online? BTW: using ubuntu 10.04
Logged

http://arenafinland.tk a finnish OpenArena server and clan.
grey matter
Member


Cakes 7
Posts: 327

>9k


« Reply #1 on: December 08, 2011, 02:08:55 pm »

Send an abuse mail to the attacker's ISP.
Logged

This space is for rent.
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #2 on: December 08, 2011, 02:46:38 pm »

Why don't you just drop all incoming traffic originating from the attackers IP on your firewall? It will seem like you've closed all your ports to the attacker, so he'll probably give up the hacking after a while.

Code:
sudo iptables -I INPUT -j DROP -s <attacker-ip>
on the console should do the trick.

Edit: sudo version

Edit 2: This firewall command doesn't survive a reboot, so after a reboot your server will allow traffic from the attacker again. I don't know how to save the firewall state on Ubuntu so it will be loaded during the boot process (I can tell you exactly how it works on Red Hat and Fedora though: sudo service iptables save). Someone else?
« Last Edit: December 08, 2011, 03:23:37 pm by 7 » Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
WingedPanther
Member


Cakes 4
Posts: 190



« Reply #3 on: December 08, 2011, 08:28:39 pm »

Firewall Configuration utility may make it easier to make the rule permanent.
Logged

Programming is a branch of mathematics.
WaspKiller
Bigger member


Cakes 7
Posts: 159



WWW
« Reply #4 on: December 08, 2011, 10:30:47 pm »

...1 month ago started the problems, some bot found the open default port of oa and the computer started to send massive amounts of stuff outbound...


MikanFIN, this problem is starting to surface more and more with Q3 based engines.  I have read about the same/similar issues in Q3 and UrT.


Quote
It seems like some clever Botnet finally discovered Excessive Plus (or Quake 3). Our servers have been used for constant DoS-attacks and generated huge traffic.

I have setup some rules to ignore those flood requests but while our servers can't be used for DDoS attacks, we still receive millions of spoofed requests.. To get delisted, our servers will be inaccessible for a day or two.


Here is a long thread at the Q3 ExcessivePlus site: http://www.excessiveplus.net/forums/thread/servers-warp


There were some solutions given (sorry, I did not read the entire thread but here is one from Easy (an E+ Dev).

Read the 2nd post: http://www.excessiveplus.net/forums/thread/servers-warp?page=3  

.
Logged



Calm is for LOSERS!  ANGER fuels my game and btw you're NEXT!
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #5 on: December 09, 2011, 02:41:24 am »

Wait a minute, is MilanFIN's server getting flooded/ddossed or hacked? If the server is getting flooded the incoming packets will have a spoofed source address and you need WaspKiller's solution, but if your server gets hacked and it starts sending outbound traffic without any incoming traffic to cause it (flood other IP's) the hacker probably won't mask his source address by spoofing during the hack and you need my solution. There is no harm in combining the solutions either way, so...

By the way, if your server is getting flooded chances are you know the attacker and he's got issues with your server (kicked/banned), with you personally (argument), or both.

Edit: tried to clarify. Again: If your server starts sending out huge amounts of traffic when there is incoming traffic from a lot of sources, your server is being flooded/ddossed by a botnet or by someone with issues. On the other hand, if your server starts sending out huge amounts of traffic without a lot of incoming traffic, your server got hacked and probably is now part of a botnet itself.
« Last Edit: December 09, 2011, 03:05:08 am by 7 » Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
grey matter
Member


Cakes 7
Posts: 327

>9k


« Reply #6 on: December 09, 2011, 06:46:16 am »

The server can also send a rather large amount of outgoing traffic for little incoming traffic, e.g. the getStatusResponse is several times longer than the getStatus query. This should already be taken care of in ioq3ded itself, but there might be more scenarios like that.

Edit: After reading the e+ thread, it's exactly how I said Smiley Your only hope is to either use iptables to limit those requests, or disable them all together, which would be very annoying to players.
« Last Edit: December 09, 2011, 06:59:37 am by grey matter » Logged

This space is for rent.
MilanFIN
Nub


Cakes 1
Posts: 41


WWW
« Reply #7 on: December 09, 2011, 07:40:23 am »

Some details about the attacks:
-attacks aren't made by players playing in the server....
-the attacks come from different ip adresses, at least 10min from one and then from another, and another and another...., so I think it is hard to be blocked via firewall.
-When the bot or whatever finds the port, it will start sending stuff with 50kt/s to the computer, even if oa is not running...
-after couple of days after it will start sending packages with 2x of the speed that my router can handle.

Is this some kind of botnet attack?

BTW: I cant see the bot's trying all the ports sestematically, they just spam to the previous ports and suddenly it will just find the server port...
IS IT possible that bot would get the port from somewhere in internet, master server maybe, cause the master server port wasn't found for any time... Tongue

+ could it go off if I keep the server off for few weeks Huh
Logged

http://arenafinland.tk a finnish OpenArena server and clan.
grey matter
Member


Cakes 7
Posts: 327

>9k


« Reply #8 on: December 09, 2011, 07:53:28 am »

The approach mentioned in the e+ thread above does not involve banning single IPs or subnets, but goes about rate limiting the number of requests in general.
It's unclear whether the different IPs are due to a botnet being the source or the attacker just spoofing the packets' source address.

If your server is listed in OA's masterserver (+set dedicated 2), then the attacker can just query that list to obtain your OA server's IP and port.

There is usually some reason for botnets to attack a single server. Did you piss off anyone, e.g. banned him from your server?

Is "the attack" just a shitload of incoming traffic or are those valid commands for your OA server, i.e. the "getstatus" command mentioned above?
Logged

This space is for rent.
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #9 on: December 09, 2011, 09:41:18 am »

The server can also send a rather large amount of outgoing traffic for little incoming traffic, e.g. the getStatusResponse is several times longer than the getStatus query. This should already be taken care of in ioq3ded itself, but there might be more scenarios like that.

Edit: After reading the e+ thread, it's exactly how I said Smiley Your only hope is to either use iptables to limit those requests, or disable them all together, which would be very annoying to players.

No it's not exactly as you said. Firstly, the e+ thread is talking about 500 packets per second and that's a considerable flood.

Secondly, the solution presented in the e+ thread doesn't throttle the getstatus requests but completely drops them as soon as there are al least 5 getstatus requests within 2 seconds coming from the same IP.

-the attacks come from different ip adresses, at least 10min from one and then from another, and another and another...., so I think it is hard to be blocked via firewall.
-When the bot or whatever finds the port, it will start sending stuff with 50kt/s to the computer, even if oa is not running...

Yes, this is a botnet attack, you need WaspKillers solution.

What's happening is the botnet is sending your server lots of requests for status information, but by faking the source address of the requests, your server won't send the requested status info to the bot that requested it but somebody else. Because the server status is a lot of information, you will send lots of data for every small getsatus packet you receive from the bots to that unlucky somebody who's IP the bots were spoofing and this will flood/ddos him.

So the attacks are not originating from the IP's you see but those are IPs of the victims. Basically the botnet is using Q3/OA servers to amplify its ddos attacks, and your server is one of those amplifiers.

Quote
+ could it go off if I keep the server off for few weeks Huh

Probably not, once you're on the botnet's list of servers, it's very hard to get off again. You really need WaspKillers solution.
Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
grey matter
Member


Cakes 7
Posts: 327

>9k


« Reply #10 on: December 09, 2011, 10:06:46 am »

I recommend using a newer ioquake3 build, since they've put code in place to prevent exacatly this scenario.
I don't know how easy you can replace oaded with ioq3ded, but it should not require much more than a few +set com_gamename options and such.
Logged

This space is for rent.
MilanFIN
Nub


Cakes 1
Posts: 41


WWW
« Reply #11 on: December 09, 2011, 10:53:30 am »

The attacks happen when there is no-one in the server, and might come when the oa-server-program is not even running.
And I have only KICKED 1 person from my server 8-9 months ago, when he was spamming and abusing other players, only kicked, no ban...

Uhm. Can someone give step-by-step to make the ip-banning system for too many requestes, still this is not about the oa, the strikes come ALWAYS, not only when oa-server is running.
Logged

http://arenafinland.tk a finnish OpenArena server and clan.
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #12 on: December 09, 2011, 11:40:03 am »

The attacks happen when there is no-one in the server, and might come when the oa-server-program is not even running.
And I have only KICKED 1 person from my server 8-9 months ago, when he was spamming and abusing other players, only kicked, no ban...

The bot software is not smart enough to check if each Q3/OA server is actually online before it starts abusing it to ddos its victims, or it does this only once a day or so. It's nothing personal, they're just abusing your OA server. The 10 minute periods are typical for ddossing dsl and cable modems off the internet, so I'm pretty sure its a botnet and nothing personal. If it were personal and you were the ddos target instead of the amplifier, the source addresses would be much more random.

Quote
Uhm. Can someone give step-by-step to make the ip-banning system for too many requestes, still this is not about the oa, the strikes come ALWAYS, not only when oa-server is running.

You can't do much about the incoming traffic from the bots, the only thing you can do is make sure your server is not flooding other people off the internet.

I think the easiest way to go about the ip-banning system is to make a shell script and to run it every time your server boots. The script should contain the commands specified in the post on the e+ forum WaspKiller is talking about. But first you should check if your iptables has the 'u32' and 'recent' modules and if not you should install them.

I'll write the script for you if you need more help, but I can't help you with installing the modules and such (I'm not an Ubuntu man).

Edit: rewrote the script to be more generic

Code:
#!/bin/bash

function numcommas () {
local S="${1//[^,]/}"
echo ${#S}
}

PORTS=${1:-27960}
NUMPORTS=$(($(numcommas ${PORTS}) + 1))
HITS=$((${2:-5} * ${NUMPORTS}))
SECONDS=${3:-2}

iptables -N quake3_ddos
iptables -A quake3_ddos -m u32 ! --u32 "0x1c=0xffffffff" -j ACCEPT
iptables -A quake3_ddos -m u32 --u32 "0x20=0x67657473&&0x24=0x74617475&&0x25&0xff=0x73" -m recent --name getstatus --set
iptables -A quake3_ddos -m recent --update --name getstatus --hitcount ${HITS} --seconds ${SECONDS} -j DROP
iptables -A quake3_ddos -j ACCEPT

if [ ${NUMPORTS} -eq 1 ]; then
iptables -I INPUT -p udp --dport ${PORTS} -j quake3_ddos
else
iptables -I INPUT -p udp --dports ${PORTS} -j quake3_ddos
fi

Store this somewhere as root (as /usr/local/bin/block_q3dd for example), make it executable and call it on boot (in /etc/rc.local for example).

The script has 3 optional parameters: a port or list of ports separated by comma's (default: 27960), how many requests per port are allowed (default: 5), and within what number of seconds (default:2). So if you've got 2 servers on port 27960 and 27961 and you want to drop getstatus packets at 3 requests per server per second or more you should call it this way in /etc/rc.local:

Code:
/usr/local/bin/block_q3dd 27960,27961 3 1

Edit 2: forgot a line in the script, it's correct now.
« Last Edit: December 09, 2011, 02:00:14 pm by 7 » Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
MilanFIN
Nub


Cakes 1
Posts: 41


WWW
« Reply #13 on: December 09, 2011, 02:53:16 pm »

What does that script exactly do? ban the ip from connecting to the server computer?
The attack ip-s have to be banned from connecting to the pc itself not only on oa btw.
Logged

http://arenafinland.tk a finnish OpenArena server and clan.
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #14 on: December 09, 2011, 03:14:52 pm »

What does that script exactly do? ban the ip from connecting to the server computer?
The attack ip-s have to be banned from connecting to the pc itself not only on oa btw.

The script sets up some firewall rules, which check for IPs sending packets with the Q3/OA server query "getstatus" on the ports you specify. If an IP sends a getstatus query it is put on a list, and if it occurs on that list more than the number of times you specify in the number of seconds you specify, further traffic from that IP is blocked (until te traffic from that IP drops under the limits you set).

It's no use banning the attacking IPs completely, because:
  • You're not really attacked from those IPs (they are spoofed) but those are actually the IPs of the hosts the botnet is attacking (and it's using your server to attack them).
  • The incoming packets are on your computer already before your iptables firewall can filter them, so blocking them with iptables won't do you any good. You're blocking to avoid sending all the outbound traffic to the botnet's victims, because this could get you into trouble with the victims and/or with your ISP, and because hopefully the botnet will give up on using your server as an attack vector/amplifier after a while.
Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
MilanFIN
Nub


Cakes 1
Posts: 41


WWW
« Reply #15 on: December 09, 2011, 03:58:13 pm »

Gonna try that our tomorrow->clock is 24:00  Rest In PEACE!
I hope that it works Tongue
Logged

http://arenafinland.tk a finnish OpenArena server and clan.
Gig
In the year 3000
***

Cakes 45
Posts: 3194


WWW
« Reply #16 on: December 09, 2011, 05:53:23 pm »

I recommend using a newer ioquake3 build, since they've put code in place to prevent exacatly this scenario.
I don't know how easy you can replace oaded with ioq3ded, but it should not require much more than a few +set com_gamename options and such.

Uhm... depending from when ioquake3 added that defensive features, they could already have been ported to openarena: try using latest oa executables (at the moment, they are version 25: http://openarena.ws/board/index.php?topic=1933.msg41587#msg41587)... but if it is a very recent feature (if these DDoS attacks are new for q3), probably they won't be here yet.
« Last Edit: December 10, 2011, 06:26:03 am by Gig » Logged

I never want to be aggressive, offending or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
grey matter
Member


Cakes 7
Posts: 327

>9k


« Reply #17 on: December 10, 2011, 06:20:48 am »

Quote from: ioquake3 log
Revision: 1762
Time: 03.01.2010 23:12
Author: tma
Path: code/server/sv_main.c (trunk)
Message: * Rate limit getstatus and rcon connectionless requests

From what I gather around the net, the attack mechanism has been well know for ages, but is getting actively used this year.
Logged

This space is for rent.
Gig
In the year 3000
***

Cakes 45
Posts: 3194


WWW
« Reply #18 on: December 10, 2011, 06:39:01 am »

January 2010? Then probably, using version 25 executables may work (but Sago could be sure)... Milan_fin, you may try and let us know if your outgoing bandwith consumption gets some positive effects from it.
Do you know if that ioquake3 defensive measure is always enabled, or is it controlled by a cvar?

Another thing.. Milan said "The attacks happen when there is no-one in the server".... uhm... maybe they do this way to avoid your users experience lag when playing (trying to make you do not notice there is a problem and start investigating)... Maybe, as a workaround, you may try to set bot_minplayers 1 or 2, to never have the server empty (but if they are smart enough to check for human players only, this workaround may not work)...
Logged

I never want to be aggressive, offending or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
grey matter
Member


Cakes 7
Posts: 327

>9k


« Reply #19 on: December 10, 2011, 06:54:09 am »

The rate limiting code is not configurable in any way and enabled by default. Just take a look at the file mentioned as "Path" in my previous post and look for SVC_RateLimit*.

The linked #25 version of OA's code does already include the rate limiting code.
Logged

This space is for rent.
MilanFIN
Nub


Cakes 1
Posts: 41


WWW
« Reply #20 on: December 10, 2011, 09:15:04 am »

Could someone just tell what I have to do... Rest In PEACE!
Step-by-step if possible, I am noob in server side.


@Gig:
There was 2 bots per team all the time when the server was on and no-one playing... no help from that.
Logged

http://arenafinland.tk a finnish OpenArena server and clan.
Gig
In the year 3000
***

Cakes 45
Posts: 3194


WWW
« Reply #21 on: December 10, 2011, 06:05:03 pm »

Do you mean what you have to do to test with the new executables?
Well, go to the post linked above, and download the zip file of version 25 binaries. Then unzip its content inside your OpenArena installation directory. It will overwrite some old files (if you do not want to lose original executables, you may manually rename them or copy them somewhere before overwriting them). Then launch the game as usual.
Logged

I never want to be aggressive, offending or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #22 on: December 11, 2011, 02:18:05 am »

This has me wondering why a developer was posting firewall rules on the E+ forum 3 weeks ago when the flood protection has been in the server code for almost 2 years? /me confused.
Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
Gig
In the year 3000
***

Cakes 45
Posts: 3194


WWW
« Reply #23 on: December 11, 2011, 04:40:05 am »

I don't know, but...
- The fact it is in ioquake3 svn since 2010 does not imply it is in current official ioquake3 executables (1.36 are dated ....?), nor in official OpenArena executables (OA 085 executables are dated.... ? And based upon ioquake3 dated ....?)
- The fix is however a way to limit the problem (limiting the maximum status packets you may accept/send)... but cannot resolve it completely (your incoming bandwidth will still be used by the attacks, and you will still answer to a part of the requests, right?). If I understood correctly.
I don't know if it exists some way to identify the real sender of spoofed udp packets, and to send police @ his home.
Logged

I never want to be aggressive, offending or ironic with my posts. If you find something offending in my posts, read them again searching for a different mood there. If you still see something bad with them, please ask me infos. I can be wrong at times, but I never want to upset anyone.
7
Member


Cakes 7
Posts: 278


Is 7 up?


« Reply #24 on: December 11, 2011, 05:15:35 am »

I know the distributed ioq3 executables are rather old, but I figured a developer would point people to a more recent version rather than reinvent the wheel with firewall rules.

A receiving host can't identify the real sender of spoofed UDP packets, only upstream providers with enough routing information are capable of doing that. Contacting your ISP is not always a good idea though, chances are your ISP will block all UDP traffic to your OA server ports (so your server will become unreachable to everybody) or worse, especially if you're not allowed to run servers over your connection in the first place (read the terms).
Logged

I'm on the ten most wanted list, I've got it dead in the groove.
My face is on every wanted poster in town, for the way I move.
Pages: [1] 2
  Print  
 
Jump to: