Articles‎ > ‎Apache‎ > ‎

Socket backlog tuning for apache

With more clients moving to the mobile world web traffic changes somewhat and new challenges rise. One particular challenge related to the fact that mobile customers are not as fast as broadband ones and they create all kinds of problems related to big roundtrip latency. One of the problems can be detected in form of occasional 3 second connection timeout to the web server. That connection timeout happens because of SYN packet loss which happens because backlog queue for the web server is filled up with connections sending SYN+ACK to slow mobile customers. The fist thing you may naturally try is to raise net.ipv4.tcp_max_syn_backlog to 2k, but surprisingly that does nothing. Its because this parameter controls global backlog but backlog per port is limited by net.core.somaxconn. You can see number of queue with this command: netstat -ant | grep -c SYN_REC. You can manually emulate SYN backlog congestion with hping3, dont forget to forge source address otherwise server will get RSTs and backlog wont fill up.

Somaxconn by default set to very small number like 128 or 256. we need to set it to something more sane like 1024 then restart apache (it needs to reopen listening socket for parameter to take effect). But wait - why backlog queue does not go past 512? It is because apache caps it by default to that number, and it needs to be increased. Also global backlog needs to be higher than somaxconn or you wont be able to log in to your server if backlog fills up unless you enable syn cookies.

Speaking about syn cookies - many argue that it violates TCP standards by not using timestamps and scaling... but syn cookies kick in only if back log fills up, in the situation where you would loose SYN packets otherwise, it is better to serve something to the customer than serve nothing. Also scaling is possible with syn cookies now in new linux kernels.

So the final configuration is:
# /etc/sysctl.conf :
net.ipv4.tcp_max_syn_backlog = 2048
net.core.somaxconn = 1024
net.ipv4.tcp_syncookies = 1
(dont forget to reload sysctl with sysctl -f /etc/sysctl.conf)

Then increase backlog in apache config via
ListenBacklog 1023

Then restart apache and you should not see 3 second timeouts anymore.

Comments