RyanFrantz

Apache TCP Backlog

NOTE: I wrote this on an internal company blog this week. It's been modified for posting here.

For about the last year, I've had this little niggle pestering me: how to capture the value of a TCP socket's listen backlog, specifically Apache's, with the goal of helping to monitor pressure on the service in the form of unhandled connections. I've poked at this problem from time to time but had never found a solution, until now.

TCP Listen Backlog

For some background, know that whenever a process opens a TCP socket, it defines a listen backlog value. This value determines the number of fully acknowledged (SYN -> SYN/ACK -> ACK) connections that are waiting to be accept()ed by the process. When requests are being processed quickly, this value should be 0, if not really low.

Apache ListenBacklog

By default, Apache sets its listen backlog to 511 based on the ListenBacklog directive. However, the Linux kernel has a completely different idea in mind: if the value of a socket's listen backlog exceeds that of net.core.somaxconn sysctl value (defaults to 128 on stock builds), the kernel quietly shrinks the socket's listen backlog to net.core.somaxconn. Thanks, Linus.

Tuning Apache and ss

While working on a project I needed to tune a small number of Apache boxes to handle a decent chunk of traffic (~3000 requests per second across 2, maybe 4 boxes). Our production web cluster contains a large number of servers so each box is configured to handle a small percentage of overall traffic. For this project, with fewer servers, I needed to tune Apache and possibly the kernel.

I set about tweaking Apache's config and benchmarking performance. I launched several sieges [1] against a few poor, unsuspecting Apache boxes and crushed them. I knew about the TCP listen backlog and tuned net.core.somaxconn along with Apache's Server and *Workers directives. I had hoped that ss [2] would show me the state of the TCP listen backlog during these trials but I could never surface the information.

Despite this, I was able to find a decent balance among the net.core.somaxconn setting and Apache's config so that requests were handled without connections dropping. At a minimum, I was able to monitor the global ListenDrops and ListenOverflow counters provided by netstat. And so it came to be that we could serve lots of traffic on a small number of boxes.

Still, the TCP listen backlog taunted me, coming to me in my dreams. I may have even crafted what I imagined the backlog to look like out of mashed potatoes...

Trawling the Kernel and ss Source

I had enough. I started digging into kernel source this week to find the damned thing. The more I dug, the more I found traces. First, I landed a direct hit in include/net/sock.h with the following comment: [3]

*        @sk_ack_backlog: current listen backlog

I'd caught the scent and it wafted, nay stang, with an impudence that dared me further. Heady with anticipation I trudged further into the kernel.

net/ipv4/tcp.c contains a function called tcp_get_info() that exposes TCP information in a struct called tcp_info. When called, if a socket is in the LISTEN state, it assigns the value of the listen backlog to a property called tcpi_unacked: [4]

    if (sk->sk_state == TCP_LISTEN) {
        info->tcpi_unacked = sk->sk_ack_backlog;

Hoping that ss could finally be of some use again here (I never gave up on this workhorse) I dug into its source (from iproute2-3.10) and found that it does, in fact poll sockets for that same struct and will display the backlog:

     if (info->tcpi_unacked)
         printf(" unacked:%u", info->tcpi_unacked);

See, I never found this before because it made sense to me that any variables tracking a listen backlog would have the string 'backlog' somewhere in their names. Naming is especially hard when you give something a name that's a bit overloaded.

Displaying Backlogged Connections with ss

Now that we know backlogged connections are defined as unacked, we can call on ss to determine if the HTTP socket is backing up:

[root@apache01:~] $ ss -lti '( sport = :http )'
State      Recv-Q Send-Q                                                       Local Address:Port                                                           Peer Address:Port
LISTEN     223    511                                                                      *:http                                                                      *:*
          rto:1000 mss:536 cwnd:10 unacked:223

The output above was output when siegeing Apache.

Finally, Mutha$%^&@n' Graphs!

After a year, we've got TCP listen backlog graphs for Apache!

Apache TCP backlog under pressure

This graph was generated by reducing Apache's available workers, variously reducing net.core.somaxconn, and pummeling the box with siege. Look at allllll those connections sitting, waiting to pulled off the backlog to be processed by Apache. It's GLORIOUS!

Typically, this graph should remain at 0 or very, very low. If it creeps up and is sustained, it demonstrates that Apache is unable process the volume of requests it's receiving.

Gawd I love graphs!

References

[1] siege is a handy tool for assailing an HTTP server with requests. Check it out.

[2] ss is a handy tool that provides lots of information on sockets on system. Check it out.

[3] See https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/include/net/sock.h#n265

[4] See https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/net/ipv4/tcp.c#n2673

For a good introduction to how the TCP backlog works, give http://veithen.github.io/2014/01/01/how-tcp-backlog-works-in-linux.html a read.