Tuning FreeBSD to serve 100-200 thousands of connections

I also use nginx as reverse-proxy and load balancer in my project.

mbuf clusters

FreeBSD stores the network data in the mbuf clusters 2Kb each, but only 1500B are used in each cluster (the size of the Ethernet packet)

mbufs

FreeBSD - How to reduce TIME_WAIT connections

Routinely, I did a "netstat -an" on a FreeBSD box, a DNS server. The screen then shower with hundreds of "TIME_WAIT" connections. Seems like some malware infected clients are querying the server and causes the terminated TCP socket waiting to be shutdown, but not fast enough, to be efficient. Fortunately, the numbers of TIME_WAIT sockets accumulated are insignificant. 

In order to reduce the number of socket waiting, tune the system value :

net.inet.tcp.msl

to a shorter time. By default, TIME_WAIT status connections will have to wait for at least 60 seconds (if no reply from the destination that this connection can be terminated) to terminate the connection. This value is based on the RFC 793. But the problem is the RFC was drafted at year 1981. IMHO, the equipments & bandwidth of that time wasn't as fast as the current one. Which means 60 seconds of waiting, an inadequate long time.

The formula to calculate the value (net.inet.tcp.msl) to time of seconds is 2 times of the net.inet.tcp.msl value. which means the value of net.inet.tcp.msl with 30000 means 60000ms (because 2x30000), thus 60 seconds. In order set net.inet.tcp.msl to 15 seconds, change the value of net.inet.tcp.msl to 7500.
E.g.

sysctl net.inet.tcp.msl=7500

This will cause the TIME_WAIT sockets to terminate after waiting for 15 seconds, if no reply from the destination that this connection can be terminated. 

For more info, refer to RFC 793 (search for "Maximum Segment Lifetime").