FreeBSD TCP Performance Tuning

To enable RFC 1323 Window Scaling and increase the TCP window size to 1 MB on FreeBSD, add the following lines to /etc/sysctl.conf and reboot.

net.inet.tcp.rfc1323=1
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576

You can make these changes on the fly via the sysctl command. As always, the '$' represents the shell prompt and should not be typed.

$ sudo sysctl net.inet.tcp.rfc1323=1
$ sudo sysctl kern.ipc.maxsockbuf=16777216
$ sudo sysctl net.inet.tcp.sendspace=1048576
$ sudo sysctl net.inet.tcp.recvspace=1048576

In addition, FreeBSD may have a low number of network memory buffers (mbufs) by default. You can view the current mbuf configuration by running netstat -m. If your mbuf value is too low, it may cause your system to become unresponsive to the network. Increase the number of mbufs by adding the line below to /boot/loader.conf and rebooting.

kern.ipc.nmbclusters="16384"

Tuning FreeBSD to serve 100-200 thousands of connections

I also use nginx as reverse-proxy and load balancer in my project.

mbuf clusters

FreeBSD stores the network data in the mbuf clusters 2Kb each, but only 1500B are used in each cluster (the size of the Ethernet packet)

mbufs

FreeBSD - How to reduce TIME_WAIT connections

Routinely, I did a "netstat -an" on a FreeBSD box, a DNS server. The screen then shower with hundreds of "TIME_WAIT" connections. Seems like some malware infected clients are querying the server and causes the terminated TCP socket waiting to be shutdown, but not fast enough, to be efficient. Fortunately, the numbers of TIME_WAIT sockets accumulated are insignificant. 

In order to reduce the number of socket waiting, tune the system value :

net.inet.tcp.msl

to a shorter time. By default, TIME_WAIT status connections will have to wait for at least 60 seconds (if no reply from the destination that this connection can be terminated) to terminate the connection. This value is based on the RFC 793. But the problem is the RFC was drafted at year 1981. IMHO, the equipments & bandwidth of that time wasn't as fast as the current one. Which means 60 seconds of waiting, an inadequate long time.

The formula to calculate the value (net.inet.tcp.msl) to time of seconds is 2 times of the net.inet.tcp.msl value. which means the value of net.inet.tcp.msl with 30000 means 60000ms (because 2x30000), thus 60 seconds. In order set net.inet.tcp.msl to 15 seconds, change the value of net.inet.tcp.msl to 7500.
E.g.

sysctl net.inet.tcp.msl=7500

This will cause the TIME_WAIT sockets to terminate after waiting for 15 seconds, if no reply from the destination that this connection can be terminated. 

For more info, refer to RFC 793 (search for "Maximum Segment Lifetime").