Load Balancing with HAProxy for MQTT

Need to cluster

IBM has a Redbook for its own IOT service environment. The book also described the topologies of enterprise MQTT service architectures. So, to cluster some mosquitto brokers, we need some other tools.

Mosquitto is a single thread process. To scale out, we need tools to better monitor and manage them.

Cue in load balancing

I followed the steps in this nice tutorial, others you googled might be just as good, as HAProxy is very easy to install, configure and run.

Use HAProxy to load balance 300k concurrent tcp socket connections: Port Exhaustion, Keep-alive and others

I’m trying to build up a push system recently. To increase the scalability of the system, the best practice is to make each connection as stateless as possible. Therefore when bottleneck appears, the capacity of the whole system can be easily expanded by adding more machines. Speaking of load balancing and reverse proxying, Nginx is probably the most famous and acknowledged one. However, TCP proxying is a rather recent thing. Nginx introduced TCP load balancing and reverse proxying from v1.9, which is released in late May this year with a lot of missing features. On the other hand, HAProxy, as the pioneer of TCP loading balacing, is rather mature and stable. I chose to use HAProxy to build up the system and eventually I reached a result of 300k concurrent tcp socket connections. I could have achieved a higher number if it were not for my rather outdated client PC.

Step 1. Tuning the Linux system

CLEAN URL REWRITES USING NGINX

This article will cover how to easily implement Clean URLs (also known as Semantic URLs, RESTful URLs, User-Friendly URLs and Search Engine-Friendly URLs) using NGINX Web Server; currently the second most popular web server platform worldwide.

Not using NGINX web server? No problem, Apache users can check out my article Clean URL Rewrites Using Apache.

A Clean URL is a URL that does not contain query strings or parameters. This makes the URL easier to read and more understandable to users. 

Clean URLs are a high ranking factor for many search engines, but there are many other reasons why having Clean URLs can be important to your website. Check out our article Are your URLs Letting Your Website Down for more information.

Nginx rewrite rules for MyBB SEO friendly URLs

I've recently switched a forum I run from Wordpress and bbPress to MyBB. Overall I've very happy with it but there are a few bits to watch out for with MyBB.

If you are running Nginx instead of Apache as your webserver then one of those things to watch out for is the need to manually update the rewrite rules needed for SEO friendly URLs.

MyBB SEO friendly URLs

Below is the full set of Nginx rewrite rules for rewriting MyBB's SEO friendly URLs. To activate this option in MyBB you can do so from the Admin CP:

Admin CP > Configuration > Settings > Server and Optimization Options > Enable search engine friendly URLs? = Enabled

You can also enable the "Enable search engine friendly URLs in Archive?" option - it's the option under the "Enable search engine friendly URLs?".

If you decide to enable just the "Enable search engine friendly URLs?" option you can use the following nginx rewrite rules:

rewrite ^/forums/forum-([0-9]+)\.html$ /forums/forumdisplay.php?fid=$1;
rewrite ^/forums/forum-([0-9]+)-page-([0-9]+)\.html$ /forums/forumdisplay.php?fid=$1&page=$2;
rewrite ^/forums/thread-([0-9]+)\.html$ /forums/showthread.php?tid=$1;
rewrite ^/forums/thread-([0-9]+)-page-([0-9]+)\.html$ /forums/showthread.php?tid=$1&page=$2;
rewrite ^/forums/thread-([0-9]+)-lastpost\.html$ /forums/showthread.php?tid=$1&action=lastpost;
rewrite ^/forums/thread-([0-9]+)-nextnewest\.html$ /forums/showthread.php?tid=$1&action=nextnewest;
rewrite ^/forums/thread-([0-9]+)-nextoldest\.html$ /forums/showthread.php?tid=$1&action=nextoldest;
rewrite ^/forums/thread-([0-9]+)-newpost\.html$ /forums/showthread.php?tid=$1&action=newpost;
rewrite ^/forums/thread-([0-9]+)-post-([0-9]+)\.html$ /forums/showthread.php?tid=$1&pid=$2;
rewrite ^/forums/post-([0-9]+)\.html$ /forums/showthread.php?pid=$1;
rewrite ^/forums/announcement-([0-9]+)\.html$ /forums/announcements.php?aid=$1;
rewrite ^/forums/user-([0-9]+)\.html$ /forums/member.php?action=profile&uid=$1;

Activating SEO friendly URLs in Archive

If you activate the "Enable search engine friendly URLs in Archive?" option you'll need the following extra rewrite rules:

rewrite ^/forums/archive/index.php/forum-([0-9]+)\.html$ /forums/archive/index.php?forum-$1.html;
rewrite ^/forums/archive/index.php/thread-([0-9]+)\.html$ /forums/archive/index.php?thread-$1.html;
rewrite ^/forums/archive/index.php/forum-([0-9]+)-([0-9]+)\.html$ /forums/archive/index.php?forum-$1-$2.html;
rewrite ^/forums/archive/index.php/thread-([0-9]+)-([0-9]+)\.html$ /forums/archive/index.php?thread-$1-$2.html;

At first glance the second set of rules might not seem essential, but it's those rules that enable pagination. Without them you'll just get 404 errors for page 2 or above on any of the archive forum or thread lists.

Mysql Database Performance tuning

Mysql provides a configuration file located in /etc/my.cnf. From here you can set all of the memory, table, and connection limits as well as a host of other options.
You can get the default buffer sizes used by the mysqld server with this command:

shell> mysqld –help

This command produces a list of all mysqld options and configurable variables. The output includes the default values and looks something like this:

mysql multiple instances on CentOS 7

mkdir -p /data/db/mysql

./mysql_install_db --datadir=/data/db/mysql/mysql-instance-01 ./mysql_install_db --datadir=/data/db/mysql/mysql-instance-02 ./mysql_install_db --datadir=/data/db/mysql/mysql-instance-03 ./mysql_install_db --datadir=/data/db/mysql/mysql-instance-04

cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-01/my.cnf cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-02/my.cnf cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-03/my.cnf cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-04/my.cnf

vi /data/db/mysql/mysql-instance-01/my.cnf

[mysqld]
port = 44001
socket = /var/lib/mysql/mysql-instance-01.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
innodb_file_per_table

datadir = /data/db/mysql/mysql-instance-01/
log-error = /data/db/mysql/mysql-instance-01/mysql-instance-01.log
innodb_data_home_dir = /data/db/mysql/mysql-instance-01/
innodb_log_group_home_dir = /data/db/mysql/mysql-instance-01/

chown -R mysql:mysql /data/db/mysql

vim /usr/lib/systemd/system/mariadb01.service

[Unit]
Description=MariaDB database server instance-01
After=syslog.target
After=network.target

[Service]
Type=simple
User=mysql
Group=mysql

ExecStart=/usr/bin/mysqld_safe --defaults-file=/data/db/mysql/mysql-instance-01/my.cnf --user=mysql --basedir=/usr

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300

# Place temp files in a secure directory, not /tmp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

systemctl start mariadb01 systemctl enable mariadb01

Connect command:

mysql -P 44001 -h 127.0.0.1 -u root