MQTT Common Questions

Common Questions

If you are familiar with the web and email then you will probably find, as I did, that MQTT is very different. These are some of the questions I had, and saw on other sites and forums that may clear things up a little.

Q, Can multiple client publish to the same topic?

A- Yes

Q- Is is possible to know the identity of the client that published a message?

A- No not unless the client includes that information in the topic or payload.

Q- What happens to messages that get published to topics that no one subscribes to?

A- They are discarded by the broker.

Q-How can I find out what topics have been published?

A- You can’t do this easily as the broker doesn’t seem to keep a list of published topics as they aren’t permanent.

Q- Can I subscribe to a topic that no one is publishing to?

A- Yes

Q- Are messages stored on the broker?

A- Yes but only temporarily. Once they have been sent to all subscribers they are then discarded. But see next question.

Q- What are retained messages?

A- When you publish a message you can have the broker store the last published message. This message will be the first message that new subscribers see when they subscribe to that topic. MQTT only retains 1 message.

MQTT (Message Queuing Telemetry Transport) is an extremely lightweight connectivity protocol designed to cater to the Internet of Things. It is a machine-to-machine (M2M) protocol which works on publish/subscribe principle of transport of message. The small code footprint for its implementation and open-endedness makes it an ideal candidate for its application in IoT.

There are two types of devices participating in the protocol – brokers and clients. Brokers are the devices which host and relays data. Clients are the devices which access and modify data by publishing topics and receive other devices’ topics by subscribing. Let us take an example of the bulletin board. On the notice board, you might go through the topics which you have interest in. If you have something interesting to put up on the notice board, you are free to do that. To get the attention of the people who are interested in the topic, it is usually done in big bold letters. This is exactly how MQTT works.
Quality of Service (QoS)

The MQTT also gives a provision for three Quality of Service (QoS) – 0, 1 and 2.

MQTT QoS

    For QoS = 0, it ensures at most one delivery of the topic. It is equivalent to a ‘fire and forget’ concept.
    For QoS = 1, it ensures at least one delivery of the topic. It receives an acknowledgment. It makes sure that the broker has received the message.
    For QoS = 2, it ensures exactly one delivery of the topic. It gives us an assurance to the delivery of the message.

Load Testing HAProxy (Part 1)

This is the first post in a 3 part series on load testing HAProxy, which is a reliable, high performant TCP/HTTP load balancer.

Load Testing? HAProxy? If all this seems greek to you, don’t worry. I will provide inline links to read up on everything I’m talking about in this blog post.

Load Balancing with HAProxy for MQTT

Need to cluster

IBM has a Redbook for its own IOT service environment. The book also described the topologies of enterprise MQTT service architectures. So, to cluster some mosquitto brokers, we need some other tools.

Mosquitto is a single thread process. To scale out, we need tools to better monitor and manage them.

Cue in load balancing

I followed the steps in this nice tutorial, others you googled might be just as good, as HAProxy is very easy to install, configure and run.

Use HAProxy to load balance 300k concurrent tcp socket connections: Port Exhaustion, Keep-alive and others

I’m trying to build up a push system recently. To increase the scalability of the system, the best practice is to make each connection as stateless as possible. Therefore when bottleneck appears, the capacity of the whole system can be easily expanded by adding more machines. Speaking of load balancing and reverse proxying, Nginx is probably the most famous and acknowledged one. However, TCP proxying is a rather recent thing. Nginx introduced TCP load balancing and reverse proxying from v1.9, which is released in late May this year with a lot of missing features. On the other hand, HAProxy, as the pioneer of TCP loading balacing, is rather mature and stable. I chose to use HAProxy to build up the system and eventually I reached a result of 300k concurrent tcp socket connections. I could have achieved a higher number if it were not for my rather outdated client PC.

Step 1. Tuning the Linux system

Unexpected DDOS: Blocking China with ipset and iptables

Set up iptables

If you already have iptables set up and in use, skip this section and go straight to the ipset section.

Create a file where we can declare some rules to use:

sudo nano /etc/iptables.firewall.rules 

Inside there you'll want to paste the following:

 

CLEAN URL REWRITES USING NGINX

This article will cover how to easily implement Clean URLs (also known as Semantic URLs, RESTful URLs, User-Friendly URLs and Search Engine-Friendly URLs) using NGINX Web Server; currently the second most popular web server platform worldwide.

Not using NGINX web server? No problem, Apache users can check out my article Clean URL Rewrites Using Apache.

A Clean URL is a URL that does not contain query strings or parameters. This makes the URL easier to read and more understandable to users. 

Clean URLs are a high ranking factor for many search engines, but there are many other reasons why having Clean URLs can be important to your website. Check out our article Are your URLs Letting Your Website Down for more information.

Nginx rewrite rules for MyBB SEO friendly URLs

I've recently switched a forum I run from Wordpress and bbPress to MyBB. Overall I've very happy with it but there are a few bits to watch out for with MyBB.

If you are running Nginx instead of Apache as your webserver then one of those things to watch out for is the need to manually update the rewrite rules needed for SEO friendly URLs.

MyBB SEO friendly URLs

Below is the full set of Nginx rewrite rules for rewriting MyBB's SEO friendly URLs. To activate this option in MyBB you can do so from the Admin CP:

Admin CP > Configuration > Settings > Server and Optimization Options > Enable search engine friendly URLs? = Enabled

You can also enable the "Enable search engine friendly URLs in Archive?" option - it's the option under the "Enable search engine friendly URLs?".

If you decide to enable just the "Enable search engine friendly URLs?" option you can use the following nginx rewrite rules:

rewrite ^/forums/forum-([0-9]+)\.html$ /forums/forumdisplay.php?fid=$1;
rewrite ^/forums/forum-([0-9]+)-page-([0-9]+)\.html$ /forums/forumdisplay.php?fid=$1&page=$2;
rewrite ^/forums/thread-([0-9]+)\.html$ /forums/showthread.php?tid=$1;
rewrite ^/forums/thread-([0-9]+)-page-([0-9]+)\.html$ /forums/showthread.php?tid=$1&page=$2;
rewrite ^/forums/thread-([0-9]+)-lastpost\.html$ /forums/showthread.php?tid=$1&action=lastpost;
rewrite ^/forums/thread-([0-9]+)-nextnewest\.html$ /forums/showthread.php?tid=$1&action=nextnewest;
rewrite ^/forums/thread-([0-9]+)-nextoldest\.html$ /forums/showthread.php?tid=$1&action=nextoldest;
rewrite ^/forums/thread-([0-9]+)-newpost\.html$ /forums/showthread.php?tid=$1&action=newpost;
rewrite ^/forums/thread-([0-9]+)-post-([0-9]+)\.html$ /forums/showthread.php?tid=$1&pid=$2;
rewrite ^/forums/post-([0-9]+)\.html$ /forums/showthread.php?pid=$1;
rewrite ^/forums/announcement-([0-9]+)\.html$ /forums/announcements.php?aid=$1;
rewrite ^/forums/user-([0-9]+)\.html$ /forums/member.php?action=profile&uid=$1;

Activating SEO friendly URLs in Archive

If you activate the "Enable search engine friendly URLs in Archive?" option you'll need the following extra rewrite rules:

rewrite ^/forums/archive/index.php/forum-([0-9]+)\.html$ /forums/archive/index.php?forum-$1.html;
rewrite ^/forums/archive/index.php/thread-([0-9]+)\.html$ /forums/archive/index.php?thread-$1.html;
rewrite ^/forums/archive/index.php/forum-([0-9]+)-([0-9]+)\.html$ /forums/archive/index.php?forum-$1-$2.html;
rewrite ^/forums/archive/index.php/thread-([0-9]+)-([0-9]+)\.html$ /forums/archive/index.php?thread-$1-$2.html;

At first glance the second set of rules might not seem essential, but it's those rules that enable pagination. Without them you'll just get 404 errors for page 2 or above on any of the archive forum or thread lists.

Mysql Database Performance tuning

Mysql provides a configuration file located in /etc/my.cnf. From here you can set all of the memory, table, and connection limits as well as a host of other options.
You can get the default buffer sizes used by the mysqld server with this command:

shell> mysqld –help

This command produces a list of all mysqld options and configurable variables. The output includes the default values and looks something like this:

mysql multiple instances on CentOS 7

mkdir -p /data/db/mysql

./mysql_install_db --datadir=/data/db/mysql/mysql-instance-01 ./mysql_install_db --datadir=/data/db/mysql/mysql-instance-02 ./mysql_install_db --datadir=/data/db/mysql/mysql-instance-03 ./mysql_install_db --datadir=/data/db/mysql/mysql-instance-04

cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-01/my.cnf cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-02/my.cnf cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-03/my.cnf cp /usr/share/mysql/my-medium.cnf /data/db/mysql/mysql-instance-04/my.cnf

vi /data/db/mysql/mysql-instance-01/my.cnf

[mysqld]
port = 44001
socket = /var/lib/mysql/mysql-instance-01.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
innodb_file_per_table

datadir = /data/db/mysql/mysql-instance-01/
log-error = /data/db/mysql/mysql-instance-01/mysql-instance-01.log
innodb_data_home_dir = /data/db/mysql/mysql-instance-01/
innodb_log_group_home_dir = /data/db/mysql/mysql-instance-01/

chown -R mysql:mysql /data/db/mysql

vim /usr/lib/systemd/system/mariadb01.service

[Unit]
Description=MariaDB database server instance-01
After=syslog.target
After=network.target

[Service]
Type=simple
User=mysql
Group=mysql

ExecStart=/usr/bin/mysqld_safe --defaults-file=/data/db/mysql/mysql-instance-01/my.cnf --user=mysql --basedir=/usr

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300

# Place temp files in a secure directory, not /tmp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

systemctl start mariadb01 systemctl enable mariadb01

Connect command:

mysql -P 44001 -h 127.0.0.1 -u root