Page 1 of 5
COMP3310/ENGN3539/etc Assignment 3 – Testing MQTT
Introduction:
• This assignment is worth 15% of the final mark
• It is due by Tuesday 23 May 23.55 AEST (Canberra time)
• Late submissions will not be accepted, except in special circumstances.
• Any extensions should be requested well before the due date, with appropriate evidence. Please use
the extension-request link on wattle rather than direct emails.
Assignment 3
MQTT is the most common open IoT protocol being deployed today. It provides a publisher/subscriber
model, using a broker or server. It allows for an almost-unbounded number of sources to publish
information, each at their own rate, and subscribers to receive information (and act on it). As such, it is
designed to provide high-performance communication mechanisms, with minimal delays and excellent
scaling performance. We’ll use it to monitor the performance of some imaginary system: say counting the
total kilograms of minerals rushing by on a conveyor belt, that you can control. This assignment will look at
the functionality and performance of the publishers, brokers, the network and the subscribers.
This is a coding, analysis and writing assignment. You may code in C/Java/Python or any programming
language that a tutor can assess (hope that’s enough for everyone), and yes, you may use MQTT and other
helper libraries. The assessment will not rely solely on running on your code, but more on the data
gathering and your analysis. However, we will review the code and may briefly test it against our own
broker running in the usual lab-type environments or similar. You will need to identify in your
README/code any libraries you are using.
Please note, you will be working in pairs or small groups for a key part of this assignment, to test each
other’s systems across the Internet. If you don’t have somebody to work with, ask your tutors NOW to help
identify somebody in your tutorial group, or post to the Wattle Forum under the pinned Find-a-Partner
topic. You do not need to be in the same location at any time.
Submitting
You will be submitting two things: your code and your analysis report. Note that there will be two
submission links on the Wattle course-site:
1. Your code must be submitted to wattle as a zip file, with README instructions on how to
compile/run the components as appropriate.
2. Your analysis report (pdf) must be submitted via TurnItIn on the wattle site, so ensure you quote
and cite sources properly.
Each question has an indication of how much (rough maximum) you should write.
Outcomes
We’re assessing your understanding of MQTT, your code’s functionality in subscribing/publishing to a
broker, dealing with high message rates from a number of sources, measuring message performance, and
your insight to interpret what you’re seeing. You will be exploring MQTT functionality, the quality-of-service
(QoS) levels, describing how they work, why you would use different levels, and how they perform in realworld deployments given various publishers.
Page 2 of 5
Resources
You will need to set up your own MQTT server/broker for you to connect to as per the specifications below.
You and your partner can share the same broker, if that suits. The Mosquitto broker is perhaps the best
choice to run on your own computer, it has binaries and/or packages for most operating systems, but if you
find a better one you can use that. It must be at least MQTT v3.1.1 compliant (MQTT v5 is not necessary).
A key consideration is that you will need to allow connection to it over the Internet for part of the
assignment. If running it at home where you have a home modem/router that does NAT you will need to
work out how to configure it to allow remote connections (think about the port(s) you need to forward,
and how to manage any changing IP addresses). If you have a firewall running on your homenetwork/computer, you may need to unblock the appropriate ports there too. There are a few other ways
to solve this, e.g. a VPN service on your modem or a local computer, or tools like Cloudflare-tunnels or
Twingate-proxy. If it proves too difficult you might consider setting up a cloud server with a public IP
address on e.g. Amazon, Azure, Alibaba, etc. and installing a broker there. Yet another option is to set up a
(free) fully-hosted broker through a provider such as EMQX.io. Public brokers may have performance vary
non-deterministically due to other people hammering at random times.
Your broker should be configured for some basic authentication and access control, so try to use:
• username student and password 33102023
• require the client-id to start with 3310-
(Public brokers may limit your ability to configure this, so note your approach in your code/README.)
You can test the broker works by subscribing to the $SYS topics, which describe the server, and help get you
familiar with the information presented there - you will be using them for your analysis later.
Assignment programming:
You need to write three small programs1
. The diagram at the end might help to clarify the following.
• A Publisher:
o You will have 1, 2 or 3 instances of the Publisher running2
. This will help stress the broker
and the network. Each publisher needs its own unique client-id.
o Each Publisher will send a simple message to the broker, namely an incrementing counter
(0, 1, 2, 3, …). It will send those messages at a specific MQTT QoS level (0, 1 or 2), and with
a specific delay between messages (0ms, 1ms, 2ms, 4ms, 8ms, 16ms, 32ms).
o Each will publish to the topic counter///, so e.g. counter/1/1/16
implies Publisher-instance-1 sending at qos=1 and delay=16. It must only publish the one
stream at any given time, based on input from the Controller. At 0ms delay and qos=0 it
should be able to publish very quickly, potentially several thousand messages per second.
• A Controller: This will listen to the broker on three topics:
o request/qos
o request/delay
o request/instancecount
1 The Controller and Publisher could be in the same program, but then it needs to be efficiently listening and sending
potentially at high data rates at the same time. 2 It could also be a single multi-threaded program, though that may have some additional complexities and
performance limitations.
Page 3 of 5
Whenever the Controller gets a new message on any of these topics it needs to modify the
Publisher’s behaviour accordingly. All 1-3 publisher instances will run the same qos/delay
parameters. One way is to stop any current publisher processes (e.g. send a KILL signal, or at
worst put a 1.5-minute time limit on it running, see below) and start a new one with the new
parameters, say via command line arguments. Another approach might be through eventhandling and multi-threading within a combined publisher/controller, but it is up to you. Please
explain in your README file or code comments what approach you are taking.
• An Analyser: This will listen to the counter(s) on the broker and take measurements (below) to
report statistics on the performance of the publisher/broker/network combination. Those
measurements should be taken across the range of qos, delay and instance-count values as above,
so that you can compare them. Your Analyser will publish to the request/qos, request/delay and
request/instancecount topics, asking the Controller to get some number of Publishers to deliver
what your Analyser needs at that time. For the QoS, subscribe at the same QoS as requested from
the Publisher (i.e. if Publisher->broker qos=2 then Analyser->broker should have qos=2 as well). The
Analyser does not need to take all (7 delay * 3 qos * 3 instances) measurements sequentially in one
run, but the closer in time the measurements occur the better, as network (and broker) conditions
do change all the time.
You can test all of these at home on a local broker. It is only in the final measurements that your broker will
allow another student’s Analyser to access your accessible broker/Controller, and vice-versa.
Analysis and Reporting
In your report, please address the following:
1. Subscribe to some broker, you can use any mqtt client you like for this, to retrieve some $SYS/#
value. Wireshark the handshake for one example of each of the differing QoS-levels (0,1,2), include
screenshots in your report that show the wireshark capture (filter for mqtt), and briefly explain how
each QoS transfer works, and what it implies for message duplication and message order. Discuss
briefly in your report in which circumstances would you choose each QoS level, and offer an
example of a sensor that would use each of them. [around 1 page]
2. Run your Analyser against your partner’s broker, have it tell the broker (and so their Controller)
what you want their Publisher to send, and record information for 1 minute for each set of the
qos/delay/instance-count parameters.
(Tips: (i) only print to screen for debugging, otherwise it will slow your code down a lot. (ii) Check
with the broker owner what delay there might be between your request and their publisher
responding. (iii) Use the counter values to tell you what messages are arriving, or are not arriving as
expected, and what values they start/stop at when you subscribe to calculate the rates below. (iv)
try publishing at a very low rate (i.e. high-delay) to start with, to check your code)
a. Collect statistics, for each instance-count, delay and QoS level, and measure over the 1-
minute period:
i. The overall average rate of messages you actually receive from all publishers across
the period [messages/second].
ii. The rate of message loss you see [percentage].
(how many messages did you see, versus how many should you have seen)
iii. The rate of any out-of-order messages you see [percentage]
(i.e. how often do you get a smaller number after a larger number)
Page 4 of 5
iv. The mean and median inter-message-gap you see, compared to the requested
delay [milliseconds].
Only measure for actually consecutive counter-value messages, ignore the
gap if you miss any messages in between.
b. While measuring the above also
i. Subscribe to and record the $SYS topics, and identify what, if anything, on the
broker do any loss/misordered-rates correlate with. (Look at measurements under
e.g. ‘load’, as well as the ‘heap’ and ‘active clients’ and ‘messages’, anything that
seems relevant. See https://mosquitto.org/man/mosquitto-8.html though it may
not quite match your broker)
ii. Record a traceroute between the Analyser and the broker, and between the
Publisher and the broker (you’ll need the broker owner to provide the second one)
3. In your report: [around 2-3 pages of text, plus any charts]
a. Summarise your measurements. Explain what you expected to see, especially in relation to
the different QoS levels, and whether your expectations were matched.
b. Describe what correlations of measured rates with $SYS topics you expect to see and why,
and whether you do, or do not.
c. Report the traceroutes captured above.
4. Consider the broader end-to-end (internet-wide maybe) network environment, in a situation with
millions of sensors publishing regularly/irregularly to thousands of subscribers. Explain in your
report [around 1-2 pages]
a. What (cpu/memory/network) performance challenges there might be when using MQTT in
general, from the source publishing its messages, all the way through the network and
broker to your subscribing client application. If you lose messages, where might they be
lost?
b. How the different QoS levels may help, or not, in dealing with the challenges.
Amendments and Assessment
Any amendments to the assignment specification will be published in the wattle forum.
Your assignment will be assessed on
• Your code clarity, and documentation/comments (15%)
• your code subscribing and properly publishing/listening to the broker (15%), and
• your analysis report addressing the questions above (70%)
Page 5 of 5
Publisher
3
Publisher
2
Broker
Publisher
1 Controller
Analyser
MQTT:
QoS, Delay, Count
MQTT:
QoS, Delay, Count
MQTT:
1-3 streams of numbers
Step 1
Step 2
Step 3
Step 4
Step 5