MQTT Complete Guide for ESP32, ESP8266 and Raspberry Pi
MQTT is a lightweight publish/subscribe messaging protocol used to exchange data between connected devices and applications. It is useful when sensor boards, automation servers and dashboards need to communicate without maintaining a direct connection to every other device.
This SolderHub guide moves from fundamentals to practical implementations: understand brokers and topics, test a connection without hardware, connect an ESP32 or ESP8266, design state and availability messages, then harden the system with access control, encrypted transport and reconnect logic.
The central idea: an MQTT publisher sends a message to a topic on a broker. Clients subscribed to a matching topic can receive it. The sender does not need to know each subscriber's address, which makes the pattern useful for a house full of sensors, dashboards and automations.
1. The broker, topic, publisher and subscriber
The broker is the server that accepts connections and routes messages. Eclipse Mosquitto is a common self-hosted broker; managed MQTT services are alternatives if you do not want to administer a server. A publisher sends data, and a subscriber requests messages from one or more topics. A device can be both publisher and subscriber.
A topic is a hierarchical string such as home/livingroom/temperature. A slash separates levels. Choose names that express device identity and meaning instead of encoding an entire sentence into the topic.
Examples:
- solderhub/lab/esp32-01/temperature — telemetry
- solderhub/lab/esp32-01/command/led — command
- solderhub/lab/esp32-01/state/led — reported output state
- solderhub/lab/esp32-01/status — availability
Keep command and state topics separate. A message that requests ON is not evidence that the output actually turned on.
2. QoS, retained messages and Last Will
MQTT defines three Quality of Service levels:
| QoS | Meaning | Typical consideration |
|---|---|---|
| 0 | At most once | Low overhead; a message may be lost |
| 1 | At least once | Delivery can be repeated, so consumers should tolerate duplicates |
| 2 | Exactly once at the MQTT protocol exchange level | More overhead; not a guarantee that a physical action happens exactly once |
QoS is not a substitute for application-level acknowledgement or durable storage.
A retained message stores the latest retained publication for a topic so a new subscriber can receive the current value on subscription. It is useful for current readings, configuration and reported state. Be deliberate with command topics: retaining TOGGLE or OPEN may repeat an action when a device reconnects.
The Last Will and Testament is registered by a client when it connects. If that client disconnects unexpectedly, the broker can publish the configured will message. It is commonly used for availability, such as an offline status. A will is not a precise heartbeat timer; choose keep-alive and status logic carefully.
3. Run Mosquitto on Raspberry Pi 5
For a home lab, Raspberry Pi 5 can host the broker and Node-RED. On Raspberry Pi OS, install the packages and client tools:
- sudo apt update
- sudo apt install -y mosquitto mosquitto-clients
- sudo systemctl enable --now mosquitto
- sudo systemctl is-active mosquitto
Do not assume the broker is remotely reachable just because the service is active. Mosquitto 2.0 changed listener behavior: without explicit listener configuration, local-only loopback access is used for testing. Another computer or ESP32 cannot connect through the Pi's LAN address until a listener and suitable authentication are configured.
Follow Install Mosquitto on Raspberry Pi 5. Consult the official Mosquitto 2.0 migration notes and configuration manual when adapting configuration to your release.
4. Prove the broker works before adding hardware
Install the Mosquitto clients and open two terminals. In the first, subscribe to a test topic:
mosquitto_sub -h 127.0.0.1 -t 'solderhub/test' -v
In the second, publish a message:
mosquitto_pub -h 127.0.0.1 -t 'solderhub/test' -m 'hello from SolderHub'
The subscriber should display the topic and message. This local test checks the broker and client tools without involving Wi-Fi credentials, firmware or sensors. If authentication is configured, use a test account and take care not to expose real secrets in shell history or screenshots.
For a desktop client, MQTT Explorer can make topic trees and payloads easier to inspect. Use a unique test topic and disconnect tools that would otherwise keep a retained test value alive.
5. Connect an ESP32 or ESP8266
A microcontroller typically connects to Wi-Fi, opens an MQTT session to the broker, publishes sensor telemetry and subscribes to command topics. On Arduino-compatible ESP32/ESP8266 projects, PubSubClient is commonly used for basic MQTT workflows. Check the currently maintained library documentation and supported features before designing around a specific capability.
The published ESP32 MQTT Tutorial: Publish and Subscribe with Arduino IDE covers the core example. The NodeMCU ESP8266 MQTT LED Control applies the same model to the ESP8266 family.
Give each client a unique ID. On each successful connection, subscribe again to required topics. Keep callbacks short: parse and validate the command, update requested state and return. Avoid long delays while processing messages, because the MQTT client must be serviced regularly.
6. Publish multi-sensor data as JSON
When a device reports several measurements, a compact JSON payload can keep related values together:
{"temperature_c": 24.8, "pressure_hpa": 1007.2, "humidity_pct": 51.0}
Include units in property names or document them unambiguously. Do not use non-finite values or undocumented abbreviations. Use a JSON parser such as ArduinoJson when needed, and check parsing succeeds before reading properties. Consider whether one message per sensor or one structured object fits the subscriber architecture better.
The SolderHub multi-sensor guide will show DHT11 and BMP280 readings published from an ESP32. Keep it separate from the existing DHT22-to-Home-Assistant article: one is a hardware-focused MQTT/JSON pattern; the other is a Home Assistant integration workflow.
7. Design command, state and availability properly
Use different topics for what the controller wants and what the device reports. A controller publishes ON to a command topic. Firmware validates the payload, applies the output, then publishes actual state. If command processing can fail, publish an error or acknowledgement separately.
For availability, configure a retained online publication after a successful connection and a Last Will publication for unexpected disconnects. Test how a new subscriber sees status after the device has already connected.
For relays, consider boot policy before retaining any command. A retained ON command can energize an output during startup when a user expected a safe/off state. Retain desired state only when the firmware's safe-state policy and load design explicitly support it. Never rely on an MQTT packet alone as a safety interlock for hazardous machinery or mains equipment.
8. Troubleshoot connection errors systematically
Start at the network layer and work outward: confirm the broker service is active, verify listening address and port, check host routing/firewall rules, inspect credentials and ACLs, then review the library's return code and broker logs. A localhost address means the local machine of the connecting client, not the machine you wish it would reach.
The guide MQTT Troubleshooting: Connection Refused, rc=-2 and rc=5 separates TCP failures from MQTT authentication and protocol failures. Error codes vary by library; the rc examples in that article specifically refer to PubSubClient state values, not all MQTT clients.
9. Reconnect without freezing the microcontroller
Wi-Fi and broker connections can disappear during router changes, access-point reboots or broker maintenance. Firmware should detect disconnection, retry after a controlled interval, re-subscribe after success and continue essential local work.
Use millis()-based scheduling rather than a long blocking delay in the reconnect loop. Back off between failed attempts to avoid hammering the broker. Reset the interval after a successful connection. Avoid assigning the same client ID to two devices, because brokers commonly disconnect the previous client when another connects with the same ID.
See Non-blocking MQTT Auto-Reconnect on ESP32 for a reusable connection state pattern. Local sensing and safety logic should not stop simply because cloud messaging is unavailable.
10. Secure a broker before remote or multi-user use
Do not expose an unauthenticated broker directly to the internet. Configure explicit listeners, individual credentials, topic ACLs and TLS for networks you do not trust. Use a VPN or secure network gateway when appropriate. Credentials sent over unencrypted MQTT can be observed by someone able to inspect the path.
Grant each device only the topics it needs. A sensor that publishes temperature should not automatically be allowed to subscribe to every relay command. Keep broker configuration backups and test password rotation. Check current Mosquitto authentication guidance instead of copying a configuration written for an older major release: Authentication methods.
11. The SolderHub MQTT learning path
Beginner
- MQTT Explained: Brokers, Topics, QoS and Retained Messages
- Install Mosquitto on Raspberry Pi 5
- Test MQTT without Hardware: mosquitto_pub, mosquitto_sub and MQTT Explorer
- ESP32 MQTT Tutorial: Publish and Subscribe with Arduino IDE
- NodeMCU ESP8266 MQTT LED Control
Intermediate
- MQTT Topic Naming Best Practices
- MQTT Troubleshooting: Connection Refused, rc=-2 and rc=5
- Retained Messages and Last Will with ESP32
- Non-blocking MQTT Auto-Reconnect on ESP32
- Publish DHT11 and BMP280 as JSON over MQTT
- Relay Control over MQTT with State Feedback
- ESP32 and ESP8266 Communicating through MQTT
Advanced
- Secure Mosquitto: Users, ACLs and TLS
- MQTT over WebSockets with MQTT.js
- MQTT with Python on Raspberry Pi 5: Log to SQLite or CSV
Connected guides
Frequently asked questions
Does MQTT require the internet? No. Clients can exchange messages through a broker on the same local network without internet access if routing and firewall rules allow it.
Is MQTT the same as Mosquitto? No. MQTT is the protocol; Mosquitto is one broker implementation.
Does QoS 2 guarantee a physical relay switches exactly once? No. QoS 2 manages protocol delivery semantics, not every possible duplicate action after a restart or firmware fault. Design application-level state and acknowledgement.
Should I retain every message? No. Retain values that represent the latest state when a new subscriber needs it. Retained messages are not a replacement for an event log.
Can ESP32 and ESP8266 communicate through MQTT? Both connect to the same broker and exchange messages through shared topics; they do not need a direct socket between boards.
Official references
- Eclipse Mosquitto documentation
- Mosquitto 2.0 migration notes
- Mosquitto configuration manual
- Mosquitto authentication methods
- Node-RED getting started
- Arduino ESP32 Wi-Fi API
A reliable MQTT system has a deliberate topic model, clear ownership of command and reported state, an offline policy, meaningful diagnostics, and security appropriate to its network.