In an email to the Personal Clouds list, Johannes wrote:
Let’s say I’d like to use MQTT to make the doorbell in my house communicate with the living room lights. I think what would have to happen is this:
- the doorbell and the living room lights would have to be an MQTT client each
- somewhere in my house I’d run an MQTT server
- doorbell and living room lights need to find that server, and register with it, one as a “producer” of information, one as a “consumer”
- some piece of code that runs the logic (“If somebody rings the doorbell like …—…, flash the living room lights in red”) must run somewhere in my house
- that piece of code would subscribe to appropriate topics as producer and consumer on that MQTT server
Am I getting this about right?
My response outgrew an email so I’m posting it here.
At a high level, yes that’s about right. There are some nuances worth pondering, though.
First, there is always a broker but there need not necessarily be a central rules engine. It is possible for the devices to interact directly, although this is more for the simple cases. For example, the light might subscribe to an actuation topic and the doorbell might publish directly on that topic. That said, my ideal implementation includes a rules engine and a standard set of actuation topics. This is similar to Windows events where you don’t need to wire the default sound up to everything that might produce it. You wire the default sound clip to the default sound event and things publish to that event. Also, as in Windows, it would be possible to plug custom events into the event framework. So there would most likely be standard event topics defined for, say, fire alarm, but you might add a custom one for “motion at bird feeder.”
So I see a rules engine in there for most useful implementations. Maybe it’s in the house (preferably) but I’m sure vendors will supply a default version so new device owners don’t need to set anything up. I also imagine people selling secondhand routers with Tomato firmware burned in and running a broker with zero configuration setup. The device publishes to the same address as the TCP Gateway.
Another thing to consider is, as a device programmer, what is involved in reporting state using REST? Right now it’s common for devices to talk directly to manufacturers over HTTPS. Now let’s say that manufacturer wants to allow the device to talk to two things. Two REST calls for each event notification? Three things = three REST calls? Four things = four REST calls? It doesn’t scale. However, with a true pub/sub like MQTT the device publishes each event once and it is capable of being subscribed by an unlimited number of other things.
Now imagine you with to subscribe to state of something using HTTP. Poll. Poll again. Poll again. Poll and got a value! Poll again. Poll again.
Now subscribe to three events. Poll, poll, poll. Poll, poll, poll again. Poll, poll, poll again. Poll, poll, poll again.
With MQTT you maintain a single connection, subscribe to as many topics as you want, and the broker pushes events to you as they are published. For most hardware, you will saturate your bandwidth before exhausting the broker’s capacity.
My use case for this is that something like an LED bulb should publish first to the local MQTT hub. The homeowner can then have their rules engine subscribe to that publication. Because it is a topic hierarchy, it is possible to subscribe or publish to all lights at once, arbitrary groups of lights, or individual lights. So out of the box you get addressability with tunable levels of granularity as simply as changing a string in the publish or subscribe call. No complicated logic required to make groups of things. Also no configuration changes to add or decommission a device. Since the rules engine subscribes to “lights” it doesn’t need to know about “light_127” specifically, it just gets any publications that light_127 makes after you screw the bulb in somewhere. Take that bulb out and no need to tell the rules engine it’s gone.
Now consider the case that you decide to publish the device events to the vendor because they offer some compelling functionality. You simply make an administrative subscription that wires up the topic the device publishes to with the URL provided by the vendor. Done. In a reasonably functional dashboard app, you just click a radio button.
Let us further suppose that some other vendor provides functionality based on the same events. Rather than relying on the device vendor for the integration, they can do it themselves and advertise the service. You simply wire the URL they give you to the topic the device publishes on. For example, LED bulbs, fans and other devices that can be dimmed, slowed or stopped might participate in discretionary load control. The utility tells you which devices it understands and you wire those up to the URL they provide. Remember, the device publishes once and only once and requires no firmware updates to do any of this, no matter how many integrations can use its data.
Want to publish to your Kynetx cloud? Wire a topic up to the URL they provide.
Want to publish to IFTTT.com? Wire a topic up to the URL they provide.
You can do any or all of these things without changing the device and without requiring the vendor too invest in the integration. It is this ability to scale that gives us the connection density which fulfills the promise of Internet of Things.
Imagine you were trying to simulate a brain and had to wire up all 10,000 connections from each node and doing it with REST polling. Now imagine how you want to change the configuration dynamically.
Now imagine the same thing with pub/sub where each node makes one publication and subscribes to the 10,ooo other nodes it talks to and it has the ability to aggregate those subscriptions by subscribing to high level topics like “lights” instead of to light_01, light_02, etc. Now imagine changing that dynamically. You add a new light and the subscription…well, it doesn’t even need to change because it already subscribes to “lights.” Take a light away and also no change.
This is why we cannot scale to trillions of devices using point-to-point protocols and polling but pub/sub and push does the trick. I don’t know that MQTT will be the thing that wins, but I’m certain whatever wins will be pub/sub for this reason. We’ll need to add topics to DNS or make a parallel global namespace like DNS, and we’ll need to route publications over the Internet. Once we do that, devices and REST will be history.
I believe that once we have some basic problems addresses – like on-boarding devices and a decent dashboard app for the home user, the pub/sub integration will become much more obviously compelling. The problem is that if only the initial implementation is considered – much as the problem statement posed by Johannes posited a single point-to-point connection – pub/sub seems more complicated and it is new whereas REST solves the problem and is well known. But once you begin to consider many-to-many integrations, pub/sub and push becomes the clear winner and the on-ramp for the connection density that delivers on the promise of Internet of Things.
Does it have a chance of winning? Charles Brett at GigaOM wrote:
It would be nice to believe the right solution will win, especially given the cost of today’s mobile data and the wastage consumed by synchronous mobile communications. But don’t hold your breath. History has shown that synchronous is the default, even when asynchronous should be chosen. The danger is that yet again we will not learn from history.
Time will tell.
Hi T. Rob,
While I totally agree that Pub/Sub is a better solution than REST for IoT style devices I’m not sure about the concept of a global namespace for devices. In many cases (in face, almost all) you do not want your actuators and sensors exposed to the public internet.
The concept I’m slowly building up at home is more of an ‘Intranet of Things’ – sensors and actuators tied to an internal MQTT server with a rules engine that has access to data sources on the internet as well. Rather than making all the internal events available to the internet at large (even with security settings) it must be explicitly done through the rules engine.
In the example at the start of the post the rules engine could send out a PushBullet notification as well as flick the lights. It could also flick the lights in response to a text message matching certain criteria as well. This gives you the same flexibility in control without being dependent on internet access to work.
Of course there is nothing stopping you from making part of your topic tree public or from connecting the rules engine to an external MQTT server as well and including messages from that source in the rules evaluation process. For things like crowd sourced data collection this would make sense but is there really any need to publicly publish when your lights are on and off?
About the only need to have access to the data over the internet is to access it from other locations via your mobile phone or, as in the example you gave, to allow someone else to monitor the inputs on your behalf. In many cases this can be down with existing services such as PushBullet, IFTTT, text messages, etc. In this situation it is more of a negotiation service – a central point you can go to that would set up a connection between your internal MQTT server and the other persons internal MQTT server and then gets out of the way.
Sorry if that sounds a little confusing – one of the big problems I’m having with this project is actually describing it in an understandable manner 🙂
Regards,
Shane
Nor am I. My intention there was that there should be a global namespace for topics, not devices. For example, I own t-rob.net which implies I also have control of all the subdomains below that. I can define iot.t-rob.net, for example. My proposal is that we need a global pub/sub topic namespace such that there’s a topic equivalent to my domain and I could choose to publish updates at /t-rob.net/iot/external/* or /t-rob.net/whereami/* or whatever. Currently if you go to MQTT.org and use their broker, it’s up to you to make sure you don’t step on anyone else’s topic and they don’t step on yours. That can’t scale and there’s no resolver service.
Similarly, some device vendor could preconfigure their devices to publish direct to a topic on which they subscribe. As a device owner I get to decide which topics get out the door. If I want the device to publish to my security monitoring company or IFTTT, I map the device to their specified topic. But, again, this assumes that IFTTT owns part of the topic namespace under IFTTT.com (or more likely .net), publish their certificate using DNSSEC, Keybase, cert pinning or whatever, that somethign resolves that topic to their server, and that the broker or dashboard I use understands all these bits and presents it to me as a pretty picture with icons that I drag and drop.
I’m right there with you as to “intranet of things”. One reason I left IBM is that internally certain people told me “yeah, we also think that people will eventually have the option to sequester their device data but we can’t sell that to vendors today.” Others simply said no outright. That’s a dead-end for vendors in my opinion. If nobody in the commercial sector builds software that respects users and caters to our needs, we are to the point we can build it ourselves. If it comes to that, vendors whose model does not anticipate this control shift will find themselves suffering a data drought. We end users are having a block party and vendors are invited, if they have vision and foresight to understand why it’s important for them to participate.
What I imagine is all my devices on their own dedicated network, behind a firewall with, since I’ll be using “Expert Mode”, default zero external access. The “Easy Mode” is the opposite. It defaults to letting the devices talk to the vendor because for many people that’s all they want or need. None of these things publish on my global t-rob.net namespace unless I explicitly tell them to. They either publish direct to the vendor or inside the firewall only. Since nobody can subscribe to my broker, nobody can subscribe to my publications unless I explicitly provision that they are publicly available from my broker. So you could not, for example, subscribe to /Google.com/Nest and get my device publications because neither I nor Google permit it. However I might map them to /t-rob.net/iot/nest01/* and expose that to the public if, for example, I wanted to do a live demo. But the resolver needs to know how to translate /t-rob.net to an IP address to do that, hence the global namespace.
Does that make sense? Also, have you seen my Slideshare on this? Happy to review any documents or slides, time permitting.
Hi T.Rob. Could you elaborate on how and why Pub/Sub and REST exclude each other? (And what’s wrong about approaches like http://www.pubnub.com/tutorial/http-rest-push-api). Regards, Thomas
Hi Thomas, I didn’t mean to imply that Pub/Sub and REST are mutually exclusive in their usage. Certainly something could use both. However, one of the six fundamental constraints of REST is that each client request contains all information necessary to process the request and no session state is maintained at the server. The pub/sub style is the opposite. A single subscribe request may result in many messages being delivered and this is because the server maintains client state (e.g. the subscription details). So the two are mutually exclusive as architectural styles by definition. For that matter, “REST Push” is an oxymoron because it also breaks the same constraint.
That PubNub provides a pub/sub style over a REST API doesn’t imply that Pub/Sub is RESTful. Rather,it is the modern-day Frankenstein’s monster of grafting REST and Pub/Sub together. Since Pub/Sub and REST are by definition mutually exclusive, it is probably more appropriate to say that PubNub provides an HTTP API and not a REST API but it’s not my project and not for me to impose as the REST police. Or the pub/sub police.
That said, all the marketing copy on the Pub/Nub site seems compatible with my thesis that pub/sub is the architectural style needed for Internet of Things. They talk about scalability, dynamic connect/disconnect from the network, efficiency, and the general benefits of pub/sub as a communication style. They then provide a couple of the things I said were missing – an externally routable protocol and a globally managed namespace. In the PubNub case, they provide it in 9 data centers and I believe that you are not free to set up your own PubNub server in the home or on the net so it isn’t quite the vision I have but it’s pretty damned good just the same.
What I’m looking for is a lot more like NNTP but in near-real time. Open protocol, open source and commercial client and broker implementations, and every ISP hosts a broker. You register your own topic space and can publish or subscribe within that space. You can elect to publish on those topics externally if what you have to publish is of interest. For example, I might subscribe to presence detection and alarms at my elderly relative’s house and set up their system to publish those things externally. If privacy is a concern, their system encrypts them prior to publication. Unlike NNTP, the messages only route only to subscribers and not to every broker at every ISP. When I make my subscription, the topic DNS resolves the source and registers the subscription from there. If there are topics that are extremely popular like stock tickers or emergency alerts, ISPs could register subscriptions and your personal subscription request would resolve locally.
Does that help? Hopefully nothing controversial in all of that. 😉