As vendors, coders and users inch toward the smart-devices-everywhere future colloquially known as the Internet of Things, they're starting to address a whole bunch of nitty-gritty issues. For starters, there's the question of just how the various devices connected to the Internet of Things will talk to each other.
Beyond that, though, lies an even larger challenge. Which is, once we've got devices on the Internet of Things speaking a common language, what in the world are they actually going to say to one another?
Talk Talk Talk, But Nothing To Say
See also: What's Holding Up The Internet Of Things
As I outlined in an earlier piece on the subject, the Internet of Things is currently limited by the fact that manufacturers have embraced a profusion of incompatible protocols for communicating with their devices. These protocols are effectively different languages. As a result, devices connected to the Internet often can't speak to each other even if they wanted to.
It's a good assumption, though, that such barriers to communication will eventually fall as companies converge on standard protocols. Others will work to create centralized "commons" that act as interpreters, making it possible to coordinate a multitude of devices that otherwise couldn't cooperate.
Let's assume that problem is dealt with. Now, whatever will these devices—smartphones and TVs and refrigerators and cars and watches and smart clothing and what-have-you—talk about? Sounds like a trivial question, but a simple example demonstrates how tricky it will be to solve.
Come Into My Parlor
Let's go back to our earlier thought experiment: A "smart" living room with three Internet-connected devices—a Nest thermostat, a Spark-enabled light and Makita automated drapes. It's possible to control each device remotely. And for the sake of this example, let's assume each of these gadgets can report various sensor readings directly to one other.
The thermostat could report a few things: the room temperature, its target temperature and the time of day. The lamp can tell its fellow devices whether it's on or off and how bright the room is. The drapes are relatively dumb by comparison; they'll report whether they're open, closed, or somewhere in-between.
So the thermostat is busily reporting all of its data to the lamp and the drapes. First question: What the heck do they care? I keep envisioning a scenario similar to one from The Hitchhiker's Guide to the Galaxy, where doors on the Heart of Gold keep opening cheerily to the murderous dismay of one Marvin the Paranoid Android.
Seriously, how would draperies have any way of knowing that the room temperature is important to them? (Much less caring about it.) Likewise for the lamp. Were it sentient, I could imagine the lamp overloading in a fit of frustration, catching the drapes on fire and burning the whole house down.
Of course if they were sentient, the devices would probably intuitively understand that things like temperature and time of day are relevant to their basic functions of temperature regulation and providing light. The question is, how do we get that meaning across?
A Living Room Fabric Of Knowledge
Stefan Weitz and Matt Wallaert are, without a doubt, geeks. When I spoke to them, they'd just come out of a big celebration for the engineers who added the Klingon language to Bing in advance of the latest Star Trek premiere. Wallaert, a behavioral scientist with Bing, had recently shaved the Klingon phrase Qapla'—"success"—on his head. (That's him to the right.)
It's not just their love of Trek that puts the pair squarely in the geek category. They're also thinking deeply about the future of Bing, and search engines in general. And while they wouldn't mind seeing such a future kick Google's butt, they have a lot of other things on their minds.
To these guys, the increasing number of devices connected to the Internet represents a platform that will create a ton of data and a big opportunity for Bing and any other service that can tap into it. “Any control surface can provide data,” Weitz, a senior director of search at Microsoft, told me. “Any signal from any device. Any input.”
That sounds like a real cacophony, one that would be way too noisy to yield much actual information. But the engineers at Bing figure their search engine can play a big role cleaning up a lot of that noise even as the devices keep churning out more and more data.
Weitz and Wallaert willingly tackled our hypothetical smart living room. Assuming the thermostat is programmed to pay attention to such things, they suggested, it will note that at certain times every day, the room gets hot.
Ordinarily, the thermostat's response to that kind of event would be to turn up the air conditioning or turn down the furnace. But Weitz suggests that the thermostat could also be programmed to search for reasons that the room is getting hot at certain times of the day.
This would be a significant change in how the object is currently designed. Adapting to temperature changes is one thing—that's just controlling the home's heating and cooling systems. This would be seeking out solutions beyond the mission parameters of a thermostat.
Eventually, the thermostat might correlate the increasing room temperature with the fact that sunlight happens to be streaming in through the windows at the same time, Wallaert suggested. Then it could search for possible solutions on an Internet data platform (like, say, Bing).
Then it would have to sift through those solutions, whereupon it could discard the impractical—nuking the Sun with an anti-fusion device, for instance, would be right out. But closing the drapes at 2 p.m. for a few hours in the summer? Now there's an idea.
Finally, the thermostat sends a signal to the sullen drapes across the room, which dutifully respond. The lamp, noticing the decrease in brightness, might then turn on—but only if there's a person in the room.
Looked at one way, all that seems pretty straightforward, because people can figure out solutions like this pretty fast. But stop for a moment and think about what just happened. Using information and data from humans and other devices on the Internet, a completely blind thermostat learned about the existence of a heat source that can be blocked by another device that's cooperating with it.
One Damn Smart Thermostat
Or did it? This thermostat would have to be ridiculously sophisticated, even with the resources of the Internet behind it. The ability to independently look for alternatives and to learn from knowledge available on the Internet borders on sentience.
More likely, the thermostat's programming would consider a constrained set of options to make judgment calls like this. It might take in information like local weather data, ambient light readings, power consumption (perhaps someone is running a dumb space heater), and maybe even CO2 concentrations (to measure occupancy) and use it to make decisions about heating and cooling. If it is a sunny day, close the drapes. If there are a lot of people in the room, kick in the air conditioner and leave the drapes open.
Any device programmed to gather information to make on-the-spot decisions like this would still benefit from the availability of data from the Internet, some of which would help it make relevant correlations. Data scientists refer to this sort of repository as a knowledge fabric or mesh network, Weitz said. “It would be an entirely new information architecture.”
It would be a data translation service that would behave like a “Babel fish for things,” Wallaert added. Devices that need information to effect change would tap into the knowledge fabric based on their programming and adjust their controls accordingly.
For Bing's engineers, creating a knowledge fabric represents a much bigger goal than seeing who has the best search results. Unlike Google, Wallaert argues, Bing is "building the layer that understands what's going on.” Doing this will eventually enable Bing to apply data to actions instead of just search results.
Which is just the thing to get an Internet of Things to work.
Command And Control
Ask Mike Harris, CEO of Zonoff, about this particular outlook for the Internet of Things, and he will carefully raise some objections.
For one thing, Harris has trouble imagining that a host of competing smart-device makers will make cooperative use of a single knowledge fabric. Corporate rivalries being what they are, sharp elbows are likely to be out in force.
"There are a lot of real companies out there that are saying 'I want to own this' about their data," Harris said. And the resulting friction could well impede technical advances like those that the team at Bing describes.
But Harris figures the Internet of Things can get rolling anyway. That's no big surprise when you discover that Zonoff is a significant player in the connected home space, as a white-box manufacturer of home automation control systems for a variety of vendors.
Harris finds the concept of a knowledge fabric with machine learning interesting, but for now he's leading his company toward what he considers a more practical approach: Building software platforms that can handle centralized command and control systems for home automated devices.
The metaphor Harris uses to describe Zonoff's home control systems isn't too different from the one that the Bing team uses These systems serve as the brain of the home, connecting to devices via standard Wi-Fi or Z-Wave (a low-bandwidth frequency intended for home automation) networks, sorting out protocols and coordinating commands and responses.
This latter task could be daunting, given the sheer number of devices out there and the vast number of ways in which they could conceivably work together. But here lies a bit of cleverness: Zonoff systems can be easily programmed by non-technical homeowners to introduce their smart devices to one another, and then to monitor their actual usage in order to provide home users with more educated feedback for tweaking their setting.
This combination of data analysis and crowdsourcing data has produced, for instance, a "goodnight" command set that locks doors, set alarms and put the home's systems in energy-saving mode.
Such systems can even ask the homeowner lifestyle questions—do they have kids? pets?—and apply relevant rules to the home automation system that will in turn spark better feedback.
Here Comes Skynet?
While the "knowledge fabric" approach to the Internet of Things isn't out of reach, control hubs such as Zonoff's are already on the market and seem likely to dominate markets like home automation for the near future.
But the vision of an machine-learning knowledge fabric is a compelling one. So long, that is, as machines don't learn too much and form their own too-clever-by-half ideas about what to do with these humans running around the planet.
We shouldn't dismiss such concerns out of hand, but there are a lot of technical challenges that have to be solved. As economist Andrew McAfee noted sardonically in a recent Ted talk:
There's no shortage of dystopian visions about what happens when our machines become self-aware and they decide to rise up and coordinate attacks against us.... I'm going to start worrying about those the day my computer becomes aware of my printer.
Lead image modified from Shutterstock by Madeleine Weiss for ReadWrite. Fair-use images of the Jetsons screencapped from the TV show "The Jetsons"