4 Challenges for IoT Data Compression
Data compression — reducing the number of bits needed to represent data electronically — has been around for as long as there’s been data. It’s an important tool for increasing transmission speeds, saving storage space, and reducing hardware and network transport costs.
But compression doesn’t offer a free ride. And when sensor-based data from billions of "things" begins to flood networks as the Internet of Things (IoT) ramps up, the industry will run up against some painful truths about data compression for the IoT.
Four converging challenges threaten to become obstacles to the IoT, and overcoming them means taking a fresh approach to data compression.
Challenge #1: The high cost of data networks.
IoT sensors and devices have limited computing, memory, and storage resources. They typically communicate using many "bursts" of small amounts of data. Traditional compression technologies don’t work with frequent bursty messages; compression has been designed for big chunks of data, making the small data bursts essentially uncompressable.
Then networking protocols add "overhead" to the data, in the form of headers and other transmission instructions. A 100-byte message could carry a 200-byte header, expanding the message to 3 times the expected size. Multiplied over many messages a day, these bloated bursts of uncompressed IoT data can send wireless costs spiraling out of control.
Challenge #2: The short battery life of IoT sensors.
The batteries in IoT devices such as smart meters, location trackers, or remote sensors, might need to stay on in the field for months or years at a time. Radio use drains batteries, so using the radio less can help preserve battery life.
Unfortunately, IoT device makers rarely consider the implications of the communication protocols they choose. Unfortunately, a "chatty" protocol takes a lot of extra bits to do its job, which means that battery radios stay on longer, and end up consuming more battery power for communications.
Challenge #3: The inefficiency of network operations.
IoT applications often communicate over slower wireless networks, based on the assumptions that small IoT devices don’t require high speeds or throughput for their communications, especially over short distances, and that latency isn’t an issue.
But just because IoT devices might not require the network capacity or extremely low latency of an application such as video streaming, they must still be able to transmit actionable data fast enough to perform basic functions. Emerging wireless networks such as Bluetooth LE offer many benefits, but if the IoT data being transmitted exceeds the anticipated capacity, it is less useful, or actionable. Wireless networks focused on extending IoT device battery life by reducing the size and frequency of messages can cause messages to get backed up, introducing often unacceptable latency.
Challenge #4: The tradeoffs of today’s low-power wireless networks.
Mainly for cost reasons, IoT applications are increasingly being run over low-power wireless networks, which have significant bandwidth constraints. Some impose actual traffic limits. For instance, the SIGFOX wireless protocol allows each device a maximum of 140 messages per day, with each message limited to only 12 bytes of data — for a grand total of 1.68kB of data per IoT device per day.
Other emerging wireless networks, including LoRaWAN and Cat 1 LTE, also require careful balancing of traffic over their narrow bandwidth. Trying to transmit meaningful IoT data from a device to the network back end under the constraints of narrowband network implementations can become an exercise in frustration.
Reinventing Data Compression for the IoT
It turns out that all 4 of the challenges outlined here can be addressed by reinventing data compression and optimizing it for the IoT. In particular, the IoT industry can look to a methodology created and patented for automobile navigation systems — which have size, power, memory, and bandwidth constraints very similar to IoT devices.
This IoT data optimization approach has 3 stages: compaction, compression, and conversion.
First, the data is pre-processed to compact it. Before encoding takes place, compaction analyzes the data payload to determine the data type, how it is organized, which elements can be reduced, and how to maximize compression.
Second, an encoder selects appropriate algorithms, based on the payload information of the compaction step, to optimally compress the data. The end result is up to a 20:1 compression ratio.
The third step is to convert the message to operate over the best networking protocol available. This step has 2 components: 1) determining what protocols are available for use, and 2) determining whether there are traffic limitations imposed by the protocols. For example, if SIGFOX is one of the protocol options, that protocol’s 12-byte message limitation must be considered. The conversion step also determines if there is a less chatty protocol available before the message is parsed accordingly.
Here’s how this new 3-stage approach to IoT data compression addresses the 4 challenges described above:
1. Reduces data network costs by compressing IoT data by as much as 20x compared to traditional compression.
2. Reduces the size and frequency of data transmissions, which means less radio usage, less battery drain, and longer battery life.
3. Optimizes data to overcome bandwidth, latency, and power issues.
4. Compresses and optimizes data so that devices can efficiently operate, regardless of network constraints, over any existing or new wireless networking protocol that enters the market.
Data compression reinvented in this way enables IoT data to be sent successfully over any kind of network, without regard to IoT devices’ limited computing capabilities, storage, or processing abilities — all while fitting into a small space consuming just 1kB or 2kB of memory and less than 5kB of storage.
With optimized data compression, companies developing IoT solutions won’t be stopped in their tracks as they run up against previously insurmountable data challenges.
