In a market that’s expected to grow to almost $950 billion by 2025, it’s now safe to say the Industrial Internet of Things (IIoT) has moved from pilot projects to larger-scale production deployments. This growth has, in turn, triggered the need to ensure that these deployments run smoothly, particularly as more IoT systems are becoming critical to the business operations of a company.
Testing these systems for resiliency and performance has become a critical element in delivering reliable systems. In order to accomplish this at scale, our industry needs a new generation of testing technologies to deal with the scope and complexity of today’s IoT systems.
Although it’s not talked about much, testing IoT systems is an important part of the ROI for IoT. Proper testing significantly reduces the number of errors found in production, errors that are often difficult and expensive to diagnose and fix. In addition, IoT devices that have been deployed in the field are often difficult to access and update; testing them beforehand spares some challenging fixes later.
Unfortunately, many of solutions were not designed for testing IoT use cases, so testers of IoT systems are left with insufficient tooling and can be forced to “kludge” together solutions meant for web-service testing. The complexity of testing IoT systems is different from testing web applications or server applications. As a result, existing testing tools targeted at these types of applications often fall short when applied to IoT systems. IoT systems are massively distributed applications. They can have millions of connected devices that connect to complicated enterprise services that each, independently, execute application behavior.
For example, our customers are deploying connected-car platforms that involve millions of connected cars, each of which is simultaneously executing different behavior, while publishing and subscribing to data from a variety of connected-car services running in the cloud. To properly test the resiliency of this type of system, our customers need to simulate millions of connections that are executing a variety of different behaviors.
Setting up a testing environment for an IoT system that replicates a production environment is also a big challenge. For example, testing 1,000 simultaneous connections is very different from a production environment of 100,000 connections. At HiveMQ, we have seen customers test for 1,000 connections and try to extrapolate their results to 100,000 connections with inconsistent results. Similarly, if IoT devices in a system are deployed around the world, how do you simulate global operations for testing?
Finally, production data generated from IoT devices can have a high degree of variability, so replicating this diversity in the test environment is very important.
To meet these demands, a new category of testing tools needs to be defined to support the load and stress-testing of IoT systems. These new IoT testing tools need to be able to do the following:
● Create millions of unique network connections from anywhere in the world to simulate deployed IoT devices. For example, the tool needs to be able to create client connections on an Alibaba cloud deployment in China, an AWS data center in the USA, and an Azure data center in Germany. These client connections need to be able to execute different behaviors during the testing simulation.
● Create very high throughput of messaging from a small or large number of clients. Many manufacturing use cases can have tens of thousands of messages per second sent through the system. Future autonomous-driving examples may see hundreds of thousands of messages per second. A proper IoT-testing tool needs to be able to generate this amount of traffic but with a degree of variability to truly reflect a production deployment.
● Each device behavior can be modeled and executed by a simulation environment so all functionality of the system is tested.
● Custom-data generators can be created to generate variable production data.
Luckily, new modern cloud-native technologies have made this new category of testing tools a possibility. Hyperscale cloud platforms from Microsoft, AWS and Google have made it possible to create distributed-testing environments. Docker and Kubernetes make it possible to setup distribution-testing clients around the world on cloud platforms. Modern time-series databases and dashboard technologies make it possible to collect the enormous amount of testing data required to visualize the results.
It is time for the testing community to embrace these new technologies to deliver IoT-testing tools that meet the requirements of IoT systems.
By Dominik Obermaier, CTO and co-founder of HiveMQ
