The top three challenges of preparing IoT data

For instance, sensor devices might be found on a production line to track the readiness of the machines and automate predictive maintenance. Or, a hospital might use IoT devices for remote patient monitoring, robotic surgery or dispensing medication.

All of these growing sensors, devices, and other connected ‘things’ ultimately mean more data. And lots of it. But with more data come more complex challenges in preparing it. To harness the value of IoT and big data, and deliver innovation-driving insights, industrial organisations must quickly prepare all of this disparate, unstructured data. Below, we’ve named some of the top three challenges in preparing IoT data to leverage it for analysis.

1. Huge volumes of data
International Data Corporation (IDC) market research estimates that IoT devices will create 40,000 exabytes of data by 2020. To keep this in perspective, in the year 2000, three exabytes of information were created globally. That is a lot of data to prepare, and under many current processes, organisations won’t be able to keep up. This is particularly challenging in the industrial world, where manufacturers and other large industrial organisations typically collect billions of data sets from machines, sensors and internal business applications.

Data preparation still accounts for up to 80% of the time and resources involved in any data project, and the more data you add, the more time-intensive that process will become. As organisations take on new IoT data initiatives, it’s important for them to consider new technologies and processes that will allow them to keep up with this huge influx of data.

2. Complexity
Another challenge in preparing IoT data is its complex nature. Often, organisations must not only prepare timestamp or geotag data, but combine it with more structured sources, such as csv files. This complexity is only multiplied when factoring in the rate at which this data is being generated.

Finding a solution to this problem is tricky. The technical resources within an organisation that could handle this complexity are typically limited, and scaling out those resources are costly. Using common data preparation tools like Excel can’t handle this complexity, which leaves skilled analysts locked out of working with this data. Today’s organisations must figure out a way to leverage the resources they have in order to prepare the increasingly complex IoT data.

3. Interoperability 
Business computer systems, both hardware and software, aren’t made to exchange or process the vast amounts of complex information pulled from sensors and connected devices. It’s difficult to quickly integrate and enrich machine generated data with data from business applications such as Salesforce and Marketo for example, and other data repositories. Therefore today’s organisations must look for solutions that better allow data to talk to each other, so that the entirety of an organisation’s data can be leveraged.

Data preparation platforms for IoT initiatives
Many organisations that are spearheading IoT initiatives have turned to modern data preparation platforms to ease these challenges. With an intelligent data preparation platform, some of Trifacta’s customers have seen time spent preparing data reduced as much as 90%, while also allowing nontechnical resources to prepare large quantities of complex data themselves. In addition, we have partnered with Sumo Logic to offer clients a solution to prepare complex log data with business application data.

For instance, a large European rail company is using Trifacta to prepare sensor data generated from monitoring 8,000 locomotives across 32,000 miles of rail tracks in order to predict when they required maintenance. Prior to adopting Trifacta, the company was preparing this data ad-hoc across multiple people and with many different tools, which ultimately delayed analysis and their responsiveness to necessary repairs. Now, this company can prepare 100% of complex sensor data and has rapidly decreased their time spent preparing data.

Another customer, Kuecker Logistics Group (KLG), is using the Trifacta platform to prepare a multitude of sensor data generated from warehouses owned by the world’s largest retailers. These customers have extensive, complex supply chain operations and one faulty or inefficient link in the chain can cause a ripple effect downstream. By using an intelligent data preparation platform, Kuecker has been able to scale data preparation processes without hiring costly developers, which has dramatically improved their efficiency. Now, they’re preparing customer warehouse data and more quickly identifying the necessary changes that need to be made within the warehouses.

Conclusion
IoT data is an exciting opportunity, but its benefits can only be realised with a proper data preparation strategy in place. Organisations must equip their team with data preparation platforms that can handle the volume and complexity of IoT data, as well as understand how this data can and will be joined with other sources across the organisation. By adopting intelligent data preparation solutions, the universe of IoT and big data no longer overwhelms. Sensor data becomes the key to innovation, not an impediment to it.