Big Data projects reduce defects to take on the IoT

An increasing number of organisations are leveraging big data to realise efficiencies in their business processes and are using analytics to track customer behaviour and campaign efficiency. With the volume of data being handled by organisations growing exponentially, Big Data technology is critical for analysis of data that is too diverse, fast-changing or voluminous to address with conventional technology. Also, software quality for projects that handle Big Data is becoming more important for enterprises to consider.

The IoT is having a significant effect on Big Data. The IoT will deliver $6.2 trillion of revenue by 2025 and an explosion of data is expected to be generated in the process, according to the McKinsey Global Institute. To leverage this data, organisations must be able to efficiently harvest, store and analyse it. Many of the leading open source projects and technologies that enable the Big Data movement and support the IoT are using the Coverity Scan service, indicating a drive to improve software quality and security.

Tim Hall, Vice President, Product Management, Hortonworks, commented: "Enterprise Hadoop must meet the requirements established by corporate security officers around the core tenants of authentication, authorisation, auditing and data protection. The Hortonworks Data Platform is built on 100% open source Apache Hadoop and we support the Coverity Scan open source software scanning service to help focus the community efforts and ensure the numerous Apache Hadoop projects are meeting those enterprise requirements."

In the 2013 Coverity Scan Report, the average defect density rate for Java projects was 2.72. Ten out of the 16 Big Data projects in the Coverity Scan Project Spotlight report sample have a lower defect density rate than that average, but many still remain higher than those in C and C++ code bases. Additionally, analysis of the results of the Big Data projects found that contributors are fixing more critical issues, like resource leaks, null pointer dereferences and concurrent data access violations. However, the Coverity Scan Project report shows that project contributors do have some 'Open Web Application Security Project (OWASP) Top 10' issues and need to better examine security to assure sensitive data from IoT devices is secured.

• Apache Hadoop has made steady progress in eliminating key defects since Coverity Scan profiled the project in the Coverity Scan 2013 Report. In that report, Hadoop had a defect density rate of 1.71. Since that time, they have reduced it to 1.67.
• Since the 2013 Scan report, almost 200,000 lines of code have been added to the Apache HBase project, and the defect density rate has been lowered from 2.33 to 2.22.
• Apache Cassandra has also made progress in eliminating key defects since being profiled in the 2013 Scan report. Previously, Cassandra had a defect density rate of 1.95. Since that time, they have lowered the rate to 1.61.

"Early efforts of the Big Data projects tracked by Coverity Scan are showing interesting and actionable results. IoT and big data have the power to transform lives and our economy. There's a great deal riding on these foundational technologies, and these organisations are taking that responsibility seriously. It's encouraging to see their commitment to addressing critical defects and to taking the appropriate steps to deliver higher quality software to the market," said Zack Samocha, Director of Marketing, Coverity business group, Synopsys.

During the past eight years, the Coverity Scan service has analysed several hundreds of millions of lines of code from more than 3,000 open source projects - including C/C++ projects such as NetBSD, FreeBSD, LibreOffice and Linux, and Java projects such as Apache Hadoop, HBase and Cassandra. Coverity Scan has helped developers find and fix more than 94,000 defects since 2006. Nearly 50,000 defects were fixed in 2013 alone - the largest number of defects fixed in a single year by Coverity Scan users. More than 11,000 of these defects were fixed by the four largest projects in the service: NetBSD, FreeBSD, LibreOffice and Linux.