Benchmark targets machine learning inference on edge devices

Creating relevant standards for neural net inference behaviour has been challenging however, given the pace of advancement and the wide variety of SDKs and compute libraries.

MLMark addresses this confusion by focusing on three measurements deemed significant in the majority of commercial ML applications:

• Throughput - Measured in inferences per second.
• Latency - Defined as single-inference processing time.
• Accuracy - Comparing network performance to human-annotated ‘ground truth’.

“Hardware platforms for running AI are proliferating, in a wide variety of architectures—both repurposed and custom-built—and the industry has had a hard time finding a common baseline for measuring their performance and accuracy,” said Sujeeth Joseph, Chief Product Officer at Ignitarium. “The release of MLMark comes at just the right time to fill a crucial gap in the AI inference toolkit. It’s a welcome addition to EEMBC’s highly respected suite of benchmarks.”

The largest single category of application for ML today is image recognition, so MLMark’s measurements focus on visual object detection and image classification tasks. It comes with three vision models ready for immediate use: ResNet-50 v1, Mobilenet v1.0 and SSDMobileNet v1.0. In addition to a hardware-independent Tensorflow implementation, MLMark also includes C++ code for a variety of target hardware:

• Intel CPUs, GPUs and neural-compute devices using OpenVINO.
• NVIDIA GPUs using TensorRT.
• Arm Cortex-A CPUs and Arm Mali family of GPUs using Neon technology and OpenCL, respectively.

“With this benchmark, we’re helping to solve the ‘scavenger hunt’ problem that faces newcomers to Machine Learning, there are so many competing technologies that it’s hard to know where to start,” added EEMBC President Peter Torelli. “MLMark includes a collection of models, ground truths, and source code right out of the box, giving recent entrants a useful overview, and helping the entire industry move toward a more coherent, collaborative footing.”

In addition to this learning function, MLMark also enables engineers to create better performance models, helping them identify bottlenecks in ML-specific tasks when developing new products. And as the market expands, the MLMark score database provides both a snapshot of the current market, and a historic record (and predictor) of industry trends.

By creating strict requirements for gathering data, disclosing results and publishing scores, MLMark creates a standardised environment where competing technologies can be evaluated on merits rather than marketing messages. This raises the quality of the ML space as a whole, by incentivising the market to focus on improving performance rather than messaging. 

Torelli continued: “EEMBC has a 20 year track record of delivering intuitive benchmarks that enable the embedded industry, and we’re excited to add machine learning to our repertoire. It’s a crucial technology, and ultimately it should be a capability, not a contest.”