The time for autonomy in mobile machines is now

In anticipation of the International VDI Congress ELIV 2024, Hindman answers questions about the technological advancements, applications, and challenges of implementing autonomous off-road machines, particularly in industries like agriculture and construction.

Why do you think that now is the perfect time for autonomous off-road machines?

There are three technology developments that have matured in the last decade to make autonomy for some off-road applications possible where it had not been previously.

Those three areas are:

Pattern recognition algorithms like convolutional neural networks and more recently, transformer networks have been developed that have allowed for pattern identification to occur in complex data sets, like data from stereo cameras, radar, LiDAR, etc., that provide the perception senses for an autonomous machine.

These algorithms are computationally intensive, particularly when tasked with the sufficient degrees of resolution required in autonomous machine operation.

The second area is the advancement of high performance compute, often through GPU hardware advancements, that has enabled these algorithms to run “on the Edge” in mobile off-road machines with sufficient performance for the application.

The fuel for the new algorithms is data, and they require a lot of it to perform well in a generalised sense. Acquiring that data has required significant effort historically but that burden is decreasing with the improvement in connectivity to off-road machines. Connected machines can support communicating important data necessary to improve algorithm performance on an ongoing basis. Data collection from an unconnected machine wasn’t tractable at scale. As machines are connected through terrestrial cell infrastructure or satellite connectivity, the data collection effort can scale as necessary to support model training and performance improvements.

Which typical applications do you expect to see first?

I think it makes sense to start at the intersection of technical possibility and customer value creation. In the markets that John Deere serves, the agricultural applications have certain benefits over others. First, farmers have been using GNSS for path plan guidance for nearly 20 years. They are accustomed to a high degree of machine automation with the current solutions. The environment that agricultural machines operate in is relatively homogeneous at certain times of the year. The work that the machines accomplish follows a predictable pattern. These factors lend themselves to an application that improves the technical feasibility. At the same time, the work that is accomplished in agriculture is episodic. There are significant demands for labour during the planting and harvest times. This creates demand from the customer to solve for these short duration, high magnitude labour demands. The opportunity for autonomy to create value for these customers is high.

And what do you expect as the next milestones in technological development?

There will be many milestones needed to accomplish autonomy in the off-road space. Some that I am particularly interested in are improved/new sensor modalities that are tailored to the operating environments we are interested in. Radar development that improves near-source target separation, the capability to sense through environmental factors like rain, standing crop, and soil, will open up new operating design domains. I also think there is significant opportunity in expanding the pattern identification of new algorithms to an “end to end” approach as opposed to “sensor perception” only approach. Said differently, we have the opportunity to train models to not only identify the machine operating environment, but to use that environment information to make machine control (braking, steering, implement, etc.) decisions. This of course increases the computational requirements but the current trajectory of on-board compute makes this more tractable over time.

How challenging is it for John Deere as a global concern to fulfil different regulations worldwide? Would you like to see more harmonised standards here?

Similarities in standards across markets allows solutions to scale more rapidly and efficiently. In that sense, commonality of regulations is a good thing. On the other hand, if the regulations standardise towards solutions that drive unwarranted system complexity for many markets, then the standards will limit the ability of customers in those markets to adopt technology that is more costly than it otherwise would need to be.

In what aspects is it easier to implement autonomy off-road than on the road – and what other or additional challenges do you face?

The differences in off-road autonomy don’t always make the implementation easier than on-road. A primary difference that increases the difficulty in off-road is that the off-road machines are not simply tasked with traveling from point A to point B. They must also do a job along the way. That job might be an agricultural job like tillage, planting, spraying or harvesting, or it might be a construction job like digging, grading, or paving. That said, there are several key differences that make off-road autonomy more tractable than on-road. The slower speeds often seen in off-highway reduce the distance a perception system must be able to operate around the machine. This also reduces the computational needs of the system given there is more time available to make machine control decisions. In many off-highway applications, a stopped machine is seen as a safe-state. This is a significant benefit as stopping the machine is a productivity loss but doesn’t necessarily create a safety issue. Finally, in many of the off-highway applications the environment is more homogenous with fewer objects to identify and classify than the on-highway case.

Autonomy is seen mainly as an answer to understaffing. Can these new technologies also have a positive impact in terms of increased sustainability?

I think so. We’ve seen in agriculture applications that the machine-derived path plan is often a more efficient path than the one a human would have created on their own. This results in fewer passes in the field, less overlap of work, and less energy consumed in the activity.

How would you complete the following statement: Anyone who does not switch to autonomy now ...

… represents an opportunity for future solutions to create enough value for them to reconsider!