Ensuring Code Quality In Mission Critical Systems

Embedded systems are now widely used in mission critical systems. The obvious applications include avionics and medical, while less obvious might be a diesel engine management system. Complex multicore engine management systems now enable highly efficient diesel engine development, enabled by the latest debug technologies available to engineers that can help ensure code quality and performance.

Proving code meets the target specification is not easy; even something as simple as proving code has actually executed (code coverage) is rarely undertaken by development teams. The tracking of obscure or random bugs can be so difficult that when a code alteration ‘fixes’ the problem, the developer moves on without any concrete proof that the original problem was found and fixed.

Another challenge is that debugging code properly is often harder than it needs to be because very few teams think about what debug capability will be needed when they define the original system design parameters. Parameters such as measuring best and worst case response times may be considered, but the operation of the cache and obtaining details of code victims and misses is also invaluable if the system is expected to run at its optimum level. There are many similar interlinked aspects to analysing the operation of the final embedded system and proving it is fit for purpose.

An Essential Investment

A popular Abraham Lincoln quote reads: ‘Give me six hours to chop down a tree and I will spend the first four sharpening the axe,’ and is an excellent analogy to debug tools. The more capable the debug environment, the better the development experience for all concerned, resulting in better code, faster development and a reliable in-field performance.

At the start of every project, engineers get to choose their hardware platform, their software solution, compiler and their debug solution.

More important than the choice of tools, compiler or operating system is the choice of processor. In order to get non-instrumented coverage information from the system, a processor with an off-chip trace port is essential. For systems that require safety certification, a good quality trace port is required. The bandwidth at the trace port will prove a very important consideration.

For multi-core devices, the trace port must be capable of providing reliable, error free trace streams from all cores. This means that the trace port should be running at a good percentage of CPU clock and be wide enough to get the data off-chip fast enough to prevent overflows.

Required Hardware

Off-chip trace hardware will be required to interface the processor to the development environment. It should be capable of collecting data with the core running at its final clock rate. Many solutions are available, ranging in capability and price. Some include important features such as a logic analyser and high-speed serial trace functionality. For older processors without a JTAG port, an In-Circuit-Emulator (ICE) can be used.



Many reference design boards come with low cost debug solutions. These have the advantage of being easy to set up and use, but can prove to have a very limited capability as code development progresses. The more powerful solutions may be more costly and may even require some training, but the longer-term benefits will quickly prove a good return on investment.

The big changes that debug tools and code analysis developers are seeing is that it is no longer enough to simply test a system. A new trend is an increased requirement for engineers to document and prove code behaviour and performance. In some instances companies are requesting engineers to be certified and qualified in software writing or mandate some form of specified code quality. With the right choice of debug tools, detailed information about code behaviour can be generated to provide the necessary proof, showing such aspects as the route taken by the code and time taken. In this respect, program flow trace can go a long way to helping companies prove code behaviour and in achieving safety specifications such as ISO26262.

Long-term trace is the name given to streaming a trace to a hard drive to overcome the limitations on internal buffer size in most debug tools. This provides the ability to collect massive amounts of code performance information from a running embedded system for the developer to detect and analyse the most unpredictable and transient of bugs. Demand for long-term trace debugging is being driven by market sectors such as automotive, medical and aerospace, where microprocessor based systems are becoming increasingly complex and in need of more rigorous testing to comply with safety and performance criteria.

One example is to imagine an engine management system. This longer code coverage enables engineers to analyse the software from a cold start, then up to temperature, through acceleration and deceleration and then to shut down. Using long-term trace technologies, engineers are now able to capture code over much longer periods that can extend to hours or even days. This has been a very important break-through for those developing complex or safety critical code.

High Speed Serial Trace

'High Speed Serial' trace technology is a serial transmission technology that has a transmission rate of over 6Gbits on each of up to 4 lines from the target core. To put it in perspective this data rate is such that it could transmit the entire contents of a DVD in 3 seconds, making it ideal for collecting data when debugging and developing very high speed multicore embedded systems and systems requiring mission critical analysis.



Traditionally, the trace interface used by target processors to deliver the detailed information on the operation of their inner processes had been a parallel port. Over recent years this approach has struggled to keep up with the growing flood of information as processors have become more complex, faster and with multiple cores. The step to high speed serial solved these problems and had the side effect of reducing the pin count, which is always good news. For many developers of embedded systems it would have been unthinkable to undertake a development without their valued trace information, so a lot of effort by silicon designers and tool vendors has been made to increase the data throughput of the trace interface.

In recent times, ARM has implemented High Speed Serial Trace with its High Speed Serial Trace Port (HSSTP). This has been followed by AMCC with the Titan, Freescale with the QorIQ processors P4040 and P4080, and Marvell with the SETM3.

Engineers can now source a hardware interface for serial trace; a universal pre-processor has been developed on the basis of the Aurora protocol. Only the firmware and software have to be changed to record any of the alternative protocols. This means that existing systems will need minimal reconfiguration for further variants of serial trace protocols.

An important consideration of this technology is that the large volume of trace data generated obviously requires a correspondingly large trace memory and careful consideration should be given to the tool choice with regards to memory size and the ability to stream to the host.

Multi-core Debugging

Multi-core development is an area of embedded systems development that is becoming more prevalent. It started with mobile phone development back in 2003/4, then moved into telecoms (base stations, routers) about five years ago, and today, it is finding its place in the automotive sector as future emissions and fuel usage specifications demand far greater analysis and control.

With multi-core, the data is extracted through a shared trace port. There may well be four or more cores with processes running in parallel and some interacting across core boundaries. The fact is, two cores are not simply ‘twice’ as complicated to debug as a single core; they are several times more complicated, because of these interactions.

Getting to the point, budget debug technology is not up to the job of multi-core development. For multi-core you will need a powerful trace tool. One with a lot of memory and one that can process data fast. In multi-core you will have a whole new level of problems to avoid, such as race conditions, uncontrolled interactions and out of sequence code behaviour. The toolset will need to be able to collect a large amount of accurate data to give the detail and visibility into the code needed to understand the behaviour and interactions of the processes running on the cores.

Code optimisation

In mission critical systems where fast and predictable performance is essential, cache analysis has proven to be an important development. Cache memory is on-chip memory that can be accessed very quickly by the CPU. Access to cache memory can be in the order of 10-100 times faster than access to off-chip memory.

This fast memory acts as a temporary storage for code or data variables that are accessed on a repeated basis, thereby enabling better system performance. However, having cache memory does not necessarily equate to better performance from a system. In order to get the best performance from cache-based architecture, it is important to understand how a cache-based architecture works and how to get the best from a particular cache architecture. Trace technology enables analysis tools to confirm the effectiveness of cache usage and can have an important impact on the overall performance of the software.

Also, these days, safety critical equipment may be a wireless portable device, where battery life and predictable power usage may be an important factor. With this in mind, power management is another area that is being influenced by the use of trace tools.

Increasingly, engineers are relying on trace technology to not only enable the location of bugs, but to also create better code and provide clearer visibility and understanding of code behaviour. It is strongly recommended that such tools are explored and evaluated at the earliest stage of a new project.