Get the most out of your image

 ISP configuration should be calibrated and tuned for optimal image quality while preventing unwanted effects in the image. By taking a structured end-to-end approach, the right ISP tuning process ensures the complete package is acting in unison.

The Vision industry utilises a diverse range of products in the areas of camera technologies (visible, hyperspectral, 3D, etc.), optics and lighting, vision processing (embedded vision, IPC, GPU), software and deep learning. However, a vision system is much more than the sum of its parts. The image quality is determined by the complex interplay of many contributing factors. Users or camera manufacturers do not always achieve optimum results right away.

The image signal processor (ISP) is a central component of the vision system. It determines how an image is transmitted and displayed. Carefully tuned settings can considerably improve the image quality. FRAMOS can help with such tuning processes while supporting customers on their pathway ‘from concept to production’. For example, the FRAMOS Sensor Module (FSM) Ecosystem is paired with service features for ISP parameters tuning for NVIDIA platforms, that have been prepared within our modern in-house IQ (Image Quality) Laboratory.

True colour images are essential for AI Edge computing

Using AI (Artificial Intelligence) techniques in object and colour classification, for example in security applications, have increased the demands for colour accuracy. Trained AI models use the most original reproduction possible to decide whether a person or an object is actually recognised. Deviations beyond a particular tolerance threshold in the colour display when the image is reproduced, often caused by changing lighting conditions for example, the AI algorithm, can decrease the changes of a correct match. In these situations, the recognition performance deteriorates rapidly. ISP tuning can considerably improve their results for these vision applications.

Another scenario where the best possible reproduction is important would be the use of the vision systems in smart farming. These purpose build systems can assess the ripeness of fruit or vegetables so that automatic harvesting machines know when to best harvest them. For this application, the colour accuracy over the full light spectrum is important because the lighting conditions can change constantly in outdoor environments depending on the sun’s position and the clouds present.

Only reasonably accurate colour reproduction can ensure that AI edge systems are able to reliably recognise the ripeness of produce. The camera’s recognition performance must be optimised to ensure faster and more precise operations in these applications. ISP tuning therefore has a major role in colour dependent applications like this.

FRAMOS – An NVIDIA Elite partner

Edge intelligence is the combination of edge computing and AI with algorithms running locally on the hardware devices (‘on-device AI’). This results in rapid response times, low latency, high security, and higher robustness while also helping to ensure that network bandwidth is used more efficiently.  FRAMOS has been supporting such AI-based edge applications for some time now.

As FRAMOS is an Elite member of the NVIDIA Partner Programme, it can support customers utilising the NVIDIA Jetson platform. It can effectively support customers employing AI-edge computing solutions in vision applications like robotics, automation, IoT-networked production, just to name a few. Customers implementing FRAMOS Sensor Module Ecosystem components can simply connect these imaging modules with the cutting-edge NVIDIA AI platforms, like those based on Jetson and Xavier, and apply ISP tuning configurations, included with these components, immediately.

ISP calibration and tuning

Each part of an imaging system, such as image sensors, lenses, optical filters, illumination sources, etc., have very specific properties that influence the overall image quality and, in turn, the ISP configuration needed. To achieve optimum image and colour quality while avoiding undesirable side-effects, the ISP must be calibrated and tuned. The ISP block on NVIDIA platforms plays a key role as they can implement corrections and improvements to the image data without loading the CPU and GPU, maximising the overall system efficiency.

The calibration of the ISP and its successive tuning across a wide range of operating conditions is a complex, time-consuming task. It requires specialised measuring equipment and subject-specific expertise, with a deep understanding of the platforms in question. FRAMOS engineering team has established an image quality tuning lab that is equipped with the necessary tools that complements their expertise in calibrating and tuning the ISP to achieve optimum image quality for every application.

To calibrate a given configuration that comprises of an image sensor module, a lens, and optical filters; the imaging experts recreate a customer’s setup in the lab to carry out the necessary measurements to produce the optimum ISP configuration values for subsequent stages. A wide range of different parameters are reviewed and corrected during this process, like the color correction matrix values, black level compensation settings, auto white/colour balance ranges, lens shading falloff correction parameters, noise reduction setting, sharpening algorithm values, to name a few.

Creating colour images from RAW data

Most colour image sensors use a Bayer filter pattern based on an R-G-G-B colour filter arrangement. Each pixel has a unique colour filter placed over top of it. The sensor captures a modified monochrome, aka RAW, image that contains the filtered pixel data with the colour information. The resulting colour image is produced using various colour interpolation algorithms implemented in the ISP. This is known as demosaicing the image and is a digital process for reconstructing a full colour image from the RAW data from the colour filter array.

The transmission spectrum and total transmissivity for these colour channels are different with each CMOS sensor used. The colour image produced after the demosaicing process is far from optimal and does not align with what the human eye sees under the same conditions. To correct for this mismatch in response, a Colour Correction Matrix (CCM) is calculated and is applied to correct the subsequent images to provide a truer colour response to the user and the application.

Left: Relative Sensitivity of color filters for IMX290 sensor

Right: Uncorrected demosaiced and interpolated image from a color sensor with balanced light source

Colour images are influenced by external conditions especially by the lighting.  It will affect the brightness and the hue of the resulting image. Changes in the illumination level affect the brightness of the captured images which can produce poor results if not addressed prior to them being analysed. Moreover, the colour temperature of the illumination has a major impact on the colour casting of the image. Therefore, it is necessary to apply operations, such as Auto Exposure/Auto Gain to achieve necessary image brightness, and Auto White Balance to compensate for illumination colour temperature to achieve the best results.

Left: Two images showing effects of the varying illumination on image brightness

Right: Influence of illumination color temperature on captured images

A well-tuned ISP, with the automatic white balancing, can accommodate such diverse lighting conditions (incandescent bulb to afternoon sunlight to fluorescent light sources).

Tuning the demosaicing, CCM and white balance algorithms ISP blocks together result in considerably improved colour response in captured images, as shown below.

Sensors in the FRAMOS Sensor Module Ecosystem

When developing a vision application, it is necessary to experiment and validate important characteristics of the image sensors such as resolution, frame rate, optical format, shutter technology, low-light or HDR performance, for example. The FRAMOS Sensor Module Ecosystem allows users to test various sensors and quickly choose the right components for their application. The image sensor is responsible for capturing and digitising the available light before this image starts along its digital signal chain of events starting with and led by the ISP. However, it is important to choose a sensor that will provide the best images that fulfill the application requirements.

Using the comprehensive product portfolio that is part of the FRAMOS Ecosystem, vision engineers and developers can evaluate many different image sensors and efficiently assemble a proof of concept to validate their theories. In collaboration with carrier board partners and availability of the companion board schematics, a custom solution can be designed and manufactured for production prototyping, beta testing and, ultimately, mass production using these same FRAMOS modules.

The Ecosystem includes a wide range of sensor modules (FSM) with rolling or global shutters sensors with resolutions from 0.4 to 24MP from manufacturers like Sony and Pyxalis. Each FSM can be directly connected to the NVIDIA Jetson SOM development boards directly through FRAMOS’s unique PixelMate connector and/or with the applicable mating FSAs (sensor adaptors) and FPAs (processor adapters). The development kits contain all the necessary hardware and software components needed to stream images quickly and easily, right out of the box. Using this solution, developers can achieve the best performance without expert knowledge of camera design and development, while focusing on their value and core skills to launch their vision applications into the market more quickly.

ISP tuning: beyond the illumination and colour reproduction

The ISP tuning process has many steps with the validation of the module driver being the first. Once done and images are flowing from the sensor, the focus shifts to analysing the sensor controls and register settings. In this stage, the FRAMOS team characterises the black level setup, lens shading (vignetting) correction, colour reproduction, white balance, noise reduction and image sharpness. Once done, they look at customer specific imaging condition requirements to complete the fine-tuning the ISP configuration for final mass production.

Making the blacks black and the whites white

The black level of the sensor has prominent effect on not only the darker regions of the image but, in some cases, the brighter ones as well. This parameter is fine-tuned based on the sensor manufacturer’s recommendation and fine-tuned during the ISP tuning process. This is done by capturing raw images in absence of light at different exposure settings. After analysing all these images, the black level is calibrated to the ideal value for each exposure setting.

Tuning the ISP for the lens and filter

Sensor modules need a lens, and sometimes a filter, to see correctly. The lens focuses light on the imaging plane of the sensor and controls the field of view (FOV) of the imaging system. Optical filters such as NIR-Cut (Near InfraRed-Cut) filters are necessary to reproduce high colour fidelity images. However, these optical components also affect the image and colour quality. Therefore, several ISP parameters have been implemented to compensate for these components and mitigate some of their adverse effects. Here are some of the parameters that can be tuned:

• Lens Shading Correction: Removes the vignetting (shading) effect of the lens by mathematically accounting for the fall-off effect of the light intensity across the entire image 

• White Balance Algorithm: Adjusts the color levels used by the sensor to compensate for the different lighting conditions, NIR-Cut filters, and lens colour effects to ensure accurate white/colour representation 

• CCM Values: Calculates the necessary matrix elements for a given light source. A X-Rite Macbeth ColourChecker and/or grey cards are used in light-box with the system being calibrated for the specific application 

• Final Parameter Settings: Fine tunes the remaining parameters, such asauto exposure, auto/manual white balance, sharpness, tone curve, flicker, falloff, and saturation

ISP tuning at various levels

FRAMOS offers sensor-specific ISP tuning files created with their in-house image quality laboratory (IQ Lab) that can be used to maximise the imaging quality of an NVIDIA Jetson based system. Every lens, sensor and illumination combination are calibrated and tuned to achieve optimum performance. The IQ Lab uses cutting-edge measuring systems and tool chains to tune the image quality from industry-standard image sensors and individual components, up to the entire camera system. The ISP tuning services are offered at various levels (Level 0 to Level 3), depending on budget, need and performance.

Level 0 is included in the Development Kit, right out of the box. Using these parameters, colour reproduction is already very good, but it is primarily aimed for well-balanced light source near 5000K colour temperature. When using light sources outside of this range, a shift in the hue is noticeable in the images: At 2,885K (incandescent light bulb), the image’s hue is distorted and pushed towards the yellow/red. At 6,500K (sunlight), a clear blue cast can be seen. Lens falloff corrections are not applied, which is easily seen in the illustrations above.

With Level 1, FRAMOS team optimises and corrects the negative effects of the lens falloff. The Auto White Balance is further tuned to improve colour balance and colour reproduction. The standard FRAMOS lens and IR-Cut filter supplied with the FRAMOS module dev kits are used during the tuning process.

Custom tuning is achieved with Level 2 where the customer’s exact hardware setup is used to define the calculate the ISP parameters from scratch. In this scenario, customer’s selected lens and IR-cut filter are used to generate the ISP file. The process is like the one performed with Level 1 however is tuned to the customer’s specific components.

The highest level, Level 3, includes the customer’s feedback to achieve the best results and highest quality images. Application-specific requirements are implemented through a feedback loop. The ISP parameter setting is tested in the field and optimised in stages until the maximum quality is achieved. If needed, further tuning of each individual ISP parameter can be performed according to subjective application requirements making this vision product unique in the market.

Conclusion

More vision applications are looking to ISPs to augment and improve image quality and do analytics at the edge. These devices can offload much of the rough processing from the host CPU/GPU ensuring the most efficient system overall and, in some cases, removing their need entirely. To use these ISPs effectively and get the most out of their vision system, these companies are looking to FRAMOS to provide modules and components to speed up development times and tune their systems to get the best images possible. This is allowing them to provide custom vision solutions to their customers that is unique in the marketplace and differentiates them from their competition. FRAMOS’s one stop shop approach to providing modules, ISP tuning services, and imaging expertise, means that they can have a quality product quickly that means, or exceeds, their vision system requirements while reducing their development cycles and getting to market as fast as possible.

For more information, visit: https://go.framos.com/l/857823/2022-07-11/2t8xjk