Hardware-in-the-loop End-to-end Optimization of
Camera Image Processing Pipelines

Ali Mosleh
Avinash Sharma
Emmanuel Onzon
Fahim Mannan
Nicolas Robidoux
Felix Heide

CVPR 2020 (Oral)




(a)
(b)

We demonstrate that it is possible to directly optimize low-level hardware ISP pipelines for an end-to-end domain-specific loss. (a) An overview of the proposed hardware-in-the-loop method for multi-objective end-to-end ISP optimization. (b) Hyperparameter optimization of the ARM Mali-C71 hardware ISP for automotive object detection using the proposed hardware-in-the-loop method; left: sample images ISP outputs during the optimization proces followed by automotive object detection using [1], right: mAP and mAR measured on the test set at each hyperparameter setting explored during the optimmization.


Commodity imaging systems rely on hardware image signal processing (ISP) pipelines. These low-level pipelines consist of a sequence of processing blocks that, depending on their hyperparameters, reconstruct a color image from RAW sensor measurements. Hardware ISP hyperparameters have a complex interaction with the output image, and therefore with the downstream application ingesting these images. Traditionally, ISPs are manually tuned in isolation by imaging experts without an end-to-end objective. Very recently, ISPs have been optimized with 1st-order methods that require differentiable approximations of the hardware ISP. Departing from such approximations, we present a hardware-in-the-loop method that directly optimizes hardware image processing pipelines for end-to-end domainspecific losses by solving a nonlinear multi-objective optimization problem with a novel 0th-order stochastic solver directly interfaced with the hardware ISP. We validate the proposed method with recent hardware ISPs and 2D object detection, segmentation, and human viewing as end-to-end downstream tasks. For automotive 2D object detection, the proposed method outperforms manual expert tuning by 30% mean average precision (mAP) and recent methods using ISP approximations by 18% mAP.




Paper

Ali Mosleh, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, Felix Heide

Hardware-in-the-loop End-to-end Optimization of Camera Image Processing Pipelines

CVPR 2020 (Oral)

[Paper]
[Supplement]
[Bibtex]


Selected Results

Hyperparameter Optimization for Software ISPs on RAWs synthesized from existing RGB datasets
Default ISP Hyperparameters
ISP Optimized for Perceptual Image Quality
Hardware-in-the-loop End-to-end Optimization (this work)












Automotive object detection on KITTI using [1]

The imagees obtained with hyperparameters optimized with the proposed hardware-in-the-loop method have a lower perceptual quality compared to those obtained with the ISP optimized for perceptual quality, however, more cars are correctly detected.



Default ISP Hyperparameters
ISP Optimized for Perceptual Image Quality
Hardware-in-the-loop End-to-end Optimization (this work)


















Instance segmentation on COCO using [3]

While the task-specific-optimized ISP do not achieve the same perceptual quality as the one optimized for perceptual image quality, it substantially improves the downstream image understanding task.


Hyperparameter Optimization of Hardware ISPs on Captured RAW Data

Default ISP Hyperparameters
Expert-tuned ISP Hyperparameters for Perceptual Image Quality
End-to-end ISP Optimization with ISP Approximation from Tseng et al. [4]
Hardware-in-the-loop ISP End-to-end Optimization (this paper)
























Automotive object detection using [1] on real-world Sony IMX249 captures processed with the ARM Mali-C71 hardware ISP.

The proposed hardware-in-the-loop end-to-end detection loss optimization approach outperforms default ISP hyperparameters, ISP expert-tuned for perceptual image quality, and ISP optimized using a differentiable approximation [4].



Hyperparameter Optimization of Hardware ISPs for Perceptual Image Quality
Expert Manually Tuned ISP Hyperparameters
ISP Optimized for Perceptual Image Quality using ISP Approximation Approach [4]
ISP Optimized for Perceptual Image Quality using the Proposed Hardware-in-the-loop End-to-end Optimization



AR0231AT CMOS sensor captures processed with the OnSemi AP0202AT hardware ISP for perceptual image quality.

The proposed hardware-in-the-loop optimization approach outperforms default ISP hyperparameters, and ISP optimized using a differentiable approximation [4].




References