File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1145/3446791
- Scopus: eid_2-s2.0-85108668941
- WOS: WOS:000667456500009
- Find via
Supplementary
- Citations:
- Appears in Collections:
Article: Differentiable Compound Optics and Processing Pipeline Optimization for End-To-end Camera Design
Title | Differentiable Compound Optics and Processing Pipeline Optimization for End-To-end Camera Design |
---|---|
Authors | |
Keywords | Compound optics computational imaging deep learning end-To-end image processing optics design |
Issue Date | 2021 |
Citation | ACM Transactions on Graphics, 2021, v. 40, n. 2, article no. 3446791 How to Cite? |
Abstract | Most modern commodity imaging systems we use directly for photography-or indirectly rely on for downstream applications-employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today's compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss' linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-To-end losses. To this end, we propose a learned, differentiable forward model for compound optics and an alternating proximal optimization method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as Zemax. We can thus assess our method across many camera system designs and end-To-end applications. We validate our approach in an automotive camera optics setting-together with hardware ISP post processing and detection-outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-Tuning methods qualitatively and quantitatively, across all domain-specific applications tested. |
Persistent Identifier | http://hdl.handle.net/10722/315195 |
ISSN | 2023 Impact Factor: 7.8 2023 SCImago Journal Rankings: 7.766 |
ISI Accession Number ID |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Tseng, Ethan | - |
dc.contributor.author | Mosleh, Ali | - |
dc.contributor.author | Mannan, Fahim | - |
dc.contributor.author | St-Arnaud, Karl | - |
dc.contributor.author | Sharma, Avinash | - |
dc.contributor.author | Peng, Yifan | - |
dc.contributor.author | Braun, Alexander | - |
dc.contributor.author | Nowrouzezahrai, Derek | - |
dc.contributor.author | Lalonde, Jean François | - |
dc.contributor.author | Heide, Felix | - |
dc.date.accessioned | 2022-08-05T10:18:00Z | - |
dc.date.available | 2022-08-05T10:18:00Z | - |
dc.date.issued | 2021 | - |
dc.identifier.citation | ACM Transactions on Graphics, 2021, v. 40, n. 2, article no. 3446791 | - |
dc.identifier.issn | 0730-0301 | - |
dc.identifier.uri | http://hdl.handle.net/10722/315195 | - |
dc.description.abstract | Most modern commodity imaging systems we use directly for photography-or indirectly rely on for downstream applications-employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today's compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss' linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-To-end losses. To this end, we propose a learned, differentiable forward model for compound optics and an alternating proximal optimization method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as Zemax. We can thus assess our method across many camera system designs and end-To-end applications. We validate our approach in an automotive camera optics setting-together with hardware ISP post processing and detection-outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-Tuning methods qualitatively and quantitatively, across all domain-specific applications tested. | - |
dc.language | eng | - |
dc.relation.ispartof | ACM Transactions on Graphics | - |
dc.subject | Compound optics | - |
dc.subject | computational imaging | - |
dc.subject | deep learning | - |
dc.subject | end-To-end image processing | - |
dc.subject | optics design | - |
dc.title | Differentiable Compound Optics and Processing Pipeline Optimization for End-To-end Camera Design | - |
dc.type | Article | - |
dc.description.nature | link_to_subscribed_fulltext | - |
dc.identifier.doi | 10.1145/3446791 | - |
dc.identifier.scopus | eid_2-s2.0-85108668941 | - |
dc.identifier.volume | 40 | - |
dc.identifier.issue | 2 | - |
dc.identifier.spage | article no. 3446791 | - |
dc.identifier.epage | article no. 3446791 | - |
dc.identifier.eissn | 1557-7368 | - |
dc.identifier.isi | WOS:000667456500009 | - |