Neural Fields in Robotics: A Survey

1Toyota Research Institute 2University of Bristol 3Nvidia 4University of Freiburg 5Georgia Institute of Technology
Image description

Timeline of Neural Fields in Robotics paper showing key papers over the years divided into 5 application areas i.e. Pose Estimation, Manipulation, Navigation, Physics and Autonomous Driving. This survey paper presents a comprehensive summary of over 200+ papers covering 4 key fields i.e. Occupancy networks, Signed Distance Function, Neural Radiance Fields and 3D Gaussian Splatting.

Abstract

Neural Fields have emerged as a transformative approach for 3D scene representation in computer vision and robotics, enabling accurate inference of geometry, 3D semantics, and dynamics from posed 2D data. Leveraging differentiable rendering, Neural Fields encompass both continuous implicit and explicit neural representations enabling high-fidelity 3D reconstruction, integration of multi-modal sensor data, and generation of novel viewpoints.

This survey explores their applications in robotics, emphasizing their potential to enhance perception, planning, and control. Their compactness, memory efficiency, and differentiability, along with seamless integration with foundation and generative models, make them ideal for real-time applications, improving robot adaptability and decision-making. This paper provides a thorough review of Neural Fields in robotics, categorizing applications across various domains and evaluating their strengths and limitations, based on over 200 papers.

First, we present four key Neural Fields frameworks: Occupancy Networks, Signed Distance Fields, Neural Radiance Fields, and Gaussian Splatting. Second, we detail Neural Fields' applications in five major robotics domains: pose estimation, manipulation, navigation, physics, and autonomous driving, highlighting key works and discussing takeaways and open challenges. Finally, we outline the current limitations of Neural Fields in robotics and propose promising directions for future research.


Overview

This survey paper discusses a large variety of state-of-the-art Neural Field methods that enable robotics applications
in pose estimation, manipulation, navigation, physics, and autonomous driving. Images adapted from individual papers,
please see our paper for references.

Image description

Taxonomy

Neural Fields in Robotics taxonomy of selected key neural fields papers divided based on robotics application areas.
Please see our paper pdf for detailed references.

Image description

BibTeX

If you find our survey paper useful, please consider citing:

@misc{irshad2024neuralfieldsroboticssurvey,
          title={Neural Fields in Robotics: A Survey}, 
          author={Muhammad Zubair Irshad and Mauro Comi and Yen-Chen Lin and Nick Heppert and Abhinav Valada and Rares Ambrus and Zsolt Kira and Jonathan Tremblay},
          year={2024},
          eprint={2410.20220},
          archivePrefix={arXiv},
          primaryClass={cs.RO},
          url={https://arxiv.org/abs/2410.20220}, 
    }