GPT4Point++: Advancing Unified Point-Language Understanding and Generation

Abstract

<p>Multimodal Large Language Models (MLLMs) have made significant progress in 2D image-text tasks, but the 3D domain remains challenging. To bridge this gap, we introduce GPT4Point and its enhanced version, GPT4Point++, both of which are pioneering point-language multimodal models designed for 3D object understanding and generation. They excel in tasks such as 3D object recognition, 3D point cloud captioning and question answering. Additionally, GPT4Point is equipped with advanced capabilities for controllable 3D generation, and it can get high-quality results through a low-quality point-text feature that maintains geometric shapes and colors. GPT4Point's training consists of two stages: first, aligning point-text features, followed by integrating the LLM. Our advanced version GPT4Point++ simplifies this with a single, unified end-to-end training approach for improved performance. To support the substantial demand for 3D object-text pairs, we have developed Capverse, a point-language dataset annotation engine. Capverse constructs a large-scale database with diverse levels of text granularity by leveraging the Objaverse dataset. We established a comprehensive benchmark to assess 3D point-language understanding. Extensive evaluations show that GPT4Point and GPT4Point++ excel in both understanding and generation tasks. Additionally, GPT4Point effectively evaluates 3D object generation methods and demonstrates strong understanding of both individual objects and indoor scenes, highlighting its robustness.<br></p>