Новая система DAAAM обеспечивает автономным роботам точное понимание пространства и времени

Enabling autonomous systems to answer spatiotemporal queries like “where did you last see the red screwdriver?” requires internal memory representations that are geometrically precise, semantically rich, and computationally efficient. Two dominant paradigms have emerged to address this. Metric-semantic maps, such as three-dimensional (3D) scene graphs, ground objects in three-dimensional (3D) space but typically sacrifice semantic detail for real-time performance, or rely on prohibitively slow per-object vision-language model queries for richer descriptions. No time to finish this now? Download a free PDF here! Conversely, view-based methods annotate video frames using multimodal large language models (LLMs), achieving expressive detail but lacking sufficient 3D grounding, resulting in spatial inconsistencies across frames.  To bridge this gap, the paper introduces DAAAM, a framework that constructs a hierarchical 4D scene graph combining real-time metric-semantic mapping with highly detailed, geometrically grounded natural language descriptions, achieving both spatial consistency and online performance. Building the 4D Scene Graph DAAAM constructs a real-time 4D scene graph from red, green, blue, depth (RGB-D) input and camera poses, serving as a spatiotemporal metric-semantic memory for embodied agents. The pipeline begins with an active window that segments and tracks objects across frames using the fast segment anything model (Fast-SAM) and Bot-Sort, while Khronos lifts these fragments into 3D at sensor rate. To efficiently generate detailed descriptions without sacrificing speed, a two-stage optimization first finds the minimum frames needed to observe all fragments, then maximizes view quality for annotation. Selected frames are batched and processed through the describe anything model (DAM) to produce rich natural-language descriptions, along with contrastive language-image pre-training (CLIP) and sentence embeddings for semantic search. Background places are similarly extracted based on traversability and semantically annotated. A backend globally optimizes node positions using factor graphs and reconciles fragments with similar geometry and features into temporally consistent histories. Regions are formed by clustering the places graph and summarizing object descriptions using farthest-point sampling and an LLM. Finally, a tool-calling agent leverages this 4D scene graph for retrieval-augmented reasoning, enabling spatial and temporal queries over the memory representation. Experimental Validation and Analysis DAAAM is evaluated on spatiotemporal question answering (SQA) and sequential task grounding to demonstrate its flexibility and real-time capability. For SQA, the NaVQA benchmark on the CODa dataset is used, comprising 210 question-answer pairs across binary, spatial, and temporal categories over short, medium, and long sequences. The model outperforms baselines including ReMEmbR and ConceptGraphs, particularly excelling in long-sequence and temporal reasoning, confirming that geometrically structured 4D scene graphs enhance spatiotemporal understanding. However, limitations in the NaVQA benchmark were identified, including confounded in-context examples, view-based rather than object-centric position annotations, and noisy labels. To address these, an improved object-centric NaVQA (OC-NaVQA) dataset was curated with actual 3D object positions and full-sequence evaluation up to 35.8 minutes and 1.64 km. On OC-NaVQA, DAAAM achieves state-of-the-art results, improving question accuracy by 53.6%, position errors by 21.9%, and temporal errors by 21.6% over baselines, while scaling effectively to large environments. For sequential task grounding on the SG3D dataset, DAAAM significantly outperforms Hydra and the specialized ASHiTA method, achieving a 27.8% improvement in task accuracy, demonstrating that detailed open-vocabulary descriptions combined with precise spatial grounding generalize well to task planning. Ablation studies validate the benefit of concatenating CLIP and sentence embeddings for retrieval, showing complementary information capture. Explicit DAM descriptions aid compositional reasoning, particularly for positional and temporal queries, while hierarchical region clustering improves all metrics. Runtime analysis confirms DAAAM operates in real time at a 10 Hz sensor rate on CODa sequences using a single NVIDIA RTX 5090 GPU. Batch processing accelerated DAM inference by an order of magnitude, making it suitable for mobile robotics deployment. Closing the Gap in Spatial Memory DAAAM presents a significant advancement in spatiotemporal memory for embodied agents by successfully bridging the longstanding gap between rich semantic understanding and real-time geometric grounding. By decoupling geometric tracking from expensive semantic annotation through an optimization-based frame selection and batch inference strategy, the framework constructs hierarchical 4D scene graphs with detailed natural-language descriptions while maintaining real-time performance at 10 Hz. Experimental results show strong performance in both spatiotemporal question answering and sequential task grounding, with improvements over existing methods.  While current limitations include occasional description inaccuracies from the underlying DAM model and throughput constraints for highly dynamic platforms, the open-source release of DAAAM provides a practical foundation for future research into large-scale, long-horizon scene understanding. As multimodal models continue to evolve, frameworks like DAAAM will be instrumental in enabling autonomous systems to understand and interact with complex, large-scale, dynamic environments over extended time periods. Journal Reference Gorlo, N., Schmid, L., and Carlone, L. (2025). Describe Anything Anywhere At Any Moment. ArXiv.org. https://arxiv.org/pdf/2512.00565. Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.