EgoLife: Towards Egocentric Life Assistant

🔼 This 3D model of the EgoLife house shows the placement of 15 standard cameras and two mmWave radar devices within the house. The cameras are spread throughout the common areas to capture third-person perspectives of participant activities. The mmWave radar devices, highlighted in red, are located on the second floor for additional motion capture data. Color-coded lines represent 10-minute movement tracks for each participant, indicating their movements within the space during the study.

read the caption

Figure 2: 3D reconstruction of the shared house using Aria Multi-MPS [1], showcasing the locations of 15 Exo cameras in the common area and 2 mmWave devices (highlighted in red) on the second floor. Color-coded 10-minute participant traces are also displayed.

🔼 This figure presents a visual overview of the activities undertaken by six participants over a week-long period within the EgoLife project. Each horizontal band represents a single participant’s activities, and the entire timeline spans seven days. The timeline is divided into 20-minute intervals, with each interval color-coded and labeled with an icon to represent the type of activity (e.g., social, housekeeping, cooking, etc.). A legend provides a complete list of the 14 activity categories used, along with the total number of times each activity occurred throughout the week. Notably, the activity categorization was automatically determined using GPT-4, which analyzed both the audio and video data associated with each 20-minute interval.

read the caption

Figure 3: The Activity Timeline of the EgoLife Dataset. It visualizes the activity timeline of six participants over one week. Each block represents a 20-minute interval, color-coded and marked with icons for different activities. The legend shows 14 activity categories with their total occurrence counts. The categorization is automatically performed using GPT-4o on visual-audio captions with timestamps.

🔼 The figure illustrates the data processing pipeline for the EgoLife project. It begins with the synchronization of data streams from Meta Aria glasses and GoPro cameras using the EgoSync codebase. This synchronized data (video, audio, and IMU) then undergoes privacy protection via the EgoBlur module. Next, the data is processed by EgoCaption for dense caption generation and by EgoTranscript for transcription. Finally, the processed and annotated data is fed into the EgoLifeQA system for further analysis and task completion.

read the caption

Figure 4: The Overview of Data Process Pipeline. The pipeline synchronizes multi-source data (video, audio, IMU) from Aria glasses and GoPro cameras using EgoSync codebase, processes them through privacy protection (EgoBlur), dense captioning (EgoCaption), and transcription (EgoTranscript) modules, ultimately feeding into the EgoLifeQA system.

🔼 This figure showcases the five question types within the EgoLifeQA benchmark designed to assess an egocentric AI assistant’s capabilities. These question types cover entity logging (tracking objects), event recall (remembering past events), task tracking (monitoring ongoing tasks), and two human-centric problem areas: habit analysis (identifying personal habits) and relationship understanding (analyzing social interactions). Each example displays a multiple-choice question with its answer, along with timestamps linking the answer back to relevant evidence in the recorded data. The timestamps are at least 5 minutes before the question to demonstrate the AI’s ability to use long-term contextual information. Vertical black lines mark question timestamps; colored arcs connect to relevant evidence timestamps.

read the caption

Figure 5: Question Types and Examples in the EgoLifeQA Benchmark. We design five types of questions to evaluate egocentric assistants’ capabilities in entity logging, event recall, task tracking, and human-centric problems (habit analysis and relationship understanding). Each example includes a multiple-choice Q&A with supporting evidence from timestamps at least 5 minutes prior to the question. Black vertical lines indicate question timestamps, while colored curved lines connect to relevant evidence timestamps.

🔼 The figure presents a statistical overview of the EgoLifeQA dataset, showing the distribution of different question types and their temporal characteristics. It features two visualizations: a pie chart illustrating the proportion of questions that require audio input and a bar chart depicting the daily distribution of questions across question types, with color intensity indicating the certification length (time span between the question and relevant evidence). The dataset comprises 3000 QAs (500 per participant).

read the caption

Figure 6: Statistics of EgoLifeQA. We gathered 500 long-context QAs per participant, totaling 3K QAs. The sum of QAs for each question type is reported. In the pie chart, darker segments indicate the proportion of questions requiring audio. The bar chart presents the daily count of QAs per question type, with brightness levels reflecting 4-level certification length [11] (from <<<2h to >>>24h).

🔼 EgoButler is a two-stage system. First, EgoGPT processes egocentric video clips to generate detailed visual-audio captions. Then, EgoRAG retrieves relevant information from these captions based on user queries, enabling it to answer questions requiring temporal reasoning across multiple days. The figure shows an example of how EgoButler answers a breakfast-related question by extracting keywords, retrieving evidence from past days’ captions, and generating a context-aware response.

read the caption

Figure 7: The EgoBulter Architecture. The system comprises (a) a Captioning Stage powered by EgoGPT for dense visual-audio understanding of egocentric clips, and (b) a Question Answering Stage utilizing EgoRAG for memory retrieval and response generation. The example demonstrates temporal reasoning across multiple days, with keyword extraction, evidence retrieval, and context-aware answer generation for a breakfast-related query.

🔼 Figure 8 demonstrates a qualitative comparison of EgoGPT and Gemini-1.5-Pro within the EgoButler framework. The top half shows a side-by-side comparison of captions generated by each model for the same 30-second egocentric video clip. This highlights EgoGPT’s superior performance in generating personalized captions that accurately reflect the content of the video, while also exhibiting fewer hallucinations (inventions of facts not present in the source video). The bottom half presents a question-answering scenario. It illustrates how EgoRAG, a retrieval-augmented generation component, leverages temporal information from the video to correctly identify and utilize relevant time segments to answer the question accurately.

read the caption

Figure 8: Qualitative Comparison of EgoGPT and Gemini-1.5-Pro under the EgoButler Framework. The top section compares captions from two models on a 30-second clip: EgoGPT excels in personalization and hallucinates less on the egocentric videos. The bottom section features a question that is answered by the clip, showcasing EgoRAG’s skill in pinpointing relevant time slots and key clues.

🔼 This figure provides a comprehensive overview of existing egocentric datasets. It presents a timeline of dataset releases, highlighting the evolution of data modalities (video, RGB-D, gaze, IMU, 3D scans, touch, point clouds), annotation types (activity labels, bounding boxes, pixel-level masks, timestamps, narrations, QA pairs), and the expansion of domains captured (sports, kitchen activities, assembly, daily life, interactions). The figure also summarizes key features and statistics of each dataset, such as the number of videos, duration, and types of annotations. This visualization helps readers understand the context of the EgoLife dataset by showing its position relative to other egocentric datasets in terms of scope, complexity, and features.

read the caption

Figure A1: The Overview of Egocentric Datasets. The figure summarizes the domain, modality, annotation type, release time, dataset statistics, and other aspects of datasets, providing a comprehensive view of existing egocentric datasets.

🔼 This table compares the EgoLifeQA benchmark with other related egocentric video benchmarks. The key differentiator of EgoLifeQA is its use of ultra-long video footage (spanning a week, totaling 266 hours across six participants), which allows for the exploration of novel tasks such as habit discovery and the analysis of complex relational interactions between people over extended periods. A ‘clip’ is defined as a continuous segment of video with a narrative thread. The duration per clip for EgoLife is derived from the total footage length and is not a fixed value.

read the caption

Table 2: Related Work for EgoLifeQA Benchmark. The EgoLifeQA dataset is distinguished by its ultra-long video footage and certificate length, facilitating novel tasks such as habit discovery and relational interaction pattern analysis (see Figure 5 for details). Note on Dur./Clip: A clip is defined as a session with narrative continuity. For the EgoLife dataset, this value is derived from 266 hours of retained footage distributed across six participants.

🔼 The EgoIT-99K dataset is a curated collection of nine classic egocentric video datasets used for fine-tuning the EgoGPT model. These datasets were selected to provide a wide range of activities, contexts, and interaction types, building upon the LLaVA-OneVision base model. The table details the composition of EgoIT-99K, specifying for each source dataset: its duration, the number of videos (including those with audio), and the types of question-answering tasks used (Video Captioning (VC), Audio-Video Captioning (AVC), Multiple Choice Questions (MCQ), Multi-Round Questions (MRC), and Image Question-Answering (IQA)). The inclusion of diverse question types is intended to comprehensively evaluate the model’s capabilities.

read the caption

Table 3: Dataset Composition of EgoIT-99K. We curated 9 classic egocentric video datasets and leveraged their annotations to generate captioning and QA instruction-tuning data for fine-tuning EgoGPT, building on the LLaVA-OneVision base model [55]. #AV means the number of videos with audio used for training. QAs include multiple types - VC: Video Captioning, AVC: Audio-Video Captioning, MCQ: Multiple Choice Questions, MRC: Multi-Round Questions, IQA: Image Question-Answering.

🔼 This table presents a quantitative comparison of the EgoGPT model’s performance against other state-of-the-art video-language models on several established egocentric vision benchmarks. The benchmarks evaluate different aspects of video understanding, including video captioning and question answering. The results highlight EgoGPT’s strengths and weaknesses relative to these other models, offering insights into its capabilities and limitations in processing egocentric video data.

read the caption

Table 4: Performance of EgoGPT. The table compares EgoGPT with state-of-the-art commercial and open-source models on existing egocentric benchmarks.

🔼 This table presents a quantitative comparison of EgoGPT’s performance against other state-of-the-art models on the EgoLifeQA benchmark. To ensure a fair comparison, the EgoGPT model was swapped out with each competing model within the EgoButler pipeline, keeping all other parameters consistent. The results are presented in terms of accuracy across five different question types within the EgoLifeQA benchmark: EntityLog, EventRecall, HabitInsight, RelationMap, and TaskMaster. A crucial consideration is that models providing captions for the EgoLifeQA benchmark utilized a video sampling rate of 1 frame per second (FPS). The table highlights the relative strengths and weaknesses of each model across various question types, revealing which models perform better at certain aspects of egocentric life assistance.

read the caption

Table 5: Performance comparison of EgoGPT with state-of-the-art models on EgoLifeQA benchmarks. For a fair comparison on EgoLifeQA, EgoGPT was replaced with the corresponding models in the EgoButler pipeline to evaluate their performance under the same conditions. Models that provide captions for EgoLifeQA use 1 FPS for video sampling.

🔼 This table demonstrates the significant performance improvement achieved by integrating EgoRAG (Retrieval-Augmented Generation) into a video-language model for long-context question answering. It compares the performance of EgoRAG combined with EgoGPT (a multimodal video-language model) against the performance of Gemini-1.5-Pro and EgoGPT alone, all evaluated on the task of answering questions requiring various lengths of contextual information (certification lengths). The results show EgoRAG’s substantial benefit, particularly for questions demanding ultra-long temporal context understanding (over 24 hours). The comparison with Gemini-1.5-Pro and EgoGPT alone on shorter video segments (half-hour) highlights EgoRAG’s effectiveness in handling long-context scenarios.

read the caption

Table 6: Effectiveness of EgoRAG. Integrating EgoRAG significantly enhances video-language models’ performance in long-context question answering, especially for questions requiring longer certification lengths. For comparison, we evaluate Gemini-1.5-Pro and EgoGPT on a half-hour video segment, limiting their answers to this timeframe.

🔼 This ablation study investigates the impact of caption quality on the EgoButler system’s performance. Different EgoRAG memory banks were constructed using captions generated by various EgoGPT models, including human-annotated captions as a baseline. All response generation models used the same EgoGPT (EgoIT-99K+D1) to maintain a fair comparison. The results demonstrate the effect of caption quality on the overall performance of EgoButler.

read the caption

Table 7: Ablation Study on EgoGPT. We construct different EgoRAG memory banks using generated captions from EgoGPT variants. The first three rows use captions from human annotations as a reference. All response generation models utilize EgoGPT (EgoIT-99K+D1) to ensure fair comparison. The result indicates how caption quality affects of EgoBulter performance.