TeleEgo-Source
Source Videos and Time-Aligned Transcripts for TeleEgo

Official source release for
TeleEgo: Benchmarking Egocentric AI Assistants in the Wild

Overview

TeleEgo is a multimodal benchmark for evaluating egocentric AI assistants in realistic, long-duration settings. It contains recordings from five participants over three days and covers four broad themes: Work & Study, Lifestyle & Routines, Social Activities, and Outings & Culture.

This repository complements the main TeleEgo benchmark release by providing the pre-merge, per-session source videos together with the official synchronized textual annotations:

• *_speech.srt: spoken conversations and utterances, with speaker labels when available.
• *_narration.srt: visual narrations describing activities, attended objects, scene content, and other salient events.

Dataset Summary

Item	Count
Participants	5
Recording days per participant	3
Source video sessions	51
Speech transcript files	51
Narration transcript files	51
Total repository size	approximately 340.5 GiB

Session counts by participant:

Participant	Videos	Speech SRTs	Narration SRTs
P1	9	9	9
P2	10	10	10
P3	11	11	11
P4	11	11	11
P5	10	10	10

Repository Structure

TeleEgo-Source/
├── source_videos/
│   ├── P1/
│   │   ├── D1-P1-am-ppt_video.mp4
│   │   ├── D1-P1-pm-1-meeting-1_video.mp4
│   │   └── ...
│   ├── P2/
│   ├── P3/
│   ├── P4/
│   └── P5/
└── teleego_srts/
    ├── P1/
    │   ├── D1-P1-am-ppt/
    │   │   ├── D1-P1-am-ppt_narration.srt
    │   │   └── D1-p1-am-ppt_speech.srt
    │   └── ...
    ├── P2/
    ├── P3/
    ├── P4/
    └── P5/

The session identifier generally follows this pattern:

D<day>-P<participant>-<time-of-day>-<activity>

For example, D1-P1-am-ppt denotes participant P1's presentation-related session on the morning of day 1.

Annotation Format

Both annotation streams use the standard SubRip (.srt) format:

1
00:00:41,533 --> 00:00:43,800
P1: 去打印一份吧

Each entry contains a sequence number, a start/end time, and annotation text. Timestamps are aligned to the corresponding source video session.

Speech transcripts

Speech transcripts capture verbal communication in the recordings. According to the TeleEgo paper, the speech was automatically transcribed and then manually verified and annotated with speaker identities. These files are intended for audiovisual understanding, conversation analysis, speaker-aware retrieval, and long-term memory research.

Visual narrations

Visual narrations describe ongoing activities and salient environmental details. They cover actions, attended objects, spatial relationships, scene context, and other visually grounded events. Each narration is timestamped and aligned with its corresponding video segment.

Matching Videos and Transcripts

Use the participant, day, and activity/session tokens to associate a video with its transcript directory. For example:

source_videos/P1/D1-P1-am-ppt_video.mp4
teleego_srts/P1/D1-P1-am-ppt/D1-P1-am-ppt_narration.srt
teleego_srts/P1/D1-P1-am-ppt/D1-p1-am-ppt_speech.srt

Download

The repository is large. We recommend using the Hugging Face CLI and downloading only the subset you need.

Install or update the client:

pip install -U "huggingface_hub[cli]"

Download the complete dataset:

hf download H-oliday/TeleEgo-Source \
  --repo-type dataset \
  --local-dir TeleEgo-Source

Relationship to the Main TeleEgo Release

Repository	Contents	Recommended use
TeleEgo-Source	Per-session source videos, speech SRTs, and narration SRTs	Transcript-based research, customized preprocessing, and rebuilding continuous streams
TeleEgo	Merged participant videos, QA annotations, and timeline files	Reproducing the official benchmark and evaluation protocol
TeleEgo GitHub	Evaluation code, metrics, and model runners	Running benchmark evaluations

The source repository does not replace the main benchmark release. Use the two datasets together when you need both the official QA benchmark and the source-level transcript annotations.

Intended Uses

TeleEgo-Source is intended for research on topics including:

• Long-form egocentric video understanding
• Multimodal and cross-modal retrieval
• Speech-aware video understanding
• Long-term and ultra-long-term memory
• Streaming video-language models and agents
• Temporal grounding and event localization
• Dialogue, interaction, and activity understanding

The data should not be used to identify participants, infer sensitive personal attributes, or support surveillance or other harmful applications.

Privacy and Responsible Use

As described in the paper, the recordings were de-identified by blurring faces, removing speech from non-participants, and masking sensitive visual or audio content. Nevertheless, this dataset contains real-world first-person recordings and natural conversations. Users are responsible for handling the data carefully, respecting participant privacy, and complying with applicable laws and institutional requirements. Do not attempt to reverse de-identification or re-identify any individual.

Citation

If you use TeleEgo or TeleEgo-Source in your research, please cite the paper:

@article{yan2025teleego,
  title   = {TeleEgo: Benchmarking Egocentric AI Assistants in the Wild},
  author  = {Yan, Jiaqi and Ren, Ruilong and Liu, Jingren and Xu, Shuning and Wang, Ling and Wang, Yiheng and Zhong, Xinlin and Wang, Yun and Zhang, Long and Chen, Xiangyu and Sun, Changzhi and Luo, Jixiang and Zhang, Dell and Sun, Hao and Zhang, Chi and Li, Xuelong},
  journal = {arXiv preprint arXiv:2510.23981},
  year    = {2025}
}

License

This repository is currently marked as Apache-2.0 in its Hugging Face dataset metadata. Please also review the licensing and usage terms published with the official TeleEgo project before using or redistributing the data.

Contact

We welcome questions and feedback about the dataset, annotations, and usage. For general questions or reproducible issues, please open an issue in the official TeleEgo repository. For inquiries that are better handled privately, please contact us at kakibluee@gmail.com.