LVDU — Long Video Data for Understanding

Working storage for the video-understanding datasets used across this PhD. One top-level directory per dataset; every dataset follows the same shape so tooling written against one works against all of them.

Datasets

Convention

<dataset>/
├── README.md            # dataset card: provenance, how it was built, usage
├── annotations/         # json/jsonl, upstream schema kept unchanged
├── metadata/
│   ├── manifest.parquet     # record -> media + stratification columns
│   └── shard_index.parquet  # member -> (shard, offset, nbytes)
├── shards/<kind>/*.tar  # uncompressed tar, ~2 GiB each
└── <extras>/            # embeddings, oversized loose media, ...

Rules that make this work:

• Shards are uncompressed. A member is then a contiguous byte range, so any file is one pread away and can be decoded from memory. Compression would force sequential scans.
• Member names are the upstream relative paths. Annotations never need rewriting, and a shard can be unpacked with plain tar if something goes wrong.
• shard_index.parquet is the contract. Anything that reads media goes through it; nothing hardcodes shard filenames.
• Oversized files stay loose. Anything big enough that buffering it in RAM would hurt (>64 MiB for video) sits outside the shards, with an index row whose shard is empty.

Buckets are not versioned — files here can be overwritten or deleted in place. The citable, version-pinned copy of the annotations lives in a separate dataset repo per dataset; this bucket holds the bulk media.