Brian PRO

CC's own CDX server does not index CC-NEWS at all.

that's why I'm doing this

I've tried all kinds of things on the hugging face space, I'm not going to try any new permutations of options. If I put 1 or -1 in max_open_files it still hangs in the same way (I don't think I literally tried 1). I reproduced locally using hf-mount and it hangs in the same way on the first query that tries to hit the db files.

here is a regenerated file with the new cdx-rocks-build command
https://huggingface.co/buckets/brian-learns/cdx-rocks-monthly

I've got this small demo index I'll use for testing with the hugging face. Once I get the space working, I'll see how large I can get the demo database
https://huggingface.co/buckets/brian-learns/cdx-rocks-demo

Common Crawl file naming conventions are part of their spec and documented on their dataset page on amazon, but the file sorting requirements seem worth documenting better in the dataset info.

/extent is only on the docker version, I'm not sure when I'll update the space. I started over in a new git repository and I think I need to do a force push, I might just build a new space. The current space hangs after about 40 consecutive hits because the files are in a bucket and I think the remote file system starts blocking up.

I'm not sure I agree about the two kinds of not found, I'm not sure how the index could know which the case is. https://index.commoncrawl.org does not have that, and https://archive.org/help/wayback_api.php is over their whole web archive (but does a whole different thing with robot.txt know as the Oakland Archive Policy). I don't think wayback tells if the URL got OAPed or if it was never crawled. I haven't looked at webrecorder's cdx index lookup, but that indexes your local WARC files from what I gather.

I wrote up a little fantasy spec for the cdx-rocks database definition https://github.com/brian-learns/cdx_rocks/wiki/database_definition -- I guess it could not hurt to include the information from the /extent endpoint formatted in the same json -- and it's not expensive to include, although to my mind it's redundant.

Looks like there might be some updates to the news crawler soon https://groups.google.com/g/common-crawl/c/SrAlK9l0lPA

I added /extent yesterday to the API and ccnget. It returns this from the one published index.

{
  "file_extent": 51101,
  "file_oldest": "crawl-data/CC-NEWS/2016/08/CC-NEWS-20160826124520-00000.warc.gz",
  "file_newest": "crawl-data/CC-NEWS/2026/07/CC-NEWS-20260731214950-00313.warc.gz"
}

The catalog file and the rocks db are implicitly versioned and need to match. I've only published one version really of the rocksdb, the first one was a test and I didn't announce it. It seems really rather neither here nor there if it gets recompressed or if the new warc_paths gets concatenated to the end, zstd should be able to be cat files together. If I publish more indexes I was thinking I'll create a little mini format for the index with the catalog in the root dir and the rocksdb in a subdir. Then it will just be one sync command for the whole directory with both artifacts.

Why would you make up WARC files that don't exist to inject in the middle? The WARC files are produced in chronological order and lexically sort in chronological order, as well the warc path files they publish have this same characteristic.

I don't understand what you are saying about the two type of misses. I think at a certain point news publishers started blocking the news crawler with robots.txt.

ccnget is just supposed to be dumb client for the API that also knows how to grab the content out of the WARC file with the byte offset method.

Thanks for your feedback. Please extend to me a generous reading, as I do to you. this is a tiny box in a tiny font and not easy to write in.

I think "!" is network byte order (big-endian), and the struct padding is only relevant to RAM

I think I'll change it to "!IQI".

Creating the cdxj files is what took awhile (almost a week on a c7g.2xlarge)-- I don't have any notes, but I don't think running the rocks index script took very long, maybe 30 to 60 minutes at the most?

I think I'll create two rocksdb -- one rocksdb with a subset of records that will fit in around 30G so it can run in the hugging face space. The second rocksdb will be the full size 75G and growing, but one has to run the docker to use it, and then uv run ccnget config set cdx-url http://0.0.0.0:7860/lookup. This is why I don't want ccnget to have to know about index versions.

Should probably pick a different port that the default for a hugging face space.

# --- RocksDB value format ---
# Each value is a 16-byte big-endian struct:
#   H (uint16) — WARC file ID (index into the catalog)
#   Q (uint64) — Byte offset within the WARC file
#   I (uint32) — Record length in bytes
# Total: 2 + 8 + 4 = 14 bytes (struct alignment pads to 16)
VALUE_FORMAT = "!HQI"
VALUE_SIZE = struct.calcsize(VALUE_FORMAT)

• max offset 1,073,731,048 -- 30 bits needed
• max length 1,048,157 -- 20 bits needed

seems like I could turn it to

#   I (uint32) — WARC file ID (index into the catalog)
#   Q (uint64) — Byte offset within the WARC file
#   I (uint32) — Record length in bytes
# Total: 4 + 8 + 4 = 16 bytes with no padding

and it would not really change the size of the db?