Datasets:

hf-internal-testing
/

tokenizers-test-data

Modalities:

Formats:

Size:

Libraries:

Dataset card Data Studio Files Files and versions

tokenizers-test-data

358 MB

4 contributors

History: 16 commits

lysandre's picture

lysandre HF Staff

Add reproducible tokbench v1 model inputs

1214884 verified about 2 hours ago

fixtures
Add sentence-transformers all-MiniLM-L6-v2 and all-mpnet-base-v2 tokenizer.json fixtures (#11) 1 day ago
models
Add reproducible tokbench v1 model inputs about 2 hours ago
.gitattributes

3.38 kB
Add reproducible tokbench v1 model inputs about 2 hours ago
README.md

1.66 kB
Add README (repo layout) and xet/LFS rule for fixtures/ (#4) 2 months ago
albert-base-v1-tokenizer.json

1.31 MB
Add all tokenizers test artifacts 5 months ago
bert-base-uncased-vocab.txt

232 kB
Add all tokenizers test artifacts 5 months ago
bert-base-uncased.json

466 kB
xet

Xet-track full tokenizer configs (pipeline bench) (#5) 2 months ago
bert-wiki-slim.json

8.14 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
bert-wiki.json

439 kB
Add all tokenizers test artifacts 5 months ago
big.txt

6.49 MB
Add all tokenizers test artifacts 5 months ago
deepseek-v4-slim.json

305 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
deepseek-v4.json

6.37 MB
xet

Xet-track full tokenizer configs (pipeline bench) (#5) 2 months ago
dummy-unigram-special_tokens-train.txt

1.11 kB
Add all tokenizers test artifacts 5 months ago
gemma-4-slim.json

170 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
gemma-4.json

32.2 MB
xet

Make bench tokenizer configs byte-identical to their source models (#8) about 1 month ago
glm-5.2-slim.json

116 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
glm-5.2.json

20.2 MB
xet

Make bench tokenizer configs byte-identical to their source models (#8) about 1 month ago
gpt-oss-slim.json

112 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
gpt-oss.json

27.9 MB
xet

Make bench tokenizer configs byte-identical to their source models (#8) about 1 month ago
gpt2-merges.txt

456 kB
Add all tokenizers test artifacts 5 months ago
gpt2-slim.json

106 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
gpt2-vocab.json

1.04 MB
Add all tokenizers test artifacts 5 months ago
gpt2.json

1.36 MB
xet

xet-track llama-2.json and gpt2.json (#7) about 2 months ago
llama-2.json

1.84 MB
xet

Make bench tokenizer configs byte-identical to their source models (#8) about 1 month ago
llama-3-slim.json

150 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
llama-3-tokenizer.json

9.09 MB
xet

Make bench tokenizer configs byte-identical to their source models (#8) about 1 month ago
mistral-small-4.json

17.1 MB
xet

Make bench tokenizer configs byte-identical to their source models (#8) about 1 month ago
openai-gpt-merges.txt

458 kB
Add all tokenizers test artifacts 5 months ago
openai-gpt-vocab.json

816 kB
Add all tokenizers test artifacts 5 months ago
roberta-base-merges.txt

456 kB
Add all tokenizers test artifacts 5 months ago
roberta-base-vocab.json

899 kB
Add all tokenizers test artifacts 5 months ago
roberta-slim.json

107 kB
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 3 months ago
roberta.json

1.36 MB
Add all tokenizers test artifacts 5 months ago
small.txt

7.44 kB
Add all tokenizers test artifacts 5 months ago
t5-base.json

1.39 MB
xet

Xet-track full tokenizer configs (pipeline bench) (#5) 2 months ago
tokenizer-wiki.json

668 kB
Add all tokenizers test artifacts 5 months ago
tokenizer.json

4.1 kB
Add all tokenizers test artifacts 5 months ago
unigram.json

259 kB
Add all tokenizers test artifacts 5 months ago
unigram_wagahaiwa_nekodearu.txt

1.12 MB
Add all tokenizers test artifacts 5 months ago
xnli.txt

18.4 MB
xet

feat: upload `xnli.txt` to assert validity of the whitespace split (#2) 5 months ago