Add reproducible tokbench v1 model inputs 1214884 verified lysandre HF Staff commited on about 21 hours ago
Add sentence-transformers all-MiniLM-L6-v2 and all-mpnet-base-v2 tokenizer.json fixtures (#11) 943303b sbrandeis HF Staff commited on 2 days ago
Add meta-llama/Llama-3.2-1B tokenizer.json fixture (#10) 54d012f sbrandeis HF Staff commited on 4 days ago
Make bench tokenizer configs byte-identical to their source models (#8) e20ac59 sbrandeis HF Staff commited on Jul 27
Upload fixtures/modalities/added_normalized_dense.txt with huggingface_hub 5fbfaa6 verified ArthurZ HF Staff commited on Jul 8
Upload fixtures/modalities/added_normalized_sparse.txt with huggingface_hub bfb22f0 verified ArthurZ HF Staff commited on Jul 8
Upload fixtures/modalities/added_special_dense.txt with huggingface_hub 91a83ec verified ArthurZ HF Staff commited on Jul 8
Upload fixtures/modalities/added_special_sparse.txt with huggingface_hub 7d74239 verified ArthurZ HF Staff commited on Jul 8
Add llama-2-7b-chat-hf tokenizer.json (canonical, array-safe merges) for pipeline oracle (#6) a59a428 sbrandeis HF Staff commited on Jul 7
Add README (repo layout) and xet/LFS rule for fixtures/ (#4) 400d310 sbrandeis HF Staff commited on Jul 2
Add slim byte-level tokenizer fixtures for tokenizers byte_level_fast tests (#3) 2853ff2 sbrandeis HF Staff commited on Jun 19
feat: upload `xnli.txt` to assert validity of the whitespace split (#2) bf38880 ArthurZ HF Staff mcpotato HF Staff commited on Apr 20
