Instructions to use josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1")
model = AutoModelForMultimodalLM.from_pretrained("josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1

SGLang

How to use josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1 with Docker Model Runner:
```
docker model run hf.co/josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1
```

ThinkingCap-Qwen3.6-27B INT4 AutoRound

A fast INT4 / W4A16 AutoRound quantization of bottlecapai/ThinkingCap-Qwen3.6-27B, prepared for local long-context agent serving with vLLM.

This is a practical first release: small enough to run comfortably on a 32 GB RTX 5090-class card, with Qwen3.6 thinking/tooling behavior preserved and Multi-Token Prediction (MTP) tensors kept in BF16 for speculative decoding.

A higher-quality v2 quant is planned later. The likely direction is a more quality-oriented recipe and/or stronger calibration, after we finish comparing the speed / context / quality tradeoffs.

Credits

Huge thanks to BottleCap AI for releasing ThinkingCap. Their checkpoint is the model here; this repo is only a community quantization of it.

Also, personally: BottleCap are local friends/neighbours — their offices are about 200 meters from me, and I have known them for years — so this one is especially fun to package properly. Please give the original model and team credit:

Base model: bottlecapai/ThinkingCap-Qwen3.6-27B
BottleCap AI: https://www.bottlecapai.com/

What is ThinkingCap?

From the original BottleCap release: ThinkingCap keeps the capability and style of Qwen3.6-27B while using substantially fewer thinking tokens on average. See the original model card for the full evaluation table, methodology, citation, and intended sampling settings.

This quant keeps the base/default sampling recipe:

temperature = 1.0
top_p       = 0.95
top_k       = 20
min_p       = 0.0

Quantization recipe

Field	Value
Source model	`bottlecapai/ThinkingCap-Qwen3.6-27B`
Quantizer	Intel AutoRound
AutoRound version	`0.14.0`
Scheme	W4A16 / INT4 weight-only
Bits	4
Group size	128
Symmetric	true
Packing format	`auto_round:auto_gptq`
Quantized block	`model.language_model.layers`
MTP tensors	restored/preserved as BF16
Local safetensors size	~19.0 GiB / ~20.4 GB
Disk usage	~20 GB

Notes:

This is the group-size 128 release. It prioritizes speed, smaller resident VRAM, and long-context KV headroom.
Smaller group sizes such as GS32/GS16 may improve quality but increase metadata size and reduce KV-cache room. A later v2 may explore that direction.
MTP/speculative-decoding tensors were restored from the BF16 source auxiliary tensors instead of leaving them packed/quantized.

Recommended long-context / agent serving config

This is the more tuned single-GPU configuration used during local testing:

vllm serve "$MODEL_ID" \
  --dtype half \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype turboquant_4bit_nc \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --max-long-partial-prefills 1 \
  --long-prefill-token-threshold 4096 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,3,4],"max_cudagraph_capture_size":4}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
  --port 8888 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Notes:

turboquant_4bit_nc is a vLLM KV-cache quantization mode, not the model weight quantization. It is useful for fitting long context.
--max-num-seqs 16 allows many shorter concurrent requests; it does not mean sixteen full 262k requests fit simultaneously.
--compilation-config ... PIECEWISE ... [1,2,3,4] keeps CUDA graphs enabled but caps graph capture sizes to save VRAM.
vLLM 0.24 rejected non-default concurrent partial prefill in this environment, so the command intentionally does not pass --max-num-partial-prefills 4.

BenchLocal snapshots

Local BenchLocal screenshots; preliminary smoke/quality signals, not a complete public evaluation suite. Run target: vllm 5090 on the local W4G128 checkpoint.

Hermes Agent benchmark

The main benchmark for this release: a Hermes-style agent suite covering memory, scheduling, delegation/recovery, procedural skills, and workspace orchestration.

Overall	memory_recall	scheduling_delivery	delegation_recovery_boundaries	skills_procedural_memory	workspace_orchestration
81	93	75	76	80	83

Outcome	Cases
Pass, 13/20	HA-01, HA-02, HA-04, HA-05, HA-06, HA-08, HA-09, HA-10, HA-11, HA-13, HA-15, HA-18, HA-20
Fail, 5/20	HA-07, HA-12, HA-14, HA-16, HA-17
Provider warning, 2/20	HA-03, HA-19

Other local BenchLocal runs

BenchLocal set	Overall
TC-15 / tool-call-style set	90
instructfollow-15	89
bugfind-15	73
cli-40	65

Performance notes

Local inference-benchmarker tests against vLLM used ~256 input tokens and ~64 output tokens with thinking disabled for content-throughput measurement. The tuned server reached approximately:

Arrival rate	Actual QPS	Output tok/s	Avg TTFT	Avg ITL	Success
1 req/s	~1.01	~65 tok/s	~79 ms	~9 ms	29/29
3 req/s	~2.96	~187 tok/s	~110 ms	~11 ms	87/87
6 req/s	~5.77	~354 tok/s	~111 ms	~18 ms	172/172
8 req/s	~7.29	~439 tok/s	~128 ms	~28 ms	217/217

These are local measurements on one machine and are not a formal quality benchmark. They are included to show that the quant is quick and usable under concurrent short-request traffic.

Compatibility

Known-good path:

vLLM 0.24.0
CUDA 13.x on RTX 5090
--kv-cache-dtype turboquant_4bit_nc
MTP speculative decoding with num_speculative_tokens=1

The model should also be usable through Transformers/GPTQModel-compatible AutoRound loaders, but the most exercised runtime for this artifact is vLLM.

Limitations

This is an INT4 quantization; it is not expected to exactly match BF16 quality.
Group size 128 is chosen for practicality. Future v2 variants may trade more disk/VRAM for quality.
The included performance numbers are local serving benchmarks, not broad model evaluations.
For long-context serving, KV-cache settings matter as much as model size. If you disable KV quantization, reduce context/concurrency accordingly.

Citation

If you use this quant, please cite and credit the original BottleCap model. Original citation from their model card:

@misc{,
  title  = {bottlecapai/ThinkingCap-Qwen3.6-27B},
  author = {Lasocki, Karol and Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Mikolov, Tomas},
  year   = {2026},
}

Downloads last month: 9,530

Safetensors

Model size

6B params

Tensor type

I32

BF16

F16

Model tree for josefprusa/ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1

Base model

Qwen/Qwen3.6-27B

Finetuned

bottlecapai/ThinkingCap-Qwen3.6-27B

Quantized

(40)

this model