GGUF

GGUF is a single-file format used to store models for inference with GGML, containing the model metadata and tensors. It supports many quantized data types (refer to the quantization type table), which saves a significant amount of memory.

Load GGUF models

Install kernels, otherwise the packed path falls back to full dequantization at load.

pip install kernels

Weights stay packed when the Hub kernel transformers-community/ggml-quantization is available. The loader defaults to MPS when that kernel is present and runs matmuls directly on the packed blocks. If the kernel isn’t available, the model is dequantized at load.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)

The same load works for Qwen3.5 MoE.

moe_model_id = "unsloth/Qwen3.5-35B-A3B-GGUF"
moe_filename = "Qwen3.5-35B-A3B-Q4_K_M.gguf"

moe_model = AutoModelForCausalLM.from_pretrained(moe_model_id, gguf_file=moe_filename)
moe_tokenizer = AutoTokenizer.from_pretrained(moe_model_id, gguf_file=moe_filename)

The packed path currently supports Qwen3.5 and Qwen3.5 MoE. Packed loads use float32 automatically because it’s faster on MPS. If you set another dtype, the loader returns a warning. Other architectures go through the legacy loader.

Attention

On Metal, with kernels installed, attention can use ggml-attn, the same flash-attention kernel llama.cpp uses for decode and prefill.

model = AutoModelForCausalLM.from_pretrained(
    model_id, gguf_file=filename, attn_implementation="transformers-community/ggml-attn"
)

Dequantize

Dequantizing unpacks every weight at load time and gives back a plain dense model. It is the fallback whenever the fast, compressed path doesn’t apply. You can also ask for it explicitly with GgufConfig.

import torch

from transformers import AutoModelForCausalLM, GgufConfig

quantization_config = GgufConfig(dequantize=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, gguf_file=filename, quantization_config=quantization_config, dtype=torch.bfloat16
)

You get a regular dense model in the dtype you passed.

Architectures other than Qwen3.5 and Qwen3.5 MoE go through the legacy loader, which always dequantizes.

The legacy loader supports Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, and more.

Serve

transformers serve lists each .gguf file in a repository as its own model. A GGUF model is named <repo>:<file>.gguf, since a repository holds several quantizations and the id has to say which one to load. Requests name it the same way.

transformers serve unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf