What it is
QLoRA, short for quantised LoRA, combines two ideas. Quantisation stores each number in fewer bits. LoRA freezes the base model and trains a small adapter beside it. Because the base is frozen, it never needs gradients or optimizer state, so it can be stored compactly; only the adapter has to stay in a precision suited to learning. QLoRA keeps the frozen base in 4 bits, a quarter of the usual 16, and trains the same LoRA adapter on top in 16-bit.
It was introduced by Dettmers et al. in 2023. Their paper fine-tuned a 65-billion-parameter model on a single 48 GB GPU, where regular 16-bit fine-tuning would have needed more than 780 GB, while preserving the task performance of 16-bit fine-tuning.
How it works
A 16-bit weight takes 2 bytes; a 4-bit one takes half a byte. Squeezing a trained model that hard without wrecking it relies on three ingredients from the paper:
- 4-bit NormalFloat (NF4). Trained weights roughly follow a bell curve centred on zero. NF4 places its 16 possible values to match that shape, so more of them fall where most weights are. In the paper it beat ordinary 4-bit integers and 4-bit floats.
- Double quantisation. Weights are quantised in small blocks, each with its own scaling constant, and those constants add up. QLoRA quantises the constants too, cutting their cost from about 0.5 to about 0.13 bits per weight: roughly 3 GB saved on a 65B model.
- Paged optimizers. When a long example causes a sudden memory spike, optimizer state is moved to ordinary CPU memory for a moment instead of the run crashing.
During the forward and backward passes, each layer's 4-bit weights are unpacked (dequantised) into 16-bit, used for that layer's arithmetic, then dropped. Gradients flow through the frozen base into the adapter, which is the only part that changes. At any moment, memory holds the whole model in 4 bits plus one layer's worth of 16-bit weights.
bytes per weight of the frozen base
16-bit (bf16) 2 bytes
4-bit NF4 + double-quantised constants ≈ 0.52 bytes (≈ 4.13 bits)
an 8B model, frozen base only
16-bit 8 × 10⁹ × 2 bytes ≈ 16 GB
4-bit 8 × 10⁹ × 0.52 bytes ≈ 4.1 GB- LoRA on a 16-bit base: about 16 GB of frozen weights alone. With the adapter, its optimizer state and activations, a common rule of thumb gives about 19 GB. It doesn't fit.
- QLoRA: the base shrinks to about 4–5 GB. A few parts, such as normalisation layers and often the embedding and output layers, usually stay in 16-bit. With the adapter and working memory the total is about 9 GB, which fits with room to spare.
- The adapter is exactly as large as a LoRA adapter of the same rank: tens of MB.
- The trade: each step is somewhat slower, because every layer is unpacked on the fly.
| Model | LoRA (16-bit base) | QLoRA (4-bit base) |
|---|---|---|
| 8B | ~19 GB | ~9 GB |
| 14B | ~33 GB | ~13 GB |
| 27–32B | ~60–70 GB | ~22–26 GB |
These are totals to train, including working memory, which is why they sit above the 4–5 GB the full fine-tuning page gives for a 7B model's 4-bit base before activations. A 36 GB laptop with shared memory, with about 27 GB free once the operating system has its share, can train a 27–32B model with QLoRA, against about 10B with plain LoRA.
What it costs, and when to use it
- Speed. Unpacking every layer on every step adds work, so training is somewhat slower than LoRA on the same model.
- Quality. Results are very close to LoRA's. The paper matched 16-bit fine-tuning across its benchmarks; small differences can still show on your own evaluation, so compare with an evaluation run rather than assuming.
- The result. It is still an adapter. Load it on the normal 16-bit base model or on its 4-bit copy; the adapter itself doesn't change size.
- When. Use QLoRA when the model you want doesn't fit with LoRA. If it fits, plain LoRA is faster for the same result.
The settings
Whether the frozen base is loaded in 4 bits. Off trains a normal LoRA adapter on a 16-bit base: faster per step, but it needs about three times the memory. On trains QLoRA, the choice when the model you want wouldn't otherwise fit. It applies to adapters only: a full fine-tune changes every weight, so every weight has to stay in a trainable precision.
The number format the 4-bit weights are unpacked into for each layer's arithmetic, and the format the adapter trains in. bf16 (bfloat16) keeps the range of 32-bit floats in 16 bits, so values rarely overflow; NVIDIA GPUs support it from the Ampere generation onwards. Older GPUs, such as the T4, usually compute in fp16 instead, which has less range.
Common mistakes
- Choosing QLoRA when the model already fits with LoRA. It only adds training time.
- Expecting a smaller result file. The adapter is the same size as a LoRA adapter; only training memory shrinks.
- Trying to use QLoRA to make a full fine-tune fit. It only applies to adapters.
- Starting from a model that was published already compressed (FP8, MXFP4). QLoRA quantises a full-precision model itself, so it starts from the BF16 version.
- Loading the adapter on a different model, or a different version of it, than the one it was trained on.
References and further reading
External sources open in a new tab.
- 01QLoRA: Efficient Finetuning of Quantized LLMsDettmers et al., 2023 · arXiv · arxiv.orgThe original paper: NF4, double quantisation, paged optimizers and the 65B-on-one-GPU result.
- 02Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRAHugging Face blog, 2023 · huggingface.coA readable walk-through of 4-bit loading and QLoRA training in practice.
- 03BitsandbytesHugging Face Transformers docs · huggingface.coThe 4-bit loading options: NF4, double quantisation and the compute dtype.
- 04QuantizationHugging Face PEFT docs · huggingface.co
- 05LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleDettmers et al., 2022 · arXiv · arxiv.orgThe earlier 8-bit work that QLoRA's quantisation library grew out of.