# What is LoRA: how low-rank adaptation fine-tunes AI models cheaply

> LoRA freezes a model and trains two thin matrices per layer, cutting GPT-3's fine-tuning checkpoint from 350 GB to 35 MB. QLoRA and a new AWS test show its reach and its limits.

URL: https://claudegpt.si/news/what-is-lora  
Published: 2026-11-19T02:58:08.488Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: Fine-tuning, Research papers, Language models, Microsoft, AWS, Quantization  
Publisher: ClaudeGPT Si

## Key takeaways

- LoRA freezes a model's weights and trains two small matrices per layer whose product is the change to that layer.
- On GPT-3 175B, the 2021 paper reports about 10,000 times fewer trainable parameters, three times less GPU memory, and accuracy that matched or beat full fine-tuning.
- QLoRA stores the frozen model in 4 bits, bringing fine-tuning of a 65B model from more than 780 GB of GPU memory to 48 GB.

The short answer to what is LoRA: low-rank adaptation is a way to fine-tune a large AI model by training a small add-on instead of the whole model. The original weights stay frozen, and LoRA learns two thin matrices per layer whose product is the change to that layer. In the [original 2021 paper](https://arxiv.org/abs/2106.09685) from Microsoft researchers, that shrank a fine-tuned GPT-3 175B checkpoint from 350 GB to about 35 MB.

That makes fine-tuning possible without a cluster of GPUs. [QLoRA](https://arxiv.org/abs/2305.14314) pushed it further in 2023 by fine-tuning a 65-billion-parameter model on a single 48 GB GPU. And on November 18, 2026, AWS published [a set of experiments](https://aws.amazon.com/blogs/machine-learning/full-fine-tuning-fft-on-sagemaker-ai-serverless-model-customization/) showing where full fine-tuning still does better.

This is LoRA the machine learning technique. LoRa, the long-range radio standard behind LoRaWAN, is unrelated.

## The problem LoRA solves

[Fine-tuning](https://claudegpt.si/news/what-is-fine-tuning) means continuing to train a pretrained model on your own examples so it gets better at one task, domain or style. Full fine-tuning updates every parameter, which has two costs.

The first is storage. Every fine-tuned copy is as big as the original: for GPT-3, 175 billion parameters per task.

The second is memory. The Adam optimizer keeps extra numbers for every parameter it updates. The paper puts full fine-tuning of GPT-3 175B at 1.2 TB of GPU memory.

Earlier shortcuts had their own problems. Adapter layers, small extra layers inserted into the network, slow down every answer: on GPT-2 medium at batch size one, the paper measured them adding 20.7% to 30.3% to a forward pass. Prefix tuning, which learns special tokens added to the prompt, eats part of the context window.

## What is LoRA doing inside the model

A layer in a [transformer model](https://claudegpt.si/news/what-is-a-transformer-model) is mostly a big grid of numbers, a weight matrix. Fine-tuning learns a change to that grid. The LoRA authors bet that this change is simple: that it has a low "rank", meaning it can be described with far fewer numbers than the grid holds.

So LoRA learns the change as the product of two thin matrices, B and A. If the layer's matrix is d by k, B is d by r and A is r by k, where r, the rank, is a small number you choose.

The savings are large. GPT-3's layers are 12,288 numbers wide, so one full matrix holds about 151 million numbers. With a rank of 4, B and A together hold 98,304.

Three details make it work:

- **It starts from the original model.** B starts at zero, so the add-on changes nothing until training moves it.
- **Only A and B are trained.** Frozen weights need no gradients and no optimizer state, which is where the memory saving comes from.
- **It costs nothing at inference.** After training, B times A can be added into the original matrix, and the model runs as fast as before. Or you keep adapters separate and swap them per task: 100 adapted GPT-3 models would need about 354 GB instead of 35 TB.

The paper applied LoRA only to the attention layers' query and value matrices. For GPT-3 on its test tasks, a rank as low as 1 or 2 was enough. The authors warn that a small rank won't suit every task, such as one in a language the model wasn't trained on.

## What the original paper found

The Microsoft team, with Edward Hu as first author, tested LoRA on RoBERTa, DeBERTa, GPT-2 and GPT-3 175B, and released code at [github.com/microsoft/LoRA](https://github.com/microsoft/LoRA). On GPT-3 175B, LoRA matched or beat full fine-tuning on all three test sets:

| Method on GPT-3 175B | Trainable parameters | WikiSQL accuracy | MultiNLI-matched accuracy |
| --- | --- | --- | --- |
| Full fine-tuning | 175,255.8M | 73.8% | 89.5% |
| LoRA | 4.7M | 73.4% | 91.7% |
| LoRA | 37.7M | 74.0% | 91.6% |

The paper also reports about 10,000 times fewer trainable parameters, GPU memory down from 1.2 TB to 350 GB, and training about 25% faster: 43.1 tokens per second per V100 GPU against 32.5. One limit it names: once adapters are merged into the weights, serving several tasks in one batch gets harder.

## QLoRA: LoRA on a 4-bit model

LoRA still needs the full model in GPU memory. QLoRA, from Tim Dettmers and colleagues at the University of Washington in May 2023, shrank that part.

It stores the frozen model in 4 bits per weight instead of 16, a form of [quantization](https://claudegpt.si/news/what-is-quantization-in-ai), and trains LoRA adapters on top. The paper says this brings fine-tuning a 65B LLaMA model from more than 780 GB of GPU memory to 48 GB. It adds a 4-bit number format suited to model weights (NF4), "double quantization" that saves about 3 GB on a 65B model, and paged optimizers that spill to CPU memory when the GPU runs out.

It also corrected a LoRA habit. Adapters on only the query and value matrices did not match full fine-tuning on large models. Adapters on every linear layer did.

The team's Guanaco models reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours on one GPU. Read that with care: GPT-4 judges that benchmark, and the authors write that current chatbot benchmarks can't be trusted to rank chatbots accurately. They also did not show that QLoRA matches full 16-bit fine-tuning at 33B and 65B. Code is at [github.com/artidoro/qlora](https://github.com/artidoro/qlora).

## Where LoRA falls short

AWS's post, by solutions architect Sandeep Raveesh-Babu, launches full fine-tuning in SageMaker AI Serverless Model Customization. It argues that as datasets grow past tens of thousands of examples, LoRA hits a ceiling a higher rank doesn't fix.

AWS fine-tuned Qwen3-4B on 99,000 coding examples from the Magicoder-Evol-Instruct-110K dataset. On 1,000 code tasks graded by Amazon Nova Pro, the base model scored 82.0%, LoRA 86.0% and full fine-tuning 90.0%.

The same post shows the other side. In another comparison, LoRA scored 81.5% and full fine-tuning 79.5%. Full fine-tuning with a learning rate of 5e-5 or higher dropped MMLU, a general-knowledge test, to 3.6%. Both methods scored below the base model on MATH. AWS concludes that full fine-tuning "isn't universally superior to LoRA".

That fits the original paper's caveat. LoRA bets the change you need is simple, and teaching a model a lot of new material can break that bet. But this is one 4-billion-parameter model, one coding dataset and an Amazon judge model, in a post launching a full fine-tuning product.

## When to use LoRA

LoRA or QLoRA is the sensible default when GPUs are scarce, when you want many variants of one base model, or when your dataset is small or medium. Each variant is megabytes, not gigabytes.

Full fine-tuning is worth testing with a large, clean dataset, a task far from what the model knows, and budget to tune learning rates. In SageMaker, switching is one setting, `TrainingType.LORA` or `TrainingType.FULL`.

If the model needs to answer from your documents, [retrieval-augmented generation](https://claudegpt.si/news/what-is-rag-retrieval-augmented-generation) looks facts up at question time without retraining. To shrink a big model into a small one, look at [distillation](https://claudegpt.si/news/what-is-distillation-ai).

## Frequently asked questions

### What is LoRA in AI?

LoRA, or low-rank adaptation, is a fine-tuning method that keeps a model's original weights frozen and trains small add-on matrices instead. The result is a file of megabytes that changes the model's behavior, which can be merged into the weights so it adds no delay at inference.

### What is the difference between LoRA and QLoRA?

QLoRA is LoRA trained on top of a base model compressed to 4 bits per weight. That cuts the memory needed to hold the model, so the QLoRA paper fine-tuned a 65B model on a single 48 GB GPU.

### Is LoRA better than full fine-tuning?

Not always. The original paper found LoRA matched or beat full fine-tuning on GPT-3 175B, while an AWS test in November 2026 found full fine-tuning scored higher on a large coding dataset, though it also damaged general knowledge when the learning rate was too high.

### What is LoRA vs RAG?

LoRA changes how the model behaves by training it on examples. Retrieval-augmented generation leaves the model alone and looks up relevant documents when a question comes in, which suits facts that change often.

## Sources

- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685), arXiv (Hu et al., Microsoft)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314), arXiv (Dettmers et al., University of Washington)
- [Full Fine-Tuning (FFT) on SageMaker AI Serverless Model Customization](https://aws.amazon.com/blogs/machine-learning/full-fine-tuning-fft-on-sagemaker-ai-serverless-model-customization/), AWS
- [microsoft/LoRA](https://github.com/microsoft/LoRA), Microsoft
- [artidoro/qlora](https://github.com/artidoro/qlora), GitHub
