# How to run Gemma 4 locally with Ollama, Transformers or vLLM

> Install steps from the official docs, plus the memory each Gemma 4 size needs and what to check when a model won't load.

URL: https://claudegpt.si/news/gemma-4-how-to-run-locally  
Published: 2026-10-30T18:50:50.410Z  
Author: ClaudeGPT Si Newsroom  
Section: Guides  
Topics: Google, Gemma, Ollama, Open-weight models, Local AI  
Publisher: ClaudeGPT Si

## Key takeaways

- The shortest route is Ollama: install it, then run `ollama run gemma4`.
- Gemma 4 comes in E2B, E4B, 12B, 26B A4B and 31B sizes, and Google lists 4-bit weights from 2.9 GB to 17.5 GB.
- Hugging Face Transformers and vLLM are the options for Python code and GPU servers.

To run Gemma 4 locally, install Ollama and run `ollama run gemma4`. That downloads the default model, which Ollama lists at 6.6 GB to 9.5 GB, and opens a chat in your terminal. If you want a Python setup, [Google's Hugging Face model cards](https://huggingface.co/google/gemma-4-31B-it) show how to load the models with Transformers. For a server, vLLM has a recipe. Which one you pick depends mostly on how much memory you have.

This guide is for developers and curious users with a laptop or a single GPU. By the end you will have a Gemma 4 model answering prompts on your own machine. Steps come from the official Gemma docs, the Hugging Face model card, the Ollama library page and the vLLM recipe, as read on October 30, 2026. We did not run the commands ourselves.

## Pick a Gemma 4 size first

[Google's Gemma docs](https://ai.google.dev/gemma/docs/core) list five sizes: E2B and E4B (small "effective parameter" models for phones and laptops), a 12B model, a 26B A4B mixture-of-experts model, and a 31B dense model. A mixture-of-experts model uses only part of its network for each word, but all of its weights still have to be loaded. The small models have a 128K-token context window and the larger ones 256K.

Google gives these approximate memory figures for loading the weights. They include about 20% overhead, but not the memory the context window needs.

| Model | BF16 (full) | 8-bit (SFP8) | 4-bit (Q4_0) |
| --- | --- | --- | --- |
| E2B | 11.4 GB | 5.7 GB | 2.9 GB |
| E4B | 17.9 GB | 8.9 GB | 4.5 GB |
| 12B | 26.7 GB | 13.4 GB | 6.7 GB |
| 26B A4B | 57.7 GB | 28.8 GB | 14.4 GB |
| 31B | 69.9 GB | 34.9 GB | 17.5 GB |

Pick the largest model whose 4-bit figure fits in your RAM or video memory with room to spare. Ollama downloads quantized (compressed) files, so its sizes are close to the 4-bit column.

## How to run Gemma 4 with Ollama

This is the shortest path, and the one [Google's Ollama page](https://ai.google.dev/gemma/docs/integrations/ollama) describes.

1. Install Ollama from [ollama.com/download](https://ollama.com/download). On Windows, run the installer. On Mac, move the app to Applications. On Linux, use the install script.
2. Check the install with `ollama --version`. If the command isn't found, add the Ollama executable to your system path. That is the only fix the Gemma docs list.
3. Download a model: `ollama pull gemma4`, then `ollama list` to confirm.
4. Chat with it: `ollama run gemma4 "roses are red"`. Leave the prompt out for an interactive session.

To pick a size, add a tag. The [Ollama library page](https://ollama.com/library/gemma4) lists these, with the download sizes it shows:

| Command | Download size | Context |
| --- | --- | --- |
| `ollama run gemma4:e2b` | 4.6 GB to 7.5 GB | 128K |
| `ollama run gemma4:e4b` | 6.6 GB to 9.5 GB | 128K |
| `ollama run gemma4:12b` | 7.7 GB to 8.0 GB | 256K |
| `ollama run gemma4:26b` | 16 GB to 19 GB | 256K |
| `ollama run gemma4:31b` | 19 GB to 20 GB | 256K |

Ollama also serves a local API on port 11434:

```bash
curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "roses are red"
}'
```

For image input, the Gemma docs say to include the file path in the prompt on the command line, or to add a list of base64-encoded images named `images` to the API request.

## How to run Gemma 4 with Hugging Face Transformers

The model card for `google/gemma-4-31B-it` gives this setup. Swap the model ID for a smaller one if the 31B model won't fit.

```bash
pip install -U transformers torch accelerate
```

```python
from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "google/gemma-4-31B-it"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short joke about saving RAM."},
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=False
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=1024)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
processor.parse_response(response, prefix=inputs["input_ids"])
```

The card recommends temperature 1.0, top_p 0.95 and top_k 64. To turn on thinking mode, set `enable_thinking=True` in the chat template. The weights are stored in BF16, so the full-precision memory figures in the table above apply here.

## How to serve Gemma 4 with vLLM

[The vLLM recipe](https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html) targets GPU servers. It installs a pre-release build of vLLM on NVIDIA GPUs:

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match
```

Then start a server. For a single GPU, the recipe's quick start uses the E4B model:

```bash
vllm serve google/gemma-4-E4B-it \
  --max-model-len <n_of_tokens>
```

The recipe says E2B and E4B need one GPU with 24 GB or more, the 12B needs one with 40 GB or more, and the 26B A4B and 31B need one with 80 GB. It runs the 31B on two GPUs with `--tensor-parallel-size 2` when each has less memory. To use thinking mode and tool calling, it adds `--enable-auto-tool-choice`, `--reasoning-parser gemma4`, `--tool-call-parser gemma4` and a chat template file. Lower `--max-model-len` if you run out of memory, since the context window takes memory too.

The recipe page is marked as a historical reference. The current recipes are at recipes.vllm.ai.

## Gemma 4 not working in LM Studio or Ollama

Google's docs list LM Studio as a way to run Gemma 4, but give no LM Studio steps and no troubleshooting for it. The official pages cover these causes of a model that won't load or runs badly.

- Not enough memory. Compare the model's size with the table above. Google's figures exclude the context window, so a model that just fits may still fail with a long prompt. Pick a smaller size or a 4-bit version.
- A very long context. Memory use grows with the context length. In vLLM, lower `--max-model-len`.
- The command isn't found. Run `ollama --version`. If it fails, fix your path.
- Thinking or tool calls missing in vLLM. The recipe says the reasoning and tool-call parsers are required for those features.
- Odd output in Transformers. Use the sampling settings from the model card: temperature 1.0, top_p 0.95, top_k 64.

If the error is specific to an app, check that app's own documentation or issue tracker. We did not find an official fix for LM Studio.

## What Gemma 4 can do once it runs

All sizes take text and image input. The E2B, E4B and 12B models also take audio and video. Google lists configurable thinking modes, built-in function calling and native system prompts. The 31B model card lists the license as Apache 2.0, and Google's docs say the family permits commercial use.

Google's Gemma docs say they are deprecated and point to gemma.ai for the new version, so page layouts may change.

## Frequently asked questions

### How much memory does Gemma 4 need?

Google lists 4-bit weights at 2.9 GB for E2B, 4.5 GB for E4B, 6.7 GB for 12B, 14.4 GB for 26B A4B and 17.5 GB for 31B. These include about 20% overhead but not the context window.

### Is Gemma 4 free to use locally?

The 31B model card lists the Apache 2.0 license, and Google's docs say Gemma 4 permits responsible commercial use. Running it locally has no per-token fee.

### Can Gemma 4 run without a GPU?

Google's Ollama page says Gemma can run on a laptop or small device without a GPU using quantized models. It gives no RAM or speed figures for that.

### Why is Gemma 4 not working in LM Studio?

Google's docs list LM Studio as an option but give no troubleshooting for it. The documented causes of failure are too little memory for the model size and a context window that is too long.

## Sources

- [Gemma 4 model overview](https://ai.google.dev/gemma/docs/core), Google AI for Developers
- [Run Gemma with Ollama](https://ai.google.dev/gemma/docs/integrations/ollama), Google AI for Developers
- [google/gemma-4-31B-it model card](https://huggingface.co/google/gemma-4-31B-it), Hugging Face
- [gemma4](https://ollama.com/library/gemma4), Ollama
- [Gemma 4 usage guide](https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html), vLLM