How to run Gemma 4 locally with Ollama, Transformers or vLLM

To run Gemma 4 locally, install Ollama and run ollama run gemma4. That downloads the default model, which Ollama lists at 6.6 GB to 9.5 GB, and opens a chat in your terminal. If you want a Python setup, Google's Hugging Face model cards show how to load the models with Transformers. For a server, vLLM has a recipe. Which one you pick depends mostly on how much memory you have.

This guide is for developers and curious users with a laptop or a single GPU. By the end you will have a Gemma 4 model answering prompts on your own machine. Steps come from the official Gemma docs, the Hugging Face model card, the Ollama library page and the vLLM recipe, as read on October 30, 2026. We did not run the commands ourselves.

Pick a Gemma 4 size first

Google's Gemma docs list five sizes: E2B and E4B (small "effective parameter" models for phones and laptops), a 12B model, a 26B A4B mixture-of-experts model, and a 31B dense model. A mixture-of-experts model uses only part of its network for each word, but all of its weights still have to be loaded. The small models have a 128K-token context window and the larger ones 256K.

Google gives these approximate memory figures for loading the weights. They include about 20% overhead, but not the memory the context window needs.

Pick the largest model whose 4-bit figure fits in your RAM or video memory with room to spare. Ollama downloads quantized (compressed) files, so its sizes are close to the 4-bit column.

How to run Gemma 4 with Ollama

This is the shortest path, and the one Google's Ollama page describes.

1. Install Ollama from ollama.com/download. On Windows, run the installer. On Mac, move the app to Applications. On Linux, use the install script.
2. Check the install with ollama --version. If the command isn't found, add the Ollama executable to your system path. That is the only fix the Gemma docs list.
3. Download a model: ollama pull gemma4, then ollama list to confirm.
4. Chat with it: ollama run gemma4 "roses are red". Leave the prompt out for an interactive session.

To pick a size, add a tag. The Ollama library page lists these, with the download sizes it shows:

Ollama also serves a local API on port 11434:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "roses are red"
}'

For image input, the Gemma docs say to include the file path in the prompt on the command line, or to add a list of base64-encoded images named images to the API request.

How to run Gemma 4 with Hugging Face Transformers

The model card for google/gemma-4-31B-it gives this setup. Swap the model ID for a smaller one if the 31B model won't fit.

pip install -U transformers torch accelerate

from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "google/gemma-4-31B-it"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short joke about saving RAM."},
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=False
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=1024)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
processor.parse_response(response, prefix=inputs["input_ids"])

The card recommends temperature 1.0, top_p 0.95 and top_k 64. To turn on thinking mode, set enable_thinking=True in the chat template. The weights are stored in BF16, so the full-precision memory figures in the table above apply here.

How to serve Gemma 4 with vLLM

The vLLM recipe targets GPU servers. It installs a pre-release build of vLLM on NVIDIA GPUs:

uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

Then start a server. For a single GPU, the recipe's quick start uses the E4B model:

vllm serve google/gemma-4-E4B-it \
  --max-model-len <n_of_tokens>

The recipe says E2B and E4B need one GPU with 24 GB or more, the 12B needs one with 40 GB or more, and the 26B A4B and 31B need one with 80 GB. It runs the 31B on two GPUs with --tensor-parallel-size 2 when each has less memory. To use thinking mode and tool calling, it adds --enable-auto-tool-choice, --reasoning-parser gemma4, --tool-call-parser gemma4 and a chat template file. Lower --max-model-len if you run out of memory, since the context window takes memory too.

The recipe page is marked as a historical reference. The current recipes are at recipes.vllm.ai.

Gemma 4 not working in LM Studio or Ollama

Google's docs list LM Studio as a way to run Gemma 4, but give no LM Studio steps and no troubleshooting for it. The official pages cover these causes of a model that won't load or runs badly.

• Not enough memory. Compare the model's size with the table above. Google's figures exclude the context window, so a model that just fits may still fail with a long prompt. Pick a smaller size or a 4-bit version.
• A very long context. Memory use grows with the context length. In vLLM, lower --max-model-len.
• The command isn't found. Run ollama --version. If it fails, fix your path.
• Thinking or tool calls missing in vLLM. The recipe says the reasoning and tool-call parsers are required for those features.
• Odd output in Transformers. Use the sampling settings from the model card: temperature 1.0, top_p 0.95, top_k 64.

If the error is specific to an app, check that app's own documentation or issue tracker. We did not find an official fix for LM Studio.

What Gemma 4 can do once it runs

All sizes take text and image input. The E2B, E4B and 12B models also take audio and video. Google lists configurable thinking modes, built-in function calling and native system prompts. The 31B model card lists the license as Apache 2.0, and Google's docs say the family permits commercial use.

Google's Gemma docs say they are deprecated and point to gemma.ai for the new version, so page layouts may change.