Qwen 3.8 how to run locally: vLLM and GGUF steps

Qwen 3.8 how to run locally comes down to one choice: download Qwen3.8-27B from Hugging Face and serve it with vLLM, or pick a quantized GGUF file if you have less GPU memory. This guide follows the model card and the vLLM recipe as read on October 30, 2026. We did not run these commands ourselves.

The model is open: Qwen3.8-27B uses the Apache 2.0 license, as we explain in Is Qwen 3.8 open source.

What you need first

• A GPU with enough memory. The vLLM recipe says the NVFP4 version needs 24.6 GiB. A single RTX 5090 has 31.4 GiB usable. The recipe also lists two RTX 5090 cards, an FP8 setup across four GPUs, and a single Ascend 950PR.
• vLLM 0.17.0 or later and transformers 5.8.0 or later, per the recipe.
• An OpenAI-compatible client if you want to call the server, such as the OpenAI Python SDK.

Option 1: serve it with vLLM

The model card recommends a dedicated serving engine such as SGLang, vLLM or TokenSpeed for real workloads. For one RTX 5090, the vLLM recipe gives this command:

vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enforce-eager \
  --reasoning-parser qwen3

The recipe says the NVFP4 build on a single 5090 needs --enforce-eager, or startup fails with an out-of-memory error. With two RTX 5090 cards it uses unsloth/Qwen3.8-27B-NVFP4, --tensor-parallel-size 2 and the full --max-model-len 262144.

The recipe calls --reasoning-parser qwen3 effectively required, because it separates the model's reasoning from its answer. Add --enable-auto-tool-choice --tool-call-parser qwen3_coder if you want tool calling.

Option 2: use a GGUF file

If you use llama.cpp-style tools, Unsloth publishes Qwen3.8-27B-GGUF. It lists many sizes, from IQ1 and Q2 files up to Q8 and BF16. Smaller files use less memory and lose more accuracy. Unsloth links its own run guide at unsloth.ai/docs/models/qwen3.8 for the exact commands.

Option 3: call it through the API

Point the OpenAI SDK at your server:

pip install -U openai
export OPENAI_BASE_URL='your-base-url'
export OPENAI_API_KEY='your-api-key'

Then call client.chat.completions.create with model="Qwen/Qwen3.8-27B". The card says thinking is on by default and reasoning_effort accepts xhigh (default), medium or low.

Which option to pick

Pick vLLM if you want an API for several users or for coding agents, and you have a recent NVIDIA card. Pick a GGUF file if you want to run the model in a desktop tool on whatever memory you have. Pick the API route if you already run a server and only need a client.

Check that the server works

Once vLLM prints that it is serving, send one request with the OpenAI SDK and a short prompt, using the model name you passed to vllm serve. If the reply arrives with the reasoning separated from the answer, the reasoning parser is working. If the reasoning text shows up inside the answer, the --reasoning-parser qwen3 flag is missing. Start with a short context length, such as 32,768, and raise it once the first request succeeds, because the key-value cache grows with context and can push a card out of memory.

Settings the card recommends

To turn thinking off, the recipe passes {"enable_thinking": false} in chat_template_kwargs.

Common problems

• Out-of-memory at startup on one GPU: add --enforce-eager, shorten --max-model-len, or choose a smaller quantization. The recipe notes that --gpu-memory-utilization does not fix the single-5090 case.
• Slower answers at low effort: the card warns that in multi-turn agent tasks, a lower reasoning_effort can cause more failures and retries, and raise total time.
• Context beyond 262,144 tokens: the card says to enable YaRN scaling, which vLLM, SGLang and TokenSpeed support.

Alibaba also says a hosted Qwen3.8-27B is coming to Qwen Cloud, with no date given. Until then, local serving or a third-party host is the way to run it.