# Qwen 3.8 how to run locally: vLLM and GGUF steps

> Serve Qwen3.8-27B with vLLM on one or two RTX 5090 cards, or use a GGUF quantization, with the settings Alibaba recommends.

URL: https://claudegpt.si/news/qwen-3-8-how-to-run-locally  
Published: 2026-10-30T10:50:51.704Z  
Author: ClaudeGPT Si Newsroom  
Section: Guides  
Topics: Qwen, Alibaba, Open-weight models, vLLM  
Publisher: ClaudeGPT Si

## Key takeaways

- The vLLM recipe serves Qwen3.8-27B in NVFP4 on one RTX 5090, which needs the --enforce-eager flag.
- The NVFP4 build needs 24.6 GiB of GPU memory, per the recipe.
- Unsloth publishes GGUF files from IQ1 up to BF16 for tools that read that format.

Qwen 3.8 how to run locally comes down to one choice: download Qwen3.8-27B from Hugging Face and serve it with vLLM, or pick a quantized GGUF file if you have less GPU memory. This guide follows the [model card](https://huggingface.co/Qwen/Qwen3.8-27B) and the [vLLM recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-27B) as read on October 30, 2026. We did not run these commands ourselves.

The model is open: Qwen3.8-27B uses the Apache 2.0 license, as we explain in [Is Qwen 3.8 open source](https://claudegpt.si/news/is-qwen-3-8-open-source).

## What you need first

- A GPU with enough memory. The vLLM recipe says the NVFP4 version needs 24.6 GiB. A single RTX 5090 has 31.4 GiB usable. The recipe also lists two RTX 5090 cards, an FP8 setup across four GPUs, and a single Ascend 950PR.
- vLLM 0.17.0 or later and transformers 5.8.0 or later, per the recipe.
- An OpenAI-compatible client if you want to call the server, such as the OpenAI Python SDK.

## Option 1: serve it with vLLM

The model card recommends a dedicated serving engine such as SGLang, vLLM or TokenSpeed for real workloads. For one RTX 5090, the vLLM recipe gives this command:

```bash
vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enforce-eager \
  --reasoning-parser qwen3
```

The recipe says the NVFP4 build on a single 5090 needs `--enforce-eager`, or startup fails with an out-of-memory error. With two RTX 5090 cards it uses `unsloth/Qwen3.8-27B-NVFP4`, `--tensor-parallel-size 2` and the full `--max-model-len 262144`.

The recipe calls `--reasoning-parser qwen3` effectively required, because it separates the model's reasoning from its answer. Add `--enable-auto-tool-choice --tool-call-parser qwen3_coder` if you want tool calling.

## Option 2: use a GGUF file

If you use llama.cpp-style tools, Unsloth publishes [Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF). It lists many sizes, from IQ1 and Q2 files up to Q8 and BF16. Smaller files use less memory and lose more accuracy. Unsloth links its own run guide at unsloth.ai/docs/models/qwen3.8 for the exact commands.

## Option 3: call it through the API

Point the OpenAI SDK at your server:

```bash
pip install -U openai
export OPENAI_BASE_URL='your-base-url'
export OPENAI_API_KEY='your-api-key'
```

Then call `client.chat.completions.create` with `model="Qwen/Qwen3.8-27B"`. The card says thinking is on by default and `reasoning_effort` accepts xhigh (default), medium or low.

## Which option to pick

Pick vLLM if you want an API for several users or for coding agents, and you have a recent NVIDIA card. Pick a GGUF file if you want to run the model in a desktop tool on whatever memory you have. Pick the API route if you already run a server and only need a client.

## Check that the server works

Once vLLM prints that it is serving, send one request with the OpenAI SDK and a short prompt, using the model name you passed to `vllm serve`. If the reply arrives with the reasoning separated from the answer, the reasoning parser is working. If the reasoning text shows up inside the answer, the `--reasoning-parser qwen3` flag is missing. Start with a short context length, such as 32,768, and raise it once the first request succeeds, because the key-value cache grows with context and can push a card out of memory.

## Settings the card recommends

| Mode | Temperature | top_p | top_k | presence_penalty |
| --- | --- | --- | --- | --- |
| Thinking | 1.0 | 0.95 | 20 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 1.5 |

To turn thinking off, the recipe passes `{"enable_thinking": false}` in `chat_template_kwargs`.

## Common problems

- Out-of-memory at startup on one GPU: add `--enforce-eager`, shorten `--max-model-len`, or choose a smaller quantization. The recipe notes that `--gpu-memory-utilization` does not fix the single-5090 case.
- Slower answers at low effort: the card warns that in multi-turn agent tasks, a lower `reasoning_effort` can cause more failures and retries, and raise total time.
- Context beyond 262,144 tokens: the card says to enable YaRN scaling, which vLLM, SGLang and TokenSpeed support.

Alibaba also says a hosted Qwen3.8-27B is coming to Qwen Cloud, with no date given. Until then, local serving or a third-party host is the way to run it.

## Frequently asked questions

### Qwen 3.8 how to run locally?

Serve Qwen3.8-27B with vLLM or SGLang, or load a quantized GGUF file from Unsloth. The vLLM recipe has commands for one and two RTX 5090 cards.

### Is Qwen 3.8 27B multimodal?

Yes. The model card describes native image and video understanding in addition to text.

### Is Qwen 3.8 27B released?

Yes. The weights are on Hugging Face under Apache 2.0. A hosted version on Qwen Cloud is listed as coming soon.

## Sources

- [Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B), Hugging Face (Qwen)
- [Qwen3.8-27B recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-27B), vLLM
- [Qwen3.8-27B GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF), Hugging Face (Unsloth)