Key takeaways
- Post-training is the work done on a language model after pretraining: supervised fine-tuning, RLHF, preference optimization such as DPO, and reinforcement learning with verifiable rewards.
- In OpenAI's InstructGPT paper, people preferred answers from a 1.3 billion parameter post-trained model over the 175 billion parameter GPT-3.
- On November 18, 2026, Palo Alto Networks agreed to buy Applied Compute, a post-training and inference platform for open-weight models, in an all-stock deal valued at $4.5 billion.
Post-training is everything done to a language model after pretraining, the long first phase in which it learns to predict the next word from a huge pile of text. It is what turns a raw text predictor into a model that follows instructions and solves problems. The main tools are supervised fine-tuning, reinforcement learning from human feedback (RLHF), preference optimization such as DPO, and reinforcement learning with verifiable rewards.
It is also where money is moving this week. On November 18, 2026, Palo Alto Networks agreed to buy Applied Compute, which its release calls "a leading platform for post-training and inference of open-weight AI models," in an all-stock deal valued at $4.5 billion. A day earlier, Paris-based Remanence announced a $30 million seed round to post-train open-weight models for companies. And RadixArk released DaisyRose-27B, an open model made by post-training someone else's model.
Why pretraining is not enough
A pretrained model has read a lot, but it was only ever asked to continue text. OpenAI's InstructGPT paper, published in March 2022, made the point directly: predicting the next token on a web page is a different goal from following a user's instructions helpfully and safely.
InstructGPT closed that gap in three steps: show the model good answers, have people rank its own answers, then train it to give the answers people rank higher.
Step one: supervised fine-tuning
Supervised fine-tuning (SFT) trains the model on prompts paired with good responses, so it learns to imitate them. For InstructGPT, about 40 contractors wrote the demonstrations, and the SFT dataset held about 13,000 prompts.
OpenAI's model optimization docs recommend it for classification, output in a fixed format and fixing instruction-following failures. The same page now says OpenAI is winding down its fine-tuning platform, and new users can no longer sign up for it.
Step two: reinforcement learning from human feedback
Examples only teach a model to copy. RLHF teaches it from judgments: people compare the model's own answers, and the model learns from the comparison.
In InstructGPT, labelers ranked 4 to 9 answers to each prompt. Rankings from about 33,000 prompts trained a separate reward model, a network that scores an answer by how much a person would like it. Then reinforcement learning, with an algorithm called PPO, ran on about 31,000 prompts to push the model toward answers the reward model scored highly.
A model trained against a scorer can learn to please the scorer instead of the people behind it. InstructGPT added a penalty for drifting too far from the SFT model to limit that.
The result is the paper's best-known finding. People preferred answers from the 1.3 billion parameter InstructGPT model over those of the 175 billion parameter GPT-3, a model 100 times larger.
It had costs. Scores on some public benchmarks, such as SQuAD and DROP, fell after RLHF; the authors called this an alignment tax and reduced it by mixing in training on the original pretraining data. The model also still made simple mistakes, the abstract says.
Preference optimization: DPO skips the reward model
RLHF is heavy: a reward model, a reinforcement learning loop, and sampling from the model while it trains. Direct Preference Optimization (DPO), published in May 2023 by Rafael Rafailov and five co-authors, removes most of that.
DPO uses the same data, pairs of a preferred and a rejected answer, and trains on it directly with a simple classification loss. The paper shows this solves the same problem RLHF does, with the language model itself acting as the reward model. No separate scorer, no sampling during training.
In the paper's tests, DPO beat PPO-based RLHF at controlling the sentiment of generated text and matched or improved quality on summarization and single-turn dialogue, while being simpler to train. OpenAI's fine-tuning docs offer it, and Hugging Face's TRL library ships a DPO trainer.
Reinforcement learning with verifiable rewards
Human preferences are a fuzzy signal. For some tasks there is a sharper one: check whether the answer is right.
The Tulu 3 paper, released in November 2024 by Nathan Lambert and 21 co-authors, named this Reinforcement Learning with Verifiable Rewards (RLVR). The model gets a fixed reward only when its output is verified correct, such as the right answer to a math problem or an instruction constraint a program can check. Tulu 3 ran SFT, then DPO, then RLVR on Llama 3.1 base models at 8, 70 and 405 billion parameters, and released the data, code and weights.
DeepSeek-R1, published in January 2025 and later in Nature, pushed the idea further. For DeepSeek-R1-Zero, the team skipped SFT and applied reinforcement learning straight to the base model. The rewards were rules: one checked math answers and ran code through compilers, another required the reasoning to sit inside think tags. The team avoided learned reward models because they "are susceptible to reward hacking during large-scale reinforcement learning."
During training, R1-Zero's average pass@1 on the AIME 2024 math exam rose from 15.6% to 77.9%, and the model began to check its own work without being shown how. That written-out reasoning is the chain of thought reasoning models now produce.
What a post-training run looks like now
RadixArk's DaisyRose-27B, published November 18, 2026, shows the RL step aimed at agents. It is Qwen3.8-27B with the same architecture, tokenizer and 256K-token context window. Only the post-training is new.
RadixArk trained it with reinforcement learning on about 1,200 long agent tasks built with Eigent AI, each aimed at a mistake the base model makes and graded in a sandbox. The run took 18 days on 144 Nvidia H200 GPUs, about 63,000 GPU-hours. The weights are out under the Apache 2.0 license.
RadixArk reports that Terminal Bench 4.1 pass@1 rose from 5.6% to 12.0% over the base model. Other gains were small: GPQA Diamond went from 89.2 to 90.7, and Humanity's Last Exam (text only) from 33.9 to 34.3. These are the company's own numbers, not independent tests.
That is the honest shape of post-training. It moves the skill you train for a lot, and the rest a little.
Why companies are paying for it
With a closed model, post-training happens on the vendor's terms. With open-weight models, anyone with GPUs can run every step above on their own data.
Palo Alto Networks says the deal will let enterprises train, adapt and deploy open-weight models with security built in. The deal is expected to close in the company's fiscal third quarter of 2027.
Remanence's round, reported by Axios Pro and Tech.eu on November 17, was co-led by EQT Ventures and FirstMark. Its website describes RLVR applied to one company's work: trace where its agents fail, build tasks and verifiers in sandboxes that mirror its software, then post-train an open-weight model that runs in the customer's cloud or on its own servers.
Both bets rest on the same claim: a smaller open model, post-trained on one company's tasks, can do them well for less than a frontier model costs. DaisyRose's Terminal Bench jump shows it can work on a narrow task. Whether it holds across a real company's messy work is what these customers are about to test.
Frequently asked questions
What is the difference between pretraining and post-training?
Pretraining teaches a model to predict the next word from a huge amount of text. Post-training comes after and teaches it to follow instructions and solve tasks, using examples, human rankings or automatic checks of its answers.
Is fine-tuning the same as post-training?
Fine-tuning is one part of post-training. Supervised fine-tuning trains the model on example answers, while post-training also covers preference methods such as RLHF and DPO and reinforcement learning with verifiable rewards.
What is the difference between RLHF and DPO?
RLHF trains a separate reward model on human rankings and then uses reinforcement learning to push the model toward high-scoring answers. DPO trains directly on pairs of preferred and rejected answers with a simple classification loss, with no reward model and no sampling during training.
What is RLVR?
Reinforcement learning with verifiable rewards gives the model a reward only when its answer can be checked as correct, such as a right math answer or code that passes. The Tulu 3 paper named it, and DeepSeek-R1 used rule-based rewards of this kind to train reasoning.
Sources
- Training language models to follow instructions with human feedbackarXiv (Ouyang et al., OpenAI)
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelarXiv (Rafailov et al.)
- Tulu 3: Pushing Frontiers in Open Language Model Post-TrainingarXiv (Lambert et al.)
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningarXiv (DeepSeek-AI)
- Palo Alto Networks to Acquire Applied Compute, Bringing Secure Open-Weight AI to the EnterprisePalo Alto Networks
- DaisyRose-27B: Pushing the frontier of post-trainingRadixArk
- Model optimizationOpenAI
- TRL documentationHugging Face
- RemanenceRemanence
- Synthesia and Datadog CEOs back AI startup Remanence in $30M funding roundTech.eu
- Exclusive: Remanence raises $30M to help companies train their own modelsAxios



