CoworkOS agent harness leads two of four domains on 12 benchmarks

CoworkOS is an open-source harness for AI agents that do long, multi-step office work, described in a paper posted to arXiv by the team behind the RUC-NLPIR/CoworkOS repository. A harness is the software around a model: it decides what the model sees, which tools it can call and how its work is tracked. The authors report that, with the same model underneath, CoworkOS scores at or near the top of three rival harnesses on 12 benchmarks. The margins are small, and in two of four domains it is not first.

The problem: work that takes hours, not one prompt

Most agent harnesses are built for one job at a time, such as fixing a bug or answering a question. The paper argues that real coworker tasks have two demands at once. The output has to be good on complex tasks like a legal memo or a slide deck. And the agent has to keep watching and deciding over a long stretch of time, with a person able to step in.

CoworkOS splits the work in two. The model handles strategy: what to do, what to hand off, and whether a result is good enough. The runtime handles bookkeeping: state, logs and feedback.

How it is built

The paper describes three layers.

• Main Agent. The one the user talks to. It plans, manages and executes, and it makes the final call on accepting work.
• Coordinator Runtime. A control layer that dispatches work, tracks each task's lifecycle and saves state.
• Subagents. Optional helpers with their own isolated context. They can be called synchronously, so the result comes straight back, or asynchronously, so the result lands later in the Main Agent's inbox.

Apps share one resource protocol with revision checks. If an agent edits a file based on an out-of-date version, it gets a conflict error instead of silently overwriting someone else's change. Native apps cover text documents, presentations, parametric CAD and Blender, and external tools such as FreeCAD connect through MCP, the Model Context Protocol for plugging tools into agents. Long-running operations run as logged jobs with checkpoints, and the agent keeps a task memory document and compacts its context as it goes.

The benchmark results

The team compared CoworkOS with OpenHands, Codex and DeepSeek Harness (DSH) on 12 benchmarks in four domains. All ran with GLM-5.3 at high thinking effort, a 256K context window and up to 64K output tokens per response, according to the paper. Scores below are from its Table 1.

CoworkOS has the best average in productivity and ML coding. It is second to DSH in tool use by 0.09 points and second to OpenHands in search by 0.08 points. Those last two gaps are far too small to call a loss or a win.

Per benchmark, it posts the top score on SkillsBench (64.34), JobBench (59.88), WideSearch (82.47), MLE-Bench-Lite (76.90) and MLS-Bench (35.62). It trails on others: on tau-cubed Banking it scores 49.48 against DSH's 55.67, and on BrowseComp 81.80 against OpenHands' 83.00.

The ML coding lead over DSH is 56.26 to 56.14, and on MLE-Bench-Lite only 22 tasks are used, so these are not strong separations. The clearest gaps are over Codex, and in productivity over the field.

The paper also shows 24 side-by-side cases, such as websites, games, slide decks, CAD and Blender scenes, against Codex and DSH. These are visual judgments, not scores, and the authors say results are case-dependent. In one cutaway case, Codex separated interior layers more clearly than CoworkOS did.

Self-improving a small model

The second half of the paper uses the harness to train a model. GLM-5.3 acts as controller and data teacher, and Qwen3.6-35B-A3B is the model being trained. It starts from supervised fine-tuning on 1,345 tool-use trajectories, then goes through five feedback rounds.

The average over eight benchmarks rises from 33.19 to 42.95, a gain of 9.76 points. AutomationBench jumps from 11.00 to 44.15 and SkillsBench from 20.85 to 35.76. But JobBench falls from 22.66 to 20.53, and round two dipped (36.94 to 35.90) after a data-reuse mistake the authors describe: 42 trajectories supplied 84.2 percent of the training tokens. The team filtered training tasks against test tasks with a text-similarity check at a 0.8 threshold, which only catches near-duplicates.

Limits to keep in mind

• There is no ablation study. The training changes in each round were made together, so the paper cannot say which one helped.
• The authors do not claim their CAD and Blender outputs are manufacturable, correctly lit or physically sound. The visual comparisons are not user studies.
• External apps that do not expose revision data, such as FreeCAD and Blender windows, get no conflict-safety guarantee.
• The paper is inconsistent on its own numbers: the abstract and introduction say 12 benchmarks, while one figure caption says 10. The GDPVal column in Table 1 is shown as Elo, but the Productivity average appears to use a win rate instead.
• All head-to-head results use one model, GLM-5.3. The paper does not show whether the ranking holds with other models.
• The results come from the authors themselves. We found no independent replication.

Code and weights

The code is public under an MIT license, with third-party exceptions noted in the repository. It installs as the npm package coworkos-app and needs Node.js 22.19.0 or newer; a macOS desktop app and a command-line version for macOS and Linux are listed. As of November 3, 2026, the repository had 17 commits, 61 stars and no tagged releases. We saw no trained model weights or training dataset in the repository, and the paper does not say they will be released. There is also a project page with a demo gallery.

What to watch: whether outside groups rerun the 12 benchmarks with other models, and whether the post-training trajectories and checkpoints are released so the 9.76-point gain can be checked.