VibeWorldingCan Multimodal Agents Construct 3D Open Worlds End-to-End?

Technical Report
Yansong Ning1*, Jingwen Ye2, Zhongkai Wu2, Yang Sun2, Yiqin Zhu2, Xingyi Li2, Weidong Zhang2, Hao Liu1†
1 AI Thrust, HKUST(GZ) 2 TEG AIPD, Tencent
* Work done during internship at Tencent.    † Corresponding author.
Code 🤗 VWE-Bench Dataset 🤗 VibeWorlder Models
The VibeWorlding framework: a vibe worlding agent plans, invokes 3D tools, observes rendered feedback, and iterates; VWE-Bench provides evaluation data and VibeWorlding-Gym provides the RL training framework.
The VibeWorlding framework. A vibe worlding agent plans autonomously, invokes 3D tools through MCP in a Blender sandbox, then observes and reflects on multimodal feedback — the 3D map plus five rendered views — over multi-turn interaction. VWE-Bench (bottom left) is the evaluation suite; VibeWorlding-Gym (bottom right) is the joint multimodal RL post-training framework, sharing one sandbox and one rubric-based verifier.
2,616
Curated 3D assets
323
Human-annotated seed worlds
6,828
Reverse-synthesized queries
59.3%
Best overall Pass@1 (ours)
64.5 / 55.9
Pass@1, Verified / Unverified
Illustrative assets from small to large, and illustrative 3D seed worlds with asset counts from low to high.
Left: illustrative assets, size class from small to large. Right: illustrative seed 3D worlds, asset count from low to high.

How the Agent Works

Each turn the agent sees the current 3D map and five rendered views, calls a tool, and the scene is re-rendered — so it can see what it built and fix it.
1

Autonomous planning

Infer the user's intent from the query, plan a scene layout, and decide which 3D tools to invoke.

2

Tool use via MCP

Call retrieve_assets, add, delete, and rotation_and_translation inside the Blender sandbox.

3

Observe & reflect

Read back the updated 3D map plus five re-rendered views, spot mistakes, and repair them next turn.

4

Interactive 3D world

Output a simulation-ready world, scored by a rubric verifier on physical feasibility and intent fulfillment.

Two task families

3D world construction builds a world from scratch given only a text query — for example, "Create a simple farm with a barn, crop fields, fences, and a few trees." 3D world refinement edits an existing world — for example, "Remove the car in the middle of the road and delete the two green buildings."

The same rubric-based verifier serves double duty: it scores the leaderboard and acts as the reward service for multimodal RL post-training, so evaluation and training are measured on identical criteria.

Seed 3D Worlds

Human-annotated seed worlds from VWE-Bench, spanning themes from ancient cities to desert oases. Below are six representative examples you can explore live — drag to orbit, scroll to zoom.

Six representative worlds, streamed as real GLB scenes and rendered live in your browser.

Representative examples from the 323 seed worlds

Rendered in the Blender sandbox, grouped by theme family.

What the Agent Sees

Every turn, the sandbox returns five rendered views — front, back, left, right, and top-down — alongside the textual 3D map. This is the exact visual feedback the agent reflects on.

Assets in 360°

Representative examples from the 2,616-asset library, spanning the type taxonomy. Each spins automatically — drag to take control, scroll to zoom.

VWE-Bench

6,828 reverse-synthesized multimodal queries across two task families. Only asset-level edit (precise) has a ground-truth map and is rule-verified; the rest are scored by an MLLM judge against our rubrics.
Query typeSub-typeCount
3D world constructionTheme only322
Theme + elements620
Full blueprint302
Distractor120
3D world refinementAsset-level edit (precise) verified1,710
Asset-level edit (fuzzy)1,462
Scene critique553
Scene guidance757
Scene restatement477
Complex description505
Total6,828

Data splits

The three splits use completely disjoint seed worlds (264 SFT / 47 RL / 12 Test, summing to 323), so leaderboard numbers measure generalization rather than seed memorization.

SplitContentsSeeds
Evaluation254 cases (46 construction + 208 refinement)12
SFT5,567 cases (1,129 construction + 4,438 refinement)264
RL1,007 rollout prompts (189 construction + 818 refinement)47

Results

Pass@1 on the 254-case evaluation split, reported separately on the Verified and Unverified query families.
VWE-Bench leaderboard: Pass@1 on Verified and Unverified query sets. VibeWorlder-30B-A3B leads both panels.
VWE-Bench leaderboard (Pass@1). Verified (left) is rule-based and tests whether a model can hit a known target map. Unverified (right) is judged by an MLLM against intent rubrics, and is where post-training pays off most. Our VibeWorlder-30B-A3B attains the best Pass@1 on both splits (64.5 / 55.9).

Findings

Frontier MLLMs are far from solving this task. Even GPT-5.5 (57.3% overall) and Qwen3.8-Max (56.9%) stay below ~60% Pass@1. The bottleneck traces to precise 3D world editing — hitting an exact target map, not merely producing something plausible. Untrained open backbones fare far worse: Qwen3-VL-8B reaches 5.3% and Qwen3-VL-30B-A3B 13.6%.

Post-training closes the gap to the frontier. Cold-start SFT and joint multimodal RL together lift the 30B-A3B backbone from 13.6% (base) to 34.5% (SFT) to 59.3% after RL — the best overall Pass@1 of any model evaluated, edging out GPT-5.5. Its lead is widest on the Verified track (64.5 vs. 60.4).

The 8B model punches above its weight. VibeWorlder-8B reaches 41.4% overall, matching Gemini-3.1-pro (42.7%) while clearly surpassing it on Verified (59.3 vs. 44.4), exactly where rule-checkable editing is required.

The two stages improve different axes. Cold-start SFT installs foundational physical and ecological competence, but delivers gains almost exclusively in 3D understanding (0.02 → 0.17). Multimodal RL is what unlocks spatial capability: 3D understanding climbs 0.17 → 0.80 and 3D reasoning 0.04 → 0.69 — though reasoning stays below understanding, so it is only partially unlocked.

Reward curves on the validation set and per query type during RL post-training, comparing the run initialized from the cold-start SFT policy against RL applied directly to the base backbone.
Reward during RL post-training (VibeWorlder-8B). Solid curves are the flagship run with cold-start — RL from the cold-start SFT policy; dashed curves are the w/o cold-start ablation, applying RL directly to the base backbone. The ablation learns slowly and flattens early, while the cold-started run climbs steadily and pulls ahead on every split — most dramatically on the verified queries, where the reward more than doubles.

BibTeX

If you find VibeWorlding useful, please consider citing our work.
@article{vibeworlding2026,
  title   = {VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?},
  author  = {Ning, Yansong and Ye, Jingwen and Wu, Zhongkai and Sun, Yang and
             Zhu, Yiqin and Li, Xingyi and Zhang, Weidong and Liu, Hao},
  journal = {arXiv preprint},
  year    = {2026}
}