VibeWorldingCan Multimodal Agents Construct 3D Open Worlds End-to-End?

Technical Report
Yansong Ning1*, Jingwen Ye2, Zhongkai Wu2, Yang Sun2, Yiqin Zhu2, Xingyi Li2, Weidong Zhang2, Hao Liu1†
1 AI Thrust, HKUST(GZ) 2 TEG AIPD, Tencent
* Work done during internship at Tencent.    † Corresponding author.
Code 🤗 VWE-Bench Dataset 🤗 VibeWorlder Models
The VibeWorlding framework: a vibe worlding agent plans, invokes 3D tools, observes rendered feedback, and iterates; VWE-Bench provides evaluation data and VibeWorlding-Gym provides the RL training framework.
The VibeWorlding framework. A vibe worlding agent plans autonomously, invokes 3D tools through MCP in a Blender sandbox, then observes and reflects on multimodal feedback — the 3D map plus five rendered views — over multi-turn interaction. VWE-Bench (bottom left) is the evaluation suite; VibeWorlding-Gym (bottom right) is the joint multimodal RL post-training framework, sharing one sandbox and one rubric-based verifier.
2,616
Curated 3D assets
323
Human-annotated seed worlds
6,828
Reverse-synthesized queries
59.3%
Best overall Pass@1 (ours)
64.5 / 55.9
Pass@1, Verified / Unverified
Illustrative assets from small to large, and illustrative 3D seed worlds with asset counts from low to high.
Left: illustrative assets, size class from small to large. Right: illustrative seed 3D worlds, asset count from low to high.

How the Agent Works

Each turn the agent sees the current 3D map and five rendered views, calls a tool, and the scene is re-rendered — so it can see what it built and fix it.
1

Autonomous planning

Infer the user's intent from the query, plan a scene layout, and decide which 3D tools to invoke.

2

Tool use via MCP

Call retrieve_assets, add, delete, and rotation_and_translation inside the Blender sandbox.

3

Observe & reflect

Read back the updated 3D map plus five re-rendered views, spot mistakes, and repair them next turn.

4

Interactive 3D world

Output a simulation-ready world, scored by a rubric verifier on physical feasibility and intent fulfillment.

Two task families

3D world construction builds a world from scratch given only a text query — for example, "Create a simple farm with a barn, crop fields, fences, and a few trees." 3D world refinement edits an existing world — for example, "Remove the car in the middle of the road and delete the two green buildings."

The same rubric-based verifier serves double duty: it scores the leaderboard and acts as the reward service for multimodal RL post-training, so evaluation and training are measured on identical criteria.

Seed 3D Worlds

Human-annotated seed worlds from VWE-Bench, spanning themes from ancient cities to desert oases. Below are six representative examples you can explore live — drag to orbit, scroll to zoom.

Six representative worlds, streamed as real GLB scenes on demand. Further down: the wider gallery of rendered worlds.

What the Agent Sees

Every turn, the sandbox returns five rendered views — front, back, left, right, and top-down — alongside the textual 3D map. This is the exact visual feedback the agent reflects on.

Assets in 360°

Representative examples from the 2,616-asset library, spanning the type taxonomy. Each spins automatically — drag to take control, scroll to zoom.

Full Asset Library

Thumbnail previews of the curated 2,616-GLB library — each asset carries a 5-digit type_id, display name, size class, and per-asset orientation correction. These are static renders for browsing; see Assets in 360° for the interactive models.

VWE-Bench

6,828 reverse-synthesized multimodal queries across two task families. Only asset-level edit (precise) has a ground-truth map and is rule-verified; the rest are scored by an MLLM judge against our rubrics.
Query typeSub-typeCount
3D world constructionTheme only322
Theme + elements620
Full blueprint302
Distractor120
3D world refinementAsset-level edit (precise) verified1,710
Asset-level edit (fuzzy)1,462
Scene critique553
Scene guidance757
Scene restatement477
Complex description505
Total6,828

Data splits

The three splits use completely disjoint seed worlds (264 SFT / 47 RL / 12 Test, summing to 323), so leaderboard numbers measure generalization rather than seed memorization.

SplitContentsSeeds
Evaluation254 cases (46 construction + 208 refinement)12
SFT5,567 cases (1,129 construction + 4,438 refinement)264
RL1,007 rollout prompts (189 construction + 818 refinement)47

Results

Pass@1 on the 254-case evaluation split, reported separately on the Verified and Unverified query families.
VWE-Bench leaderboard: Pass@1 on Verified and Unverified query sets. VibeWorlder-30B-A3B leads both panels.
VWE-Bench leaderboard (Pass@1). Verified (left) is rule-based and tests whether a model can hit a known target map. Unverified (right) is judged by an MLLM against intent rubrics, and is where post-training pays off most. Our VibeWorlder-30B-A3B attains the best Pass@1 on both splits (64.5 / 55.9).

Findings

Frontier MLLMs are far from solving this task. Even GPT-5.5 (57.3% overall) and Qwen3.8-Max (56.9%) stay below ~60% Pass@1. The bottleneck traces to precise 3D world editing — hitting an exact target map, not merely producing something plausible. Untrained open backbones fare far worse: Qwen3-VL-8B reaches 5.3% and Qwen3-VL-30B-A3B 13.6%.

Post-training closes the gap to the frontier. Cold-start SFT and joint multimodal RL together lift the 30B-A3B backbone from 13.6% (base) to 34.5% (SFT) to 59.3% after RL — the best overall Pass@1 of any model evaluated, edging out GPT-5.5. Its lead is widest on the Verified track (64.5 vs. 60.4).

The 8B model punches above its weight. VibeWorlder-8B reaches 41.4% overall, matching Gemini-3.1-pro (42.7%) while clearly surpassing it on Verified (59.3 vs. 44.4), exactly where rule-checkable editing is required.

The two stages improve different axes. Cold-start SFT installs foundational physical and ecological competence, but delivers gains almost exclusively in 3D understanding (0.02 → 0.17). Multimodal RL is what unlocks spatial capability: 3D understanding climbs 0.17 → 0.80 and 3D reasoning 0.04 → 0.69 — though reasoning stays below understanding, so it is only partially unlocked.

Reward curves on the validation set and per query type during RL post-training, comparing the run initialized from the cold-start SFT policy against RL applied directly to the base backbone.
Reward during RL post-training (VibeWorlder-8B). Solid curves are the flagship run with cold-start — RL from the cold-start SFT policy; dashed curves are the w/o cold-start ablation, applying RL directly to the base backbone. The ablation learns slowly and flattens early, while the cold-started run climbs steadily and pulls ahead on every split — most dramatically on the verified queries, where the reward more than doubles.

BibTeX

If you find VibeWorlding useful, please consider citing our work.
@article{vibeworlding2026,
  title   = {VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?},
  author  = {Ning, Yansong and Ye, Jingwen and Wu, Zhongkai and Sun, Yang and
             Zhu, Yiqin and Li, Xingyi and Zhang, Weidong and Liu, Hao},
  journal = {arXiv preprint},
  year    = {2026}
}