Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments

Source: arXiv:2607.02407 · Published 2026-07-02 · By Xianhui Meng, Zirui Song, Yuchen Zhang, Li Zhang, Yongxuan Lv, Xiuying Chen et al.

TL;DR

This paper addresses the limitations of current text-driven 3D indoor scene synthesis methods that predominantly assume Manhattan environments with orthogonal, axis-aligned layouts. Such methods fail to generate plausible object arrangements in non-Manhattan indoor spaces characterized by irregular, oblique, or curved architectural boundaries common in realistic buildings. To overcome these challenges, the authors propose SPG-Layout, a novel framework that integrates Spatial Prior Guidance (SPG) and a Hierarchical Layout Strategy (HLS) to produce physically plausible, semantically coherent scene layouts in complex non-Manhattan environments. SPG incorporates explicit statistical priors capturing object-to-boundary and object-to-object spatial relationships, formulated as computable reward signals guiding reinforcement learning. HLS mimics human interior design by prioritizing large object placement followed by medium and small objects to effectively reduce layout fragmentation and collisions.

The method is trained in two stages: supervised fine-tuning for structured scene serialization adherence using the SSR format, followed by reinforcement learning via Group Relative Policy Optimization (GRPO) optimizing a multi-objective reward balancing format compliance, geometric consistency, semantic matching, and spatial prior rewards. To benchmark non-Manhattan performance, they curate a new dataset of 500 refined non-Manhattan indoor scenes extracted and manually polished from public floor plans. Experimental results show that SPG-Layout drastically reduces layout violations (up to 90% reduction vs. prior SOTA) and improves layout fidelity on both non-Manhattan and Manhattan test sets. Qualitative and user studies confirm superior perceptual quality and spatial realism. Ablations verify all components are critical, with HLS especially impactful for physical collision avoidance. This work establishes a foundational approach and benchmark for text-driven 3D indoor synthesis beyond Manhattan world assumptions.

Key findings

• SPG-Layout reduces layout violations (OOB, MBL, VBL) by over 86% in full scene synthesis compared to the runner-up method on non-Manhattan bedroom and living room splits.
• Attention-weighted Spatial Prior Guidance improves object out-of-room rate (OOR) from 63.39% to 65.71% compared to averaging prior aggregation.
• Hierarchical Layout Strategy reduces average layout violations (avg_LV) by approximately 100–113 on ‘bed’, ‘liv’, and ‘all’ splits compared to model variants without HLS.
• SPG-Layout achieves format compliance on SSR-3DFRONT validation at 99% compared to 78% with LoRA fine-tuning.
• On standard Manhattan test sets, SPG-Layout outperforms SOTA ReSpace with layout violations reduced from 88.8/124.6/213.4×10³ to 8.12/9.73/17.85×10³ (OOB/MBL/VBL) and fidelity metrics improved by 3-5%.
• User studies with 20 participants show SPG-Layout received 1362 votes vs. next best 346 votes across 100 layout comparison sets for spatial fidelity and semantic consistency.
• Reinforcement learning with multi-objective reward balancing geometric, semantic, format, and spatial prior constraints improves layout fidelity by >11% and reduces violations by over 50 points on avg.
• SPG-Layout successfully generalizes to Manhattan environments despite training on non-Manhattan priors, indicating robustness.

Threat model

The adversary is an automated text-driven 3D scene synthesis system or model that implicitly relies on the Manhattan assumption, leading to physical implausibility or semantic incoherence in non-Manhattan environments. The threat is that generated scenes will violate boundary constraints or contain object collisions, reducing utility in realistic architectural contexts. The model assumes the adversary cannot directly manipulate the internal learned spatial priors or reinforcement learning reward, nor access detailed non-Manhattan ground-truth spatial distributions.

Methodology — deep read

The paper tackles text-driven 3D indoor scene synthesis in non-Manhattan environments where architectural boundaries lack orthogonality and axis alignment.

1. Threat Model & Assumptions: They assume a user instructs the model with natural language specifying room type and objects. The adversary is avoiding is layout generation methods that fail to produce physically plausible, collision-free object arrangements in non-Manhattan spaces. Adversary capabilities include implicit reliance on Manhattan assumptions.

2. Data: Constructed a new dataset of 500 non-Manhattan indoor scenes curated from public floor plan repositories. Polygonal room boundaries were extracted, and coarse object layouts synthesized using a Manhattan pre-trained generative model (ReSpace + HLS). These were manually refined to ensure physical plausibility and semantic coherence. The dataset was split into subsets (bedroom, living room) following previous benchmarks. Additionally, standard SSR-3DFRONT was used for supervised training.

3. Architecture / Algorithm: The framework SPG-Layout first encodes the indoor scene state as a Structured Scene Representation (SSR) consisting of architectural polygonal layouts and object parameter tuples (description, scale, position, rotation). It models layout generation as conditional sequence generation with a Large Language Model (Qwen3-4B backbone). The key innovation is the Spatial Prior Guidance (SPG) module, which provides a spatial validity score by combining:

• Object-boundary prior (B), measuring how well object placement conforms to room polygon boundaries.
• Semantic-aware object-object affinity (A), capturing spatial relationships between semantically related objects (e.g., nightstand adjacent to bed). The weights for the affinities are computed via an attention mechanism with a learnable temperature to dynamically focus on stronger anchors.

They train with a two-stage approach:

• Stage 1: Supervised fine-tuning (SFT) on SSR-3DFRONT with full parameter tuning to maximize SSR format compliance (achieving 99%).
• Stage 2: Reinforcement learning using Group Relative Policy Optimization (GRPO) to optimize a multi-objective reward:
  • Format compliance reward (Rfmt) to ensure valid SSR output.
  • Geometric consistency reward (Rgeo) to penalize collisions and boundary violations measured by voxelized object/environment overlaps.
  • Semantic matching reward (Rsem) measuring recall of keywords from user prompts.
  • Spatial prior guidance reward (Rspg) incorporating the statistical spatial priors.

They also introduce the Hierarchical Layout Strategy (HLS) that divides objects by area into large, medium, and small and places them in sequence (large to small). This prioritization reduces spatial fragmentation and allows for dynamic conflict resolution by temporarily retracting small objects to re-optimize layouts on collision.

4. Training Regime: 100 epochs of supervised fine-tuning + 10 epochs of reinforcement training on SSR-3DFRONT and non-Manhattan priors with GRPO. Group size of 8 during RL. Hyperparameters: α=0.55, β=0.45 for SPG weights; γ1=0.3, γ2=0.3, γ3=0.4 for reward terms.

5. Evaluation Protocol: Metrics include layout violations (out-of-boundary (OOB), minimum boundary length (MBL), and voxelized boundary violations (VBL)) and layout fidelity metrics (object out-of-room (OOR), object arrangement ratio (OAR), attribute recall (ATR), plausible mean score (PMS)). Benchmarks are conducted on the proposed 500 scene non-Manhattan dataset split into bedroom and living room subsets, a Manhattan baseline on SSR-3DFRONT, and full scene synthesis vs. single object addition tasks. Baselines include SOTA text-driven layout generators: LayoutGPT, LayoutVLM, InstructScene, ATISS, MiDiffusion, and ReSpace. They perform ablation studies removing SPG, Rgeo, and HLS to analyze module contributions.

6. Reproducibility: Code release is planned. The SSR-3DFRONT dataset is public, but the new non-Manhattan 500-scene benchmark is curated from public data but not explicitly stated as released. Details on preprocessing and refinement are described. They use the Qwen3-4B LLM backbone with full-parameter tuning.

Example end-to-end: Given a text prompt for a bedroom with non-Manhattan polygon boundaries, the system initializes the SSR, uses the fine-tuned LLM to autoregressively generate object parameters, guided by SPG rewards to ensure spatial plausibility and HLS to place large furniture before smaller items, producing a collision-free, semantically coherent 3D layout consistent with irregular architectural constraints.

Technical innovations

• Spatial Prior Guidance (SPG) explicitly models statistical object-boundary and semantic object-object spatial priors as attention-weighted intrinsic reward signals guiding reinforcement learning.
• Hierarchical Layout Strategy (HLS) introduces a human-inspired coarse-to-fine object placement ordering by size to reduce layout fragmentation and physical collisions in irregular non-Manhattan spaces.
• Two-stage training combining supervised fine-tuning for structured scene representation compliance with reinforcement learning using multi-objective rewards balances format, geometry, semantics, and spatial priors.
• Application of Group Relative Policy Optimization (GRPO) in scene synthesis to efficiently optimize LLM policy without a value network, stabilizing spatial reasoning under spatial prior constraints.

Datasets

• Non-Manhattan Indoor Scenes Benchmark — 500 scenes — curated from publicly accessible floor plan repositories and manually refined
• SSR-3DFRONT — structured variant of 3D-FRONT — public dataset used for supervised fine-tuning

Baselines vs proposed

• LayoutGPT: OOB (bedroom full synthesis) = 2243.30×10^-3 vs SPG-Layout = 29.30×10^-3
• LayoutVLM: MBL (living room single addition) = 164.91×10^-3 vs SPG-Layout = 1.30×10^-3
• InstructScene: VBL (all full synthesis) = 1838.02×10^-3 vs SPG-Layout = 29.29×10^-3
• ReSpace: OOR (all single addition) = 60.24% vs SPG-Layout = 66.18%
• ReSpace: avg layout violation (Manhattan) OOB = 88.8×10^-3 vs SPG-Layout = 8.12×10^-3
• SPG-Layout full model improves layout fidelity avg_LF by 2-5% absolute and reduces avg_LV by up to >100 points over ablated variants (e.g., without SPG or HLS)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02407.

Fig 1: Manhattan vs. Non-Manhattan Environments.

Fig 2: Overview of the proposed SPG-Layout framework. (i) Provides an intuitive comprehension of structured scene

Fig 3: Mechanism of Spatial Prior Guidance (SPG).

Fig 4: The Illustration of HLS.

Fig 5: Qualitative Results on non-Manhattan Dataset.

Fig 6: Qualitative results of layout generation in complex non-Manhattan environments compared to the existing methods.

Fig 7: Semi-Automated Data Generation Pipeline. We extract geometric boundaries (a) and generate coarse scenes (b),

Fig 8: Expansion Capability of SPG-Layout.

Limitations

• The new non-Manhattan dataset of 500 scenes, while carefully curated, is relatively small and may not capture all architectural diversity.
• The manual refinement step introduces human bias and limits scalability of data preparation.
• Adversarial or out-of-distribution robustness tests on entirely novel non-Manhattan styles or extreme geometry are not reported.
• The impact of texture, materials, or lighting is out of scope; results focus on geometric plausibility only.
• Reinforcement learning tuning and hyperparameter selection details are limited, possibly hindering exact reproduction.
• While SPG-Layout outperforms baselines, absolute layout violation rates remain nonzero, indicating room for further improvements.

Open questions / follow-ons

• How to generalize spatial prior guidance to even more complex or curved geometries beyond polygonal boundaries?
• Can the reinforcement learning stage be extended to jointly optimize for lighting, materials, or functional usability beyond geometric plausibility?
• How can data efficiency be improved to reduce reliance on manual refinement in dataset curation?
• What are the trade-offs and potential impact of alternative hierarchical layouts or dynamic object grouping strategies in scene synthesis?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners exploring text-driven 3D scene synthesis as a challenge format or bot-detection mechanism, SPG-Layout provides insights into modeling complex spatial relationships beyond simple grid or Manhattan assumptions. Its Spatial Prior Guidance offers a concrete method to incorporate statistical spatial priors into LLM-based generative models to improve physical realism, which could inform the design of synthetic but plausible 3D scene tests requiring spatial reasoning to differentiate human users from automated scripts. The hierarchical layout method echoes cognitive patterns that could be leveraged to generate human-intuitive arrangements difficult for bots to mimic without explicit geometric priors.

Furthermore, the two-stage training with reinforcement learning against multi-factor rewards exemplifies how to align language models with strict physical and semantic constraints, vital for robust CAPTCHA challenges involving spatial semantics. The curated non-Manhattan dataset and evaluation metrics provide a foundation for benchmarking generative models on complex geometric constraints, a valuable resource for researchers designing bot-resilient 3D interaction tasks. Overall, the work underscores the importance of modeling spatial priors and hierarchical compositionality in generating realistic 3D scenes from text, directly applicable to advances in bot detection leveraging 3D scene understanding and generation.

Cite

@article{arxiv2607_02407,
  title={ Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments },
  author={ Xianhui Meng and Zirui Song and Yuchen Zhang and Li Zhang and Yongxuan Lv and Xiuying Chen and Kun Wang and Yan Luo and Kai Chen and Hangjun Ye and Long Chen and Jun Liu and Xiaoshuai Hao },
  journal={arXiv preprint arXiv:2607.02407},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02407}
}

Read the full paper

• arXiv:2607.02407 (abstract)
• arXiv:2607.02407 (PDF)