LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

08/04/2025

• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models
  0. Review Purpose
     • As part of our research to improve our LLM-based product at Spacewalk for furniture placement in apartments and houses, I will review this paper. The method used in the LayoutVLM paper is similar to ours, so I hope to understand the differences and gain insights.

       

       LayoutVLM Demo
       
       

       

       
       
       Spacewalk's PoC for LLM-based furniture placement
  
  
  1. Introduction
     • In this paper, we advance this goal by addressing open-universe layout generation, which involves creating diverse layouts based on unlabeled 3D assets and free-form language instructions.
     • Traditional scene synthesis and layout generation methods are often constrained by predefined object categories and patterns of object placements learned from synthetic scene datasets
     • LayoutVLM uses visually marked images, allowing accurate estimation of object placements within the scene, especially when existing objects exist
     • we introduce a novel scene layout representation that can be combined with differentiable optimization to generate diverse layouts . The scene representation builds on two complementary representations—numerical pose estimates and spatial relations with matching differentiable objectives.
     • More specifically, LayoutVLM first predicts numerical object poses as initialization for the optimization process. Then, LayoutVLM jointly optimizes for physics-based objectives and spatial relations, each with their corresponding differentiable objectives.
  
  
  2. Related Work
     • The advent of Large Multimodal Models (LMMs) has enabled open-vocabulary 3D scene syn- thesis, supporting the flexible generation of scenes without dependence on predefined labels or categories
     • LayoutGPT prompt language models to directly generate 3D Layouts for indoor scenes
     • Holodeck uses LLMs to generate spatial scene graphs and then uses the specified scene graph to optimize object placements
     • LayoutVLM also falls into this line of work, but instead of using LLMs, we generate scene layout representations from image and text inputs using VLMs
  
  
  3. Problem Formulation
     • The problem of open-universe 3D layout generation is to arrange any assets within a 3D environment based on natural language instructions.
     • Formally, given a layout criterion \(\ell_{\text{layout}} \) in natural language, a space defined by four walls oriented along the cardinal directions \({w_1,...,w_4} \), and a set of \(N\) 3D meshes \({m1,...,m_N}\), the goal is to create a 3d scene that faithfully represents the provided textual description.
     • We assume that the input 3D objects upright and an off-the-shelf vision-language model (VLM) (i.e., GPT-4o) is employed to determine their front-facing orientations
     • The VLM also annotates each object with a short textual description \(s_i\), and the dimensions of its axis-aligned bounding box after rotating to face the \(+x\) axis will be represented as \(b_i \in \mathbb{R}^3 \)
     • The target output of layout generation is each object's pose \(p_i = (x_i,y_i,z_i,\theta_i) \), including the object's 3D position and rotation about the \(z\)-axis
  
  
  4. LayoutVLM
     • Our approach employs vision-language models (VLMs) to generate code for our proposed scene layout representation that specifies both an initial layout \(\hat{p}_{i=1}^{\tiny{N}} \), as well as a set of spatial relations between assets (and walls).

       

       Process of generating 3D scene layout with Vision-Language Models
     
     
     • This representation is then used to produce the final object placements through optimization: \[ \begin{aligned} \,\\ &\arg\min \left( \mathcal{L}_{\text{semantic}} + \mathcal{L}_{\text{physics}} \right), \\ &{\{p_i\}_{i=1}^{\tiny{N}}} \end{aligned} \\ \quad\text{initial solution } \{\hat{p}_i\}_{i=1}^N. \,\\ \] where \(\mathcal{L}_{\text{semantic}} \) is the objective function decoded from the scene layout representation and \(\mathcal{L}_{\text{physics}} \) is the objective function we employ to ensure physical plausibility
     
     
     1. Scene Layout Representation
        • Our representation includes (a) numerical estimates of object poses \(\hat{p}_{i=1}^{\tiny{N}} \) and (b) spatial relations with differentiable objectives. Initial estimates provide a starting point for optimization, while differentiable objectives guide the process, addressing the challenge of directly predicting physically plausible layouts. The initial layout is key in the optimization process, as poor initialization can lead to a suboptimal layout

          

          Example Scene Representation
        
        
        • Differentiable Spatial Relations. The goal of these spatial relations is twofold: (a) to capture the semantics of the input language instruction and (b) to preserve these semantics during optimization for physical plausibility To design a set of spatial relations that can capture a wide range of semantics for indoor scenes, we devise five expressive spatial relations: two positional objectives (i.e., \(\text{distance}\), \(\text{on_top_of}\) ), two orientational objectives (i.e., \(\text{align_with}\), \(\text{point_towards}\)), and one wall-related objective (i.e., \(\text{against_wall}\)) that pertains to both the position and orientation of an asset The below table presents the notations and meanings of these spatial relations. Formally, we denote a set of spatial relations derived from a scene layout representation as \(\mathcal{R}\)
          
          
          \[ \begin{array}{lll} \hline \textbf{Type} & \textbf{Notation} & \textbf{Explanation} \\ \hline \text{Positional} & \mathcal{L}_{\text{distance}}(p_i, p_j, d_{\min}, d_{\max}) & \text{The distance between the two assets should fall within the range } [d_{\min}, d_{\max}] \\ \text{Positional} & \mathcal{L}_{\text{on_top_of}}(p_i, p_j, b_i, b_j) & \text{Position one asset on top of another.} \\ \text{Rotational} & \mathcal{L}_{\text{align_with}}(p_i, p_j, \phi) & \text{Align two assets at a specified angle } \phi. \\ \text{Rotational} & \mathcal{L}_{\text{point_towards}}(p_i, p_j, \phi) & \text{Orient one asset to face another with an offset angle } \phi. \\ \text{Mix} & \mathcal{L}_{\text{against_wall}}(p_i, w_j, b_i) & \text{Place an asset again wall } w_j. \\ \hline \\ \end{array} \]
          Spatial Relation Definition
     
     
     
     2. Generating Scene Layout Representation with Vision-Language Models
        • Visual Prompting. The VLM's input includes rendered images of the 3D scene and individual asset views. Prior research has shown that visual cues can improve VLMs' object recognition and spatial reasoning. We provide two types of visual annotations for layout generation: coordinate points in the 3D space spaced 2 meters apart to help the VLM gauge dimensions and scale and visualizations of coordinate frames to maintain consistent spatial references. We also annotate the front-facing orientations of each object with directional arrows, which is essential for generating rotational constraints like \(\text{aligned_with}\) or \(\text{point_towards}\).
        
        
        • Self-Consistent Decoding. One key challenge is that VLMs struggle with spatial planning; while they may successfully generate spatial relations for pairs of objects, they tend to fail at accounting for overall layout coherence. We introduce self-consistent decoding for our scene layout representation. Different from standard self-consistency, which selects the most consistent answer from multiple reasoning paths following the same format, we require the two distinct but mutually reinforcing representations predicted by the VLM to self-consistent. After self-consistent decoding, we can formally describe the semantic part of the optimization loss as: \[ \,\\ \mathcal{L}_{\text{semantic}} = \sum_{\mathcal{L} \, \in \, \mathcal{R}} \mathbb{1} \, \left[\, \mathcal{L}_i (\hat{p}_i, \hat{p}_j, \lambda) \le \epsilon \, \right] \cdot \mathcal{L}_i (p_i, p_j, \lambda) \,\\ \] where \(\hat{p}_i \) and \(\hat{p}_j \) are the initial estimated poses, \(\lambda \) represents the additional parameters in the functions (refer to Table 1), and \(\epsilon\) is a threshold value for determining whether the differentiable spatial relation \(\mathcal{L}\) is satisfied in the initial estimates
     
     
     3. Differentiable Optimization
        • To generate a scene from the estimated object poses and differentiable constraints, we jointly optimize all objects according to the specified constraints over a set number of iterations. In addition to the spatial relational functions generated by the VLM, we impose Distance-IoU loss on objects' 3D oriented bounding box for collision avoidance: \[ \,\\ \mathcal{L}_{\text{physics}} = \sum_{i=1}^N \sum_{\substack{j=1 \\ j \ne i}}^N \mathcal{L}_{\tiny{\text{DIoU}}} (p_i, p_j, b_i, b_j) \,\\ \] \(\mathcal{L}_{\text{semantic}} \) and \(\mathcal{L}_{\text{physics}} \) form the final objective function for the optimization problem. With the objective function, we can confine objects within the boundary and avoid unwanted intersections, ensuring physically valid layouts
     
     
     4. Finetuning VLMs with Scene Datasets
        • This scene representation can be automatically extracted from scene layout datasets without requiring manual annotations. Specifically, given a set of posed objects in a 3D scene, we apply the preprocessing procedure to obtain both textual descriptions and oriented bounding boxes for each object
        • The resulting scene representation includes both raw object poses and the satisfied spatial relations, which we then use to fine-tune VLMs to generate these scene representations from input objects and scene renderings
        • In our implementation, we use:
          • 3D-Front dataset
          • closed-source GPT-4o
          • open-source LLaVa-NeXT-Interleave
  
  
  5. Experiments
     1. Experimental Setup
        • Evaluation. In our evaluation, we assess models’ abilities in 1) open-vocabulary language instructions, 2) predicting layouts for objects beyond predefined categories, and 3) generating accurate object placements within boundaries while avoiding collisions
        • Evaluation Metrics. Physical plausibility is measured using the \(\text{Collision-Free Score (CF)}\) and \(\text{In-Boundary Score (IB)}\). All assets are enforced to be placed, with remaining assets randomly placed if a method fails. Semantic coherence is assessed using \(\text{Positional Coherency (Pos.)}\) and \(\text{Rotational Coherency (Rot.)}\), measuring alignment with the input prompt To evaluate semantic coherence across layouts without groundtruth, we use GPT-4o to score layouts based on top-down and side-view renderings and the language instructions. We also introduce the \(\text{Physically-Grounded Semantic Alignment Score (PSA)}\) to assess physical plausibility and semantic alignment, assigning 0 if assets cannot be feasibly placed. \(\text{PSA}\) is calculated simply the GPT-4o rating weighted by physical plausibility. Scores range from 0 to 100, with higher scores indicating better performance
     
     
     2. Ablaion Study
        • w/o Visual
        • w/o Self-Consistency
        • w/o Constraint
        • The results indicate that spatial constraints are crucial for ensuring physical validity, particularly in preventing object overlap, while a good numerical initialization contributes to better semantic scores
  
  
  6. Conclusion
     • With the scene layout representation combined with self-consistent decoding, differentiable optimization, and visual prompting, we demonstrated significant improvements over existing LLM and constraint-based approaches
     • While LAYOUTVLM shows promise, it has limitations, including occasional failures in generating valid layouts due to suboptimal VLM initializations

objective function을 agent가 스스로 정의하고 코딩해서 최적화의 목표로 삼지 않는 한 레이아웃의 자유도는 제한적일 것임. 그럼에도 개발자가 objective function을 지정하는 것 자체는 레이아웃의 baseline quality를 조절할 수 있다는 점에서 의미가 있음.

---