latentspace.

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation


  • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
    1. Abstract
      • In this work, we introduce DeepSDF, a learned continuous Signed Distance Function (SDF) representation of a class of shapes that enables high quality shape representation, interpolation and completion from partial and noisy 3D input data.
      • DeepSDF represents a shape's surface by a continuous volumetric field.
      • The magnitude of a point in the field represents the distance to the surface boundary and the sign indicates whether the region is inside \((-)\) or outside \((+)\) of the shape,
      • hence our representation implicitly encodes a shape's boundary as the zero-level-set of the learned function. (If the point in the field is perfectly on the surface, the SDF value is 0)

    2. Introduction
      • we may need to deal with an unknown number of vertices and arbitrary topology.
      • In this work, we present a novel representation and approach for generative 3D modeling that is efficient, expressive, and fully continuous.
      • Contributions include:
        1. the formulation of generative shape-conditioned 3D modeling with a continuous implicit surface,
        2. a learning method for 3D shapes based on a probabilistic auto-decoder.

    3. Related Work
      1. Representation for 3d Shape Learning
        • Representations for data-driven 3D learning approaches can be largely classified into three categories:
          1. Point-based
          2. Mesh-based
          3. Voxel-based

      2. Representation Learning Techniques
        • Modern representation learning techniques aim at automatically discovering a set of features that compactly but expressively describe data. For a more extensive review of the field, we refer to Bengio et al.
          1. Generative Adversarial Networks
          2. Auto-encoders
          3. Optimizing Latent Vectors:
            which simultaneously optimizes the latent vectors (3D 오브젝트 클래스별 임베딩) assigned to each data point and the decoder weights through back-propagation. Throughout the paper we refer to this class of networks as auto-decoders, for they are trained with reconstruction loss on decoder-only architectures

    4. Modeling SDFs with Neural Networks
      • A signed distance function is a continuous function that, for a given spatial point, outputs the point's distance to the closest surface, whose sign encodes whether the point is inside (negative) or outside (positive) of the watertight surface: \[ \,\\ SDF(x) = s: x \in \mathbb{R}^3, \, s \in \mathbb{R} \\ \]
      • The surface is implicitly represented by the \(SDF(\cdot) = 0\). A view of this implicit surface can be rendered through raycasting or rasterization of a mesh obtained with Marching Cubes.
      • Our key idea is to directly regress the continuous SDF from point samples using deep neural networks. The resulting trained network is able to predict the SDF value of a given query position, from which we can extract the zero level-set surface.
      • As a universal function approximator, deep feed-forward networks in theory can learn the fully continuous shape functions with arbitrary precision.
      • The most direct application of this approach is to train a single deep network for a given target shape.

      • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
        Difference of DeepSDF model input shape between a single shape and multi-shapes


      • Given a target shape, we prepare a set of pairs \(X\) composed of 3D point samples and their SDF values: (xyz 입력 좌표 및 sdf 레이블 값) \[ \,\\ X := {(x, s) : SDF(x) = s} \\ \]
      • We train the parameters \(\theta\) of a multi-layer fully-connected neural network \(f_\theta\) on the training set S to make \(f_\theta\) a good approximator of the given SDF in the target domain \(\Omega\): \[ \,\\ f_\theta(x) \approx SDF(x), \, \forall x \in \Omega \\ \]
      • The training is done by minimizing the sum over losses between the predicted and real SDF values of points in \(X\) under the following \(L_1\) loss function (MAE loss): \[ \,\\ \mathcal{L}(f_\theta(x), \, s) = | \, \text{clamp}\, (f_\theta(x), \, \delta) - \text{clamp}\, (s, \, \delta) \, | \\ \] where \(\text{clamp}(a, \, \delta) := \text{min}(\delta, \, \text{max}(-\delta, \, a)) \) (각각의 점이 대상 표면 주변 거리 \(-\delta \,\) ~ \(\delta \, \) 범위 내에서 예측되도록 조정)

      • In this paper, the \(\delta = 0.1\) and a feed-forward network composed of eight fully connected layers, each of them applied with dropouts. All internal layers are 512-dimensional and hae \(\text{ReLU}\) non-linearities. The output non-linearity regressing the SDF value is \(\text{tanh}\). We found training with batch-normalization to be unstable and applied the weight-normalization technique instead. For training, we use the \(\text{Adam}\) optimizer.

      • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
        DeepSDF decoder architecture


    5. Learning the Latent Space of Shapes
      • We introduce a latent vector \(z\), which can be thought of as encoding the desired shape, as a second input to the neural network. The latent vector \(z\) is mapped to a 3D shape represented by a continuous SDF.
      • For some shape indexed by \(i, \, f_\theta\) is a function of a latent code \(z_i\) and a query 3D location \(x\), and outputs the shape's SDFs: \[ \,\\ f_\theta(z_i, \, x) \approx SDF^i(x) \\ \]
      • By conditioning (임베딩) the network output on a latent vector, this formulation allows modeling multiple SDFs with a single neural network. (one 3D object has one embedding vector)

      • Motivating Encoder-less Learning
        1. In the encoder-decoder architecture, since the trained encoder is unused at test time, it is unclear whether using the encoder is the most effective use of computational resources during training. (auto-decoder, decoder only architecture for learning a shape embedding without an encoder)

      • Auto-decoder-based DeepSDf Formulation
        1. Given a dataset of \(N\) shapes represented with signed distnace function, we prepare a set of \(K\) point samples and their signed ditance values: \[ \,\\ X_i = \{ \, (x_j, \, s_j) \, : \, s_j = SDF^i(x_j) \, \} \\ \]
        2. For an auto-decoder, as there is no encoder, each latent code \(z_i\) is paired with training shape \(X_i\). The posterior over shape code \(z_i\) given the shape SDF samples \(X_i\) can be decomposed as: \[ \,\\ p_\theta(z_i\, | \, X_i) = p(z_i) \, \cdot \, \prod{(x_j, \, s_j) \in X_i} \,\, p_\theta(s_j\, | \, z_i;x_j) \\ \] where \(\theta\) parameterizes the SDF likelihood.

          • 3D shape \(X_i\) 를 가장 잘 설명하는 임베딩 벡터 \(z_i\)를 학습
          • latent vector \(z_i\)를 찾는 것은 데이터 \((x_j, s_j) = X_i\) 를 동시에 잘 설명할 수 있는 \(z_i\)를 찾는 것
          • \(\prod{(x_j, \, s_j) \in X_i} \,\, p_\theta(s_j\, | \, z_i;x_j)\) : Maximum Likelihood Estimation, \(X_i\)를 가장 잘 설명하는 \(z_i\)와 파라미터 \(\theta\) 탐색
          • In the latent shape-code space, we assume the prior distribution over codes \(p(z_i)\) to be zero-mean and \(\sigma^2\)
          • 굳이 정규분포를 따르는 \(z\, \)를 사용해야 하는 이유는? (학습 안정성? 확률적 해석을 위한?) 그냥 임베딩 하면 되는것 아닌지?

        3. In the auto-decoder-based DeepSDF formulation we express the SDF likelihood via a deep feed-forward network \(f_\theta(z_i, \, x_j )\) and, without loss of generality, assume that the likelihood takes the form: \[ \,\\ p_\theta(s_j\, | \, z_i;x_j) = \text{exp}(-\mathcal{L}(f_\theta(z_i, \, x_j), \, s_j)) \\ \] where the SDF prediction \(\tilde{s}_j\) is represented using a fully-connected network. \(\mathcal{L}(\tilde{s}_j, \, s_j)\) is a loss function penalizing the difference between network prediction and actual SDF value \(s_j\).

          • 손실함수를 \(\text{exp}\)의 입력으로 하여 손실값이 작을수록 가능도가 높은 것으로 해석

        4. In the implementation \( \text{clamped} \, \, L_1\) cost is used for back-propagation.
        5. At training time we maximize the \[ \,\\ \underset{\theta, \{z_i\}_{i=1}^N}{\arg \min} \sum_{i=1}^N \left( \sum_{j=1}^K \mathcal{L}(f_{\theta}(z_i, x_j), s_j) + \frac{1}{\sigma^2} \|z_i\|_2^2 \right) \\ \] where the term \(\frac{1}{\sigma^2} \|z_i\|_2^2\) is to 정규화

          • Minimizing loss \( \mathcal{L}(f_{\theta}(z_i, x_j), s_j) \) == maximizing the likelihood \( p_\theta(s_j | z_i; x_j) \)

    6. Results
      1. Latent Space Shape Interpolation
        • To show that our learned shape embedding is complete and continuous, we render the results of the decoder when a pair of shapes are interpolated in the latent vector space

        • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
          DeepSDF represents signed distance functions (SDFs) of shapes via latent code-conditioned feed-forward decoder networks