Polynomial Implicit Neural Representations For Large Diverse Datasets

09/19/2024

• Polynomial Implicit Neural Representations For Large Diverse Datasets
keywords to search: Frequency, Positional Encoding, Taylor Series Fourier Transform


1. Abstract
• Most INR architectures rely on sinusoidal positional encoding, which accounts for high-frequency information in data. However, the finite encoding size restricts the model's representational power.
  • 이미지 데이터에서, high-frequency는 sharp variations(edges, fine details), low-frequency는 smooth variations(bigger, less details) 의미
  • Positional encodings in the context of image data provide a way for neural networks, to understand the spatial relationships between different points or pixels in an image.


• convolution, normalization, self-attention을 사용하지 않고 이미지 생성. i.e., no interaction between the pixels


2. Introduction
• 최근 생성모델은 대부분 CNNs을 기반으로 하지만, Implicit Neural Representations(INRs)와 같은 연구는 이미지를 좌표값에 대한 연속함수로 나타내며, 각각의 픽셀은 독립적으로 합성된다.
• INRs generally consists of a positional encoding module and a MLP. The positional encoding in INRs is based on sine functions, often reffered to as Fourier features.
  • Fourier features is a positional encoding that embeds input coordinates to higher-dimensional vectors using sine functions.


• Several methods have shown that using MLP without sinusoidal positional encoding generates blurry outputs, i.e., only preserves low-frequency information.
• ReLU의 piece-wise linear nature에 의해 2차 이상의 미분이 0이므로, ReLU 기반의 네트워크는 high-frequency 함수를 표현하는데 좋지 않다.
  • In tasks requiring high precision for high-frequency information, such as image generation, signal processing, or implicit representations, ReLU-based MLP often fail to capture fine details, resulting in blurry or inaccurate outputs.


• Positional encoding을 사용하지 않고, 점진적으로 다항식의 차수를 MLP의 깊이에 따라 점진적으로 증가시키며, 이는 feature와 각 ReLU 레이어 이후에 얻어진 affine transformed coordinate location(아핀변환 + 비선형성, \( \sigma(Wx + b)\))간의 element-wise multiplication을 통해 이루어진다.
• Affine 파라미터는 알려진 분포에서 샘플링된 latent code에 의해 학습된다.


3. Method
• We are interested in a class of functions that represent an image in the form: \[ \,\\ G(x, y) = g_{00} + g_{10}x + g_{01}y \,+\, ... \,+\, g_{pq}x^{p}y^{q} \,\\ \]
  • where, \((x, y)\) is the normalized pixel location sampled from a coordinate grid of size \(H \times W\), while the coefficients of the polynomial \((g_{pq})\) are parameterized by a latent vector \(z\) sampled from a known distribution.
  • 파라미터 \(g_{pq}\) 는 \((x, y)\) 좌표를 기반으로 다항식의 각 항의 behaivor 조절하고 low-frequency & high-frequency 정보를 모두 표현가능


• Therefore, to form an image, we evaluate the generator \(G\) for all pixel locations \((x, y)\) for a given fixed \(z\): \[ \,\\ I = \{ G(x, y; z) \,|\, (x, y) \in CoordinateGrid(H, W) \} \,\\ \]
  • where, \(CoordinateGrid(H, W) = \{ ( \frac{x}{W - 1}, \frac{y}{H - 1} ) \,|\, 0 \leq x < W, 0 \leq y < H \} \).
  • By sampling different latent vectors \(z\), we generate different polynomials and represent images over a distribution of real images.


• Our model consists of two parts: a) Mapping Newtork, which takes the latent code \(z\) and maps it to affine parameters space \(\mathbf{W}\), and b) Synthesis Network, which takes the pixel location and generates the corresponding RGB value.
  
  Polynomial Implicit Neural Representation (Poly-INR) based generator architecture
  
  
  1. Mapping Network
  • The mapping network takes the latent code \(z \in \mathbb{R}^{64} \) and maps it to the space \(\mathbf{W} \in \mathbb{R}^{512} \) (This mapping network is used in StyleGAN-XL).
  • It consists of a pre-trained class embedding, which embeds the one hot class label into a 512 dimension vector and concatenates it with the latent code \(z\) Then the mapping network consists of an MLP with two layers.
    

    
        self.mapping_network = nn.Sequential(
            nn.Linear(64 + class_dim, 512)
            nn.ReLU()
            nn.Linear(512, 512)
        )
    

  
  
  • mapping_network is used to generate affine transformation matrix; hence we call \(\mathbf{W}\) as affine parameters space.
  
  
  2. Synthesis Network
  • The synthesis network generates the RGB (\(\mathbb{R}^3\)) value for the given pixel location \((x, y)\).
  • The synthesis network consists of multiple levels; at each level, it receives the affine transformation parameters from the mapping network and the pixel coordinate location
  • Leaky-ReLU layer is used with negative_slope=0.2.
  • The synthesis network can be expressed as: \[ \,\\ G_{syn} = \ldots \sigma(W_{2}((A_{2}X) \odot \sigma(W_{1}((A_{1}X) \odot \sigma(W_{0}(A_{0}X)))))) \,\\ \]
    • where \(X \in \mathbb{R}^{3 \times HW} \) is the coordinate grid of size \(H \times W\),
    • Affine transformation matrix \(A_{i} \in \mathbb{R}^{n \times 3} \).
    • element-wise multiplication을 level별로 수행하면서 다항식의 차수를 점진적으로 증가시키고 모델이 표현의 복잡도를 학습
    
    이런느낌?

    
        def forward(self, z, class_embedding, x):
            w = self.mapping_network(z, class_embedding)
    
            for level in range(len(self.affine_layers)):
                # 1x3 @ 3x3 -> 1x3
                transformed_coord = self.affine_layers[level](w, x)
    
                # 3 -> 512
                linear_output = self.linears[level](transformed_coord)
                
                # apply Leaky-ReLU
                x = self.lrelu(x)
    
                # element-wise multiplication
                x *= linear_output
    
            # 512 -> 3
            rgb = self.output_layer(x)
    
            return rgb
    

    
    


4. Experiments


5. Training details