Image Inpainting with Diffusion

#diffusion models #image inpainting #generative models #deep learning #computer vision #denoising #image processing #neural networks #python

1. Problem Definition and Use Cases

1.1 Problem Definition and Use Cases

Image inpainting refers to the task of reconstructing missing or corrupted regions in an image while maintaining coherence with the surrounding context. Mathematically, given an input image I with a masked region M, the goal is to generate a plausible completion I' such that:

$$ I' = \begin{cases} f_\theta(I) & \text{for } (x,y) \in M \\ I & \text{otherwise} \end{cases} $$

where fθ represents the inpainting model with parameters θ. Diffusion models approach this problem through iterative denoising, where the model learns to gradually reconstruct the masked region by reversing a Markov chain of noise additions.

Technical Challenges

Diffusion-based inpainting must address several key challenges:

Diffusion Formulation

The forward diffusion process gradually adds Gaussian noise to the image according to a variance schedule βt:

$$ q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t\mathbf{I}) $$

For inpainting, the reverse process is conditioned on the observed pixels outside the mask. At each denoising step t, the model predicts the noise component for the masked region while preserving the known pixels:

$$ p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t,t), \Sigma_\theta(x_t,t)) $$

Practical Applications

Diffusion-based inpainting has significant real-world applications:

Performance Metrics

Quantitative evaluation typically employs:

$$ \text{PSNR} = 10\log_{10}\left(\frac{\text{MAX}_I^2}{\text{MSE}}\right) $$
$$ \text{SSIM}(x,y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)} $$

where PSNR measures pixel-level accuracy and SSIM evaluates structural similarity. For semantic evaluation, Fréchet Inception Distance (FID) compares feature distributions between generated and real images.

Problem Definition and Use Cases – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the forward and reverse diffusion processes with a visual representation of noise addition and denoising steps, including the masked region and boundary consistency.

1.2 Traditional Inpainting Methods and Their Limitations

Diffusion-Based Inpainting

Diffusion-based methods formulate inpainting as a partial differential equation (PDE) problem, where pixel values propagate from known regions into missing areas through iterative diffusion. The anisotropic diffusion equation governs this process:

$$ \frac{\partial I}{\partial t} = \text{div}(g(|\nabla I|)\nabla I) $$

where I represents the image intensity, g(·) is a diffusion coefficient function that preserves edges, and ∇ denotes the spatial gradient. While effective for small gaps, these methods suffer from:

Exemplar-Based Techniques

Exemplar-based approaches like Criminisi's algorithm decompose the inpainting task into:

$$ \hat{\Psi} = \arg\max_{\Psi} \text{SSD}(\Psi_p, \Psi_q) $$

where Ψ represents image patches, and SSD computes sum-of-squared differences. These methods:

Variational Methods

Variational formulations minimize an energy functional combining data fidelity and regularization terms:

$$ E(I) = \lambda \int_\Omega (I-I_0)^2 dx + \int_\Omega \phi(|\nabla I|) dx $$

where φ(·) is a convex regularizer. While theoretically elegant, these approaches:

Fundamental Limitations

All traditional methods share three critical weaknesses when compared to modern diffusion-based approaches:

The inability to model complex priors over natural images fundamentally limits their application to real-world scenarios where missing regions often require semantically meaningful synthesis rather than simple interpolation or copying.

1.3 Introduction to Diffusion Models

Diffusion models are a class of generative models that learn to synthesize data by gradually denoising a normally distributed variable. The process is inspired by non-equilibrium thermodynamics, where a system evolves from a high-entropy state (noise) to a low-entropy state (structured data) through iterative refinement. Unlike GANs or VAEs, diffusion models operate by defining a fixed forward process that corrupts data with Gaussian noise and then learning a reverse process that systematically removes this noise.

Forward and Reverse Processes

The forward process is a Markov chain that gradually adds Gaussian noise to the data over T timesteps. Given an input image x0, the forward process generates a sequence x1, x2, ..., xT where:

$$ q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I}) $$

Here, βt is a noise schedule that determines how much noise is added at each step. The reverse process learns to approximate the true posterior q(xt-1 | xt) by training a neural network to predict the noise component:

$$ p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t)) $$

The model is trained to minimize the variational lower bound (VLB) on the negative log-likelihood, which simplifies to a denoising objective:

$$ \mathcal{L} = \mathbb{E}_{t, x_0, \epsilon} \left[ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 \right] $$

where ε is the noise added during the forward process and εθ is the model's noise prediction.

Denoising Diffusion Probabilistic Models (DDPM)

DDPMs formalize the diffusion process by parameterizing the reverse transitions with a neural network. The key insight is that the reverse process can be approximated by a Gaussian distribution if the forward process uses small noise steps. The mean μθ is typically reparameterized to predict the noise εθ directly:

$$ \mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \epsilon_\theta(x_t, t) \right) $$

where αt = 1 - βt and ᾱt = ∏s=1t αs.

Score-Based Interpretation

Diffusion models can also be interpreted as score-based generative models, where the score function ∇x log p(x) is approximated by the noise predictor. The score function describes the direction in which the data density increases most rapidly, guiding the denoising process:

$$ \epsilon_\theta(x_t, t) \approx -\sigma_t \nabla_{x_t} \log p(x_t) $$

This connection links diffusion models to stochastic differential equations (SDEs), where the denoising process is viewed as solving a reverse-time SDE.

Practical Considerations

Training diffusion models requires careful tuning of the noise schedule βt and the architecture of εθ. Common choices include:

Sampling from diffusion models is iterative and computationally expensive, requiring T forward passes of the network. Recent advances, such as DDIM (Denoising Diffusion Implicit Models), accelerate sampling by using non-Markovian reverse processes without sacrificing quality.

Introduction to Diffusion Models – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would visually show the forward and reverse diffusion processes, including the gradual addition and removal of noise across timesteps.

2. Denoising Diffusion Probabilistic Models (DDPM)

Denoising Diffusion Probabilistic Models (DDPM)

Denoising Diffusion Probabilistic Models (DDPM) formulate image generation as an iterative denoising process, reversing a fixed Markov chain that gradually corrupts data with Gaussian noise. The forward process q is defined as:

$$ q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t\mathbf{I}) $$

where βt is a noise schedule increasing from β1 to βT over T timesteps. The forward process admits closed-form sampling at arbitrary timesteps:

$$ q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}x_0, (1-\bar{\alpha}_t)\mathbf{I}) $$

where αt = 1 - βt and \(\bar{\alpha}_t = \prod_{s=1}^t \alpha_s\). The reverse process learns to iteratively denoise through a neural network εθ predicting noise from noisy inputs:

$$ p_θ(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_θ(x_t,t), \Sigma_θ(x_t,t)) $$

Training Objective

DDPM minimizes a reweighted variant of the ELBO, focusing on the noise prediction term:

$$ \mathcal{L}_{simple} = \mathbb{E}_{t,x_0,\epsilon} \left[ \| \epsilon - \epsilon_θ(x_t,t) \|^2 \right] $$

where \(x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon\) and \(\epsilon \sim \mathcal{N}(0,\mathbf{I})\). This simplification yields more stable training than the full variational bound.

Sampling Process

Sampling iteratively refines noise over T steps using the learned reverse transitions:

$$ x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_θ(x_t,t) \right) + \sigma_t z $$

where \(z \sim \mathcal{N}(0,\mathbf{I})\) and \(\sigma_t^2 = \beta_t\). Recent variants employ deterministic samplers (DDIM) for accelerated generation.

Connection to Score-Based Models

DDPMs implicitly learn the score function \(\nabla_{x_t} \log p(x_t)\) through the noise prediction network, as:

$$ \epsilon_θ(x_t,t) = -\sqrt{1-\bar{\alpha}_t} \nabla_{x_t} \log p(x_t) $$

This links diffusion models to score-based generative modeling, enabling techniques like annealed Langevin dynamics.

Architectural Considerations

Common architectures use U-Nets with:

Denoising Diffusion Probabilistic Models (DDPM) – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the forward and reverse diffusion processes with their respective Gaussian noise additions and denoising steps, illustrating the Markov chain transitions.

Training and Inference in Diffusion Models

Forward and Reverse Diffusion Processes

The forward diffusion process gradually adds Gaussian noise to an image x0 over T timesteps according to a predefined schedule. At each step t, the noised sample xt is generated by:

$$ q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t\mathbf{I}) $$

where βt represents the noise schedule. The reverse process learns to gradually denoise the image by estimating pθ(xt-1|xt), typically parameterized as:

$$ p_θ(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; μ_θ(x_t,t), Σ_θ(x_t,t)) $$

Training Objective

The model is trained to minimize the variational lower bound (VLB) on the negative log likelihood. In practice, this reduces to predicting either the noise component or the clean image at each step. The simplified objective for noise prediction is:

$$ \mathcal{L}_{simple} = \mathbb{E}_{t,x_0,ε}[||ε - ε_θ(x_t,t)||^2] $$

where ε is the actual noise added during the forward process and εθ is the neural network's prediction. The network architecture typically uses a U-Net with residual blocks and attention mechanisms.

Sampling and Inference

During inference, sampling starts from pure noise xT ∼ 𝒩(0,I) and iteratively applies the learned reverse process:

$$ x_{t-1} = \frac{1}{\sqrt{α_t}}(x_t - \frac{β_t}{\sqrt{1-\bar{α}_t}}ε_θ(x_t,t)) + σ_tz $$

where z ∼ 𝒩(0,I), αt = 1-βt, and σt controls the stochasticity. For image inpainting, the known regions are conditioned at each step using:

$$ x_t^{known} = \sqrt{\bar{α}_t}x_0^{known} + \sqrt{1-\bar{α}_t}ε $$

while the unknown regions are updated by the model predictions.

Practical Considerations

Several techniques improve training stability and sample quality:

The choice of architecture details like the number of residual blocks, attention heads, and channel multipliers significantly impacts both training efficiency and final performance. Recent variants employ transformer-based architectures or hybrid approaches for improved scaling.

Training and Inference in Diffusion Models – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the forward and reverse diffusion processes with step-by-step image transformations from clean to noisy and back, including the noise schedule and conditional inpainting regions.

2.3 Adapting Diffusion Models for Inpainting Tasks

Diffusion models excel at generating high-quality images by iteratively denoising random noise, but adapting them for inpainting requires conditioning the generation process on the known regions of the image. The key challenge lies in ensuring the inpainted regions remain consistent with the surrounding context while preserving structural coherence.

Conditional Denoising for Inpainting

Given a corrupted image xmasked with a binary mask m (where 1 indicates known pixels and 0 indicates missing regions), the inpainting task involves sampling from the conditional distribution p(xmissing | xknown). The denoising process is modified to incorporate the known pixels at each timestep t:

$$ \tilde{x}_t = m \odot x_{\text{masked}} + (1 - m) \odot x_t $$

where ⊙ denotes element-wise multiplication. This ensures the known pixels remain fixed while the model focuses on denoising only the masked regions. The denoising network εθ is trained to predict noise for the entire image but is evaluated only on the masked regions during sampling.

Gradient-Based Guidance

To enhance semantic consistency, gradient-based guidance can be applied by modifying the predicted noise with the gradient of a perceptual loss:

$$ \hat{ε}_θ(x_t, t) = ε_θ(x_t, t) - \lambda \nabla_{x_t} \mathcal{L}_{\text{perc}}(x_t, x_{\text{masked}}) $$

where λ controls the strength of guidance and ℒperc measures feature-level similarity between the generated and known regions using a pre-trained network like VGG. This encourages the inpainted content to align with the surrounding context.

Blended Diffusion

An alternative approach blends the denoised output with the known pixels at each step using a schedule that gradually increases blending strength:

$$ x_{t-1} = \alpha_t \cdot \text{Denoise}(x_t) + (1 - \alpha_t) \cdot \tilde{x}_t $$

The blending weights αt follow a cosine schedule, starting near 0 (strong conditioning) and approaching 1 (weak conditioning) as t decreases. This allows early steps to focus on structural alignment while later steps refine details.

Practical Considerations

Adapting Diffusion Models for Inpainting Tasks – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the step-by-step blending process of denoised output with known pixels during diffusion iterations, illustrating how the mask and blending weights interact spatially.

3. Data Preparation and Mask Generation

3.1 Data Preparation and Mask Generation

Effective image inpainting with diffusion models requires carefully curated training data and strategically generated masks. The quality of the inpainted results is directly influenced by the diversity of the dataset and the realism of the masks used during training. This section covers the key considerations for data preparation and mask generation in diffusion-based inpainting systems.

Dataset Requirements

The training dataset should encompass a wide variety of scenes, objects, and textures to ensure the diffusion model learns robust feature representations. For general-purpose inpainting, large-scale datasets like ImageNet, COCO, or Places365 are commonly used. Domain-specific applications may require custom datasets (e.g., medical images for healthcare applications). Key dataset characteristics include:

Mask Generation Strategies

The mask generation process determines what portions of the image the model must inpaint during training. Several approaches exist for creating realistic masks:

Random Geometric Masks

Simple but effective masks can be generated using random geometric shapes. The probability density function for generating rectangular masks of width w and height h follows:

$$ p(w,h) = \frac{1}{Z} \exp\left(-\frac{(w - \mu_w)^2}{2\sigma_w^2} - \frac{(h - \mu_h)^2}{2\sigma_h^2}\right) $$

where Z is the normalization constant, and μw, μh define the mean mask dimensions (typically 20-40% of image size) with σw, σh controlling size variation.

Irregular Masks

More realistic masks can be generated using random strokes or brush patterns. These better simulate real-world damage or object removal scenarios. The stroke generation process involves:

  1. Initialize mask with zeros
  2. Generate random Bézier curves with control points sampled from a normal distribution
  3. Apply varying stroke widths along each curve
  4. Combine multiple strokes until desired occlusion percentage is reached

Semantic-Aware Masks

For object removal tasks, masks should align with object boundaries. This requires either:

Preprocessing Pipeline

The complete data preparation pipeline involves several transformation steps:

def preprocess_image(image, mask):
    # Normalize pixel values
    image = image / 255.0
    # Apply data augmentation
    if training:
        image, mask = random_augmentation(image, mask)
    # Combine image and mask
    masked_image = image * (1 - mask)
    # Add noise for diffusion process
    noisy_image = add_diffusion_noise(masked_image)
    return {
        'original': image,
        'masked': masked_image,
        'noisy': noisy_image,
        'mask': mask
    }

Validation Considerations

When preparing the validation set, maintain separate mask generators to prevent data leakage. Common validation scenarios include:

Data Preparation and Mask Generation – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would physically show the different types of masks (geometric, irregular, semantic-aware) applied to sample images, demonstrating their visual characteristics and coverage patterns.

3.2 Model Architecture Choices

The effectiveness of diffusion-based image inpainting hinges on the architectural design of the underlying neural network. Two primary architectures dominate the field: U-Net-based diffusion models and transformer-based diffusion models. Each has distinct advantages in handling spatial dependencies, computational efficiency, and scalability.

U-Net-Based Diffusion Models

U-Nets are the de facto standard for diffusion models due to their ability to capture multi-scale features through skip connections. The architecture consists of an encoder-decoder structure with residual blocks and attention mechanisms. The forward process is modeled as:

$$ q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t\mathbf{I}) $$

where βt is the noise schedule. The U-Net learns to reverse this process by predicting the noise component:

$$ \epsilon_\theta(x_t, t) \approx \epsilon $$

Key modifications for inpainting include:

Transformer-Based Diffusion Models

Vision transformers (ViTs) have emerged as competitive alternatives, particularly for high-resolution inpainting. The self-attention mechanism allows global context modeling, which is critical for large missing regions. The diffusion process is reformulated in token space:

$$ z_t = \text{Tokenize}(x_t), \quad p_\theta(z_{t-1} | z_t) = \mathcal{N}(\mu_\theta(z_t, t), \Sigma_\theta(z_t, t)) $$

Architectural innovations include:

Hybrid Architectures

Recent work combines U-Nets and transformers, such as using a U-Net for local feature extraction and a transformer for global coherence. The hybrid approach achieves state-of-the-art results on benchmarks like Places2 and CelebA-HQ, with PSNR improvements of 1.5–2 dB over pure architectures.

Computational trade-offs must be considered: U-Nets are more memory-efficient for high resolutions (e.g., 1024×1024), while transformers excel at capturing long-range dependencies but require careful optimization to scale.

Model Architecture Choices – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the architectural differences between U-Net and transformer-based diffusion models, including skip connections, attention mechanisms, and token processing.

3.3 Training Strategies and Hyperparameter Tuning

Noise Scheduling and Diffusion Steps

The noise schedule determines how progressively noise is added and removed during training and inference. A well-designed schedule balances computational efficiency and generation quality. The forward process gradually corrupts an image \(x_0\) with Gaussian noise over \(T\) steps, following:

$$ q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t \mathbf{I}) $$

where \(\beta_t\) is the noise schedule. Common choices include:

The cosine schedule often outperforms linear scheduling due to smoother transitions between noise levels, reducing abrupt artifacts during denoising.

Loss Function and Training Dynamics

Diffusion models are trained to predict noise \(\epsilon\) at each timestep \(t\). The loss function is typically:

$$ \mathcal{L}(\theta) = \mathbb{E}_{t,x_0,\epsilon} \left[ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 \right] $$

where \(\epsilon_\theta\) is the neural network. Variants include:

Hyperparameter Optimization

Key hyperparameters and their tuning strategies:

Learning Rate and Batch Size

Diffusion models benefit from large batch sizes (e.g., 256–1024) and adaptive learning rates. The Adam optimizer with warmup is commonly used:

Architecture Choices

The U-Net backbone requires careful design:

Practical Considerations

Training diffusion models for inpainting introduces additional constraints:

Case Study: Stable Diffusion Inpainting

Stable Diffusion’s inpainting pipeline uses:

Empirical results show that fine-tuning the noise schedule and loss weighting reduces artifacts in high-resolution inpainting (≥512×512).

3.4 Evaluation Metrics for Inpainting Quality

Quantitative evaluation of image inpainting results requires carefully designed metrics that capture perceptual quality, structural coherence, and fidelity to the original data. While human judgment remains the gold standard, automated metrics enable scalable benchmarking of diffusion-based inpainting models.

Pixel-Level Metrics

Traditional pixel-wise comparisons measure the discrepancy between inpainted regions I and ground truth G:

$$ \text{MSE} = \frac{1}{N}\sum_{i=1}^N (I_i - G_i)^2 $$
$$ \text{PSNR} = 10 \cdot \log_{10}\left(\frac{\text{MAX}_I^2}{\text{MSE}}\right) $$

where MAXI represents the maximum possible pixel value. While computationally efficient, these metrics often correlate poorly with human perception, particularly for diffusion-based outputs where plausible hallucinations may deviate from ground truth while remaining visually convincing.

Perceptual Metrics

Feature-space metrics better align with human judgment by comparing deep representations:

$$ \text{LPIPS} = \sum_{l} \frac{1}{H_lW_l}\sum_{h,w} \|w_l \odot (\phi_l(I)_{h,w} - \phi_l(G)_{h,w})\|_2^2 $$

where φl denotes activations from layer l of a pretrained network (typically VGG or AlexNet), and wl are learned weights. The Learned Perceptual Image Patch Similarity (LPIPS) metric has demonstrated strong correlation with human rankings of inpainting quality.

Structural Similarity

The SSIM index decomposes image similarity into luminance, contrast, and structure components:

$$ \text{SSIM}(x,y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)} $$

where μ and σ represent local means and standard deviations, while C1, C2 stabilize the division. Multi-scale extensions (MS-SSIM) improve performance by evaluating similarity across resolution pyramids.

Fréchet Inception Distance

For evaluating the statistical quality of inpainted regions, FID compares distributions in Inception-v3 feature space:

$$ \text{FID} = \|\mu_r - \mu_g\|^2 + \text{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}) $$

where (μr, Σr) and (μg, Σg) are the mean and covariance of real and generated features respectively. Lower FID values indicate better alignment with natural image statistics.

Task-Specific Metrics

For semantic inpainting applications, segmentation-based metrics quantify object-level consistency:

Recent work has introduced diffusion-specific metrics like Inception Score for diversity evaluation and Precision/Recall curves for fidelity-coverage tradeoff analysis. The choice of metrics should align with the intended use case—restoration tasks prioritize fidelity metrics, while creative applications may emphasize diversity measures.

4. Conditional Diffusion Models for Guided Inpainting

4.1 Conditional Diffusion Models for Guided Inpainting

Conditional diffusion models extend standard denoising diffusion probabilistic models (DDPMs) by incorporating auxiliary information to guide the generation process. In image inpainting, this conditioning typically takes the form of a binary mask M indicating missing regions and the observed pixels xobs. The forward process remains identical to unconditional diffusion, but the reverse process learns to sample missing content xmiss conditioned on xobs.

Conditional Reverse Process

The key modification occurs in the reverse transition, where the denoising network εθ now takes both the noisy image xt and the conditioning signal as input. For inpainting, we formulate the conditional reverse process as:

$$ p_θ(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; μ_θ(x_t, t, x_{obs} \odot M, M), Σ_θ(x_t, t)) $$

where ⊙ denotes element-wise multiplication. The mean μθ is predicted by a neural network that processes the concatenation of the noisy image and the masked observed pixels.

Training Objective

The training loss extends the standard DDPM objective by focusing only on the missing regions:

$$ \mathcal{L}(\theta) = \mathbb{E}_{t,x_0,\epsilon} \left[ \| \epsilon - \epsilon_\theta(x_t, t, x_{obs} \odot M, M) \odot (1 - M) \|^2 \right] $$

This formulation ensures the model only learns to predict noise for the masked regions while preserving the observed content. The conditioning is implemented via concatenation or attention mechanisms in the UNet architecture.

Guidance Strategies

Several approaches exist to strengthen the conditioning:

Architectural Modifications

Effective conditional inpainting requires specialized architectures:

Sampling Considerations

The sampling procedure requires careful handling of noise schedules and conditioning:

$$ x_{t-1}^{miss} = \frac{1}{\sqrt{\alpha_t}} \left( x_t^{miss} - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha_t}}} \epsilon_\theta(x_t, t, c) \right) + \sigma_t z $$

where c represents the conditioning information and z ~ 𝒩(0,I). The observed pixels are resampled at each step according to the forward process to maintain consistency.

Conditional Diffusion Models for Guided Inpainting – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the conditional reverse process architecture with the UNet processing both noisy image and masked observed pixels, highlighting the concatenation/attention mechanisms.

4.2 Accelerating Diffusion Sampling for Faster Inference

The primary computational bottleneck in diffusion models lies in the iterative sampling process, which typically requires hundreds to thousands of sequential denoising steps to generate high-quality samples. For image inpainting applications where real-time or interactive performance may be desired, this slow inference speed poses significant practical limitations.

Denoising Diffusion Implicit Models (DDIM)

DDIMs reformulate the diffusion process as a non-Markovian chain while maintaining the same training objective as DDPMs. This allows for deterministic sampling along a learned trajectory, enabling high-quality generation in significantly fewer steps. The update rule for DDIM sampling is given by:

$$ x_{t-1} = \sqrt{\alpha_{t-1}} \left( \frac{x_t - \sqrt{1-\alpha_t}\epsilon_\theta(x_t,t)}{\sqrt{\alpha_t}} \right) + \sqrt{1-\alpha_{t-1}-\sigma_t^2} \cdot \epsilon_\theta(x_t,t) + \sigma_t z_t $$

where σt controls the stochasticity of the process (set to 0 for deterministic sampling) and zt ∼ N(0,I). By carefully selecting the noise schedule {αt}, DDIM can achieve comparable sample quality in 10-50 steps versus the 1000+ required by standard DDPM.

Stochastic Differential Equation (SDE) Solvers

The continuous-time interpretation of diffusion models as discretizations of an underlying SDE enables the application of advanced numerical integration techniques. The probability flow ODE corresponding to the reverse-time SDE is:

$$ dx = \left[ f(x,t) - \frac{1}{2}g(t)^2 \nabla_x \log p_t(x) \right] dt $$

where f(x,t) and g(t) are the drift and diffusion coefficients respectively. High-order Runge-Kutta methods or predictor-corrector schemes can substantially reduce the number of required function evaluations while maintaining sample quality.

Latent Space Diffusion

Operating the diffusion process in a compressed latent space rather than pixel space dramatically reduces computational requirements. The latent representation z = E(x) is obtained via a pretrained autoencoder, with the diffusion model trained to denoise in this lower-dimensional space:

$$ L = \mathbb{E}_{z,\epsilon,t} \left[ \| \epsilon - \epsilon_\theta(z_t,t) \|^2 \right] $$

After diffusion sampling in latent space, the final image is decoded via x = D(z0). This approach reduces memory usage and accelerates sampling by 5-10× while maintaining perceptual quality.

Knowledge Distillation

Multi-step diffusion samplers can be distilled into fewer-step student models through:

$$ \min_\phi \mathbb{E}_{x_0,\epsilon} \left[ \| f_\theta^{(T)}(x_T) - f_\phi^{(S)}(x_T) \|^2 \right] $$

where fθ(T) represents the teacher's T-step denoising process and fφ(S) the student's S-step approximation (S ≪ T). Progressive distillation can further compress the sampling process into very few steps (as few as 4-8) while preserving sample quality.

Adaptive Step Sizing

Dynamic adjustment of step sizes during sampling based on local curvature estimates can optimize the tradeoff between speed and accuracy. The optimal step size Δt at each point can be estimated via:

$$ \Delta t \propto \left\| \frac{\partial^2 \epsilon_\theta(x_t,t)}{\partial x_t^2} \right\|^{-1/2} $$

This allows for larger steps in flatter regions of the denoising landscape while taking smaller steps near critical points where the score function changes rapidly.

Accelerating Diffusion Sampling for Faster Inference – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the comparison between standard DDPM and accelerated DDIM sampling trajectories in latent space, illustrating how fewer steps achieve similar results.

Handling Complex Scenes and High-Resolution Images

Challenges in High-Resolution Inpainting

Diffusion models face significant computational and memory constraints when processing high-resolution images due to the quadratic growth of attention operations in transformer-based architectures. For an image of resolution H × W, the computational complexity of self-attention scales as O(H²W²), making it impractical for resolutions beyond 512×512 without optimization.

$$ \text{Memory} \propto b \cdot h \cdot l \cdot (hw)^2 $$

where b is batch size, h is attention heads, l is layers, and hw is the number of patches. This necessitates architectural innovations for feasible high-resolution inpainting.

Multi-Scale Diffusion Approaches

Hierarchical diffusion frameworks address this by decomposing the inpainting task across multiple resolution levels. The Laplacian pyramid decomposition provides a mathematical foundation:

$$ I_k = \text{Downsample}(I_{k-1}) \circledast G_\sigma $$ $$ L_k = I_k - \text{Upsample}(I_{k+1}) $$

where Gσ is a Gaussian kernel. The diffusion process operates independently at each level, with cross-scale attention mechanisms maintaining global coherence. This reduces memory usage by approximately 60% for 1024×1024 images compared to monolithic approaches.

Patch-Based Processing Strategies

For complex scenes with mixed foreground/background structures, patch-based diffusion with overlap-tile blending proves effective. The algorithm:

  1. Divides the image into overlapping 256×256 patches
  2. Processes each patch independently with 64px overlap
  3. Blends results using a cosine-weighted window function:
$$ w(x) = \frac{1}{2}(1 + \cos(\pi x/s)) $$

where s is the overlap region width. This maintains continuity while allowing parallel processing.

Attention Optimization Techniques

Sparse attention patterns and windowed attention reduce computational overhead. The shifted window approach partitions the image into non-overlapping M×M windows, with successive layers using:

$$ \text{shift} = (\lfloor M/2 \rfloor, \lfloor M/2 \rfloor) $$

This maintains a receptive field of 2M×2M while keeping complexity linear with image size. For a 1024×1024 image with M=64, this reduces attention operations by 256× compared to global attention.

Memory-Efficient Gradient Calculation

Checkpointing and reversible layers enable training with limited GPU memory. The memory savings come from recomputing activations during backpropagation rather than storing them:

$$ \text{Memory} \approx O(\sqrt{N}) $$

where N is the number of layers. This allows training diffusion models with up to 128 layers on a single GPU for 2K resolution images.

Practical Implementation Considerations

When implementing high-resolution inpainting systems:

The trade-off between patch size and global coherence becomes critical above 2048×2048 resolutions, often requiring hybrid approaches that combine patch-based processing with low-resolution global guidance.

Handling Complex Scenes and High-Resolution Images – Image Inpainting with Diffusion – Tutorial Diagram
Diagram Description: The diagram would show the multi-scale diffusion process with Laplacian pyramid decomposition and patch-based processing with overlap regions.

5. Key Research Papers on Diffusion Models

5.1 Key Research Papers on Diffusion Models

5.2 Open-Source Implementations and Toolkits

5.3 Recommended Tutorials and Courses