Explore our collection of diffusion models.
TorchDiff is a comprehensive library offering a suite of advanced diffusion models for generative tasks, particularly image synthesis. Built on PyTorch, it provides modular implementations of foundational and cutting-edge models, enabling both unconditional and conditional generation with text prompts. From classic denoising approaches to efficient latent-space techniques, TorchDiff empowers researchers and developers to explore and innovate in generative AI with flexible, high-performance tools.
Denoising Diffusion Probabilistic Models (DDPM), pioneered by Ho et al. in 2020, form the cornerstone of diffusion-based generative modeling. This approach generates high-fidelity images by iteratively reversing a noise-adding process. DDPM employs a Markov chain to gradually corrupt data in the forward process and learns to reconstruct it in the reverse process. TorchDiff's implementation supports both unconditional image generation and text-conditioned synthesis, offering modular components for customizable training and sampling.
Denoising Diffusion Implicit Models (DDIM), introduced by Song et al., enhance DDPM by accelerating the image generation process. By using a non-Markovian process, DDIM reduces the number of denoising steps while preserving image quality. This makes it ideal for efficient inference. TorchDiff's DDIM implementation integrates seamlessly with the same noise prediction architecture as DDPM, supporting both text-guided and unconditional generation with optimized performance.
Score-Based Generative Models through Stochastic Differential Equations (SDE), developed by Song et al., provide a robust framework for modeling diffusion using continuous-time processes. This approach allows flexible noise manipulation and generation. TorchDiff includes four SDE variants: Variance Exploding (VE), Variance Preserving (VP), sub-Variance Preserving (sub-VP), and a deterministic ODE solver, enabling both conditional and unconditional generation with adaptable noise schedules.
Latent Diffusion Models (LDM), proposed by Rombach et al., revolutionize diffusion by operating in a compressed latent space, significantly reducing computational demands. Using a variational autoencoder (VAE), LDM encodes images into compact representations and applies diffusion processes there, decoding results back to pixel space. TorchDiff's LDM implementation supports DDPM, DDIM, or SDE as backbones, with training enhanced by perceptual and adversarial losses for superior image quality.
unCLIP, a key component of OpenAI’s DALL-E 2, is a text-to-image diffusion model leveraging CLIP embeddings for robust text-image alignment, as detailed in "Hierarchical Text-Conditional Image Generation with CLIP Latents" by Ramesh et al. (2022). It operates in two stages: a prior generates CLIP image embeddings from text prompts, and a diffusion-based decoder transforms these into high-quality images. TorchDiff’s upcoming unCLIP implementation will enable text-guided image synthesis in pixel space, offering flexibility for creative applications.
Kandinsky, developed by Sber AI, is a text-to-image diffusion model combining latent diffusion with an image prior, as described in "Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion" by Razzhigaev et al. (2023). Kandinsky 2.0 uses a prior to map text to image embeddings, followed by latent diffusion for efficient, high-quality image generation. TorchDiff’s forthcoming Kandinsky implementation will support text-guided generation with enhanced text understanding, leveraging its modular architecture for seamless integration.
Learn how to train and sample from diffusion models using TorchDiff.
This image was generated by Ho et al., the authors of "Denoising Diffusion Probabilistic Models" (2020). It showcases the model's ability to produce high-fidelity images through iterative denoising in a Markov chain process.
This image was generated by Song et al., the authors of "Denoising Diffusion Implicit Models" (2020). It demonstrates DDIM's capability to create high-quality images with fewer denoising steps, enhancing efficiency.
This image was generated by Song et al., the authors of "Score-Based Generative Modeling through Stochastic Differential Equations" (2020). It highlights the model's flexibility in generating images using continuous-time diffusion processes.
This image was generated by Rombach et al., the authors of "High-Resolution Image Synthesis with Latent Diffusion Models" (2021). It illustrates the model's efficiency in producing high-quality images by performing diffusion in a compressed latent space.
This image was generated by Ramesh et al., the authors of "Hierarchical Text-Conditional Image Generation with CLIP Latents" (2022). It showcases unCLIP's ability to produce detailed images from text prompts using CLIP embeddings in a pixel-space diffusion process.
This image was generated by Razzhigaev et al., the authors of "Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion" (2023). It demonstrates Kandinsky's advanced text-to-image capabilities using an image prior and latent diffusion.
Articles on TorchDiff
Latent Diffusion Models (LDMs), introduced by Rombach et al. (2022), boost efficiency by performing diffusion in a compressed latent space using a variational autoencoder (VAE). This article explores LDMs’ two-stage process—perceptual and semantic compression—and their implementation in TorchDiff for scalable image synthesis with text prompts, featuring practical training and sampling examples. Read More
This article dives into score-based generative modeling using stochastic differential equations (SDEs), covering Variance Exploding, Variance Preserving, and sub-VP variants, plus deterministic ODEs. It explains their connection to diffusion models and demonstrates their implementation in TorchDiff with MNIST-based examples for training and sampling. Read More
TorchDiff is an open-source PyTorch library for diffusion models like DDPM, DDIM, SDE, and LDM. This article outlines their theoretical foundations, focusing on DDPM, and showcases their implementation with practical examples. It invites contributions to enhance the library’s development. Read More
A Python library for diffusion models
TorchDiff is an open-source PyTorch library designed to empower researchers and developers with state-of-the-art diffusion models for generative tasks. Built for flexibility and ease of use, it supports Denoising Diffusion Probabilistic Models (DDPM), Denoising Diffusion Implicit Models (DDIM), Score-Based Generative Models (SDE), and Latent Diffusion Models (LDM), with unCLIP and Kandinsky under active development. TorchDiff enables efficient image synthesis, text-to-image generation, and applications in fields like biology and medicine, such as protein structure prediction. With comprehensive documentation and practical examples, it's a gateway to advancing generative AI research and innovation.
Loghman Samani TorchDiff Developer
As a computational biologist passionate about structural biology, I, Loghman Samani, began exploring diffusion models in 2025 to unlock their potential in predicting protein structures and designing molecules for biology and medicine. Starting with a GitHub repository to study and implement original papers, I created TorchDiff, an open-source PyTorch library featuring DDPM, DDIM, SDE, and LDM, with unCLIP and Kandinsky in development. My goal is to make these powerful models accessible, fostering innovation at the intersection of AI and life sciences.
Join the TorchDiff community
