instruct-pix2pix

Implements the InstructPix2Pix diffusion model architecture, which takes a source image and natural language instruction as input and generates an edited image by iteratively denoising in the latent space while conditioning on both the instruction embedding (via CLIP text encoder) and the original image features. The model uses a UNet backbone with cross-attention layers to fuse instruction semantics with visual content, enabling semantic-aware edits without pixel-level masks or region selection.

Encodes natural language instructions using OpenAI's CLIP text encoder, converting free-form text into a 768-dimensional embedding vector that captures semantic meaning. This embedding is injected into the diffusion UNet via cross-attention mechanisms at multiple resolution levels, allowing the model to align generated pixels with instruction semantics rather than pixel-level targets. The cross-attention layers compute attention maps between instruction tokens and spatial features, enabling fine-grained semantic control.

Executes a multi-step diffusion process in the latent space (using VAE encoder/decoder), where at each timestep the model predicts noise to remove while being conditioned on both the instruction embedding and the original image's latent representation. The original image is encoded once at the start and concatenated with the noisy latent at each step, providing a strong anchor that preserves image structure while allowing semantic edits. This architecture prevents catastrophic forgetting of the source image and enables fine-grained control over edit intensity via the number of diffusion steps.

Wraps the InstructPix2Pix model in a Gradio application deployed on Hugging Face Spaces, providing a browser-based UI with image upload, instruction text input, and real-time preview of edited results. Gradio handles HTTP request routing, file I/O, and session management, while the backend runs model inference on Spaces' GPU infrastructure. The interface supports drag-and-drop image upload, text input validation, and progress indicators for long-running inference.

Supports uploading multiple images sequentially and applying the same instruction to each, with the backend maintaining instruction state across requests and applying identical CLIP embeddings to all images. The Gradio interface queues requests and processes them serially, allowing users to edit image galleries with consistent semantic edits without re-entering instructions. Results are cached in the session for comparison.

Exposes the number of diffusion steps as a user-adjustable hyperparameter, allowing control over the intensity and extent of edits. Fewer steps (e.g., 10-20) produce subtle modifications while preserving source image fidelity; more steps (e.g., 50+) enable more dramatic transformations at the cost of longer inference time and potential drift from the original. The step count directly controls the noise schedule and denoising iterations, providing a principled way to trade edit magnitude for computational cost.