min-dalle

Generates images from natural language text prompts using a three-stage neural pipeline: text tokenization via CLIP vocabulary, DALL·E Bart encoder-decoder for semantic image token generation, and VQGan detokenization to reconstruct pixel-space images. The MinDalle orchestrator class manages lazy-loading of all three models, automatic weight downloading from Hugging Face, and supports both single-image and grid-based batch generation with configurable sampling parameters (temperature, top-k, supercondition factor) to control output diversity and text-image alignment.

Exposes a generate_image_stream() iterator that yields PIL.Image objects at intermediate generation steps, enabling progressive rendering in interactive UIs without waiting for full completion. Internally, the VQGan detokenizer is called incrementally as the Bart decoder produces image tokens, allowing applications to display partial 256x256 images as they're reconstructed from token space. This pattern decouples the neural computation from UI rendering, enabling responsive feedback loops.

Provides a Jupyter notebook (min_dalle.ipynb) enabling interactive image generation with cell-by-cell execution, inline image display, and parameter experimentation. The notebook initializes MinDalle once, then enables users to generate images with different prompts and parameters in separate cells, with results displayed inline. Supports both Mega and Mini models, and enables easy parameter tuning (seed, grid_size, temperature, top_k) via notebook cell editing.

Provides a Replicate-compatible prediction interface (replicate/predict.py) enabling deployment of min-dalle on Replicate's serverless GPU platform. The Predictor class wraps MinDalle with Replicate's API contract (predict() method accepting input dict, returning output dict), handling model initialization, inference, and result serialization. Enables users to deploy min-dalle without managing infrastructure, paying only for GPU time used.

Generates multiple images in a single inference pass by producing a grid of N×N images (typically 3×3 or 4×4) from a single text prompt, enabling efficient batch processing and visual comparison. The generate_image() method accepts a grid_size parameter and internally generates grid_size² images in parallel using batched tensor operations, then stitches them into a single composite PIL.Image. This is more efficient than sequential generation because the encoder and decoder process all images in a single batch.

Enables reproducible image generation by accepting an integer seed parameter that controls all random number generation (sampling temperature, top-k selection, etc.) in the encoder and decoder. Passing the same seed produces identical image tokens and thus identical pixel-space images, enabling reproducibility for debugging, testing, and scientific validation. Seed=-1 enables random generation (no reproducibility).

Supports dynamic tensor precision selection (float32, float16, bfloat16) and device targeting (CUDA GPU or CPU) via MinDalle constructor parameters, enabling memory/speed tradeoffs without code changes. Internally, all model weights and intermediate tensors are cast to the specified dtype before inference, and device placement is handled transparently via PyTorch's .to(device) API. This enables the same codebase to run on T4 GPUs (float32), A10G GPUs (float16), and CPU-only systems (float32 with degraded performance).

Defers loading of DalleBartEncoder, DalleBartDecoder, and VQGanDetokenizer neural network weights until first use via lazy initialization pattern, reducing startup time and enabling memory-efficient multi-model scenarios. When a model is first accessed, the MinDalle class automatically downloads weights from Hugging Face Hub (if not cached locally) to a configurable models_root directory, verifies integrity, and instantiates the PyTorch module. Subsequent accesses return cached in-memory references if is_reusable=True, or reload from disk if is_reusable=False.

Converts natural language text prompts into fixed-length token sequences using the CLIP tokenizer vocabulary, enabling the DALL·E Bart encoder to process semantic meaning. The TextTokenizer class encodes text to token IDs (integers 0-49407) and pads/truncates to a fixed sequence length (typically 64 tokens), handling special tokens (BOS, EOS, padding) according to CLIP conventions. This tokenization is deterministic and language-agnostic within CLIP's vocabulary coverage, but out-of-vocabulary words are mapped to a fallback token.

Encodes tokenized text prompts into a sequence of semantic image tokens (integers 0-16383) using a transformer-based encoder-decoder architecture trained on image-text pairs. The DalleBartEncoder takes text token sequences and produces image token logits, which are then sampled using configurable temperature and top-k parameters to generate diverse outputs. The encoder is a BART variant (denoising autoencoder) with ~400M parameters (Mega) or ~200M (Mini), trained to map text semantics to DALL·E's learned image token space.

Generates a sequence of image tokens (256 tokens total, values 0-16383) from the encoder output using an autoregressive transformer decoder with causal masking. The DalleBartDecoder iteratively predicts the next token conditioned on previously generated tokens and the encoder output, similar to language model decoding. Supports temperature and top-k sampling at each step to control diversity, and includes a supercondition_factor parameter to weight the encoder output more heavily (increasing text-image alignment at the cost of diversity).

Reconstructs 256x256 RGB images from discrete image token sequences using a pre-trained VQGan decoder (vector quantized generative adversarial network). The VQGanDetokenizer maps each token (0-16383) to a learned embedding vector, then passes through a convolutional decoder to produce pixel-space images. This is a learned inverse operation to the VQGan encoder (which was used to tokenize images during DALL·E training), enabling lossless reconstruction of 256x256 images from 256 tokens.

Provides a CLI entry point (image_from_text.py) enabling non-programmatic users to generate images via shell commands with flags for text prompt, model selection (Mega vs Mini), seed, grid size, and output path. The CLI parses arguments, instantiates MinDalle with appropriate configuration, generates images, and saves to disk as PNG files. Supports batch generation via shell loops or scripting without requiring Python knowledge.

Provides a graphical user interface (tkinter_ui.py) enabling interactive image generation with real-time text input, model selection, and progressive image display. The GUI manages MinDalle instance lifecycle, handles text input validation, displays generated images in a scrollable canvas, and provides buttons for generation, cancellation, and saving. Supports both Mega and Mini models with UI-driven selection, and displays generation progress via status messages.