ChatGLM-4

Generates contextually coherent responses in Chinese and English using a GLM-based transformer architecture that maintains full conversation history through the model.chat(tokenizer, prompt, history) interface. The model processes prior exchanges as context, enabling multi-turn conversations where each response is conditioned on the complete dialogue history rather than isolated prompts. Uses relative position encoding to theoretically support unlimited context length, though training was optimized for 2048-token sequences.

Reduces model memory requirements through post-training quantization via model.quantize(bits) method supporting INT4 (4-bit) and INT8 (8-bit) precision. Quantization is applied to the ChatGLMForConditionalGeneration weights, compressing the 6.2B parameter model from 13GB (FP16) to 6GB (INT4) or 8GB (INT8) while maintaining inference quality through careful bit-width selection. This enables deployment on consumer GPUs and edge devices without retraining.

Enables model inference on CPU-only systems through INT8 quantization and memory-mapped file loading, allowing deployment on machines without GPUs. CPU inference uses PyTorch's CPU optimizations and optional ONNX Runtime acceleration for faster computation. While significantly slower than GPU inference (10-50x latency increase), CPU deployment is valuable for edge devices, development environments, and cost-sensitive scenarios where GPU access is unavailable.

Enables optimized inference on Apple Silicon (M1/M2/M3) and Intel Macs through PyTorch's Metal Performance Shaders (MPS) backend, which accelerates tensor operations using the GPU without requiring CUDA. The deployment automatically detects Mac hardware and routes computation to Metal when available, providing 2-5x speedup over CPU-only inference while maintaining compatibility with INT8 quantization. This enables ChatGLM deployment on consumer MacBooks without external GPU hardware.

Manages conversation state through a list of (prompt, response) tuples that are passed to model.chat() as the history parameter, enabling the model to condition responses on prior exchanges. The history is maintained by the application layer (not the model), allowing flexible storage backends (in-memory, database, file system). Each inference call returns both the response and updated history, enabling stateless API design where clients manage history explicitly.

Enables domain-specific model adaptation through P-Tuning v2 implementation in the ptuning/ directory, which adds learnable soft prompts to the model without modifying base weights. During fine-tuning, only the prompt embeddings and a small adapter layer are trained (typically <1% of model parameters), while the 6.2B base model parameters remain frozen. This approach reduces fine-tuning memory from 14GB (full fine-tuning) to 7GB while maintaining task-specific performance through prompt optimization.

Exposes the model through an HTTP API via api.py that accepts JSON requests and returns JSON responses, enabling integration with web applications and microservices without direct Python dependencies. The API wraps the model.chat() interface, accepting prompt and history as JSON payload and returning generated responses with updated conversation history. Supports concurrent requests through standard Python async/await patterns, making it suitable for production deployments behind load balancers.

Provides a cli_demo.py script that implements an interactive REPL for real-time model testing without code changes. The CLI maintains conversation history across turns, displays token counts and generation time, and supports configuration flags for quantization level, device selection (GPU/CPU), and model path. Users type prompts at a command prompt and receive responses with latency metrics, making it ideal for rapid prototyping and debugging model behavior.

Exposes the model through a browser-based UI via web_demo.py using Gradio framework, which automatically generates an interactive chat interface from the model.chat() function signature. The Gradio interface handles HTML rendering, session management, and client-server communication, allowing users to interact with the model through a web browser without terminal access. Supports real-time streaming of responses and maintains conversation history in the browser session.

Provides web_demo2.py as an alternative to Gradio using Streamlit framework, which renders the chat interface using Streamlit's session state management and reactive component model. Streamlit automatically reruns the entire script on each user interaction, maintaining conversation history through st.session_state dictionary. This approach is more Pythonic for developers familiar with data science workflows, though it introduces latency from full-script reruns.

Implements ChatGLMForConditionalGeneration class using a modified transformer architecture with 6.2 billion parameters that combines bidirectional and autoregressive components from the GLM framework. The architecture uses relative position encoding instead of absolute positions, enabling theoretical unlimited context length while maintaining training efficiency. The model processes input tokens through multi-head self-attention layers with GLM-specific masking patterns that support both understanding and generation tasks in a unified architecture.

Handles text encoding/decoding through ChatGLMTokenizer class that maps text to token IDs and vice versa using a learned vocabulary optimized for Chinese-English bilingual text. The tokenizer implements subword tokenization (likely BPE or SentencePiece) with special tokens for dialogue control (e.g., [gMASK], [eos_token]). Tokenization is a required preprocessing step before model inference, and detokenization reconstructs text from token IDs with proper handling of whitespace and special characters.

Supports scaling model inference and training across multiple GPUs through PyTorch's DataParallel and DistributedDataParallel mechanisms. During multi-GPU deployment, the model is replicated across GPUs with batch splitting, allowing larger batch sizes and faster throughput. Fine-tuning on multiple GPUs uses gradient accumulation and distributed gradient synchronization to maintain training stability while reducing per-GPU memory requirements.