ctransformers

Executes transformer-based causal language models (GPT-2, LLaMA, Falcon, etc.) using C/C++ implementations compiled against GGML, with automatic runtime detection of CPU instruction sets (AVX/AVX2) and GPU capabilities (CUDA, Metal) to select the optimal compiled library variant without requiring user configuration. The Python layer wraps ctypes bindings to the native implementation, delegating all tensor operations and forward passes to the optimized C/C++ backend while maintaining a unified Python API across hardware configurations.

Generates text token-by-token with support for multiple sampling algorithms (top-k, top-p/nucleus, temperature scaling) and early stopping conditions, exposing a generator interface that yields tokens as they are produced rather than buffering the full output. The native C/C++ implementation maintains internal token history for repetition penalty calculation and applies stop sequences by checking generated tokens against a user-provided list, enabling real-time streaming to clients or interactive applications.

Provides reset parameter to clear model internal state (KV cache, token history) between generations, enabling clean context boundaries for multi-turn conversations or independent prompts. The native implementation maintains KV cache and token history across generations by default (reset=False) to enable efficient context reuse, but setting reset=True clears this state before generation. This allows users to control whether context persists across multiple __call__ invocations, enabling both stateful conversations and stateless independent generations.

Supports deterministic token generation via seed parameter that initializes the random number generator used for sampling, enabling reproducible outputs across multiple runs. The native C/C++ implementation uses the seed value to initialize GGML's RNG before sampling, ensuring that identical prompts with identical seeds produce identical outputs. Setting seed=-1 (default) uses non-deterministic seeding; explicit seed values (e.g., seed=42) enable reproducibility for testing, debugging, and result verification.

Supports inference across multiple transformer architectures (GPT-2, GPT-J, LLaMA, Falcon, MPT, StarCoder, Dolly, Replit, etc.) with automatic model type detection from GGML file headers or explicit specification via model_type parameter. The native implementation uses architecture-specific forward pass kernels compiled into the GGML library, while the Python layer provides a unified LLM class interface that abstracts away architecture differences, allowing users to swap models without code changes.

Enables selective execution of transformer layers on GPU (CUDA/Metal) while keeping remaining layers on CPU, controlled via gpu_layers parameter that specifies how many layers to offload. The native implementation manages GPU memory allocation, handles data transfer between CPU and GPU memory spaces, and automatically falls back to CPU-only execution if GPU memory is exhausted or GPU support is unavailable. This approach reduces peak memory usage and latency compared to full GPU execution while avoiding the overhead of CPU-only inference.

Integrates with Hugging Face Transformers library via custom pipeline classes that accept ctransformers LLM objects as the underlying model, enabling use of Transformers' pipeline abstraction (text-generation, question-answering, etc.) with GGML-optimized inference. The integration wraps the LLM class to expose a compatible interface (generate() method, tokenizer integration) that Transformers pipelines expect, allowing users to swap HF Transformers models for ctransformers models without changing pipeline code.

Implements LangChain's BaseLLM interface to expose ctransformers models as LangChain LLM providers, enabling use in LangChain chains, agents, and memory systems. The integration wraps the LLM class to implement LangChain's required methods (_generate, _stream, _call), handles prompt formatting and token counting, and supports LangChain callbacks for monitoring generation progress. This allows ctransformers models to be used interchangeably with OpenAI, Anthropic, and other LangChain-supported providers.

Exposes a Config class that encapsulates all text generation hyperparameters (temperature, top_k, top_p, repetition_penalty, max_new_tokens, stop sequences, etc.) as a structured configuration object. The Config object is passed to the LLM's __call__ method to control generation behavior, with sensible defaults (temperature=0.8, top_p=0.95, max_new_tokens=256) that can be overridden per-generation. The native implementation applies these parameters during token sampling, with repetition penalty calculated over a configurable window (last_n_tokens) to penalize repeated tokens.

Provides utility functions to automatically download GGML models from Hugging Face Hub repositories and cache them locally, with support for specifying model names, revisions, and cache directories. The download mechanism uses Hugging Face's hf_hub_download API to fetch model files with progress tracking and automatic retry logic, storing downloaded models in a local cache directory (default ~/.cache/huggingface/hub) to avoid re-downloading on subsequent loads.

Supports multi-threaded token generation via threads parameter that controls the number of CPU threads used for evaluating tokens during inference. The native C/C++ implementation uses thread-level parallelism (via OpenMP or pthreads) to distribute matrix operations across multiple cores, with threads parameter passed to GGML's compute graph executor. Setting threads=-1 uses all available CPU cores, while explicit values (e.g., threads=4) limit parallelism to improve latency on systems with many cores or reduce CPU contention in multi-process environments.

Supports batch processing of prompt tokens via batch_size parameter that controls how many tokens are evaluated simultaneously during the prompt phase (before generation). The native implementation uses GGML's batched matrix operations to process multiple tokens in a single forward pass, reducing total compute time compared to token-by-token evaluation. Larger batch sizes improve throughput but increase memory usage; batch_size parameter allows tuning this tradeoff for specific hardware constraints.