MediaPipe

Detects human faces in images and video streams, then localizes 468 3D facial landmarks (eyes, nose, mouth, jawline, contours) using a two-stage pipeline: a lightweight face detector identifies bounding boxes, followed by a mesh-based landmark model that maps facial geometry. Runs on-device with hardware acceleration (GPU/CPU), enabling sub-100ms latency on mobile without cloud round-trips. Supports multi-face detection in single frame.

Detects hands in images/video and estimates 21 3D hand landmarks (knuckles, joints, fingertips) per hand, enabling gesture classification (thumbs up, peace sign, pointing, open palm, etc.). Uses a hand detector to locate hands, then applies a landmark model to map finger positions. Supports multi-hand detection (up to 2 hands simultaneously in typical use). Includes pre-trained gesture classifier that maps landmark configurations to semantic gestures.

Detects the language of input text and returns language code (e.g., 'en', 'es', 'fr', 'zh') with confidence score. Uses a lightweight language identification model (likely n-gram or character-level classifier) that works on short text snippets. Supports 100+ languages. Outputs top-K language predictions with confidence scores. Useful for routing text to language-specific processing pipelines.

Classifies audio clips into predefined sound categories (e.g., speech, music, dog barking, car horn, glass breaking). Uses a pre-trained audio classifier (likely CNN on mel-spectrogram features) that processes audio frames and outputs class probabilities. Supports both single-label (one class per clip) and multi-label (multiple sounds per clip) classification. Outputs top-K predictions with confidence scores. Processes variable-length audio with automatic feature extraction.

Enables fine-tuning of pre-trained MediaPipe models on custom datasets using transfer learning. Model Maker is a separate tool that takes a pre-trained model (e.g., object detector, image classifier) and a custom dataset, then outputs a fine-tuned model optimized for mobile deployment. Supports training on custom classes/categories without requiring deep ML expertise. Handles data preprocessing, augmentation, and model optimization automatically. Outputs quantized TFLite models ready for deployment.

Deploys trained/fine-tuned models across Android, iOS, Web, and Python with automatic platform-specific optimization. MediaPipe handles model quantization, compression, and hardware acceleration (GPU/CPU/NPU) per platform. Single model can be deployed to all platforms with platform-specific SDKs handling inference. Supports TFLite model format with automatic conversion and optimization. Includes platform-specific bindings for efficient native inference.

Web-based tool for evaluating and benchmarking MediaPipe solutions without coding. Upload images/videos, select a solution (face detection, pose estimation, etc.), and visualize outputs in real-time. Provides performance metrics (latency, memory, accuracy) and allows parameter tuning (confidence thresholds, etc.). Useful for testing solutions before integration, comparing model variants, and understanding model behavior on specific data.

Enables running large language models (LLMs) on-device using MediaPipe's LLM Inference API. Supports quantized/compressed LLM models optimized for mobile and edge devices. Handles tokenization, inference, and token generation. Supports streaming token output for real-time text generation. Enables chatbots, text generation, and other LLM-based features without cloud calls. ARCHITECTURAL DETAILS UNKNOWN: documentation does not specify supported model formats, quantization methods, or provider support.

Generates images from text descriptions using a pre-trained text-to-image model. Takes text prompt as input and outputs generated image. ARCHITECTURAL DETAILS UNKNOWN: documentation does not specify model architecture, inference approach, or customization options. Likely uses a diffusion model or similar generative architecture optimized for mobile.

Detects human bodies in images/video and estimates 33 3D body landmarks (joints: shoulders, elbows, wrists, hips, knees, ankles, spine, head) representing skeletal structure. Uses a person detector to locate bodies, then applies a pose landmark model to map joint positions. Outputs 3D coordinates with per-landmark visibility/confidence scores. Supports multi-person detection in single frame. Enables pose-based activity recognition (standing, sitting, running, jumping).

Detects objects in images/video and returns bounding boxes with class labels and confidence scores. Uses a pre-trained detector (likely SSD or YOLO variant) optimized for mobile inference. Supports 80+ object classes (person, car, dog, cup, etc.) from COCO dataset. Outputs per-object bounding box coordinates, class ID, and confidence. Supports multi-object detection in single frame with configurable confidence threshold.

Classifies images into predefined categories (e.g., dog breed, plant species, food type) and returns top-K predictions with confidence scores. Uses a pre-trained CNN classifier (likely MobileNet or EfficientNet variant) optimized for mobile. Supports 1000+ classes depending on model. Outputs class label and per-class confidence distribution. Single-image classification (not multi-label by default).

Segments images into semantic regions where each pixel is classified into a category (e.g., person, background, sky, grass). Uses a pre-trained segmentation model (likely DeepLab or similar) that outputs a dense per-pixel class map. Supports 150+ semantic classes depending on model. Outputs segmentation mask (same resolution as input) with class ID per pixel, plus optional confidence map. Enables background removal, scene understanding, and region-based processing.

Enables user-guided semantic segmentation where users provide hints (clicks, strokes) to refine segmentation masks. Uses a segmentation model that takes user input (point clicks or scribbles) and outputs refined segmentation mask. Supports iterative refinement: user provides hint → model outputs mask → user refines if needed → repeat. Useful for precise object isolation or background removal where automatic segmentation is imperfect.

Generates dense vector embeddings (typically 256-512 dimensions) for images that capture semantic content. Uses a pre-trained CNN encoder (likely MobileNet or similar) that maps images to embedding space. Embeddings enable similarity search: compute embedding for query image, then find nearest neighbors in embedding space using cosine distance or L2 distance. Useful for image retrieval, duplicate detection, and visual search without explicit classification.

Classifies text into predefined categories (e.g., sentiment, intent, topic, spam/ham). Uses a pre-trained text classifier (likely BERT-based or lightweight transformer) that outputs class probabilities. Supports both single-label (one class per text) and multi-label (multiple classes per text) classification. Outputs top-K predictions with confidence scores. Handles variable-length text input with automatic tokenization and padding.

Generates dense vector embeddings (typically 256-512 dimensions) for text that capture semantic meaning. Uses a pre-trained text encoder (likely BERT-based or lightweight transformer) that maps text to embedding space. Embeddings enable semantic search: compute embedding for query text, then find nearest neighbors using cosine distance. Also enables text clustering, duplicate detection, and semantic similarity without explicit classification.