OpenCV

Loads images from disk, camera streams, or memory buffers into OpenCV's core Mat (n-dimensional matrix) abstraction, supporting 100+ image formats (JPEG, PNG, TIFF, BMP, WebP, etc.) with automatic color space detection and conversion. The Mat structure is a templated C++ class that manages pixel data with reference counting and supports arbitrary channel counts and data types (uint8, float32, etc.), enabling zero-copy operations and efficient memory reuse across the processing pipeline.

Applies convolution-based filters (Gaussian blur, Sobel, Laplacian, bilateral filtering) and morphological operations (erosion, dilation, opening, closing) via optimized kernel implementations that operate directly on Mat objects. Filters are implemented as separable convolutions where possible (e.g., Gaussian blur decomposed into horizontal + vertical passes) to reduce computational complexity from O(k²) to O(2k) per pixel, with optional SIMD vectorization (SSE2, AVX) and CUDA acceleration for large images.

Separates foreground (moving objects) from background in video streams using algorithms like MOG2 (Mixture of Gaussians), KNN (K-Nearest Neighbors), or GMG (Godbehere-Matsukawa-Goldberg). These algorithms model the background as a mixture of Gaussian distributions (MOG2) or a set of nearest-neighbor samples (KNN), and classify pixels as foreground if they deviate significantly from the background model. Models are updated frame-by-frame to adapt to lighting changes and slow background motion. Output is a binary mask (foreground/background) for each frame.

Detects contours (boundaries of objects) in binary images using Moore-Neighbor contour tracing algorithm, and computes shape descriptors (area, perimeter, moments, convex hull, bounding rectangle, circularity, etc.). Contours are represented as sequences of (x, y) points forming closed curves. Shape analysis includes moment-based descriptors (centroid, orientation, eccentricity) and Hu moments (rotation-invariant shape descriptors). Used for object detection, shape classification, and image segmentation.

Searches for a template image within a larger image using correlation-based matching (normalized cross-correlation, sum of squared differences, etc.). Computes a similarity map where each pixel represents the correlation score between the template and the image region at that location. Supports multiple matching methods (CV_TM_CCOEFF, CV_TM_SQDIFF, CV_TM_CCORR) with optional normalization. Output is a 2D map of correlation scores; peaks indicate template matches. Can be used for object detection, pattern recognition, and image registration.

Computes histograms (frequency distributions of pixel intensities) for single or multi-channel images, with configurable bin ranges and counts. Supports both grayscale and color histograms. Includes histogram equalization (stretches histogram to use full intensity range) and CLAHE (Contrast Limited Adaptive Histogram Equalization, which applies equalization locally to preserve details). Histograms can be used for image analysis, thresholding, and contrast enhancement.

Detects text regions in images using EAST (Efficient and Accurate Scene Text) detector (deep learning-based) or MSER (Maximally Stable Extremal Regions) detector (traditional), and provides integration points for OCR (Optical Character Recognition) via Tesseract or other external OCR engines. EAST detector outputs bounding boxes around text regions; MSER detector outputs connected components that may contain text. OpenCV does NOT include built-in OCR—text recognition requires external libraries (Tesseract, PaddleOCR, etc.). Used for document scanning, license plate recognition, and scene text understanding.

Detects objects (faces, eyes, pedestrians, etc.) in images using pre-trained Haar or LBP (Local Binary Pattern) cascade classifiers, which are XML-serialized decision trees trained via AdaBoost. The detection algorithm uses a sliding-window approach with image pyramid multi-scale processing: the classifier is applied at multiple scales (1.05x zoom per level) to detect objects of varying sizes, with configurable overlap thresholds to merge nearby detections. Cascade classifiers are computationally efficient (O(n) per window) compared to deep learning detectors, making them suitable for real-time embedded applications.

Loads and executes pre-trained deep learning models (trained in TensorFlow, PyTorch, Caffe, ONNX) for inference tasks (object detection, image classification, semantic segmentation) via the DNN module. Models are loaded from serialized weights/architecture files (e.g., .pb for TensorFlow, .pth for PyTorch, .caffemodel for Caffe) and executed on CPU or GPU (CUDA/OpenCL). The DNN module does NOT train models—it is inference-only; users must train externally and export to a supported format. Supports popular architectures: YOLO (v2-v8), SSD, Faster R-CNN, ResNet, VGG, MobileNet, etc.

Detects keypoints (corners, blobs, edges) in images using algorithms like SIFT, SURF, ORB, AKAZE, BRISK, and computes local descriptors (SIFT, SURF, ORB, BRIEF) that characterize the appearance around each keypoint. Keypoints are scale-invariant and rotation-invariant (depending on algorithm), enabling robust matching across image transformations. Descriptors are typically binary (ORB, BRISK) or floating-point (SIFT, SURF) vectors that can be compared via Hamming distance (binary) or Euclidean distance (float) to find correspondences between images. Used for image stitching, 3D reconstruction, visual localization, and object recognition.

Stitches multiple overlapping images into a seamless panorama by detecting and matching features across images, computing homographies (perspective transformations) to align them, and blending seams. The Stitcher class automates the pipeline: feature detection → matching → homography estimation (via RANSAC) → image warping → seam finding → multi-band blending. Supports both cylindrical and planar projections for panorama generation. The seam-finding algorithm minimizes visible artifacts at image boundaries by computing optimal seam paths based on image gradients.

Estimates camera intrinsic parameters (focal length, principal point, distortion coefficients) from checkerboard calibration images, and computes camera extrinsic parameters (rotation, translation) from 2D-3D point correspondences. Uses the solvePnP algorithm (Perspective-n-Point) to estimate pose from a set of 3D world points and their 2D image projections. Supports multiple solvePnP variants (EPNP, P3P, DLS) with different accuracy/speed trade-offs. Distortion models include radial (k1, k2, k3) and tangential (p1, p2) coefficients, enabling correction of lens distortion.

Computes dense depth maps from stereo image pairs (left and right images captured from calibrated cameras) using block-matching or semi-global matching (SGM) algorithms. The stereo matching pipeline: rectifies images to align epipolar lines → computes disparity map (pixel offset between left/right images) → converts disparity to depth via triangulation. Supports multiple stereo matchers (StereoBM for real-time, StereoSGBM for higher accuracy) with configurable block sizes, disparity ranges, and post-processing (median filtering, speckle removal). Output is a disparity map (or depth map after conversion) with one depth value per pixel.

Estimates pixel-level motion (optical flow) between consecutive video frames using algorithms like Lucas-Kanade (sparse flow), Farnebäck (dense flow), or DIS (Dense Inverse Search). Lucas-Kanade computes flow at feature points by solving a least-squares problem over a local neighborhood; Farnebäck computes dense flow for every pixel using polynomial expansion. Output is a 2-channel flow map (u, v components) representing motion vectors. Used for motion detection, video stabilization, action recognition, and visual odometry.

Reads and writes video files (MP4, AVI, MOV, MKV, etc.) and camera streams frame-by-frame using the VideoCapture and VideoWriter classes. VideoCapture abstracts over multiple backends (ffmpeg on Linux/macOS, DirectShow on Windows, V4L2 for cameras) and provides a simple frame-by-frame iteration interface. VideoWriter encodes frames to video files with configurable codec (H.264, MJPEG, etc.), frame rate, and resolution. Supports both file-based and camera-based input, enabling real-time video processing pipelines.