Why Feature Matching Matters
Feature detection finds interesting points in a single image — corners, blobs, distinctive textures. Feature matching is the next step: figuring out which keypoint in image A corresponds to which keypoint in image B, meaning they represent the same physical point in the world. This correspondence is the foundation of image stitching, 3D reconstruction, visual localization, and object recognition.
Brute-Force Matching
The simplest strategy compares every descriptor in image A against every descriptor in image B and keeps the closest one. For N features in image A and M in image B, that's N×M distance computations.
The distance metric depends on the descriptor type:
- Float descriptors (SIFT, SURF) use L2 (Euclidean) distance.
- Binary descriptors (ORB, BRIEF, BRISK) use Hamming distance — the number of bits that differ — computed efficiently with bitwise XOR plus a popcount.
The cost adds up fast. With 5,000 features per image and 128-dimensional SIFT descriptors, brute-force matching means 25 million 128-dimensional distance computations. That's fine for small images but becomes a bottleneck at scale.
FLANN: Approximate Nearest Neighbors
FLANN (Fast Library for Approximate Nearest Neighbors) builds an index — typically a randomized k-d tree or a hierarchical k-means tree — over the descriptors of one image. Instead of scanning every descriptor for every query, each lookup only examines a small subset of likely candidates.
This trades exactness for speed: FLANN is typically 10–100x faster than brute-force, but it can occasionally miss the true nearest neighbor since it's approximate. For most vision tasks that tradeoff is worth it — a slightly imperfect match rarely matters once you filter matches downstream anyway. OpenCV picks an appropriate FLANN algorithm automatically based on descriptor type.
Lowe's Ratio Test
Not every feature in image A has a genuine match in image B. Parts of the scene may be occluded, backgrounds may only appear in one image, or two unrelated features may look coincidentally similar. Raw nearest-neighbor matching doesn't know the difference — it always returns something.
The ratio test, from David Lowe, filters these out. For each feature in image A, find its two nearest neighbors in image B, then compute:
distance(best match) / distance(second-best match) < threshold
Keep the match only if this ratio is below the threshold — commonly 0.75.
The logic: if a feature has one clearly closest neighbor and everything else is far away, that match is trustworthy. If the best and second-best candidates are nearly tied, the descriptor is ambiguous and the match shouldn't be trusted, regardless of which one scored slightly better.
In practice this eliminates 70–90% of false matches while keeping most true ones. It should be considered a mandatory step in any matching pipeline, not an optional refinement.
RANSAC for Outlier Rejection
Even after the ratio test, some bad matches slip through. RANSAC (Random Sample Consensus) is a robust estimation technique built to handle exactly this: separating inliers (correct matches) from outliers when a meaningful fraction of your data is wrong.
The algorithm:
- Sample — randomly pick the minimum number of matches needed to estimate the geometric model (4 points for a homography, 5 for a fundamental matrix).
- Estimate — compute the model from that sample.
- Score — count how many of the remaining matches agree with the model within some distance threshold (inliers).
- Repeat — run many iterations, keep the model with the most inliers.
- Refine — re-estimate the final model using all inliers from the winning iteration.
RANSAC is surprisingly tolerant of bad data — it can recover the correct model even when 50% or more of the matches are wrong. The number of iterations required depends on the inlier ratio and desired confidence. As a concrete example: with a 50% inlier ratio and 4-point samples, about 72 iterations get you to 99% confidence of having sampled at least one all-inlier set.
The Full Pipeline
Putting it together, a robust matching pipeline looks like:
- Detect features and compute descriptors in both images.
- Match descriptors with brute-force or FLANN.
- Filter with Lowe's ratio test (threshold ~0.75).
- Estimate a geometric model with RANSAC to reject remaining outliers.
- Refine the model using all surviving inliers.
This five-step sequence is the standard backbone behind virtually every classical feature-based vision task — stitching panoramas, aligning point clouds, localizing a camera against a map. Skipping any single stage — matching without the ratio test, or accepting matches without RANSAC — tends to produce systems that work on easy images and fall apart on real-world ones.
Learn More
This post covers the core ideas from the full lesson on Feature Matching, part of the Feature Detection & Description chapter in NeutralBlock's Computer Vision track. The full lesson, like everything on NeutralBlock, is free: https://neutralblock.com/learn/computer-vision/cv-chapter-2/feature-matching
