Detection and Spatial Predictions

Why does one object produce many boxes?

An object detector predicts locations and labels. A dense detector can produce several strong boxes around one instance. Nearby windows, anchors, proposals and multiple feature scales can all respond to the same object. Refining their coordinates does not automatically make their predictions unique.

Non-maximum suppression, or NMS, converts these candidates into a smaller set. The usual greedy rule keeps a high-scoring box and removes sufficiently overlapping competitors. The rule assumes that nearby predictions often describe the same instance. That assumption is useful, but it can fail when different objects overlap.

This note separates the three operations that the name “NMS” can obscure: selecting one candidate, changing candidate scores, and combining candidate coordinates. Start with exact geometry, then follow the candidate pool through the algorithm. The distinctions matter more than memorizing a default threshold.

Specify the rectangle before measuring it

Use continuous xyxy coordinates: [x₁,y₁,x₂,y₂], with x₁<x₂ and y₁<y₂. The first corner is the minimum coordinate, and the second is the maximum. Area is (x₂−x₁)(y₂−y₁). These formulas do not add one to widths or heights.

Some datasets or older implementations use inclusive pixel endpoints. Others store [x,y,width,height] or [center_x,center_y,width,height]. Convert explicitly before measuring overlap. Mixing conventions changes areas, especially for small boxes.

Keep boxes in the same image coordinate system. Undo image resizing, cropping, padding or test-time augmentation consistently. Reject non-finite coordinates and invalid extents. Decide how to clip boxes to the image, and handle zero-area results before a division.

A batched detector also needs ownership information: image index and class label. Geometric overlap across different images has no meaning. A production NMS call must group predictions accordingly.

Compute intersection over union

The clamping is essential. Separated intervals have zero overlap, not a negative intersection area. Touching edges also give zero area in the continuous convention. Identical boxes have IoU=1.

Take A=[0,0,100,100] and B=[40,0,140,100]. Their intersection is $60\times100=6{,}000$. Their union is $10{,}000+10{,}000-6{,}000=14{,}000$. IoU is $\frac37\approx0.4286$. IoU measures shared area relative to total covered area, not the percentage of one box covered by the other.

IoU is symmetric and unchanged by a common translation or uniform positive scale. It says nothing about labels, scores, appearance or instance identity. Two different people can have high box IoU; two boxes for one object can have modest IoU.

See overlap and center distance separately

Move B through the point where the rectangles stop overlapping. IoU becomes zero, but center separation continues to change. That difference motivates a distance-sensitive criterion later. It does not prove that distance determines object identity.

A score orders candidates; it does not identify an instance

The detector assigns a score to each candidate. The score may encode a class estimate, objectness, a product of terms, or a localization-aware quantity. Its meaning depends on the detector and training objective. A value of .9 is not automatically a calibrated 90% chance of a correct detection.

Greedy NMS only needs an ordering. A strictly increasing transformation of all scores preserves hard-NMS order when ties and prefilters remain unchanged. A score floor can break that equivalence if the floor is not transformed too.

For equal scores, specify a tie policy. The teaching trace uses original candidate order. Production CPU and GPU kernels can choose different boxes when scores tie. Reproducibility tests should allow the documented behavior or impose a consistent upstream ordering.

Keep a maximum, remove its competitors, repeat

Begin with eligible candidates sorted by descending score. Select the maximum M, append its index to the output and remove M from the pending pool. Compare M with the remaining eligible boxes. Remove a competitor when its IoU with M exceeds the declared threshold. Repeat until the pool is empty or an explicit output limit is reached.

The comparison is with a kept maximum, not every higher-scoring original candidate. A box removed earlier does not suppress other boxes later. Changing that detail changes the algorithm.

At τ=.5, the five-box trace first keeps A. A and B have IoU≈.7467, so B is removed. A and C have IoU≈.3793, so C remains eligible. D is separate. E is a bag label; class-aware processing does not compare E with the person A. The trace makes these decisions explicit.

Follow the pool, not just the final boxes

Compare hard and Gaussian modes after the first round. Hard NMS removes B immediately at τ=.5. Gaussian decay with σ=.5 instead lowers B’s score to about .295. C remains above B, so C becomes eligible for selection earlier. Coordinates remain fixed throughout this trace.

Switch the label policy to class-agnostic. E now competes with overlapping person boxes. The change is a policy decision, not a geometry change. No ground truth is supplied here, so neither policy is labeled “correct.”

Choose which boxes may compete

Class-aware NMS compares boxes within the same predicted class. It can retain a person and a bag at similar coordinates. Class-agnostic NMS lets every candidate compete. That can remove contradictory labels for one region, but can also remove different legitimate categories.

A class-aware policy does not resolve label ambiguity by itself. Two labels for the same instance may both survive because they do not compete. Conversely, an overly broad class-agnostic policy can suppress an object inside another object’s box.

Use separate image groups in either mode. Preserve indices back to the original predictions so that coordinates, labels and other fields stay aligned. If a library encodes group IDs through coordinate offsets, verify its precision and coordinate assumptions rather than copying the trick without its contract.

Count comparisons and control the candidate budget

For N candidates, naive sorting costs O(N log N). The worst-case suppression loop makes N(N−1)/2 overlap comparisons, giving O(N²) time. A streamed implementation can keep indices and current scores in O(N) working memory.

The visual trace stores snapshots to explain every round. Its diagnostic memory can grow as O(N²). That is not the memory contract of a production kernel. A fully materialized pairwise overlap matrix also uses O(N²) memory.

Prefiltering by score or keeping only the top K candidates reduces the loop’s input. Those policies can change recall. A box omitted before NMS cannot be recovered afterward. Record candidate limits and whether they apply per image or per class.

Hardware kernels, candidate density and grouping affect actual latency. An asymptotic bound is not a serving-time measurement. More parallel comparisons can cost more memory and need not outperform a smaller sequential pool.

Overlap is not object identity

Hard suppression can remove a real neighboring object when its box overlaps the selected box. A high score can also belong to a poorly localized box. Keeping that box first can eliminate a better localized candidate.

These are two different failures: instance separation and score–localization mismatch. Raising the overlap threshold may help a crowded pair survive, but may leave duplicates. Lowering it may remove duplicates more aggressively, but may remove a distinct object. Neither adjustment fixes a detector’s missing candidates.

Soft-NMS addresses the abrupt deletion decision. A distance-sensitive criterion adds geometry. Coordinate fusion combines estimates. Each responds to a different failure; none makes box overlap a complete identity test.

Replace deletion with score decay

Soft-NMS, introduced in2017, reduces scores instead of deleting every overlapping competitor. It retains a chance for a neighboring candidate to survive a later score filter. That can help in crowded scenes; it does not guarantee that every distinct object survives.

Gaussian mode does not use τ. A larger σ weakens score decay. A separate score floor can still remove candidates after rescoring. The original method changes post-processing rather than requiring detector retraining. Soft-NMS paper ↗ gives its evaluation conditions and failure examples.

Changed scores require a new maximum

After score decay, select the current maximum. Sorting once by the original scores and never reconsidering the order implements a different procedure.

In the assigned example, B starts at .9 and C at .85. After selecting A, Gaussian decay gives B≈.295 and C≈.637. D remains .8 because it does not overlap A. Thus the next selected box is D, not B. The next choices depend on the remaining scores after each round.

A box can receive several penalties from several selected boxes. Consequently, the result is not simply the original score multiplied once by its overlap with the first maximum. Keep updated scores and return them when downstream ranking uses them.

The trace keeps selected scores fixed after selection. Remaining candidates receive updates. Removed candidates do not return. These state rules are as important as the decay formula.

Use center distance as an additional cue

Distance-IoU adds center separation to area overlap. At the same IoU, a larger normalized center distance lowers the similarity. DIoU-NMS removes a comparable candidate when this similarity reaches its threshold. The paper uses ≥ at equality; the trace follows that convention.

Similarity and regression loss are not interchangeable: DIoU loss is1−DIoU similarity. Complete-IoU, or CIoU, also includes an aspect-ratio term in its regression loss. These losses train coordinates; NMS selects predictions at inference. A loss improvement does not alone establish a better suppression policy.

The distance cue may help preserve neighboring objects. It remains a geometric heuristic, not a guarantee of distinct identities. Distance-IoU paper ↗ defines the criterion and its tested detector setups.

Combining coordinates is a different operation

Hard NMS selects an existing box. Soft-NMS changes a score. A fusion method instead constructs new coordinates from a group of estimates. This is the useful idea behind the original note’s “weighted NMS” entry, but the name does not specify a unique algorithm.

For boxes [0,0,100,100] and [20,0,120,100], weights .9 and .3 give [5,0,105,100]. That is a coordinate average for an already assigned cluster. It is not a complete Weighted Boxes Fusion implementation.

WBF groups candidates against evolving fused boxes, uses confidence-weighted coordinates and has a separate score aggregation rule. Non-maximum weighting uses a different reference/weighting policy. Declare grouping, class compatibility, model weights and score normalization. Undo test-time transformations before fusion.

Averaging distinct neighboring objects can place a box between them. Averaging estimates with the same bias does not remove that bias. Use Weighted Boxes Fusion ↗ for the full ensemble method, not merely the coordinate formula.

Three thresholds answer three different questions

Values such as .45 or .5 are common examples in particular detector implementations. They are not universal settings. The same number can occur in two rows of the table without meaning the same operation.

Choose post-processing parameters on validation data under the intended metric and candidate budget. Keep test data separate. Report thresholds with the detector, class policy, output cap and coordinate convention. A “better NMS” comparison without these conditions can hide a changed evaluation setup.

Count true positives only after ground-truth matching

In an ordinary non-crowd example, sort predictions by score and match within the correct class. A ground-truth instance can match only one detection. Additional detections for that instance are duplicates, not extra true positives. Unmatched ground-truth instances are false negatives.

For one labeled object and three valid duplicate detections, matching gives TP=1, FP=2 and FN=0. Precision is $\frac13$ and $\operatorname{Recall}=1$. Keeping one correct box gives precision=recall=1. If suppression instead removes the only correctly localized candidate, it can reduce recall.

Average precision summarizes a ranked precision–recall curve. COCO also specifies IoU thresholds, detection caps, area ranges and crowd/ignore handling. It is not the average of TP/(TP+FP+FN). Use the official evaluator ↗ for benchmark results. Statistics and Evaluation owns general evaluation and uncertainty.

A higher threshold need not retain more boxes

A higher hard-NMS threshold removes fewer competitors in an individual comparison. That local fact does not imply that the final output is a superset. A newly surviving high-ranked box can remove several lower-ranked candidates.

At $\tau=0.2$, box 0 removes box 1 because their IoU is $\frac3{13}\approx0.2308$. Boxes 2 and 3 survive: output 0,2,3. At $\tau=0.3$, box 1 survives box 0. Box 1 then removes both 2 and 3 because each has $\operatorname{IoU}=\frac13$. Output 0,1 has fewer boxes despite the higher threshold.

This exact counterexample concerns retained count, not a measured recall curve. Without ground-truth identities, the count cannot tell us which output is preferable. Greedy ordering and evaluation must be considered together.

A small operator with explicit contracts

This pseudocode expresses class-aware hard NMS for one image. Inputs are positive-area continuous boxes [N,4], finite scores [N] and labels [N]. stable_score_order declares the tie rule. The result contains original indices, not merged coordinates.

pending = stable_score_order(scores, descending=True)
kept = []
while pending:
    m = pending.pop(0)
    kept.append(m)
    next_pool = []
    for i in pending:
        same_class = labels[i] == labels[m]
        if not same_class or iou(boxes[m], boxes[i]) <= threshold:
            next_pool.append(i)
    pending = next_pool
return kept

Score filtering, output caps and image grouping are separate policies around this kernel. For class-agnostic mode, remove the same-class exemption. For Soft-NMS, maintain current scores and find the current maximum again after rescoring.

Check identical, disjoint and touching boxes; equality at the threshold; nested boxes; ties; different classes; and empty input. Keep sufficient precision for coordinate differences and areas. Finite input coordinates do not guarantee representable areas or squared distances. The teaching model rejects detected overflow and underflow instead of returning NaN; its fixed scenes stay within this numerical domain. Test the selected kernel’s CPU/GPU contract. This sketch explains the operation; it is not a claim of measured optimized latency.

What changes across detection tasks?

Dense YOLO/SSD-style candidates and proposal-based Faster R-CNN predictions can need duplicate reduction. A face detector has the same duplicate issue, but scale and occlusion can change useful thresholds. Pedestrian scenes make the overlap-versus-identity failure especially important.

Text detection for optical character recognition, or OCR, can use suppression to remove repeated region predictions. Ordinary NMS does not merge text lines or establish reading order. Rotated rectangles or polygons need a compatible overlap calculation; axis-aligned IoU is a different geometry.

Suppression sits after spatial predictions, so a stronger backbone does not automatically fix the post-processing boundary. Test the detection head and selection policy together. DINOv2 and vision representations describes one representation-learning approach; Losses and Objectives owns general training objectives.

Suppression is one branch, not every detector’s final step

The progression here is greedy hard suppression → Soft-NMS score decay (2017) → distance-aware suppression and ensemble fusion (2019 preprints). These are related improvements, not a single mandatory sequence. Fusion changes coordinates; the other methods chiefly select or re-rank.

A separate branch changes the detector’s prediction contract. DETR (2020) uses set prediction and one-to-one training matching to avoid an ordinary NMS stage. Thus NMS is common in many pipelines, not necessary in every modern detector. DETR ↗ provides this counterexample.

For a new method, identify the failure it changes: proposal duplication, ranking, localization, crowd separation, assignment or evaluation. Compare methods under the same detector, candidate budget and evaluation conditions.

Sources and connected topics

• Torchvision NMS operator contract ↗: shapes, continuous xyxy, strict suppression boundary and tie caveat.
• Bodla et al. (2017), Soft-NMS ↗: linear/Gaussian score updates and evaluation.
• Zheng et al. (2019 preprint), Distance-IoU ↗: center normalization, regression losses and DIoU-NMS.
• Solovyev et al. (2019 preprint), Weighted Boxes Fusion ↗: ensemble clustering, coordinates and scores.
• Carion et al. (2020), DETR ↗: direct set prediction.
• COCO evaluation implementation ↗: ground-truth matching and ranked metrics.

Continue with Greedy and Interval Algorithms for ordered selection and its proof boundaries, Statistics and Evaluation for validation, and DINOv2 for backbone representations.