Algorithm Overview

This section describes the algorithmic ideas and the concepts utilized within BirdBox during inference.

For user-facing parameters see CLI Reference or API Reference. To compare different models, take a look at Models and Metrics. If you are interested in the file flow see File Flows.

Inference Algorithm Concept

The key idea of the BirdBox inference is to use YOLO[23] from Ultralytics[9] with a sliding window approach. Both BirdBox models are YOLO11 object detectors: Just-Bird (YOLO11s, one bird class) and Birds-With-Labels (YOLO11l, 282 species). YOLO cannot run on continuous audio, so the recording is cut into short, overlapping images:

1. Load the audio. It is mixed to mono and resampled to 32 kHz. Silence is added at the end so that the last 3 s window is complete; boxes in that silence are cut at the real end of the file.
2. Compute a PCEN mel spectrogram with STFT[1] and PCEN[18], one minute at a time to bound the memory.
3. Cut 3 s clips every 1.5 s (50% overlap) and render each as a 256×256 pixel image. BirdNET analyzes audio in 3 s segments as well [10].
4. Run YOLO on the clips in batches. Boxes below the confidence threshold are dropped (default 0.15 for Just-Bird, 0.2 for Birds-With-Labels), and overlapping boxes within a clip are reduced by non-maximum suppression [8] (--nms-iou, default 0.7).
5. Convert the boxes from pixels to seconds and Hz. The frequency axis follows the HTK mel scale from 50 Hz to 15 kHz, the inverse of how the training labels were drawn.
6. Merge the boxes of overlapping windows into one set of detections per file, with window fusion (Just-Bird) or the song merge (Birds-With-Labels), see below.
7. Write the detections in the selected output formats.

To examine the generation of the spectrograms in detail, see Spectrogram Generation. The various output data formats provided by BirdBox can be examined in Detection Output Formats.

Merging the Windows

Consecutive windows overlap by half, so almost every call is seen by two windows. Any vocalization of 1.5 s or less lies completely inside at least one window; only longer vocalizations can be cut into parts. The two views of a call rarely give exactly the same box, and the merge decides what is kept. BirdBox has two merges, and each model has its default (--merge overrides it).

Window Fusion (Just-Bird)

Window fusion keeps one tight box per call:

1. Ownership. Each window owns its middle half; the first window also owns the start of the file, the last one the end. A box is kept only from the window that owns its centre, which is the window that sees the call most centrally. The middle halves tile the file, so every call is kept once.
2. Weighted box fusion. Boxes of one window that overlap at 2D IoU ≥ 0.4 are averaged, weighted by confidence [27].
3. Join split calls. Boxes in the same frequency band that overlap in time are joined, for example a long call that two windows cut differently.
4. Drop nested boxes. A box that lies at least 80% inside a larger box is removed, for example a syllable inside the box of its whole call.

With a phrase gap above 0, step 3 instead joins boxes in the same band that are closer than the gap, so syllables become phrase boxes. Just-Bird uses 0.05 s by default; --phrase-gap 0 keeps one box per call. See Window fusion for the scores behind these settings.

Song Merge (Birds-With-Labels)

The song merge joins detections of the same species whose gap is at most the song gap (--song-gap, default 0.1 s) into one song box that spans all of them. Most of the Birds-With-Labels training labels box whole songs, and the wide song boxes match them better than single calls. For Just-Bird (--merge reconstruct), boxes are only joined if they also share the frequency band, so calls of different birds are not chained.

For details, see Song Gap Threshold.

Filter, Then Merge

The confidence threshold is applied to the raw boxes of each window, before merging. The evaluation tools work the same way: detect_birds.py --no-merge writes the raw boxes once, and F-beta analysis or Filter and Merge Detections filter them at any threshold and merge them as detection would.

References

[1] Allen, J. B., & Rabiner, L. R. (1977). "A unified approach to short-time Fourier analysis and synthesis." Proceedings of the IEEE, 65(11), 1558–1564.

[8] Hosang, J., Benenson, R., & Schiele, B. (2017). "Learning non-maximum suppression." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4507–4515.

[9] Jocher, G., & Qiu, J. (2024). Ultralytics YOLO11 (Version 11.0.0) [Computer software]. AGPL-3.0. https://github.com/ultralytics/ultralytics. There is no YOLO11 paper; this is the citation given in the YOLO11 documentation.

[10] Kahl, S., Wood, C. M., Eibl, M., & Klinck, H. (2021). "BirdNET: A deep learning solution for avian diversity monitoring." Ecological Informatics, 61:101236.

[18] Lostanlen, V. (2021). "Self-calibrating acoustic sensor networks with per-channel energy normalization." Proceedings of Euronoise 2021. Sociedade Portuguesa de Acústica.

[23] Redmon, J., et al. (2016). "You Only Look Once: Unified, Real-Time Object Detection.“ Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27] Solovyev, R., Wang, W., & Gabruseva, T. (2021). "Weighted boxes fusion: Ensembling boxes from different object detection models." Image and Vision Computing, 104117.