Advanced: Merge Hyperparameter Search

Optional grid-search of inference merge hyperparameters (--song-gap, and optionally --nms-iou) on a frozen checkpoint. Most users never need this: the defaults in run_pipeline.sh are the usual evaluation path.

Advanced / optional

Skip this page unless you are already evaluating against labeled audio and want to retune merge parameters. For detection and F-beta with stock settings, see Typical Workflows.

Each setting is scored with the same 1-minute event F-beta protocol as f_beta_score_analysis.py: filter raw detections by confidence, merge, then Hungarian 2D IoU against annotations.csv. Confidence is swept at every grid point; the reported number is the best Overall_Micro F-beta.

Learning-rate and augmentation search lives in BirdBox-Train and is also optional. Do not mix the two: freeze --nms-iou / --song-gap while training, then run this search on the winning weights.

---

Workflow

run_hpo.sh at the repository root is the intended way to run this search. Edit variables at the top and leave exactly one option enabled — the same interaction style as run_pipeline.sh, but this script is not part of the default pipeline.

Option A (default, cheap) reuses one raw_detections.json and only sweeps --song-gap. Option B (expensive) re-runs detect_birds.py --no-merge once per --nms-iou, then sweeps --song-gap on each dump. Detection dominates wall time; use B only when you need to search NMS.

Linux / macOSWindows (PowerShell)Windows (CMD)

#!/bin/bash
#
# Advanced / optional: search merge hyperparameters (song_gap, nms_iou)
# on a frozen model. Everyday detect + evaluate is run_pipeline.sh
# (or .bat / .ps1). Skip unless you have labels and want to retune merge.

# Exit immediately if a command fails
set -e

# Optional: activate local virtual environment if present.
# If .venv does not exist, the script uses the current Python on PATH.
if [ -f ".venv/bin/activate" ]; then
    echo "Activating .venv"
    # shellcheck source=/dev/null
    source ".venv/bin/activate"
fi


#### select the dataset on which inference shall be performed #####
# DATASET_NAME="All-In-One_testset"
# DATASET_NAME="Western-US"
# DATASET_NAME="Hawaii_testset"
# DATASET_NAME="Northeastern-US_testset-subset"
DATASET_NAME="Northeastern-US_testset"


#### derive base name (strip dataset suffix for model, mapping, and results) #####
DATASET_BASE="${DATASET_NAME/_testset-subset/}"
DATASET_BASE="${DATASET_BASE/_testset/}"


#### select model #####
MODEL_PATH="models/${DATASET_BASE}.pt"
# MODEL_PATH="models/Just-Bird.pt"
# MODEL_PATH="models/All-In-One-Transfer.pt"


#### select the species mapping (according to dataset and model) #####
SPECIES_MAPPING="${DATASET_BASE}"
# SPECIES_MAPPING="Just-Bird"
# SPECIES_MAPPING="All-In-One"


#### toggle single class mode #####
# USE_SINGLE_CLS=true
USE_SINGLE_CLS=false


#### select output path #####
OUTPUT_PATH="results/${DATASET_BASE}/hyperopt_merge"
# OUTPUT_PATH="results/Just-Bird/hyperopt_merge"


#### merge-parameter grid #####
SONG_GAPS=(0.0 0.1 0.2 0.5 1.0 2.0)
NMS_IOUS=(0.5 0.6 0.7 0.8)

IOU_THRESHOLD=0.25
BETA=1.0
DETECT_CONF=0.001
DETECT_WORKERS=18
FBETA_WORKERS=9


#########################################################################
# Choose exactly one of the two options below (they are alternatives,
# not sequential steps). Detect + single F-beta stay in run_pipeline.sh.
# Freeze these merge params while training (BirdBox-Train/run_hpo.sh).
#
# Runtime: Option A reuses one raw_detections.json (no new detect).
# Option B re-runs detect_birds --no-merge once per NMS_IOUS value,
# which dominates wall time. Prefer A unless you need to search nms_iou.
#########################################################################


SINGLE_CLS_FLAG=()
if [ "${USE_SINGLE_CLS}" = true ]; then
    SINGLE_CLS_FLAG+=(--single-cls)
fi


### only enable one of the two ###
# Option A (cheap): sweep song_gap only. Reuses the existing
# detect_birds --no-merge dump (results/<DATASET>/raw_detections.json).
# Does not run detection again. nms_iou is whatever that dump was built with.
echo "Option A: sweeping song_gap on existing raw detections (no new detect)..."
python src/evaluation/hyperopt_merge.py \
    --raw-detections "results/${DATASET_BASE}" \
    --labels "datasets/${DATASET_NAME}/annotations.csv" \
    --output-path "${OUTPUT_PATH}" \
    --song-gaps "${SONG_GAPS[@]}" \
    --iou-threshold "${IOU_THRESHOLD}" \
    --beta "${BETA}" \
    --num-workers "${FBETA_WORKERS}" \
    --no-plot \
    "${SINGLE_CLS_FLAG[@]}"

# Option B (expensive): sweep nms_iou and song_gap.
# nms_iou is applied inside YOLO, so detect_birds --no-merge runs once per
# NMS_IOUS value (len(NMS_IOUS) full inference passes), then song_gap is
# swept on each dump. Comment Option A if you enable this.
# echo "Option B: re-running detect once per nms_iou, then sweeping song_gap..."
# python src/evaluation/hyperopt_merge.py \
#     --model "${MODEL_PATH}" \
#     --audio "datasets/${DATASET_NAME}/soundscape_data" \
#     --species-mapping "${SPECIES_MAPPING}" \
#     --labels "datasets/${DATASET_NAME}/annotations.csv" \
#     --output-path "${OUTPUT_PATH}" \
#     --nms-ious "${NMS_IOUS[@]}" \
#     --song-gaps "${SONG_GAPS[@]}" \
#     --detect-conf "${DETECT_CONF}" \
#     --detect-workers "${DETECT_WORKERS}" \
#     --iou-threshold "${IOU_THRESHOLD}" \
#     --beta "${BETA}" \
#     --num-workers "${FBETA_WORKERS}" \
#     --no-plot \
#     "${SINGLE_CLS_FLAG[@]}"
# ### only enable one of the two ###


echo
echo "All tasks completed!"
echo "Merge HPO summary is in ${OUTPUT_PATH}/summary.csv"

# Option A (cheap): song_gap only, reuse an existing --no-merge dump (no new detect)
python src/evaluation/hyperopt_merge.py `
    --raw-detections results/Northeastern-US `
    --labels datasets/Northeastern-US_testset/annotations.csv `
    --output-path results/Northeastern-US/hyperopt_merge `
    --song-gaps 0.0 0.1 0.2 0.5 1.0 2.0 `
    --num-workers 9 `
    --no-plot

rem Option A (cheap): song_gap only, reuse an existing --no-merge dump (no new detect)
python src/evaluation/hyperopt_merge.py ^
    --raw-detections results/Northeastern-US ^
    --labels datasets/Northeastern-US_testset/annotations.csv ^
    --output-path results/Northeastern-US/hyperopt_merge ^
    --song-gaps 0.0 0.1 0.2 0.5 1.0 2.0 ^
    --num-workers 9 ^
    --no-plot

---

CLI synopsis

Direct call to hyperopt_merge.py without the shell driver:

Linux / macOSWindows (PowerShell)Windows (CMD)

python src/evaluation/hyperopt_merge.py \
    --raw-detections results/Northeastern-US \
    --labels datasets/Northeastern-US_testset/annotations.csv \
    --song-gaps 0.0 0.1 0.2 0.5 1.0 2.0

python src/evaluation/hyperopt_merge.py `
    --raw-detections results/Northeastern-US `
    --labels datasets/Northeastern-US_testset/annotations.csv `
    --song-gaps 0.0 0.1 0.2 0.5 1.0 2.0

python src/evaluation/hyperopt_merge.py ^
    --raw-detections results/Northeastern-US ^
    --labels datasets/Northeastern-US_testset/annotations.csv ^
    --song-gaps 0.0 0.1 0.2 0.5 1.0 2.0

Checkpoint Must Be Frozen

This search does not train. It only changes how existing boxes are suppressed (--nms-iou) and stitched into songs (--song-gap). Pass the same .pt you would use in detect_birds.py.

---

What Is Searched

Parameter	When it is applied	Needs a new detect pass?
--song-gap	In F-beta, after filtering by confidence (filter-then-merge)	No — reuse one raw_detections.json
--nms-iou	Inside YOLO during detect_birds.py --no-merge	Yes — one raw dump per NMS value
--conf	Swept automatically at every grid point	No

--iou-threshold (matching detections to labels) is not part of this grid. Keep it at the evaluation default (0.25) so scores stay comparable to run_pipeline.sh.

Detection is the expensive part

--song-gap is applied in F-beta on an existing dump, so many gap values are cheap. --nms-iou is applied inside YOLO, so each NMS value is a full inference pass over the testset. Option A in run_hpo.sh never re-detects. Option B runs detect len(--nms-ious) times.

---

Parameters

Parameter	Type / Default	Required?	Description
--raw-detections	PATH / —	Yes, unless --nms-ious	Existing raw detections file or results directory from detect_birds.py --no-merge. Used for a song_gap-only search.
--labels	PATH / —	Yes	Ground truth annotations.csv. Filenames are matched without extensions.
--output-path	PATH / results/hyperopt_merge	No	Root directory for per-setting F-beta runs and summary.csv.
--song-gaps	FLOAT [...] / 0.0 0.1 0.2 0.5 1.0 2.0	No	song_gap values in seconds to evaluate.
--nms-ious	FLOAT [...] / unset	No	If set, re-run detect --no-merge once per NMS IoU (the expensive path), then sweep --song-gaps on each dump. Requires --model, --audio, and --species-mapping. Omit this to reuse one dump (Option A).
--model	PATH / —	With --nms-ious	Frozen YOLO checkpoint.
--audio	PATH / —	With --nms-ious	Audio file or directory (typically the testset soundscape_data/ folder).
--species-mapping	CHOICE / —	With --nms-ious	Must match the mapping the model was trained with. See species mapping.
--detect-conf	FLOAT / 0.001	No	Confidence used when re-running detect for --nms-ious. Keep low so F-beta can sweep later.
--detect-workers	INT / 8	No	CPU workers for those detect passes.
--iou-threshold	FLOAT / 0.25	No	IoU for matching events to labels. Use the same value as f_beta_score_analysis.py.
--beta	FLOAT / 1.0	No	F-beta weighting. Same meaning as in F-beta analysis.
--conf-range	MIN MAX STEP / 0.00 1.0 0.01	No	Confidence grid at each (nms_iou, song_gap) pair.
--num-workers	INT / 8	No	Worker processes for each F-beta confidence sweep.
--no-plot	flag / off	No	Skip F-beta plots (faster; run_hpo.sh sets this).
--single-cls	flag / off	No	Collapse all species into one class.
--single-cls-name	STR / bird	No	Class name when --single-cls is set.

---

Parameter Deep-Dives

--song-gaps — merge after filtering

--song-gap is applied inside F-beta: at each confidence, surviving raw boxes of the same species are merged when the gap between them is ≤ this value. That is the same filter-then-merge policy as the app and f_beta_score_analysis.py --song-gap.

Because merging happens after detect, one --no-merge dump is enough to try every gap.

--nms-ious — suppress duplicates at detect time

NMS runs inside YOLO on each 3-second clip. Changing it changes which boxes are written to raw_detections.json, so the script calls detect_birds.py --no-merge once per value. That is the main compute cost of Option B: four NMS values means four full testset inference runs, before any song_gap sweep.

Relationship to --song-gap

--nms-iou removes overlapping boxes within a clip. --song-gap then stitches surviving boxes across time into songs. They are different stages; searching both is a product grid (nms × song_gap), each cell still sweeping confidence.

Training search vs this search

Training hyperparameter search is a separate, optional tool in BirdBox-Train. It retrains YOLO (learning rate, mixup, …) and scores each trial with one frozen pair such as nms_iou=0.7, song_gap=0.2, so trials differ by training recipe rather than merge settings.

If you later copy a winning best.pt into models/, this page is where you can retune merge values on labeled audio. Neither search is required for ordinary detection or evaluation.

---

Output Files

All files are written under --output-path. Each grid cell gets its own folder:

results/<dataset>/hyperopt_merge/
  summary.csv
  nms_0.70/song_gap_0.20/
    f1.0_score_analysis.csv
    optimal_thresholds.csv
    score.json
  nms_0.70/song_gap_0.50/
    ...

When --nms-ious is set, each NMS value also stores its detect dump at nms_<value>/detect/raw_detections.json.

File	Description
summary.csv	One row per (nms_iou, song_gap) with best micro F-beta, confidence, precision, and recall.
score.json	Same numbers for that cell.
f{beta}_score_analysis.csv	Full F-beta table for that cell (same schema as the standalone F-beta script).
optimal_thresholds.csv	Best confidence per species at that merge setting.

Open summary.csv and take the row with the highest micro_f1. Then set --nms-iou / --song-gap in run_pipeline.sh to those values for the final test evaluation.

---

Examples

song_gap only (Option A — no new detect)

Run detect_birds.py --no-merge once first (or uncomment Step 1 in run_pipeline.sh). Then reuse that dump:

CommandExpected Output

python src/evaluation/hyperopt_merge.py \
    --raw-detections results/Northeastern-US \
    --labels datasets/Northeastern-US_testset/annotations.csv \
    --output-path results/Northeastern-US/hyperopt_merge \
    --song-gaps 0.0 0.1 0.2 0.5 1.0 2.0 \
    --num-workers 9 \
    --no-plot

=== nms_iou=0.80 song_gap=0.0 ===
micro F1=0.54..  conf=0.33  P=...  R=...
...
Best: nms_iou=0.8 song_gap=0.2 micro F1=0.55.. conf=0.33
Wrote results/Northeastern-US/hyperopt_merge/summary.csv

This is the default option in run_hpo.sh. Wall time is F-beta only (one dump, many gaps).

nms_iou and song_gap together (Option B — re-detect per NMS)

Command

python src/evaluation/hyperopt_merge.py \
    --model models/Northeastern-US.pt \
    --audio datasets/Northeastern-US_testset/soundscape_data \
    --species-mapping Northeastern-US \
    --labels datasets/Northeastern-US_testset/annotations.csv \
    --output-path results/Northeastern-US/hyperopt_merge \
    --nms-ious 0.5 0.6 0.7 0.8 \
    --song-gaps 0.1 0.2 \
    --detect-workers 18 \
    --num-workers 9 \
    --no-plot

Uncomment Option B in run_hpo.sh for the same grid (and comment Option A so only one search runs). Expect one full --no-merge detect pass per --nms-ious value.

---

After the search

The usual BirdBox path stays run_pipeline.sh. Use this page only if you want different merge values than the pipeline defaults.

Usual (most users)     run_pipeline.sh     detect + F-beta + confusion matrix
Optional (this page)   run_hpo.sh          pick song_gap (and nms_iou) on a frozen model
                       then run_pipeline.sh with those values
Optional (Train repo)  run_hpo.sh          search training hyperparameters

If you did run the search:

1. Read summary.csv and pick the best nms_iou / song_gap.
2. Set those values in run_pipeline.sh (detect --nms-iou, F-beta --song-gap).
3. Run the usual evaluation steps on the testset.

---

See Also

• Typical Workflows — the usual detect and evaluate path
• Detecting Birds — --song-gap and --nms-iou
• F-Beta Score Analysis