Evaluation¶

Evaluate a trained or quantized model on a test dataset.

Basic usage¶

python -m birdnet_stm32 evaluate \
  --model_path checkpoints/my_model_quantized.tflite \
  --model_config checkpoints/my_model_model_config.json \
  --data_path_test data/test \
  --pooling lme

The command:

Loads a .keras or .tflite model.
Reads _model_config.json for frontend and chunking parameters.
Splits each test file into non-overlapping chunks (up to --max_duration).
Runs batched inference on all chunks.
Pools chunk-level scores to file-level predictions.
Reports metrics and per-class statistics.

Pooling methods¶

Chunk scores are aggregated to file-level predictions using one of:

Method	Formula	Use case
`avg`	Arithmetic mean	Balanced baseline
`max`	Element-wise maximum	Good when target is present in few chunks
`lme`	\(\frac{1}{\beta} \log \left( \frac{1}{N} \sum_{i=1}^{N} e^{\beta \cdot s_i} \right)\)	Best overall — smoothly interpolates between avg and max

LME (log-mean-exponential) uses a fixed \(\beta = 10\).

Operational INT8 release gate¶

measure-operational evaluates the converted artifact itself at a pinned operating threshold. It refuses non-TFLite and non-INT8 models, mismatched model configs, incomplete class coverage, incomplete draws, or datasets without the hard negatives required by the gate. Inference is streamed in bounded batches.

It scores whole files, not single chunks. Each draw is a class-balanced set of --num_files recordings (round robin across classes, pinned by seed); each file's first 60 s is cut into chunks every chunk_duration - --chunk_overlap seconds (default overlap: half the chunk, 1.25 s for 2.5 s chunks, as in the catalog evaluation), every chunk is scored, and the scores are pooled per file (--pooling, default max). One random chunk is a poor test of a catalog recording, because many chunks hold no call; pooling asks whether the model finds the species anywhere in the file. That also makes the hard-negative rate stricter: a noise file alarms if any of its chunks does.

Two views of the pooled scores are reported per draw:

At each threshold: detection_rate (top-1 correct and confident), false_alarm_rate (top-1 wrong but confident), their macro versions, and negative_alarm_rate on hard negatives.
As a ranked species list, without a threshold: top_k_rates and macro_top_k_rates for k = 1, 3, 5 — how often the labelled species is in the top k — plus mean_reciprocal_rank and median_rank. Ties count against the labelled species.

Quality limits are release-specific and therefore must be supplied explicitly:

{
  "threshold": 0.5,
  "min_detection_rate": 0.70,
  "min_macro_detection_rate": 0.60,
  "max_false_alarm_rate": 0.10,
  "max_macro_false_alarm_rate": 0.15,
  "max_negative_alarm_rate": 0.05,
  "min_macro_top_k_rates": {"5": 0.80}
}

min_macro_top_k_rates is optional; the other fields are required. These numbers are placeholders, not calibrated limits: set them against a reference model on the same test set.

python -m birdnet_stm32 measure-operational \
  --model_path release/model_INT8.tflite \
  --model_config release/model_model_config.json \
  --data_path_test data/test \
  --gate_profile release_gate.json \
  --report_json report/model_INT8_operational.json

The command exits nonzero if any limit is missed. Detection floors use the worst seed; alarm ceilings use the worst seed in the opposite direction. The report records model, config, class-order, manifest, gate-profile, and measured input hashes without embedding machine-local paths.

Metrics¶

Metric	Description
ROC-AUC (micro)	Area under receiver operating characteristic, averaged over all class decisions
cmAP	Class-macro average precision — mean AP over classes that have positive examples
mAP	Micro average precision over all decisions
Precision	At threshold 0.5, file-level
Recall	At threshold 0.5, file-level
F1	Harmonic mean of precision and recall at threshold 0.5

The command also prints the top-10 and bottom-10 classes ranked by average precision.

Confusion matrix¶

Use --confusion_matrix to print an ASCII confusion matrix to stdout. Use --save_cm_plot path/to/plot.png to save a matplotlib figure.

Threshold optimization¶

By default, evaluation uses a fixed threshold of 0.5. Use --optimize_thresholds to find the per-class threshold that maximizes F1 via the precision-recall curve. Optimal thresholds are printed sorted by value.

Species-level AP report¶

Use --species_report path/to/species.csv to save a per-species average precision report with bootstrap confidence intervals. The CSV includes columns:

Column	Description
`class`	Species name
`ap`	Point-estimate average precision
`ci_lower`	95% confidence interval lower bound
`ci_upper`	95% confidence interval upper bound
`n_positive`	Number of positive test files for this class
`n_total`	Total number of test files

Control the number of bootstrap resamples with --n_bootstrap (default 1000). Higher values produce tighter CI estimates but take longer.

python -m birdnet_stm32 evaluate \
  --model_path checkpoints/my_model_quantized.tflite \
  --model_config checkpoints/my_model_model_config.json \
  --data_path_test data/test \
  --species_report report/species_ap.csv \
  --n_bootstrap 2000

DET curve¶

The Detection Error Tradeoff (DET) curve plots false rejection rate (FRR) against false acceptance rate (FAR) across thresholds — a standard metric in bioacoustics evaluation.

--det_curve — print an ASCII DET curve to stdout
--save_det_plot path/to/det.png — save a matplotlib DET curve image

python -m birdnet_stm32 evaluate \
  --model_path checkpoints/my_model_quantized.tflite \
  --model_config checkpoints/my_model_model_config.json \
  --data_path_test data/test \
  --det_curve --save_det_plot report/det_curve.png

Latency measurement¶

Use --benchmark_latency to measure per-chunk inference time. When enabled, the evaluation loop wraps each model.predict() call with high-resolution timing. The following statistics are added to the metrics output:

Metric	Description
`latency_mean_ms`	Mean inference time per chunk (ms)
`latency_median_ms`	Median inference time per chunk (ms)
`latency_p95_ms`	95th percentile latency (ms)
`latency_p99_ms`	99th percentile latency (ms)
`total_chunks`	Total number of chunks processed

Host timing

Latency is measured on the host CPU/GPU, not on-device. For on-device latency, use stedgeai validate (see Deployment).

Benchmark mode¶

Use --benchmark path/to/benchmark.json to save a structured JSON report containing all metrics, per-species AP with CIs, model config, and latency stats. This is designed for experiment tracking and automated comparison.

The JSON report contains:

{
  "model_path": "checkpoints/my_model_quantized.tflite",
  "num_classes": 10,
  "num_files": 499,
  "metrics": {
    "roc-auc": 0.8521,
    "cmAP": 0.7834,
    "f1": 0.6912,
    "latency_mean_ms": 12.3,
    "latency_p95_ms": 14.1
  },
  "per_class_ap": {
    "northern_cardinal": 0.9142,
    "wind": 0.1191
  },
  "species": [ ... ],
  "config": { ... }
}

per_class_ap maps each class name to its average precision. It is keyed by name rather than index so that reordering the class list cannot silently misattribute scores, and it is written whenever --benchmark is used — --species_report adds bootstrap confidence intervals under species, but is not required for the per-class numbers.

Beyond a few dozen outputs, the macro average stops being informative on its own: cmAP over a hundred classes averages a well-supplied species against one with a few hundred recordings and says nothing about which is carrying it. Group per_class_ap by whatever stratification matters for the model — output tier, available training data, taxonomic group — and compare within strata. A regression confined to one stratum and one spread evenly across all of them call for different responses, and a single average distinguishes neither.

To include latency stats in the benchmark, combine with --benchmark_latency:

python -m birdnet_stm32 evaluate \
  --model_path checkpoints/my_model_quantized.tflite \
  --model_config checkpoints/my_model_model_config.json \
  --data_path_test data/test --pooling lme \
  --benchmark report/benchmark.json --benchmark_latency \
  --species_report report/species_ap.csv

HTML report¶

Use --report_html path/to/report.html to generate a self-contained HTML evaluation report. The report includes:

Summary metrics table
Per-species average precision table (if --species_report or --benchmark computes species data)
Confusion matrix heatmap (uses base64-embedded matplotlib image)
Inline CSS styling — no external dependencies needed to view

python -m birdnet_stm32 evaluate \
  --model_path checkpoints/my_model_quantized.tflite \
  --model_config checkpoints/my_model_model_config.json \
  --data_path_test data/test --pooling lme \
  --report_html report/eval_report.html

Saving results¶

Use --save_csv to export per-file predictions. Evaluation run CSVs are stored in report/eval_runs/ with the naming convention:

{run_number}_{frontend}_{mag}_{alpha}_{depth}_{embed}_{batch}_{maxsamples}.csv

Full argument reference¶

Argument	Default	Description
`--model_path`	(required)	Path to `.keras` or `.tflite` model
`--model_config`	(inferred)	Path to `_model_config.json`
`--data_path_test`	(required)	Test data root with class subfolders
`--max_files`	-1 (all)	Max files per class
`--batch_size`	16	Chunk inference batch size
`--pooling`	avg	`avg`, `max`, or `lme`
`--overlap`	0	Chunk overlap in seconds
`--save_csv`	None	Path to save per-file predictions as CSV
`--confusion_matrix`	False	Print ASCII confusion matrix
`--save_cm_plot`	None	Save confusion matrix plot to image file
`--optimize_thresholds`	False	Find per-class optimal F1 thresholds
`--benchmark`	None	Save structured JSON benchmark report to this path
`--benchmark_latency`	False	Measure per-chunk inference latency (mean, median, p95, p99)
`--species_report`	None	Save per-species AP report with 95% bootstrap CI to CSV
`--n_bootstrap`	1000	Number of bootstrap resamples for CI estimation
`--det_curve`	False	Print ASCII DET curve
`--save_det_plot`	None	Save DET curve plot to image file
`--report_html`	None	Generate a self-contained HTML evaluation report

Full evaluation example¶

Run a comprehensive evaluation with all reporting options:

python -m birdnet_stm32 evaluate \
  --model_path checkpoints/my_model_quantized.tflite \
  --model_config checkpoints/my_model_model_config.json \
  --data_path_test data/test \
  --pooling lme \
  --confusion_matrix --save_cm_plot report/confusion_matrix.png \
  --optimize_thresholds \
  --benchmark report/benchmark.json --benchmark_latency \
  --species_report report/species_ap.csv --n_bootstrap 2000 \
  --det_curve --save_det_plot report/det_curve.png \
  --report_html report/eval_report.html \
  --save_csv report/predictions.csv