Spectrogram Generation¶

BirdBox converts raw audio into Per-Channel Energy Normalization (PCEN) mel spectrograms before running YOLO detection. This page documents the complete preprocessing pipeline. Following these steps exactly lets you reproduce the same spectrogram representation used during both training and inference.

For the full implementation details, you can have a look at pcen_inference.py.

Pipeline at a Glance¶

The pipeline runs in six sequential stages:

Load and normalize audio to a common amplitude range
Resample to 32,000 Hz, pad the end and split the audio into one-minute segments
Compute the Short-Time Fourier Transform (STFT)
Apply a mel filterbank to compress frequency bins
Apply PCEN normalization (with warmup and left-pad strategies)
Extract fixed-length clips and render them as 256×256 pixel images

Stages 3 to 6 run once per segment. Each step is described in detail below, including all parameter values.

Stage 1: Audio Loading and Normalization¶

BirdBox reads audio using soundfile, with a librosa fallback for problematic files. Stereo recordings are mixed down to mono by averaging both channels.

The amplitude is then scaled to the integer range before any spectral processing:

audio = (audio * (2 ** 31)).astype("float32")

This maps the normalized float input, typically in [-1.0, 1.0], into the range of a 32-bit signed integer. The scaling is required to match the magnitude assumptions built into the PCEN parameters.

Stage 2: Resampling, End Padding and Segments¶

All audio is resampled to a fixed sample rate using librosa.resample. Files already at the target rate are passed through without modification.

The clips start every 1.5 s. If the file does not end on that grid, or is shorter than 3 s, silence is added at the end until the last clip is complete. Boxes that the model finds in this silence are cut at the real end of the file, and boxes that start after it are dropped.

The spectrogram is not computed for the whole file at once. To bound the memory, the audio is split into one-minute segments. Each segment is computed with 2 s of the neighbouring audio before and after it, so a clip that starts near the end of a segment is still complete. Each clip is taken from the segment in which it starts. Segments are independent, so for a single file --num-workers computes them in parallel (for several files it processes the files in parallel).

Parameter	Value
Target sample rate	`32,000 Hz`
Segment length	`60 s`
Context before and after a segment	`2 s`
End padding	silence up to the end of the last 3 s clip

Stage 3: Short-Time Fourier Transform¶

The STFT is computed using librosa.stft[20] with the following parameters. Note that the window length equals the FFT size to avoid zero-padding artifacts inside the analysis window.

Parameter	Value	Notes
`n_fft`	`2048`	FFT size; sets frequency resolution
`win_length`	`2048`	Equals `n_fft`; no zero-padding inside window
`hop_length`	`375`	Frame step in samples (~11.72 ms per frame)
`window`	`"flattop"`	Flat-top window for accurate amplitude representation
`center`	`False`	No edge-padding; frames start at sample 0

The power spectrogram is derived from the complex STFT output:

stft = librosa.stft(audio, n_fft=2048, win_length=2048,
                    hop_length=375, window="flattop", center=False)
abs2_stft = np.abs(stft) ** 2

Why flat-top?¶

A flat-top window has a very flat passband in the frequency domain. This minimizes amplitude error for tonal signals, which is important for accurately representing bird vocalizations that may be near bin boundaries. This error between bins is known as scalloping loss [5], and flat-top windows are designed to keep it small [6].

Stage 4: Mel Filterbank¶

The power spectrogram is mapped onto a mel-scale filterbank using librosa.feature.melspectrogram. BirdBox uses the HTK mel formula throughout, consistent with both training and inference.

Parameter	Value	Notes
`n_mels`	`256`	Number of mel bands (output height)
`fmin`	`50 Hz`	Lowest frequency included
`fmax`	`15,000 Hz`	Highest frequency included
`htk`	`True`	Uses the HTK formula, not the Slaney formula

melspec = librosa.feature.melspectrogram(
    S=abs2_stft, sr=32000, n_fft=2048,
    n_mels=256, fmin=50, fmax=15000, htk=True
)

Frequency Range

The 50–15,000 Hz range covers the vast majority of bird vocalizations while discarding low-frequency rumble and ultrasonic content above the range of interest. The HTK formula spaces mel bands differently than the Slaney formula. Using the wrong formula at inference will produce mismatched filterbanks and degrade detection performance.

Stage 5: PCEN Normalization¶

Per-Channel Energy Normalization (PCEN) replaces the simple log-compression step found in traditional mel spectrograms. It applies a time-varying automatic gain control followed by a static compression nonlinearity, making the output more robust to varying background noise levels. PCEN was introduced for keyword spotting [29]. On recordings of natural environments it brings the distribution of magnitudes close to a Gaussian and decorrelates the frequency bands [16], and for bird flight calls it reduced overfitting to the time of day [17].

PCEN is computed using librosa.pcen:

pcen_output = librosa.pcen(
    melspec, sr=32000, hop_length=375,
    gain=0.75,          # norm_exponent
    bias=1.0,           # delta
    power=0.35,
    time_constant=1.0
)

Parameter	librosa argument	Value	Role
Norm exponent	`gain`	`0.75`	Controls strength of the AGC smoothing
Delta (bias)	`bias`	`1.0`	Stabilizes division near silence
Compression exponent	`power`	`0.35`	Dynamic range compression applied after AGC
Time constant	`time_constant`	`1.0`	Smoothing time for the AGC filter (seconds)
Hop length	`hop_length`	`375`	Must match the STFT hop to convert time constant correctly

Warmup Loop¶

The PCEN AGC filter requires several frames to reach steady state from a cold start. A warmup sequence is prepended by looping the first frames of the mel spectrogram back onto itself:

loop_length = min(100, melspec.shape[1] // 4)
melspec_looped = np.concatenate([melspec[:, :loop_length], melspec], axis=1)

After PCEN is computed on this extended input, the prepended warmup frames are discarded:

pcen_segment = pcen_looped[:, loop_length:]

Left-Pad Strategy¶

Before the STFT, the first 0.5 s of the segment audio is copied and prepended to it. This gives the AGC filter extra context for the very start of each segment.

segment_with_pad = np.concatenate([segment_audio[:16000], segment_audio])

Parameter	Value
`left_pad_length`	`0.5 s` (16,000 samples at 32 kHz)
Pad frames dropped after PCEN	`42` (16,000 // 375)

The pad frames are dropped from the PCEN output after processing:

pcen_pad_frames = int(0.5 * 32000) // 375  # = 42
pcen_segment = pcen_segment[:, pcen_pad_frames:]

Stage 6: Clip Extraction and Image Rendering¶

Clip Parameters¶

Fixed-length clips are extracted from the PCEN output using a sliding window with 50% overlap. Each clip covers exactly 3 seconds of audio.

Parameter	Value
Clip length	`3 s` (252 frames)
Clip hop	`1.5 s` (50% overlap)
Output shape per clip	`256 × 252` (mel bands × time frames)

The 252-frame clip length follows from the hop length: 252 × 375 / 32,000 ≈ 2.953 s, which BirdBox rounds to 3 seconds.

Image Rendering¶

Each PCEN clip is rendered as a 256×256 pixel PNG using matplotlib, with all axes and labels stripped. The image is produced at exactly 100 DPI from a 2.56×2.56-inch figure.

fig = Figure(figsize=(2.56, 2.56), dpi=100)
ax = fig.add_subplot(111)
librosa.display.specshow(
    pcen_data, sr=32000, hop_length=375,
    ax=ax, cmap="inferno", vmin=0.0, vmax=100.0
)
ax.axis("off")
fig.subplots_adjust(left=0, right=1, top=1, bottom=0)

Parameter	Value
Figure size	`2.56 × 2.56 inches`
DPI	`100` (yields 256×256 pixels)
Colormap	`inferno`
`vmin`	`0.0`
`vmax`	`100.0`

Exact Parameters Required

The vmin and vmax values are baked into the model's training distribution. Using a different colormap or rescaling will shift pixel intensities relative to what the YOLO model was trained on, causing unpredictable detection results.

Complete Pipeline Reference¶

The table below summarizes every numerical parameter across all stages for quick reference.

Stage	Parameter	Value
Normalization	Amplitude scale	`× 2³¹`
Resampling	Target sample rate	`32,000 Hz`
Segments	Length	`60 s`, with `2 s` of context on each side
Segments	End padding	silence up to the end of the last clip
STFT	`n_fft`	`2048`
STFT	`win_length`	`2048`
STFT	`hop_length`	`375`
STFT	`window`	`"flattop"`
STFT	`center`	`False`
Mel filterbank	`n_mels`	`256`
Mel filterbank	`fmin`	`50 Hz`
Mel filterbank	`fmax`	`15,000 Hz`
Mel filterbank	`htk`	`True`
PCEN	`gain` (norm exponent)	`0.75`
PCEN	`bias` (delta)	`1.0`
PCEN	`power`	`0.35`
PCEN	`time_constant`	`1.0 s`
PCEN	Left pad	`0.5 s` copy of the segment start (42 frames dropped)
Clips	Clip length	`3 s` / `252 frames`
Clips	Clip hop	`1.5 s` (50% overlap)
Image	Size	`256 × 256 px`
Image	Colormap	`inferno`
Image	`vmin` / `vmax`	`0.0` / `100.0`

References¶

[5] Harris, F. J. (1978). "On the use of windows for harmonic analysis with the discrete Fourier transform." Proceedings of the IEEE, 66(1):51–83.

[6] Heinzel, G., Rüdiger, A., & Schilling, R. (2002). "Spectrum and spectral density estimation by the Discrete Fourier transform (DFT), including a comprehensive list of window functions and some new flat-top windows." Technical report, Max-Planck-Institut für Gravitationsphysik (Albert-Einstein-Institut), Hannover.

[16] Lostanlen, V., Salamon, J., Cartwright, M., McFee, B., Farnsworth, A., Kelling, S., & Bello, J. P. (2019). "Per-channel energy normalization: Why and how." IEEE Signal Processing Letters, 26(1):39–43.

[17] Lostanlen, V., Salamon, J., Farnsworth, A., Kelling, S., & Bello, J. P. (2019). "Robust sound event detection in bioacoustic sensor networks." PLOS ONE, 14(10):e0214168.

[20] McFee, B., et al. (2015). "librosa: Audio and Music Signal Analysis in Python." Proceedings of the 14th Python in Science Conference (SciPy).

[29] Wang, Y., Getreuer, P., Hughes, T., Lyon, R. F., & Saurous, R. A. (2017). "Trainable frontend for robust and far-field keyword spotting." Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5670–5674.