distillation
birdnet_stm32.training.distillation
¶
Frozen-teacher losses: QAT consistency, and online distillation from a larger model.
QAT perturbs a model that already converged. Hard labels alone do not tell the perturbed student where the original decision surface was, so it optimizes the supervised loss together with a Bernoulli KL divergence and a cosine agreement term against a frozen copy of the pre-compression checkpoint. The tail term targets the worst samples in each batch, which is where a quantized detector actually loses ranking quality.
Online distillation (OnlineDistilledModel) trains a smaller student from
scratch on its labels plus the same Bernoulli KL against a larger trained model,
run frozen on the very batch the student sees, augmentation and all, so no
offline pass over the training set is needed.
DistilledModel
¶
Bases: Model
Train a perturbed student against labels and a frozen float teacher.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
student
|
Model
|
Functional model holding the perturbed (quantized or masked) forward graph. Its variables are shared with the deployment model. |
required |
teacher
|
Model
|
Frozen copy of the pre-compression checkpoint. |
required |
distillation_weight
|
float
|
Weight of the per-output Bernoulli KL divergence. |
DEFAULT_DISTILLATION_WEIGHT
|
cosine_weight
|
float
|
Weight of the mean per-sample cosine distance. |
DEFAULT_COSINE_WEIGHT
|
cosine_tail_weight
|
float
|
Weight of the worst-sample cosine distance. |
DEFAULT_COSINE_TAIL_WEIGHT
|
cosine_tail_fraction
|
float
|
Fraction of each batch entering the tail term. |
DEFAULT_COSINE_TAIL_FRACTION
|
Source code in birdnet_stm32/training/distillation.py
compute_loss(x, y, y_pred, sample_weight=None, training=True)
¶
Add multi-label Bernoulli KL divergence to supervised BCE.
Source code in birdnet_stm32/training/distillation.py
OnlineDistilledModel
¶
Bases: Model
Train a student on its labels plus the Bernoulli KL to a frozen larger model's outputs on the same batch.
Called, it is the student: same inputs and outputs, so validation and
checkpoint selection see the deployed graph. Checkpoint the student
(train_model(training_wrapper=...) does), never this wrapper: the teacher
exists for training only.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
student
|
Model
|
The model being trained. Its variables are shared. |
required |
teacher
|
Model
|
The trained model to distil from; frozen here. It must take the student's input and score the student's classes in the same order. |
required |
weight
|
float
|
Weight of the KL divergence against the supervised loss. |
DEFAULT_ONLINE_DISTILLATION_WEIGHT
|
Source code in birdnet_stm32/training/distillation.py
distillation_loss(x, y_pred)
¶
The weighted KL term for one batch; also updates distill_kl.
Source code in birdnet_stm32/training/distillation.py
train_step(data)
¶
The default Keras step, plus the distillation term.
Source code in birdnet_stm32/training/distillation.py
test_step(data)
¶
bernoulli_kl(teacher_pred, y_pred)
¶
Mean Bernoulli KL divergence of the student's per-class probabilities from the teacher's.
Each output is its own Bernoulli variable (multi-label sigmoid outputs), so the divergence is the binary cross-entropy against the teacher minus the teacher's own entropy, averaged over classes and batch. Computed in float32 whatever the models' precision.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
teacher_pred
|
Tensor
|
|
required |
y_pred
|
Tensor
|
|
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
Scalar divergence (0 when the two agree). |
Source code in birdnet_stm32/training/distillation.py
all_layers(model)
¶
Return every nested layer of model exactly once, in graph order.
Source code in birdnet_stm32/training/distillation.py
validate_loss_weights(weights)
¶
Raise if a teacher-consistency weighting is outside its valid range.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
weights
|
dict[str, float]
|
Mapping with the four |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If any weight is negative or the tail fraction is not in (0, 1]. |
Source code in birdnet_stm32/training/distillation.py
check_distillation_teacher(student, teacher)
¶
Raise unless teacher takes the student's input and has as many outputs.
Raises:
| Type | Description |
|---|---|
ValueError
|
On a different input shape or class count. |