embedding_distillation
birdnet_stm32.training.embedding_distillation
¶
Teacher embedding distillation: pull the student's embedding toward the teacher's.
Soft labels hand the student the teacher's scores for our own classes. The teacher's embedding carries more: what a window sounds like across its whole vocabulary, including species outside our schema, and a signal for the non-bird classes and the noise recordings, for which it has no scores at all.
A training-only linear projector maps the student's pooled embedding (gap)
to the teacher's width, and a cosine loss pulls the two together. The
projector never reaches the checkpoint: the deployed graph is the student
alone, unchanged. Cosine rather than MSE because the teacher's embeddings are
non-negative pooled activations whose norm varies by a factor of ~2.6 between
windows; an MSE would spend most of its effort on the norm.
Chunks with no usable teacher window, and chunks mixup mixed, carry
valid = 0 from the loader and add nothing to this loss.
Optionally a second training-only head, linear on the same embedding, predicts
where and when the recording was made (birdnet_stm32.data.aux_targets): the
location as a unit vector and the day of year as (sin, cos), each scored by a
cosine distance over the rows where it is known. It asks the embedding to carry
the species composition of a place and season. Like the projector, it never
reaches the checkpoint.
EmbeddingDistilledModel
¶
Bases: Model
Train a student on its labels plus a cosine loss to the teacher's embedding.
Called, it is the student: same inputs, same outputs, so validation and
checkpoint selection see exactly the deployed graph. Only train_step
differs; it expects labels (y, teacher_embedding, valid).
Checkpoint the student (train_model(checkpoint_model=...)), not this
wrapper: the projector exists for training only.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
student
|
Model
|
The model being trained. Its variables are shared. |
required |
teacher_dim
|
int
|
Width of the teacher embedding. |
required |
weight
|
float
|
Weight of the cosine distance against the supervised loss. |
required |
output_teacher
|
Model | None
|
Optional frozen larger model to distil from online, on the
same batch ( |
None
|
output_weight
|
float
|
Weight of that KL term. |
0.0
|
aux_weight
|
float
|
Weight of the location/date head's loss (the sum of its two
cosine distances); 0 for none. With it, |
0.0
|
Source code in birdnet_stm32/training/embedding_distillation.py
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | |
train_step(data)
¶
The default Keras step, plus the embedding term (and the location/date term).
Source code in birdnet_stm32/training/embedding_distillation.py
test_step(data)
¶
The default step; validation carries no teacher embeddings, so no cosine (and no distillation).
Source code in birdnet_stm32/training/embedding_distillation.py
masked_cosine_distance(projected, teacher, valid)
¶
Mean 1 - cos over the valid rows, and the per-row cosine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
projected
|
Tensor
|
|
required |
teacher
|
Tensor
|
|
required |
valid
|
Tensor
|
|
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
|
Tensor
|
none are valid) and the |
Source code in birdnet_stm32/training/embedding_distillation.py
fit_projector(model, batches, holdout=0.2, ridges=(0.001, 0.01, 0.1, 1.0))
¶
Initialize the projector by ridge regression, for a student that is already trained.
A random projector on a trained student starts the cosine loss near its maximum, and its first gradients reach the backbone before the projector has learned anything. Solved in closed form on the student's current embeddings, the projector starts where the two spaces already agree.
A few thousand rows against a 1024 x 1280 map overfit easily, so the ridge strength is chosen on held-out rows, and the projector is then refitted on all of them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
EmbeddingDistilledModel
|
The wrapper whose projector is set. |
required |
batches
|
Iterable of training batches |
required | |
holdout
|
float
|
Fraction of the valid rows held out to choose the ridge strength. |
0.2
|
ridges
|
tuple[float, ...]
|
Candidate ridge strengths, relative to the mean feature variance. |
(0.001, 0.01, 0.1, 1.0)
|
Returns:
| Type | Description |
|---|---|
dict[str, float]
|
Row counts, the chosen ridge strength and its held-out mean cosine. |
Source code in birdnet_stm32/training/embedding_distillation.py
fit_aux_head(model, batches, ridges=(0.001, 0.01, 0.1, 1.0))
¶
Initialize the location/date head by ridge regression on the student's current embeddings.
As with the projector: from a trained student, a random head would open at the largest loss and push its gradients into a converged backbone. The location columns are fitted on rows with a known location, the date columns on rows with a known date, each with the ridge strength that gives the best held-out mean cosine (20% of rows held out, then refitted on all).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
EmbeddingDistilledModel
|
The wrapper whose |
required |
batches
|
Iterable of |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Row counts, chosen ridge strengths and held-out mean cosines. |