Skip to content

worker

birdnet_stm32.data.worker

Per-file loading and preprocessing, run inside the loader's worker processes.

Kept free of TensorFlow on purpose. The worker pool uses the forkserver start method, whose server preloads only this module, so workers start from a small single-threaded process instead of a fork of the TensorFlow trainer.

arena_columns(buf, n_cells, layout)

[n_cells, *shape] arrays over buf, one per chunk column, each 64-byte aligned.

layout is a list of (shape, dtype), one per element of the tuples _process_file returns: sample, label and, with teacher embeddings, the embedding and its valid flag.

Source code in birdnet_stm32/data/worker.py 
37
38
39
40
41
42
43
44
45
46
47
def arena_columns(buf, n_cells: int, layout) -> list[np.ndarray]:
    """``[n_cells, *shape]`` arrays over ``buf``, one per chunk column, each 64-byte aligned.

    ``layout`` is a list of ``(shape, dtype)``, one per element of the tuples ``_process_file``
    returns: sample, label and, with teacher embeddings, the embedding and its valid flag.
    """
    columns, offset = [], 0
    for shape, dtype in layout:
        columns.append(np.ndarray((n_cells, *shape), dtype=dtype, buffer=buf, offset=offset))
        offset += -(-columns[-1].nbytes // 64) * 64
    return columns

arena_bytes(n_cells, layout)

Size of the buffer arena_columns lays out.

Source code in birdnet_stm32/data/worker.py 
50
51
52
53
54
55
def arena_bytes(n_cells: int, layout) -> int:
    """Size of the buffer ``arena_columns`` lays out."""
    return sum(
        -(-(n_cells * int(np.prod(shape, dtype=np.int64)) * np.dtype(dtype).itemsize) // 64) * 64
        for shape, dtype in layout
    )

set_arena(columns)

Point _process_into at columns (the in-process loader's own arena).

Source code in birdnet_stm32/data/worker.py 
58
59
60
61
def set_arena(columns: list[np.ndarray]) -> None:
    """Point ``_process_into`` at ``columns`` (the in-process loader's own arena)."""
    global _arena  # noqa: PLW0603
    _arena = columns