`nexuml.data.loaders.dali_numpy`¶

nexuml.data.loaders.dali_numpy ¶

DALI NumPy pipeline for file-backed numpy datasets.

PyTorchIterator ¶

Bases: DALIGenericIterator

DALIGenericIterator with filename-based label lookup for NumPy datasets.

numpy_data_pipeline ¶

numpy_data_pipeline(
    files: list[str],
    filename_len: int,
    target_sr: int = 16000,
    target_length: int = 10,
    shuffle: bool = False,
    shard_id: int = 0,
    num_shards: int = 1,
    device: str = "cpu",
    direct_store: bool = False,
    mono: bool = True,
    rnd_crop_size: float | None = None,
    start_sec: float | None = None,
) -> Any

Load pre-encoded NumPy arrays with DALI native reader.

Returns:

Type	Description
`Any`	Tuple of audio tensor and integer file-index label.

DaliNumpyPipeline ¶

DaliNumpyPipeline(
    files: list[str],
    labels: list[list[int]],
    batch_size: int,
    target_sr: int = 16000,
    target_length: int = 10,
    num_threads: int = -1,
    prefetch_factor: int = 2,
    shuffle: bool = False,
    local_rank: int = 0,
    global_rank: int = 0,
    world_size: int = 1,
    mono: bool = True,
    random_crop_size: float | None = None,
    start_sec: float | None = None,
    direct_store: bool = False,
    **kwargs,
) -> PyTorchIterator

Build and return a DALI numpy pipeline as a PyTorchIterator.