`nexuml.data.loaders.dali_numpy_external`¶

nexuml.data.loaders.dali_numpy_external ¶

DALI external source pipeline for NumPy datasets.

ExternalInputIterator ¶

Iterator that loads NumPy files on demand for DALI external_source.

PyTorchIterator ¶

Bases: DALIGenericIterator

DALIGenericIterator with multi-label support for external source pipelines.

external_source_pipeline ¶

external_source_pipeline(
    external_source: Iterable, device: str = "cpu"
) -> Any

DALI pipeline using external source for flexible data loading.

Returns:

Type	Description
`Any`	Tuple of data tensor and label tensor from the external source.

DaliNumpyExternalPipeline ¶

DaliNumpyExternalPipeline(
    files: list[str],
    labels: list[list[int]] | TensorDict,
    batch_size: int,
    num_threads: int = -1,
    prefetch_factor: int = 2,
    shuffle: bool = False,
    local_rank: int = 0,
    global_rank: int = 0,
    world_size: int = 1,
    **kwargs,
) -> PyTorchIterator

Build and return a DALI external source pipeline as a PyTorchIterator.