`nexuml.data.export.backend`¶

nexuml.data.export.backend ¶

Export backend abstraction for SuperDataset data serialization.

ExportConfig `dataclass` ¶

Schema for config.yaml written alongside exported data.

ExportBackend ¶

Bases: ABC

Abstract base class for dataset export backends.

Each backend handles serialization of feature tensors to a specific storage format. The metadata (labels, splits, source info) is always stored as parquet alongside the backend-specific data directory.

init ¶

__init__(**kwargs: Any) -> None

Backends accept format-specific keyword arguments.

initialize `abstractmethod` ¶

initialize(
    export_dir: Path,
    num_samples: int,
    feature_shapes: dict[str, tuple[int, ...]],
    dtype: Any | None = None,
) -> None

Prepare storage (create files, allocate mmap, etc.).

save_sample `abstractmethod` ¶

save_sample(
    index: int, features: dict[str, Tensor]
) -> None

Save one sample's feature tensors.

save_batch ¶

save_batch(
    start_index: int, features: dict[str, Tensor]
) -> None

Save a batch of feature tensors.

Backends can override this for more efficient contiguous writes.

Raises:

Type	Description
`ValueError`	If batch sizes across feature tensors are inconsistent.

finalize `abstractmethod` ¶

finalize() -> dict[str, Any]

Flush/close files. Return backend-specific metadata for config.yaml.

load_sample `abstractmethod` `staticmethod` ¶

load_sample(
    export_dir: Path, index: int
) -> dict[str, torch.Tensor]

Load one sample's feature tensors from an exported directory.

register_export_backend ¶

register_export_backend(
    name: str,
) -> Callable[
    [type[_ExportBackendT]], type[_ExportBackendT]
]

Decorator to register an export backend.

Returns:

Type	Description
`Callable[[type[_ExportBackendT]], type[_ExportBackendT]]`	A decorator that registers the backend class and returns it.

get_export_backend ¶

get_export_backend(name: str) -> type[ExportBackend]

Retrieve a registered export backend by name.

Returns:

Type	Description
`type[ExportBackend]`	The backend class for the given name.

Raises:

Type	Description
`KeyError`	If no backend is registered under that name.

get_export_backend_name ¶

get_export_backend_name(
    backend_type: type[ExportBackend],
) -> str

Return the registered name for an export backend class.

Returns:

Type	Description
`str`	Registered backend name.

Raises:

Type	Description
`KeyError`	If the backend class is not registered.

list_export_backends ¶

list_export_backends() -> list[str]

Return names of all registered export backends.

nexuml.data.export.backend¶