`nexuml.data.export.webdataset`¶

nexuml.data.export.webdataset ¶

WebDataset tar-shard export backend.

WebDatasetBackend ¶

Bases: ExportBackend

Write lossless NumPy components into indexed WebDataset tar shards.

remote_uri `property` ¶

remote_uri: str | None

S3 dataset root for remote exports.

initialize ¶

initialize(
    export_dir: Path,
    num_samples: int,
    feature_shapes: dict[str, tuple[int, ...]],
    dtype: dtype[Any] | str | None = None,
) -> None

Prepare local shard staging.

Parameters:

Name	Type	Description	Default
`export_dir`	`Path`	Local export/staging directory.	required
`num_samples`	`int`	Declared number of exported samples.	required
`feature_shapes`	`dict[str, tuple[int, ...]]`	Tensor shapes supplied by the export runner.	required
`dtype`	`dtype[Any] \| str \| None`	Optional floating-point storage dtype.	`None`

start_split ¶

start_split(split: str) -> None

Start one physically isolated dataset split.

end_split ¶

end_split(split: str) -> None

Finish one dataset split and close its current shard.

Raises:

Type	Description
`RuntimeError`	If the requested split is not the active split.

save_sample ¶

save_sample(
    index: int, features: dict[str, Tensor]
) -> None

Append one tensor sample to the active tar shard.

Raises:

Type	Description
`ValueError`	If a component key contains a path separator.

finalize ¶

finalize() -> dict[str, Any]

Close the final shard and return metadata for config.yaml.

Returns:

Type	Description
`dict[str, Any]`	Backend-specific export metadata.

Raises:

Type	Description
`RuntimeError`	If the backend was never initialized.

publish_export_metadata ¶

publish_export_metadata(
    config_path: Path, metadata_path: Path
) -> None

Upload the final lightweight export metadata for an S3 export.

load_sample `staticmethod` ¶

load_sample(
    export_dir: Path, index: int
) -> dict[str, torch.Tensor]

Load one sample from a local WebDataset export.

Returns:

Type	Description
`dict[str, Tensor]`	Mapping of stored keys to tensors.

Raises:

Type	Description
`IndexError`	If the sample is absent from the export index.
`FileNotFoundError`	If a referenced tar member cannot be read.

nexuml.data.export.webdataset¶