Export a dataset¶

nexuml export-dataset persists a dataset view for reuse, analysis, shared storage, or a different loading strategy.

Use the generated CLI reference for every flag. The common workflows are below.

Export raw batches¶

nexuml export-dataset my-scenario \
  -o exported-data \
  --backend numpy \
  --split train \
  --split val

Choose feature/label keys with repeated --x-key / --y-key options when you do not want the complete batch view.

Export an intermediate pipeline view¶

nexuml export-dataset my-scenario \
  -o prepared-data \
  --backend tensor_shards \
  --preprocess \
  --preprocess-until-key embedding

With preprocessing enabled, NexuML compiles the pipeline and uses forward_until semantics until the requested x keys exist, then writes that transformed view.

The same concept can be declared in DataSpec.preprocessing so materialization becomes part of the scenario's data workflow.

Export backends¶

The built-in registry currently includes:

Backend	Intended use
`numpy`	simple per-sample NumPy files
`numpy_mmap`	contiguous memory-mapped NumPy storage
`torch`	Torch tensor payloads
`tensordict_memmap`	TensorDict-native memory-mapped storage
`webdataset`	tar shards, including S3 export support
`tensor_shards`	fixed-shape sample shards for the windowed tensor-shard loader

Inspect the runtime instead of relying on a copied list:

nexuml backend list data-export

Export metadata¶

Every export writes a config.yaml describing the format/writer, sample count, modality, logical x/y keys, label prefix, feature shapes, key-level storage metadata, source datasets, and backend-specific extra information. Dataset metadata is stored alongside it (normally Parquet, with CSV fallback where applicable).

Treat that schema as an export contract generated by NexuML rather than hand-editing it.

Reuse an export¶

The base library exposes a typed ExportedDataset definition:

from nexuml.core.types import DataSpec, LoaderSpec
from nexuml.data.loaders.definitions import TorchLoader
from nexuml_library.data.exported import ExportedDataset

DataSpec(
    source=ExportedDataset(
        root="./exported-data",
        feature_keys=["features"],
        label_keys=["target"],
    ),
    loader=LoaderSpec(backend=TorchLoader()),
)

For a view that already ran early model/preprocessing stages, use data.skip_pipeline_stages to avoid applying those stages again.

S3 WebDataset¶

The webdataset exporter can write to an s3:// destination and accepts S3 endpoint/region/profile options. Remote ExportedDataset currently supports WebDataset exports. With the DALI loader, remote shard/index files are staged to worker-local temporary storage before DALI reads them.

See Ray execution for the distributed path.

Tensor shards¶

tensor_shards are storage shards, not fixed training batches. TensorShardsLoader can window/prefetch those shards and still form the configured runtime batch size independently.

See Data loading.