nexuml.data.export.tensor_shards¶
Non-prebatched tensor-shard export backend.
Bases: ExportBackend
Write arbitrary fixed-shape tensor keys into sample shards.
A shard is a storage unit with N samples along dimension zero. It is
NOT a training batch; the runtime loader may create arbitrary batch sizes
from the shard contents.
staticmethod
¶
load_shard(
export_dir: Path, shard: int | str | dict[str, Any]
) -> dict[str, Any]
Load a complete shard for the future windowed loader.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Shard payload with metadata and CPU feature tensors. |
Raises:
| Type | Description |
|---|---|
IndexError
|
If an integer shard index is out of range. |
ValueError
|
If the shard content or metadata is invalid. |
staticmethod
¶
load_sample(
export_dir: Path, index: int
) -> dict[str, torch.Tensor]
Map-style compatibility path; the window loader should use load_shard.
Returns:
| Type | Description |
|---|---|
dict[str, Tensor]
|
dict[str, torch.Tensor]: Feature tensors for the requested sample index. |
Raises:
| Type | Description |
|---|---|
IndexError
|
If the sample index is outside available shard bounds. |