Datasets and shards¶

A dataset is stored as numbered shards, in one directory per split. This guide declares that layout once, renders the paths a writer produces, lists the shards that exist and finds the ones that are missing. It needs Python 3.14 and wend, with no extras. The examples build on each other: run them in order, in an empty directory.

Declare the layout¶

from wend import Param, Segment, counter

root = Param("root")
dataset = Segment("dataset")
split = Segment("split", choices={"train", "test"})
name = counter("shard", width=5, of_total=True, sep="-", suffix=".parquet")

shard = root / dataset / split / name

>>> name
PathExpr('shard-{idx:05d}-of-{total:05d}.parquet')
>>> shard
PathExpr('{root}/{dataset}/{split}/shard-{idx:05d}-of-{total:05d}.parquet')
>>> shard.render(root="data", dataset="tiny", split="train", idx=0, total=4)
'data/tiny/train/shard-00000-of-00004.parquet'

counter writes a counter filename. With of_total=True the name also holds the number of shards. Its parameters, idx and total, are integers formatted with 05d, so matching reads them back as int.
dataset and split are segments: each value is one directory name. split accepts only train and test.

counter returns an ordinary expression, equal to the same name written as a t-string:

>>> from wend import T
>>> idx, total = Param("idx", type=int), Param("total", type=int)
>>> name == T(t"shard-{idx:05d}-of-{total:05d}.parquet")
True

Render every expected path¶

A writer knows the root, the dataset and the number of shards before it starts. bind fixes them once:

>>> layout = shard.bind(root="data", dataset="tiny", total=4)
>>> layout
PathExpr('data/tiny/{split}/shard-{idx:05d}-of-00004.parquet')

compile generates a Python function specialised to the expression, and caches it on the expression. Its many method renders one path for each mapping of bindings:

>>> compiled = layout.compile()
>>> rows = [{"split": s, "idx": i} for s in ("train", "test") for i in range(4)]
>>> paths = compiled.many(rows)
>>> len(paths)
8
>>> for path in paths[:3]:
...     print(path)
data/tiny/train/shard-00000-of-00004.parquet
data/tiny/train/shard-00001-of-00004.parquet
data/tiny/train/shard-00002-of-00004.parquet

columns takes the same bindings column by column, and calling the compiled renderer returns one pathlib.Path, as resolve does:

>>> for path in compiled.columns(split=["test", "test"], idx=range(2)):
...     print(path)
data/tiny/test/shard-00000-of-00004.parquet
data/tiny/test/shard-00001-of-00004.parquet
>>> compiled(split="test", idx=3)
PosixPath('data/tiny/test/shard-00003-of-00004.parquet')

A compiled renderer returns what render returns for the same bindings, and raises the same error for a value that render rejects:

>>> compiled.many([{"split": "val", "idx": 0}])
Traceback (most recent call last):
  ...
wend.errors.InvalidBindingError: split='val': not one of {'test', 'train'}

Render several paths per shard¶

Each shard has a JSON file of metadata next to it, with the same name and the suffix .json. compile_many compiles several expressions into one function. In each call, every parameter is looked up, checked and formatted once, however many of the expressions use it.

>>> from wend import compile_many
>>> outputs = compile_many({"data": layout, "meta": layout.with_suffix(".json")})
>>> shard_0 = outputs.render(split="train", idx=0)
>>> shard_0["data"]
'data/tiny/train/shard-00000-of-00004.parquet'
>>> shard_0["meta"]
'data/tiny/train/shard-00000-of-00004.json'
>>> for data, meta in outputs.many(rows[:2]):
...     print(meta)
data/tiny/train/shard-00000-of-00004.json
data/tiny/train/shard-00001-of-00004.json

Given a dict, render returns a dict with the same keys. many returns one tuple per row, with the paths in the order the expressions were given.

Write some shards¶

The next examples need shards on disk. This writes five of the eight shards, then two files that do not belong to layout: one named without the counter, and one left by an earlier write of three shards.

from pathlib import Path

written = [("train", 0), ("train", 1), ("train", 3), ("test", 0), ("test", 1)]
for path in compiled.many({"split": s, "idx": i} for s, i in written):
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    Path(path).touch()

Path("data/tiny/train/shard-2.parquet").touch()
Path("data/tiny/test/shard-00002-of-00003.parquet").touch()

List the shards that exist¶

glob walks the directories along the expression's structure and yields a Match for each existing path that the expression describes, in path order:

>>> for m in layout.glob():
...     print(m["split"], m["idx"], m.path)
test 0 data/tiny/test/shard-00000-of-00004.parquet
test 1 data/tiny/test/shard-00001-of-00004.parquet
train 0 data/tiny/train/shard-00000-of-00004.parquet
train 1 data/tiny/train/shard-00001-of-00004.parquet
train 3 data/tiny/train/shard-00003-of-00004.parquet

shard-2.parquet does not have the counter's form, and shard-00002-of-00003.parquet has a total of 3 where layout requires 4, so neither is yielded. Each idx is an int, read from the filename and checked by rendering it again.

Find the gaps¶

The set of shards that should exist, minus the set that glob found, is the set of missing shards. The same layout renders their paths:

>>> expected = {(r["split"], r["idx"]) for r in rows}
>>> found = {(m["split"], m["idx"]) for m in layout.glob()}
>>> missing = sorted(expected - found)
>>> missing
[('test', 2), ('test', 3), ('train', 2)]
>>> for path in compiled.many({"split": s, "idx": i} for s, i in missing):
...     print(path)
data/tiny/test/shard-00002-of-00004.parquet
data/tiny/test/shard-00003-of-00004.parquet
data/tiny/train/shard-00002-of-00004.parquet

Find shards from another write¶

layout fixes total. shard leaves it free, so glob reads the total from each filename, and shards written with a different total can be found:

>>> for m in shard.glob(root="data", dataset="tiny"):
...     if m["total"] != 4:
...         print(m.path, m["total"])
data/tiny/test/shard-00002-of-00003.parquet 3

Bindings given to glob fix those parameters before the walk, so this call lists only data/tiny.

Numbered files without a total¶

A log directory grows one file at a time, so its names have no total. Without of_total, counter writes the index alone:

log = root / "logs" / counter("log", width=3, suffix=".jsonl")

Path("data/logs").mkdir()
for i in range(3):
    Path(log.render(root="data", idx=i)).touch()

>>> log
PathExpr('{root}/logs/log_{idx:03d}.jsonl')
>>> last = max(m["idx"] for m in log.glob(root="data"))
>>> log.render(root="data", idx=last + 1)
'data/logs/log_003.jsonl'

Without width, the index is not padded. A match must render back to the same path, so a padded name does not match an unpadded counter:

>>> part = counter("part", suffix=".csv")
>>> part
PathExpr('part_{idx:d}.csv')
>>> part.match("part_12.csv")
Match('part_12.csv', {'idx': 12})
>>> print(part.match("part_012.csv"))
None