Effects¶

from wend import Absent, Creatable, Exists, File, P, Param, Readable, T

root, day = Param("root"), Param("day")

src = (P(root) / "raw" / T(t"{day}.csv")).requires(File, Exists, Readable)
dst = (P(root) / "clean" / T(t"{day}.csv")).requires(Absent, Creatable)
dst = dst.ensures(File, Exists)

The directory data/raw/ holds one file, mon.csv:

>>> src.render(root="data", day="tue")
'data/raw/tue.csv'
>>> src.check(root="data", day="tue")
Traceback (most recent call last):
  ...
wend.errors.MissingPathError: 'data/raw/tue.csv' does not exist (required: Exists)
>>> src.check(root="data", day="mon")
PosixPath('data/raw/mon.csv')

src declares that its path must be an existing, readable file when it is used. render returns the path for tue although no such file exists, because it reads nothing from the filesystem. check renders the path, tests each condition against the filesystem, and raises for the first one that fails.

Pure operations and effects¶

Building an expression, every path operation, bind, render, resolve, match, match_all, the relations and compile are pure. Their results depend only on the expression and the bindings.

These operations touch the filesystem:

• glob lists directories to find the paths an expression describes.
• check and confirm test an expression's conditions, and a Step tests those of several paths.
• PathUnion.first returns the first alternative of a union whose path exists and meets its own conditions.
• Tree.validate reads a directory tree and reports where it differs from a schema. Tree.scaffold creates the schema's directories, and empty files with files=True. It never overwrites a path.
• Ledger records which files were made from which, with digests of their contents, and reads the files again to say what is stale.

from wend import Dir
from wend.schema import Tree


class Project(Tree):
    data = Dir("data")
    config = File("config.yaml")

>>> project = Project("project")
>>> project.scaffold()
[PosixPath('project'), PosixPath('project/data')]
>>> project.validate()
Report('project': 1 missing)
  <missing 'project/config.yaml' [config]>

Conditions¶

requires declares preconditions, which must hold before a path is used. ensures declares postconditions, which must hold after it has been produced. check tests the first and confirm the second.

Conditions are not part of the path. An expression with conditions equals the same expression without them:

>>> src == P(root) / "raw" / T(t"{day}.csv")
True
>>> dst
PathExpr('{root}/clean/{day}.csv', requires=(Absent, Creatable), ensures=(File, Exists))

Conditions that cannot hold together in one phase, such as Exists and Absent, raise PathOperationError when they are declared.

Conditions that need no filesystem¶

Two conditions are enforced from the expression and its bindings alone.

render checks Under(base) lexically: a binding cannot lead outside the base through .. or an absolute value. check also resolves symlinks.

>>> from wend import AnyPath, Under
>>> upload = (P(root) / AnyPath("rel")).requires(Under(P(root)))
>>> upload.render(root="/srv", rel="a/b.txt")
'/srv/a/b.txt'
>>> upload.render(root="/srv", rel="../etc/passwd")
Traceback (most recent call last):
  ...
wend.errors.ContainmentError: '/srv/../etc/passwd' is not under '/srv' (required: Under('{root}'))

File is a fact about the structure: nothing can be joined below a path declared to be a file, so the join raises when the expression is built.

>>> src / "x"
Traceback (most recent call last):
  ...
wend.errors.PathOperationError: '{root}/raw/{day}.csv' is declared a File: nothing can be joined below it

Steps¶

A Step groups the paths that one step of a pipeline uses. Its bindings are shared, and each path takes the parameters it uses.

>>> from wend import Step
>>> convert = Step(src=src, dst=dst)
>>> convert.effects()
[<src: read {root}/raw/{day}.csv>, <dst: create {root}/clean/{day}.csv>]
>>> with convert(root="data", day="mon") as p:
...     p["dst"].parent.mkdir()
...     _ = p["dst"].write_text(p["src"].read_text().upper())
>>> Path("data/clean/mon.csv").read_text()
'A,B\n1,2\n'

Entering the with block checks every precondition and gives the rendered paths. Leaving it without an exception checks every postcondition, here that dst now exists as a file. A second run fails before its body runs, because dst must be absent:

>>> with convert(root="data", day="mon") as p:
...     print("not reached")
Traceback (most recent call last):
  ...
wend.errors.PathExistsError: 'data/clean/mon.csv' already exists (required: Absent)

effects lists what each path declares it is used for: read, write, create, list or delete. It reports the declarations, and does not observe what the step does.

Errors¶

Each failed condition raises a ConditionError. Its path is the rendered path and its condition is the condition that failed. Each subclass also derives from the builtin exception for the same failure, so existing handlers keep working. InvalidBindingError is a ValueError.

>>> try:
...     src.check(root="data", day="tue")
... except FileNotFoundError as e:
...     print(type(e).__name__, e.path)
MissingPathError data/raw/tue.csv

Checks are point-in-time¶

A check describes the filesystem at the moment it runs. A path that exists when it is checked may be gone when it is opened, and a symlink may be replaced between the two. Checks are not atomic with any later use of the path, and Readable, Writable and the other capabilities report the permissions of the real user. Conditions state what a program expects and catch mistakes. They are not a sandbox, and neither is glob's symlink policy.

Conditions are checked when the program runs. A type checker sees a PathExpr, with no conditions in its type.

Other filesystems¶

glob and infer read a tree through a wend.fs.FileSystem: the local disk by default, or the filesystem given with fs=. With the fsspec extra, url_to_fs gives an FsspecFS for an object store and the path within it. check, confirm, Step, first, Tree.validate, Tree.scaffold and Ledger use the local filesystem.

Reference¶

• wend.conditions, Step, PathExpr.check, PathExpr.confirm, wend.errors
• SEMANTICS.md §10 (conditions), §11 (unions and first) and §12 (schemas)