Skip to content

wend.fs

Filesystems for discovery: the local disk, fsspec, and object stores.

glob and infer read a tree through FileSystem, a small protocol: - list a directory; - say what a path is; - identify a directory, to detect link cycles; - walk a tree; - give a view of one subtree for the duration of one walk.

LocalFS, the default, implements it with os. FsspecFS adapts an fsspec filesystem, so the same expressions discover keys in memory or in S3, GCS or Azure storage. render, match and Lens never touch a filesystem, so they need nothing from here.

Paths stay in the filesystem's own namespace. fsspec strips a URL's protocol: memory://data/x is /data/x and s3://bucket/x is bucket/x. url_to_fs splits a URL into a filesystem and such a path, which is then bound like any root::

fs, root = url_to_fs("s3://bucket/data")
expr.glob(root=root, fs=fs)

A URL is not an expression's head. pathlib reads s3://bucket as the path s3:/bucket, so glob refuses one with a pointer to url_to_fs.

DirEntry

Bases: Protocol

One entry of a directory listing, as os.DirEntry provides it.

name property 

name: str

is_dir 

is_dir(*, follow_symlinks: bool = True) -> bool

is_file 

is_file(*, follow_symlinks: bool = True) -> bool

is_symlink() -> bool

EntryKind 

EntryKind = Literal['file', 'dir', 'link', 'other']

FileSystem

Bases: Protocol

What glob and infer need from a filesystem.

Paths are strings in the filesystem's namespace, joined with /; . is the directory a relative path starts from.

scandir 

scandir(path: str) -> Sequence[DirEntry]

The entries of directory path sorted by name.

Empty if path is not a directory or cannot be listed.

kind 

kind(path: str, *, follow_symlinks: bool = True) -> EntryKind | None

What path is, or None if it does not exist.

With follow_symlinks=False a symlink is "link"; otherwise it is what it points to, and None if that does not exist. A path that cannot be examined (no permission, a symlink loop) is None, as os.path.isdir reads it; a failure that says nothing about the path, such as a dropped connection, raises.

dir_key 

dir_key(path: str) -> Hashable | None

An identity for directory path, equal for the same directory reached by different paths.

None on a filesystem without links, where no walk can cycle. Raises OSError if the directory cannot be identified.

walk 

walk(top: str) -> Iterator[tuple[str, list[str], list[str]]]

(dirpath, dirnames, filenames) for top and every directory below it, as os.walk(top) gives them.

The walk is top-down, does not descend symlinks, and visits only the dirnames left after the caller prunes the list in place.

subtree 

subtree(path: str) -> FileSystem

A filesystem to read path and everything below it during one walk.

LocalFS returns itself. FsspecFS returns a view that answers from what it has listed, and lists the whole subtree at once if listing="flat".

FsspecFS 

FsspecFS(fs: Any, *, listing: Literal['flat', 'tree'] = 'flat')

An fsspec filesystem: object storage (S3, GCS, Azure), memory, HTTP...

listing decides how glob and infer read a tree:

  • "flat" (default): one find for everything below the expression's literal prefix, which glob then walks in memory. On an object store that is a single prefix listing (one request per page of keys) however deep the layout, where a directory-by-directory walk makes one request per directory.
  • "tree": one ls for each directory the expression reaches, as on the local disk. Suited to filesystems with real directories (SFTP, HDFS), where find itself lists every directory, and to prefixes holding many keys that the expression never reaches.

Results are the same either way. Within one walk, a path whose parent has been listed is classified without another request.

Stores have no symlinks: no entry is a link, so glob's symlink options change nothing, and dir_key is None. A directory is a prefix of some key or an explicit marker (a key ending in /, or makedirs on the memory filesystem), so an empty directory exists only as a marker. A key that is also a prefix of other keys is a directory. Keys with an empty, . or .. segment cannot be rendered by any expression and are never listed.

fsspec's own local filesystem is refused: use LocalFS, which keeps glob's symlink policy and cycle detection.

fs instance-attribute 

fs = fs

listing instance-attribute 

listing = listing

scandir 

scandir(path: str) -> list[_Entry]

kind 

kind(path: str, *, follow_symlinks: bool = True) -> EntryKind | None

dir_key 

dir_key(path: str) -> None

walk 

walk(top: str) -> Iterator[tuple[str, list[str], list[str]]]

subtree 

subtree(path: str) -> _Subtree

LocalFS

The local disk through os, the default for glob and infer.

scandir 

scandir(path: str) -> list[DirEntry[str]]

kind 

kind(path: str, *, follow_symlinks: bool = True) -> EntryKind | None

dir_key 

dir_key(path: str) -> tuple[int, int]

walk 

walk(top: str) -> Iterator[tuple[str, list[str], list[str]]]

subtree 

subtree(path: str) -> LocalFS

url_to_fs 

url_to_fs(url: str, **storage_options: Any) -> tuple[FileSystem, str]

Split a URL into a filesystem and a path in that filesystem's namespace.

url_to_fs("s3://bucket/data") is (FsspecFS(S3FileSystem), "bucket/data"); url_to_fs("memory://data") is (FsspecFS(...), "/data"); a local path or file:// URL gives LocalFS. Bind the path as an expression's root and pass the filesystem to glob or infer. storage_options go to fsspec (credentials, anon=True, and so on). Needs fsspec (pip install "wend[fsspec]").