Skip to content

User guide

How to run the pipeline and look after a run.

  • Setup: what the machine needs, Hugging Face credentials, and disk space.
  • Build a release: from a dump's date to published datasets, with the commands for each step.
  • Monitor and resume: reading progress from the files on disk, estimating the time left, and what happens on a failure.
  • Tuning: how many chunks to process at once, group sizes, and disk limits.
  • Performance: measured timings of the release 20260928.
  • The philippesaade build: the earlier build from the philippesaade/wikidata Parquet copy, still supported.