Data Library — Design

Status: First pass · Branch: feature/user-datalib-mounts

Goal

Let a run read large or reused datasets in place, read-only, instead of uploading or copying them into every run. A data library is a set of directories in the store, either a user's own or a global one shared with every user:

users/<userid>/datalib/<dirname>/...          # a user's: any tree, one dataset or many
users/<userid>/datalib/<dirname>/README.md    # its description (optional)
datalib/<dirname>/...                         # global: same pattern, at the store root
datalib/<dirname>/README.md

A global and a user directory may share a name; they are selected and tracked separately (global_datalib_dirs and datalib_dirs in run metadata).

The top-level directories are what a run selects, any number of them; what is inside each is up to the user. A run may use data-library directories alongside uploaded or preloaded datasets, or instead of them.

Both libraries are populated out of band: the app only lists and selects them, and provides no way to write to them (see the operator guide). Every user can see and select every global directory.

One path, everywhere

A data-library directory appears at the path its key implies under the mount root, in every environment that sees it:

/mnt/data/users/<userid>/datalib/<dirname>
/mnt/data/datalib/<dirname>                   # global

This is the same rule the job containers already follow for a run's own directory (JOB_MOUNT_ROOT, see Swappable Persistence Backends): the store's layout is decided once and each environment mirrors it, rather than each inventing its own mapping.

Environment	How the directory gets there
Docker job container (`JOB_BACKEND=docker`)	A read-only bind of `$STORAGE_HOST_DIR/users/<uid>/datalib/<dir>` (or `$STORAGE_HOST_DIR/datalib/<dir>`), beside the run's own read-write bind.
Cloud Run job container (`JOB_BACKEND=gcp`)	Already there: the job definition mounts the whole bucket at `/mnt/data`.
Host process (`JOB_BACKEND=process`)	Already there: the job reads the local store in place, so the path is `$STORAGE_DIR/users/<uid>/datalib/<dir>` (the store's directory stands in for `/mnt/data`). Not read-only, like the rest of the store for this backend.
Modal sandbox (`CODE_EXECUTION_BACKEND=modal`)	A read-only `CloudShare` of the directory's prefix in the bucket (`users/<uid>/datalib/<dir>/` or `datalib/<dir>/`) at the same path.
`process` / `local` code execution	Runs in the job container, so reads the container's path directly.

On Cloud Run the mount is the bucket's job-level volume, which is neither read-only nor scoped to the user. That is an existing property of the Cloud Run job definition, not something this feature introduces; code running in a Modal sandbox sees only its read-only shares.

The Modal sandbox mounts a run's uploads by the same rule (#117), so every path a run sees, in every environment, is the store key under /mnt/data.

How a run describes the library to the agents

A dataset in metadata.json is a filename plus a free-text description, and it reaches the agents only through the prompts: the first experiment of every run is "load the dataset(s) and summarize them", with the descriptions included, and every later hypothesis builds on it.

A data-library directory can hold any number of datasets in any layout, so it is not treated as one more dataset entry (the first experiment would try to load a directory as a table). Instead the job builds a manifest of each directory at startup:

the directory's README.md (truncated), which plays the role of an upload's description;
file and subdirectory counts, total size, and counts by file extension;
a bounded view of the file tree (a few levels, a few entries per directory).

The walk is bounded by entries visited, not just by what is shown, since the directory may be very large and is often a network mount; a partial count says so. It is breadth-first, so when the budget runs out it is the deepest levels that go unlisted, never a top-level entry. The rendered tree is capped separately: it shows as many whole levels as fit in a fixed number of lines, a directory whose contents are not shown is marked ..., and a listing cut short ends with how many entries it leaves out.

The manifest is used in two places:

The first experiment gains a step to survey the directories — use the README and layout to identify the datasets, then load the relevant ones, reading selectively if they are large — and its objective includes the full manifest. For a run with no uploaded datasets this is the only loading step, and the empty dataset-metadata section is omitted.
Agent system prompts (generator, programmer, reviser) get one short paragraph naming the directories' paths, saying they are provided data and read-only, and advising selective reads. Without a data library the prompts are unchanged.

The job writes the manifests to output/datalib_manifest.json. The directories are read in place and can change between runs, so this records what a given run was shown.

Interfaces

Layer	Change
Keys	`keys.datalib_prefix`, `keys.datalib_dir` (both take `userid=None` for the global library), and `keys.validate_datalib_dirname`, which rejects anything but a single, non-dot path segment.
Persistence / `JobManager`	`list_datalib_dirs(userid)` → `DatalibDir(name, description, scope)`, global directories first; `missing_datalib_dirs(userid_or_None, names)`.
API	`GET /api/user/me/datalib` → `{"dirs": [{"name", "description", "scope": "global" \\| "user", "file_types": [{"ext", "count"}], "file_count", "file_count_capped"}]}`. File types are tallied over at most the first 2,000 files of each directory, most common first, and sent as a list because `jsonify` sorts object keys. Run metadata gains `datalib_dirs` and `global_datalib_dirs`, each `[name, ...]`.
Submit	Checks every selected user directory exists in the submitting user's library and every global one in the global library (400 otherwise), and that the run has at least one dataset or directory.
Job args	`--datalib_dir=/mnt/data/users/<uid>/datalib/<dir>` or `--datalib_dir=/mnt/data/datalib/<dir>`, repeatable. The job does not distinguish the two. With `modal`, the job mounts each from the `--bucket_path` bucket under the key its path implies, as it does the run's own data.
Fork	Keeps `datalib_dirs` when the owner forks their own run; drops them when another user forks a shared run, since the names refer to the owner's library. Keeps `global_datalib_dirs` for anyone. A run with only data-library directories has no uploads to copy, so it can be forked without them.
Dataset expiry	Expiry deletes a run's own uploads only, so a run with only data-library directories reports no expiry.
UI	A "Data library" multi-select in run setup, shown when there are any global or user directories, with global ("Shared") and user ("Yours") directories in separate groups. Each row shows the README as secondary text and a terse file-type summary such as `12 csv, 5 tsv, 2 pdf, 14 others`. The run's parameters view lists the selection the same way.

The standalone CLI takes --datalib_dir too, with any local directory.

Possible future extensions

Permissions on global directories. Every global directory is visible to every user. Self-hosted deployments may want to grant particular global directories to particular users or groups, e.g. through the auth provider's permissions, as preloaded datasets are gated today. The listing endpoint and the submit check are the two places that would enforce it.

Open questions

Whether to snapshot, rather than just record, what a run read (e.g. object generations on GCS), for stronger reproducibility.
A UI for adding to the library.
Whether preloaded datasets, which are copied into every run today, should become a shared read-only library.