Filesystem artifact storage

By default DearDiary stores every Resource artifact inline in the SQLite database. That works for kilobyte-sized configs and tiny pickled models, but a 500 MB serialised checkpoint will balloon the database file and slow every metadata query. The DearDiary.FilesystemStore backend writes artifact bytes to a directory on local disk instead, so the database stays lean and the bytes can be backed up with the rest of your storage volume.

Configuration

Set two environment variables in your .env:

DEARDIARY_ARTIFACT_BACKEND=filesystem
DEARDIARY_ARTIFACT_FS_ROOT=/var/lib/deardiary/artifacts

The root directory is created on the first write — no separate provisioning step is needed.

Layout on disk

Each artifact is written to <root>/<aa>/<uuid>, where <aa> is a two-character shard of the UUID so a single directory never grows unbounded. Two uploads of identical bytes still produce distinct files: there is no content-addressed deduplication, so deleting one Resource can never break a sibling that happened to upload the same payload.

End-to-end example

Create a project, experiment, iteration, and upload an artifact through the configured store:

julia> user = DearDiary.get_user("default");
julia> project_id, _ = create_project(user.id, "Filesystem tutorial");
julia> experiment_id, _ = create_experiment(project_id, DearDiary.IN_PROGRESS, "FS experiment");
julia> iteration_id, _ = create_iteration(experiment_id);
julia> payload = rand(UInt8, 4096);
julia> resource_id, _ = create_resource(experiment_id, "checkpoint.bin", payload);

The resource row records the new backend and the URI that points at the bytes on disk:

julia> resource = get_resource(resource_id)DearDiary.Resource
 ├ id = 1
 ├ experiment_id = 1
 ├ name = "checkpoint.bin"
 ├ description = ""
 ├ data = UInt8[]
 ├ created_date = 2026-06-06T18:35:15.557
 ├ updated_date = nothing
 ├ backend = "filesystem"
 ├ uri = "file:///tmp/jl_DuqySj/75/75ec270b-b003-4f3c-af3e-50e45fbeb610"
 ├ size_bytes = 4096
 └ content_hash = "9f1269890dc626f494f14c730fae47884e2fde6d2b969a79eb08de9e65965756"

julia> resource.backend"filesystem"

julia> resource.uri |> startswith("file://")true

The on-disk path is reachable directly when you want to inspect or stream the bytes from another process — DearDiary itself reaches them through read_resource_data:

julia> read_resource_data(resource_id) == payloadtrue

Migrating from the SQLite backend

If a project was started on the SQLite backend and you later switch DEARDIARY_ARTIFACT_BACKEND to filesystem, run migrate_artifacts! once to move the legacy inline bytes to disk:

using DearDiary
DearDiary.run(; env_file=".env")
DearDiary.migrate_artifacts!()

The call is idempotent and restartable: rows that have already been moved have a different backend value and are skipped on subsequent passes. If the disk fills up mid-migration, the failing row is left untouched and the next invocation picks up from there.