An fsspec Filesystem for Epic's Lore

I have spent a while looking for a good way to version datasets made of large binary files. Git treats a non-text file as one object and struggles past a few gigabytes. Git LFS adds pointer files, but it cannot reuse unchanged parts within a file: changing one row in a Parquet file means uploading the whole file again. I want to split files according to their contents, so small edits affect only a few chunks. If the storage system identifies chunks by their contents and reuses them across files, each dataset version only needs storage for its new bytes. I spent more time than I would like to admit looking for something that did all of this.

I tried building part of it with OpenXet*, a Rust storage server compatible with the Xet protocol* that Hugging Face uses for models and datasets. It uses Gearhash to keep chunk boundaries stable across revisions, Blake3 keyed hashes organized into Merkle trees to identify content, and HMAC-protected hash queries to find chunks it already stores. It also implements the xorb and shard binary formats used to reconstruct files. Building those pieces taught me how the storage works. I still needed version control on top of it, and Python data tools would need custom client code to read the results.

Epic Games released Lore*¹ in June. It is a Rust version control system designed for large binary files, aimed at game and multimedia teams that have struggled with Git LFS. Lore stores reusable chunks with indexed lookup. It represents repository state as Merkle trees, which identify contents through hashes, over a revision history that cannot be changed. Data downloads when needed, so reading one file does not require downloading a 200GB project. That gave me the pieces I wanted for datasets: reuse of unchanged chunks, revisions applied as a whole, and a central server that keeps the data until a client requests it.

To read Lore datasets from Python, I built lore-fsspec*. It implements the fsspec* filesystem interface used by Python data tools, following the existing GitFileSystem where it made sense.

Reading Lore with existing Python tools

pandas, pyarrow, Polars, DuckDB, zarr, and dask can use fsspec to access storage such as S3, GCS, and HTTP without implementing each storage system themselves. A filesystem implements methods such as ls, info, cat_file, and open, then registers a protocol name. Tools can use that name in a URL:

import fsspec
 
fs = fsspec.filesystem("lore", path="lore://vcs.example.com:41337/my-project", ref="main")
fs.ls("Content/Maps")
with fs.open("Content/Config/Game.ini", "rt") as f:
    print(f.read())
 
# URL form: lore://<host>:<port>/<repo>[:<ref>][@<in-repo-path>]
url = "lore://vcs.example.com:41337/my-project:main@Content/Config/Game.ini"
with fsspec.open(url, "rt") as f:
    print(f.read())

For example, duckdb.sql("SELECT ... FROM read_parquet('lore://warehouse/events/*.parquet')") reads a Parquet dataset from a Lore repository at a given revision. DuckDB's existing fsspec support handles the connection once Lore is registered.

Identifying a repository

GitFileSystem opens a local clone, reads its tree at a branch or revision, and exposes paths relative to the repository. lore-fsspec keeps those relative paths, Git's parsing of the :ref@inner form, including its quirks, and read-only access by default.

The repository address needed a different approach. Git is distributed: a local clone is a full copy of the repository history, so a directory on disk is a useful address. Lore keeps the history on a central server. A local clone holds a working copy, caches, and branch pointers for accessing that server. The repository's address is its URL, lore://host:port/repo, which is also what lore clone accepts. Requiring a manually created local clone would leave callers to manage that connection themselves.

The constructor accepts the server URL or an existing local clone:

# Remote-first: address the repo by its Lore server URL.
fs = fsspec.filesystem("lore", path="lore://vcs.example.com:41337/my-project")
 
# Or attach to an existing local clone, GitFileSystem-style.
fs = fsspec.filesystem("lore", path="/path/to/clone", ref="main")

A regular expression distinguishes the forms by the port number in host:port/. A local clone path does not have a server address such as something:41337/.

Managing the local clone

The lore client still requires a local instance. Every command takes a repository_path, and there is no remote mode that works without a clone. When a caller supplies a URL, the filesystem manages that local instance for them.

On first use, it creates a bare clone: branch and tree metadata, with no file contents checked out. The clone lives under ~/.cache/lore-fsspec, or $XDG_CACHE_HOME if set. Callers can override the location with cache_dir=. The directory name, <repo>-<sha256(url)[:12]>, is stable for each URL, so later filesystem instances reuse it. Reads fetch missing content from the server and keep the fragments locally. Reading the same content again needs no further download. Writes stage the changes, commit them, and push them back.

Keeping file contents off disk saves space but complicates moves. For a tracked file that has not been downloaded, mv reads the content from the store into the destination and stages the missing source as a removal. Moving an undownloaded directory raises an error: staging its tracked children would record deletions without adding them back at the destination.

The initial clone also has no lock shared across processes. Two processes opening the same URL for the first time compete for liblore's instance lock. Single-process use is fine, but a parallel job starting without a cached clone can fail.

Letting reads run concurrently

GitFileSystem runs calls synchronously because libgit2 does. Lore's Python bindings, lore-vcs, call the Rust liblore library through a foreign function interface (FFI). Each command returns an executor whose collect_async() resolves asyncio futures directly, without using a thread pool to imitate async behavior.

fsspec's AsyncFileSystem fits this interface. I implement async methods such as _ls, _info, and _cat_file. fsspec provides the blocking versions on a separate event-loop thread, which schedules the async work. This allows several reads to make progress together:

fs = fsspec.filesystem("lore", path="/clone", ref="main", asynchronous=True)
data = await fs._cat_file("Content/Config/Game.ini")        # awaitable
many = await fs._cat(["Content/A.ini", "Content/B.ini"])    # runs concurrently

Calls such as _cat and cat_ranges start the underlying Lore commands concurrently. The two-function _lore.py module waits for an executor, raises errors for failure events, and selects the event type the call needs. Its synchronous counterpart handles cloning and default-branch lookup during construction; those calls must run outside the event-loop thread.

Disk access uses anyio.Path. A blocking read_bytes call on the event-loop thread would hold up the other reads.

Problems I found with a live server

Two undocumented behaviors took time to track down.

File paths use the process working directory

I expected file paths to resolve relative to repository_path. Instead, lore resolves them against the process's current working directory and discovers the repository from the resulting path. Setting repository_path does not change that behavior.

Calling chdir would affect the whole process, including other filesystems using fsspec's shared thread, and could corrupt concurrent reads. I convert every repository-relative path to an absolute working-copy path with os.path.join(clone_root, inner) before passing it to the native library. The _abs helper does this for every read and write, without changing the working directory.

Missing paths arrive as generic errors

repository_dump, used by ls, reports a missing path as a generic error with the undocumented code Node not found, 101 in liblore 0.8.3. But fsspec's glob, walk, and exists expect FileNotFoundError for a missing path and treat other exceptions as failures. The generic error breaks ordinary file searches.

file_info does distinguish missing paths correctly. When repository_dump fails, _ls checks with file_info. If the path is absent, its FileNotFoundError passes through. If the path exists, the original error is raised unchanged. Only failed calls pay for that extra request.

Reading and downloading files

For the checked-out revision, a file fully present on disk can be read directly from the clone. Lore reports this with local_size == size. lore-fsspec then needs no network request, content-store connection, or native-library call to retrieve the bytes. A bare clone never uses this path because it has no checked-out files.

For other revisions or content still on the server, the filesystem uses Lore's content store. It opens one store connection when first needed, protected by a lock so concurrent reads cannot open several and lose track of all but one. It then looks up the file's content address in its metadata and fetches that fragment. With the default offline=False, missing fragments download on demand and stay in the local store. Each fragment is fetched once and reused on later reads.

With offline=True, a missing fragment raises FileNotFoundError with the instruction to run fetch(...) or allow fetching. A bare clone may also lack tree metadata. If info cannot find a path offline, the error explains that the path may still exist on the server.

Downloads through fs.get and download use storage_get_file, which writes a fragment directly to a destination without holding it all in memory. This needs its own implementation: the default _get_file in AsyncFileSystem raises NotImplementedError, so relying on it would make every get fail.

There is a memory limitation in open_async. Its file object supports seek and read, but Lore's store can only fetch a whole fragment. On first access, the file loads that fragment into memory and serves subsequent reads from the buffer. This makes one network request per file, but it does not limit memory use to the requested range. Reading only part of a fragment would require offset and length support in Lore's store, which is not available yet. A small seek does not avoid the initial full download.

Committing writes together

Lore writes content, stages it, and commits a revision. fsspec's Transaction lets the filesystem group those steps so everything in a block becomes one revision, or none of it does:

fs = fsspec.filesystem("lore", path="lore://vcs.example.com:41337/my-project", writable=True)
with fs.transaction(message="Import baked lighting", metadata={"job": "bake-4417"}):
    fs.pipe_file("Content/Lighting/Baked.bin", data)
    fs.pipe_file("Content/Config/Game.ini", ini_bytes)

As with GitFileSystem, callers must enable writes with writable=True. File writes also require an open transaction. Calling pipe_file outside a with block raises an error, since staging a file without committing would leave the working copy partly changed.

A successful transaction issues one revision_commit for all the files and pushes unless offline. If an exception occurs, it unstages and resets only the paths it touched. The metadata mapping is saved on the revision as string values. A generated commit can carry its job ID, source partition, or pipeline run without putting those details in the message and parsing them later.

Creating, switching, and merging branches are separate methods because merges may conflict and should happen explicitly. A transaction can write to a selected branch and restore the original branch on exit. A later merge() publishes the changes. If paths conflict, it aborts and reports them without resolving them automatically:

with fs.transaction(message="ingest Feb partition", branch="ingest-2026-02", create=True):
    fs.put_file("/local/part-new.parquet", "warehouse/events/part-new.parquet")
assert fs.ref == "main"   # the block restored the original branch on exit
 
fs.merge("ingest-2026-02")  # clean merge → one revision; conflict → abort + raise

This supports a write-audit-publish process: write new data on a separate branch, check it without exposing it on main, then publish the whole revision at once.

What the adapter makes possible

With Lore registered in fsspec, duckdb.register_filesystem(fs) and read_parquet('lore://...') work with existing DuckDB support. Polars reads DataFrames from fs.open() objects while still selecting only the requested columns. The branch-and-merge process above supports the same write-audit-publish approach used with object storage. Roughly fifteen hundred lines connect a month-old version control system for game assets to these Python tools.

Following GitFileSystem saved work where the behavior matched: relative paths, the :ref@inner syntax, and read-only defaults. Using a server URL for Lore let the adapter manage its local clone without making callers do it.

This is the closest I have come to the dataset setup I wanted. OpenXet taught me the cost of building chunk reuse, Lore supplies version control, and lore-fsspec lets Python tools read the data. Both Lore and the adapter are still pre-1.0, and fetching whole fragments remains a hard limit. Even so, this is the first time I have had all the pieces working together. fsspec itself needed no Lore-specific changes; registering the lore protocol was enough to use its existing tooling.

lore-fsspec is pre-1.0 and follows Lore, whose on-disk formats and APIs are also explicitly unstable. Install it with pip install lore-fsspec (Python 3.12+, MIT). The source is on GitHub*.