git is the most commonly used version control system for software development1, also thanks to popular central hubs like GitHub, GitLab, BitBucket, etc.2 which greatly simplify collaboration for OSS projects or provide a central place for proprietary software development in IT companies.

There are scenarios in software development that may require or can benefit from versioning large files into the development cycle, for example:

• game development (graphics, soundtracks or sound effects, video clips, etc.)

• machine learning (training data, processed data, model networks, etc.)

• audio processing applications (audio plugins, eg. VSTs, sound assets, etc.)

Sourcing large files or vendoring external assets in the same repository where the rest of the codebase lives can greatly reduce integration friction and simplify deployment reproducibility, since all source code and related assets updates can be kept in sync and committed together3.

However, while git excels at handling thousands of small source text files, its design struggles when it comes to handling large binary files. There have been attempts at improving performance in this aspect4, but git remains a suboptimal choice for large file management when used alone.

External solutions have thus been developed to handle large files by extending git's abilities. Their common principle of operation relies on git hooks, specifically the “smudge/clean interface” that allows to automatically swap between the actual file at tree checkout and a replacement “pointer file” that gets stored into git’s database at commit time. The actual large files get offloaded and managed separately by the git extension, essentially creating a side-versioning system to git for large files only. Using this approach, dealing with large files becomes almost transparent for the user who can continue to follow their usual git versioning workflow.

The most two popular options include Git LFS (developed by GitHub, since 2014) and git-annex (developed by Joey Hess, since 2010)5. I want to focus and promote git-annex over LFS for the following reasons:

• git-annex predates LFS, hence is more battle tested given both projects are still being actively maintained6.

• git-annex is more flexible than LFS when it comes to where to store large files and when to retrieve them - LFS behaves more automagically7, and while this may make it easier to use at the beginning, it becomes a hard wall when more flexibility is needed. git-annex may need more tuning, but its flexibility pays off in the long term.

• git-annex embraces the distributed nature of git, preventing any lock-in to a specific remote or hosting provider, and instead allowing data replication among clones and "special remotes" at will8. LFS is tightly integrated into the most common git hosting providers (GitHub, GitLab, Bitbucket) which also provide storage for your large files, enforcing a centralised architecture and binding you to their pricing policies9.

• git-annex’s multitude of "special remotes"10 allow for more flexible configurations including storing files in AWS S3, Google Drive, Dropbox, your external hard drive, you name it - among them, it also has an LFS backend, making migration seamless.

Introduction to git-annex

Before diving into a concrete example of how to setup git-annex for a project, let me introduce a foundation for how to understand git-annex in 2025. Specifically, this article is based on git v2.39.5 and git-annex v10.20250416 so be mindful of features and deprecations in case you read this article far in the future.

git-annex is a relatively complex system, and it went over several best practices over the years, mostly to accomodate with specific file-system constraints (e.g. support for symlinks) and improvements in git hooks (e.g. “direct mode”, now deprecated in favour of “unlocked files”). Moreover, Joey attempted to promote git-annex as a mechanism for cloud storage (a-la-Dropbox), introducing a bunch of commands that behave so automagically that even bypass vanilla git behaviour (i.e. auto-commit, auto-pull, auto-push, auto-merge, etc.), so you may want to stay away from git annex webapp, git annex assistant and relatives — git annex sync is probably as automatic as you want to go (but see the configuration and caveats in the following section).

If however we put the “cloud storage” interface aside, git-annex is pretty straightforward and robust when it comes to its lower-level commands, which are probably the only commands you ever need to care about:

• git annex add

• git annex get

• git annex drop

• git annex copy

• git annex move

In git-annex lingo, large files are referred to as "annexed files" since the rules to define what goes to regular git and what goes to the annex are arbitrary and totally up to you11, so for the rest of the rest of the article I'll simply refer to them as "annexed files".

How are annexed files stored

git-annex offloads annexed files for a repository by placing them in its .git/annex/objects/ directory according to their hash (i.e. their key)12. This is convenient, because it means that a git-annex-enabled repository is self-contained, you can move it around in the file system without breaking its integrity.

Another implication of this model is that annexed files get attached to a clone. This is in contrast with Git LFS, where there’s a single source-of-truth for large files which would be the central repository (e.g. GitHub), while local copies are usually considered as a cache.

In git-annex, each clone is treated equally, and annexed files can be copied or moved among clones arbitrarily: this is achieved by assigning a uuid to each clone and keeping a log of which clone uuid contains which annexed file key via a special orphan branch13 named “git-annex”. Any given clone can thus have a set of “present” and “missing” keys.

The default policy for git-annex is to guarantee that at least one clone in the network contains any given annexed file, so that dropping files from a clone can be done securely by checking the “git-annex” branch and explicitly communicating with available remotes to ensure the file to be dropped is replicated elsewhere14.

Like for regular git, the contents stored in the internal database (as objects) and the contents of the checked-out working tree are independent. In git-annex’s case, the working tree can reference to the annexed object in two ways: as “unlocked” file or “locked” file.

• locked form: the commited tree is saved as an absolute symlink that starts with /annex/objects/..., while the checked out file is a relative symlink that points to the file inside of the .git/annex/objects/ directory. If the annexed file is not present in the current clone, then the symlink appears broken on checkout. Since the file inside of the objects directory is write-protected, this mechanism creates a “read-only” access to annexed files (hence why it’s called “locked form”)15.

• unlocked form: the commited tree is still saved as an absolute symlink like above, but its file mode bits are set differently to differentiate the form, while the checked out file is a copy of the file inside of the .git/annex/objects/ directory, allowing you to perform edits over the file (which can then get committed under a new key). If the annexed file is not present in the current clone, then you get a little text file on checkout containing the key of the annexed file, instead of a broken symlink, which can be a bit confusing but can behave more transparently for programs that don’t like broken symlinks.

Example case: sourcing trained ML models and VSTs for an audio application

For simplicity, we'll start from a classic centralised configuration where developers work on their clones and sync back to a single repository representing the source of truth of the codebase. This is the typical scenario for projects hosted on GitHub or similar. As for annexed files instead, since GitHub doesn't provide support for annex storage, we'll use an S3 bucket — given the pricing is quite affordable, compared to GitHub's LFS storage plan — via the S3 git-annex special remote.

Initial Setup

To start using git annex in an existing or new repository, simply issue:

git annex init

This command will create the "git-annex" branch and assign a uuid to the current clone. Next are a couple of personal taste configuration options that make git-annex more well behaved in respect to a traditional git workflow:

# for git annex add / git add
git annex config --set annex.addunlocked true

# for git annex sync
git annex config --set annex.autocommit false
git annex config --set annex.synccontent true
git annex config --set annex.synconlyannex true
git annex config --set annex.resolvemerge false

These options only need to be configured once, since they will be recorded to the “git-annex” branch and be applied to every clone. Let me explain what they do:

• By default, annexed files are added in locked form (i.e. as symbolic links, as described earlier). Some software may misbehave when symlinks are involved (especially if symlinks are broken, i.e. when annexed files are missing from the clone), so you may get more transparent behaviour when using “unlocked” files instead (which become simple small text files when annexed files are missing). The only tradeoff is that your data would get duplicated between the annex database and the working tree — but this is also true for any other file in a regular git repository (one blob in the git database, one file in the working tree) — so annexed files will take twice the amount of storage unless you use a CoW file-system (eg. btrfs, zfs, apfs).

• git annex sync is the easiest command to let git-annex manage itself (i.e. sync the git-annex branch) and transfer files among remotes as needed. You may find surprising, however, that it comes with a bunch of questionable default behaviours, specifically:

  • if your working tree is dirty, git annex sync will auto-commit your changes with an auto-generated commit message — you may prefer to have complete control over what you commit.

  • despite the command name (sync), annexed files don’t get transferred between repositories (this however may change in a future version of git-annex16)

  • sync will push/pull your checked out branch (!) and create a bunch of synced/… prefixed branches for each existing branch in your repository — synconlyannex is a must to make git-annex care only about the annex and not interfere with your traditional git workflow in any other way.

  • similarly, git annex sync will attempt to automatically resolve merge conflicts involving annexed files by adding variants of the files to the working tree rather than keeping your index in a “conflict” state (that you can later manage by hand) — to be honest, I didn’t come across a conflicting scenario yet, so I’m unsure whether this option is better set on or off (feedback appreciated).

Configure S3 remote

Next step is to configure an S3 bucket to use for storing the annexed files in a central location, so that they can be shared with your colleagues. There’s no special requirement for creating the bucket, so I’ll skip the details (just use any infra worfklow you’re already familiar with, e.g. sst, terraform, or even manually).

You will need a mechanism to export these environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN (if you use temporary credentials), when interfacing with git-annex. I use this convenience script, but you can use any other compatible workflow:

# if you use AWS profiles, specify it here
export AWS_PROFILE="<your_profile>"

# check if session is still valid, and login otherwise
# note: enforcing code-based auth so we can login from a remote ssh device
if ! aws sts get-caller-identity > /dev/null; then
    echo "Session expired, login again..."
    aws sso login --use-device-code
fi

# export AWS env vars to be used by every other tool depending on them (e.g. git-annex)
eval "$(aws configure export-credentials --format env)"
echo "AWS environment variables exported correctly"
echo "Your AWS session will expire:" $(date -d $AWS_CREDENTIAL_EXPIRATION)

which must be sourced, rather than invoked, like so:

source aws-setup

💡 Remember to run this command whenever you use a new shell or your credentials expire, otherwise git-annex operations that involve the s3 bucket will fail.

Let’s assume you have a bucked configured with the following properties:

name:    acmecompany-myproject-annex
region:  eu-west-1

To make git-annex be able to store files on it, we’ll add an S3 special remote. This operation only needs to be done once, since the configuration will be stored in the “git-annex” branch (note: “s3storage” is an abritrary name identifying the remote and it’s up to you).

git annex initremote s3storage type=S3 encryption=none datacenter=eu-west-1 bucket=acmecompany-myproject-annex protocol=https partsize=200MiB signature=v4

Additionally, to let git annex sync copy files automatically to the remote, and prevent colleagues from accidentally dropping files from the remote, we can enforce required content settings like so:

git annex group s3storage backup
git annex required s3storage standard

Committing files into the annex

Now that git-annex is configured and tamed, the easiest way to source large files into the annex is by explicitly adding them via:

git annex add assets/lsp-plugins.vst3 assets/classifier.pt

while any other regular file can be added to git in the usual way with:

git add src/classifier.py

and finally commit in the usual way:

git commit -m "feat: updated classifier to run on band-passed audio via LSP"

Push & Pull of commits and annexed files

Remember that moving files around with git-annex is a manual operation, so simply running:

git push

will only push your git commit history to the remote (i.e. GitHub), no matter the remote is a git-annex-enabled remote or not. To explicitly push the files to the S3 bucket, you can run:

git annex sync

Similarly, when your collegues will want to integrate your changes, they’ll have to run the following three steps:

git pull
git annex sync
git annex get

Local management of annexed files

The manual control of git-annex is extremely flexible. Say you’re working on a large monorepo containing several projects, each containing hundreds of MB of annexed files (e.g. test fixtures), but you only care about working on one specific project. Contrary to LFS, you’re not forced to an all-or-nothing situation, but instead can selectively git annex get <path/to/files> and git annex drop <path/to/files> according to your storage and network bandwidth constraints.

Moreover, since the annex is independent from your checked out branch, you may want to get rid of old versions of files that are not reachable by any branch anymore with git annex find --unused and git annex drop --unused (or, alternatively, git annex move --unused --to s3storage, in case the drop fails because files were not replicated).

In the scenario where you want to safely delete a clone, you can do the following:

# ensure all your commits are pushed
git push --all
git push --tags

# migrate all historical annexed files to S3, leaving the current clone with no annexed files left
git annex move -A --to s3storage

# propagate git-annex updates
git annex sync -g

# mark this clone uuid as dead to hide it from annex logs, etc.
git annex dead here

# finally delete the repository
cd .. && rm -rf <your-repo>

Dealing with annexed files in CI

git-annex assigns a uuid to each clone in order to map where annexed files are located. However, CI workflows usually run in a disposable environment, so we can avoid the overhead of letting git-annex track each one-time usage clone by adjusting the local git config before initialising git-annex on the CI cloned repo:

git config annex.private true
git annex init
git annex enableremote s3storage
...

And since we’re setting up CI, why not also checking if annexed files have been pushed to the S3 remote? This can be a handy PR check to ensure your colleagues pushed their annexed files. Here’s an example GitHub Workflow:

name: "git-annex checks"

on:
  - "pull_request"

jobs:
  check-annexed-files:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Install git-annex
        uses: awalsh128/cache-apt-pkgs-action@latest
        with:
          packages: git-annex
          version: 1.0

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::<your-company-id>:role/GitHub
          aws-region: <your-bucket-region>

      - name: Checkout code
        uses: actions/checkout@v4
    
      - name: Setup git-annex
        run: |
          git config user.name github-actions
          git config user.email github-actions@github.com
          git config annex.private true
          git fetch origin git-annex
          git annex init
          git annex enableremote s3storage

      - name: Check annexed files in s3
        run: |
          missing=$(git annex find --not --in s3storage)
          if [ -n "$missing" ]; then
             echo "Un-pushed annexed files found:" "$missing"
             exit 1
          fi

Conclusions

In this article I overviewed how to source “large files” in git thanks to git-annex, explored the basic internals of git-annex and walked through an example use case.

Hopefully, I’ve convinced you that git-annex is quite powerful and can enrich your DevOps workflow, without having to rely on Git LFS as the only option.

Let me know how’s your experience in the comments section and I can enrich the article accordingly :)

Thank you for reading!