A repository that has got too big

Q: How do I find what is taking up the space?

Start with git count-objects -vH and look at the pack size. If it is far larger than a fresh checkout, something in the history is not in your working copy. Then list the largest objects to find out what it was called. Somebody usually recognises the top entry immediately.

Clones take ten minutes, the build is slow, and somebody has noticed. The cause is nearly always one file committed once, years ago, and deleted shortly afterwards. Deleting it is what did not help.

Why deleting it did nothing

The history is the repository.

A clone brings down every version of every file that has ever been committed, because that is what makes the history usable offline. Deleting a file is a new commit saying the file is no longer present. The earlier commits still contain it, so the data is still there and still travels with every clone.

A four hundred megabyte database dump committed in 2022 and removed a week later is a four hundred megabyte cost on every clone, every build agent, and every developer's machine, for as long as the repository exists.

It also compounds. Text compresses well and git stores only what changed between versions. Binary files usually do neither, so ten commits of a large binary can cost close to ten times its size.

Find out what it actually is

Measure before you touch anything.

Start with the size of the stored history rather than of your working files:

git count-objects -vH

size-pack is the number that matters. If it is far larger than a fresh checkout of your project, something in the history is not in your working copy any more.

Then find the largest objects and what they were called:

git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1=="blob"' \
  | sort -k3 -n -r \
  | head -20

That prints the twenty biggest blobs with their sizes and paths. In almost every case somebody recognises the top entry immediately, and it is a dump, an archive, a video, a design file, or a directory of dependencies that should never have been committed.

If you have git filter-repo installed, git filter-repo --analyze writes a fuller report to .git/filter-repo/analysis without changing anything.

Try the cheap things first

Two fixes that are not a history rewrite.

A history rewrite disrupts everybody, so it is worth ruling these out first.

A shallow clone, if the problem is only that fetching is slow on build agents. It takes the recent history and not all of it, which is usually all a build needs.

git clone --depth 1 git@github.com:owner/project.git

A partial clone, if people need the full history but not every old file. Git fetches the large objects only when something actually asks for them.

git clone --filter=blob:none git@github.com:owner/project.git

Neither shrinks the repository. Both remove the pain for the people feeling it, which is often the entire requirement, and neither asks anybody to re-clone or coordinate anything.

Large files you genuinely need

Git LFS, and when it is worth the trouble.

Some projects really do need large binaries under version control. Design sources, media, test fixtures. Git Large File Storage keeps a small pointer in the repository and the real file on a server, fetched only when checked out.

git lfs install
git lfs track "*.psd"
git add .gitattributes
git add design/cover.psd
git commit -m "Track design sources with LFS"

Two things to know before adopting it.

It only helps from the moment you start. Files already in the history stay where they are. Moving existing ones across is a history rewrite with all the disruption that implies.

It is not free and it is not universal. Hosts meter LFS storage and bandwidth separately, and anybody cloning has to have the extension installed or they get pointer files instead of their assets, which is a confusing failure the first time somebody meets it.

For a file that changes rarely and is not really source, object storage with a link in the readme is often the better answer, and it is free.

If you have decided to rewrite

The same tool, and the same warnings.

Removing a large file from the whole history is the identical operation to removing a leaked password, and it carries the identical costs. Everybody re-clones, forks keep the old objects, and the host may hold on to unreachable commits until asked.

git clone --mirror git@github.com:owner/project.git project-rewrite
cd project-rewrite

# By path
git filter-repo --invert-paths --path backups/dump.sql

# Or everything above a size, which catches what you have forgotten
git filter-repo --strip-blobs-bigger-than 10M

Then measure again before you push, because a rewrite that did not shrink anything means you removed the wrong thing:

git count-objects -vH

The full account of what a rewrite does not reach is on the secrets page, and every word of it applies here.

Not again

Four lines and one habit.

Ignore the usual suspects on day one. Dependency directories, build output, archives, dumps and media, unless media is genuinely the point of the project. The ignore page has a starting file.
Never commit a database dump. Not temporarily, not to move it between machines, not at three in the morning. It is the single most common cause of this problem and it usually contains customer data as well, which makes it the other kind of incident at the same time.
Treat anything installable as not yours. Dependencies are reproducible from a lock file. Committing them costs space forever to save a command.
Look at what you are staging. git status before git commit catches nearly all of it, and a hook that refuses very large files catches the rest.

Short answers

Questions people actually ask

I deleted the big file. Why is the repository still huge?

Because a clone brings down every version of every file ever committed. Deleting adds a commit saying the file is no longer present; the earlier commits still contain it. The data travels with every clone until the history is rewritten.

How do I find what is taking up the space?

Start with git count-objects -vH and look at the pack size. If it is far larger than a fresh checkout, something in the history is not in your working copy. Then list the largest objects to find out what it was called. Somebody usually recognises the top entry immediately.

Can I make clones faster without rewriting anything?

Often, yes, and it is worth trying first because a rewrite disrupts everybody. A shallow clone takes recent history only, which is all a build needs. A partial clone keeps the full history and fetches large objects on demand. Neither shrinks the repository and both remove the pain.

Should I use Git LFS?

If you genuinely need large binaries under version control, yes. Two caveats: it only helps from the moment you adopt it, so existing files stay where they are, and anybody cloning needs the extension installed or they get pointer files instead of their assets. For a file that rarely changes and is not really source, object storage with a link in the readme is cheaper.

Is it ever all right to commit a database dump?

No. It is the most common cause of this problem, the cost is permanent, and it usually contains customer data as well, which makes it the other kind of incident at the same time.

Every question on this site, in one list