I ran du -sh */node_modules across my projects folder one evening because the
laptop was complaining about space, and the number came back in double-digit
gigabytes. Fifteen-odd side projects, most of them abandoned, and every one of
them carrying its own full copy of roughly the same dependencies. React was on
that disk more times than I could count.
I had been treating a package manager as a thing that downloads packages and gets out of the way. That evening it stopped being invisible.
npm to yarn was about speed and a lockfile I could trust
The first switch was the easy one and I barely thought about it. Early npm gave
me installs that were slow and, worse, not quite the same twice. A package.json
range like ^4.17.0 could resolve to different trees on different machines, and I
lost an afternoon once to a bug that existed only in CI because CI had installed a
patch version my laptop had not.
yarn fixed the parts that were annoying me. yarn.lock pinned the exact tree, the
install ran in parallel and felt noticeably faster, and workspaces let me keep a
few related packages in one repo without ceremony. By 2021 npm had mostly caught
up, with a real lockfile and npm ci, so this stopped being much of an argument.
If it had been the only thing, I would probably have drifted back.
yarn to pnpm was about the gigabytes, and I had the mechanism wrong
pnpm's pitch is that it stores each package once on your machine and links it into every project that needs it. I read that, pictured symlinks, decided the symlink was the trick that saved the space, and installed it. I was half right in a way that took me a while to untangle, and untangling it is the actual reason I am writing this.
There are two mechanisms doing two different jobs, and I had credited the wrong one.
The first is a content-addressable store. pnpm keeps one copy of every version
of every package you have ever installed, in a store on disk, keyed by a hash of
the file contents. pnpm store path tells you where. Install react@17.0.2 in
ten projects and it lands in the store exactly once.
The second is how that single copy reaches your project, and this is the part I
had wrong. It is not a symlink. It is a hard link. A symlink is a small file
that points at a path; follow it and you arrive somewhere else. A hard link is a
second name for the same file, the same actual bytes on disk, the same inode. The
store holds the file once, and each project's node_modules gets another name
pointing at those same bytes. No copy is made. The disk cost of the second, third,
tenth project is close to nothing.
The hard link is what saves the space. The symlink does something else. That sentence is the whole post, and I had the two backwards for longer than I would like to admit.
The symlink builds the tree. Look inside a pnpm node_modules and the layout is
strange the first time:
node_modules/
.pnpm/
react@17.0.2/node_modules/react/ <- the real files, hard-linked from the store
...
react -> .pnpm/react@17.0.2/node_modules/react <- a symlink to that
The real package sits once under .pnpm, hard-linked to the store. The thing at
the top of node_modules that your imports resolve is a symlink pointing at it.
Hard links share the bytes; symlinks arrange them into a tree. Two tools, two
jobs.
One consequence worth knowing, because it confused me the first time it happened: hard links only work within a single filesystem. If your store and your project sit on different drives, pnpm cannot link and falls back to copying. That is why the store is kept per drive, and why moving a project to an external disk quietly made an install slower than I expected.
The thing I did not go looking for: it caught a real bug
npm and yarn flatten node_modules. Every package, direct or transitive, gets
hoisted up to the top so it can be found. A side effect is that you can import
something you never put in your package.json, because some dependency of a
dependency happened to drag it up to where your code can see it. It works until
that grandchild dependency bumps a version and drops it, and then your code breaks
for a reason that is not written down anywhere in your own file.
pnpm does not flatten. The top of node_modules holds only what you actually
declared; everything else is tucked under .pnpm where your code cannot reach it
by accident. The first time I ran a real project through it, two imports failed
immediately. Both were packages I had been using without ever installing. The
tool I picked for disk space found me two dependencies I did not know I had. I
had assumed a stricter folder would be a nuisance. It was the opposite.
Which one should you use
Honestly, all three install packages and your project will run on any of them.
If you are starting something today, I would reach for pnpm, and more strongly the more projects or CI runs you have, because the store pays off every time and the strictness saves you from a class of bug you would otherwise meet at the worst moment. If your team is already on yarn and content, there is no crisis worth a migration. npm is fine, it is already installed, and for a single small project you will not feel the difference.
The larger point is the one the du command made me admit. A package manager is
not glamorous and it is not where the interesting code lives, but it is the floor
the entire project stands on, and I had never once looked at how the floor was
built.
I almost skipped learning any of this, because on the surface they all do the same boring thing. What I got instead was the first time I properly understood the difference between a hard link and a symlink, which is a filesystem idea, not a JavaScript one, and which has turned up in three unrelated places since. The package manager was just the thing that finally made me open the folder and look.