SeaweedFS, FreeBSD and a rather deep rabbit hole

At this point, reading my blog, you could be forgiven for thinking that I have become entirely about IBM midrange systems, COBOL and offline-first computing. That is still true, in a way. I genuinely enjoy those things. But I am still very much attracted to large, complex architectures, distributed systems, and the sort of infrastructure where several apparently innocent design decisions eventually turn into a whiteboard covered in arrows.

This particular rabbit hole started somewhere else.

A few months ago, after designing an architecture for a Swiss trading firm, and again during my current engagement, I found myself looking for possible replacements for MinIO for some specific use cases. I had already written a little about MinIO earlier this year, and I was curious about what else existed in that space without immediately reaching for Ceph.

A friend mentioned VersityGW. Someone else suggested GarageHQ. Another pointer led me to SeaweedFS, initially through Stefano’s article about using it for Mastodon media storage.

So I built a small virtualised lab under VirtualBox and started poking at them.

VersityGW came first, running as a single node on FreeBSD. Have I mentioned that I like ZFS? GarageHQ followed, again on FreeBSD. What started as a single node eventually became two replicated nodes, because apparently I am incapable of leaving distributed storage alone once it starts behaving.

Then came SeaweedFS.

The FreeBSD port, together with quite a few examples and blog posts, naturally leads you towards the all-in-one weed server setup. That worked perfectly well. I had S3, I could put objects into it, and for a while everything looked reassuringly simple.

Then the little bell in my head rang.

SeaweedFS was not really the same sort of thing I had been comparing it with. It was rather like walking down the biscuit aisle and discovering that one of the packets, despite having “biscuit” written on it, actually contains a three-storey decorated ceremonial cake.

What really gave the game away was the weed command itself. While looking around, I noticed subcommands called master, volume and filer. I already knew that weed server could enable S3 with a flag, but seeing master made something click. A master implies that there may be more than one.

That was the point where I stopped thinking of SeaweedFS as another small S3 implementation and started seeing it as a proper software-defined storage system.

I went back to the project wiki and started reading more carefully. The architecture was there, although I would not say that its implications were immediately obvious from the quick-start material I had originally followed. SeaweedFS was designed as a set of independent roles which could be combined on one machine or spread across several of them.

Naturally, I wanted to try the whole thing.

I started with one FreeBSD VM, but instead of using the all-in-one weed server command I separated the roles. First came the master, then the volume server, then the filer with its embedded S3 gateway.

That immediately created another small problem. The FreeBSD port provides an rc.d script for the convenient all-in-one setup, but not separate service scripts for the individual roles, so I wrote them: one for the master, one for the volume server and one for the filer. Later I added the admin service as well. I did not strictly need the admin UI for the experiment, but a nice dashboard is rarely unwelcome when one is already voluntarily staring into a distributed-storage rabbit hole.

The first version of the lab was still very small. One machine ran the separated stack, and I added a second VM running only a volume server. I specifically wanted to see what SeaweedFS felt like when a node was just storage rather than another copy of the complete stack.

That worked, but it also made the next limitation rather obvious. With only one master, rebooting things during experiments could produce confusing moments where volumes temporarily appeared or disappeared from the topology. More importantly, I was no longer really testing the architecture that had caught my attention in the first place. A system designed to have multiple masters deserves to be tested with multiple masters.

The FreeBSD package was also behind the SeaweedFS version I wanted to explore, so instead of continuing to hand-build disposable VMs I finally gave in and made the experiment reproducible.

Vagrant, three FreeBSD nodes, three masters, three volume servers, two filers with S3, and one deliberately filer-less node later, I had something resembling a tiny software-defined storage lab.

It was certainly not a performance PoC. Three virtual machines sharing one laptop are not three storage servers, however politely VirtualBox labels them. But it was enough to start asking the questions I actually cared about, and that was when the volumes started doing something I did not understand.

When free space stopped meaning free space

The virtual disks were only 40 GB each, which at the time seemed almost extravagant. I had just finished experimenting with VersityGW and GarageHQ and was still carrying roughly the same mental model with me: this was a disposable lab, I was going to upload a few ISO images and perhaps an OVA or two, so 40 GB per node was plenty. In terms of actual data, it was.

My first SeaweedFS scripts also left most of the defaults alone. The volume size was roughly 30 GB and the volume server maximum was eight slots. I was not even using replication yet. Everything was defined as replication 000.

Then I started running into allocation problems. The disks were mostly empty, yet SeaweedFS was telling me, in effect, that it could not find somewhere to create another volume.

So I did the obvious thing: I read the documentation and started tuning.

First I reduced the volume size to 10 GB and set the maximum number of volumes to zero, letting SeaweedFS calculate the available slots from the disk size. That seemed sensible, except that on a 40 GB VM disk it gave me roughly three usable volume slots per node.

I had managed to make the problem worse.

I put the maximum back to eight, which gave me more room to experiment and allowed me to upload more files, although by then I had started changing enough variables that the picture was becoming rather muddy.

Then I added replication 001, because I wanted to see how SeaweedFS distributed the physical copies between volume servers. That was useful, but it also added another multiplier to something I still did not properly understand.

After a few more tests I stopped. There were simply too many knobs moving at once.

I wanted something small enough that I could look at the topology and reason about it almost as an allocation unit, rather than constantly translating large nominal volume sizes in my head. So I reduced the volume size to 1 GB and set the maximum back to zero.

One gigabyte became my little disk allocation quantum. Not quite a sector, obviously, but close enough as a mental model for the experiment.

That finally gave the tiny lab plenty of headroom. I could create more buckets, upload more files and actually watch volumes being distributed across the three nodes.

But the original mystery was still there.

By then I understood that every S3 bucket maps to its own collection. What I did not understand was why a collection containing almost nothing needed several volumes. Even more confusingly, some of those volumes had FileCount:0, and if I waited, SeaweedFS could allocate still more volumes to the same collection, also with FileCount:0.

I changed the volume size. I changed the slot count. I removed replication and added it again. I created fresh buckets. I uploaded tiny files and large files. I destroyed things and started again.

The extra volumes kept coming back.

Opening the lid

After several days, tuning parameters was no longer teaching me anything. I could make the symptom larger or smaller, but I still did not know what was causing it, and the documentation I had found did not explain the behaviour I was seeing.

I have been here before. Years ago, when I wanted to understand something sufficiently obscure in Solaris networking, eventually the documentation stopped being enough and I went to the source. I have done the same with the Linux kernel, OpenStack and Ceph more than once.

There is a point where another search result is less useful than opening the lid of the engine.

So I cloned the SeaweedFS source tree and started following the volume-allocation code.

The source code is truth.

And that was where the numbers finally started to make sense.

I eventually found some less obvious pages in the SeaweedFS wiki discussing optimisation and many-small-bucket workloads. They mentioned a set of master volume-growth parameters that I had not encountered in the documentation I had been following.

Then I found the defaults in the volume-growth source:

VolumeGrowStrategy = volumeGrowthStrategy{
        Copy1Count:     7,
        Copy2Count:     6,
        Copy3Count:     3,
        CopyOtherCount: 1,
        Threshold:      0.9,
}

That was the first click. A little further down was the function deciding which value to use:

func VolumeGrowthCountForCopies(copyCount int) uint32 {
        switch copyCount {
        case 1:
                return VolumeGrowStrategy.Copy1Count
        case 2:
                return VolumeGrowStrategy.Copy2Count
        case 3:
                return VolumeGrowStrategy.Copy3Count
        default:
                return VolumeGrowStrategy.CopyOtherCount
        }
}

The comment in SeaweedFS 4.40 is particularly useful: these are the number of logical volumes to create at once for a given replica copy count.

Suddenly my original tests with replication 000 made sense. One copy selects Copy1Count, whose default value is seven.

I uploaded one file to one S3 bucket and wondered why SeaweedFS was creating several volumes. It was not confused at all. It was doing exactly what the volume-growth strategy told it to do: create a pool of seven writable logical volumes for that collection.

Remember that every S3 bucket maps to a separate collection. That little detail had just acquired considerably more significance.

Then I added replication 001 to my experiments.

SeaweedFS calculates the physical copy count from the placement rule. In simplified form, the code eventually comes down to:

func (rp *ReplicaPlacement) GetCopyCount() int {
        return rp.DiffDataCenterCount +
               rp.DiffRackCount +
               rp.SameRackCount + 1
}

So 001 means two copies. Two copies select Copy2Count, whose default is six.

Six logical volume IDs, each stored twice, means twelve physical volume instances.

That was the multiplication I had been seeing in the lab. Nothing was randomly eating my volume slots. The allocator was behaving deterministically according to defaults that made much more sense once I knew they existed.

But that still did not explain everything. There was another behaviour that had been driving me slightly mad: I could create a collection, see writable volumes with FileCount:0, wait for a while, and then discover that more empty volumes had appeared.

I had assumed that some operation was triggering them.

It turned out that time itself was triggering them.

In the master source there is another rather important constant:

const (
        volumeGrowStepCount = 2
)

There is also a background loop in the master running on the current leader. After its first pass, it sleeps for roughly five minutes, with a small random delay added, and then walks through the volume layouts again.

The interesting part is the decision that follows. The master looks at how many writable volumes the collection currently has, how many are considered crowded, how many were created during the previous growth event, and the growth threshold.

One of the conditions in SeaweedFS 4.40 is essentially:

lastGrowCount > 0 &&
writable < lastGrowCount*2 &&
float64(crowded+volumeGrowStepCount) >
        float64(writable)*VolumeGrowStrategy.Threshold

If that condition is met, another growth step is requested. The step count is two.

There was my timer, and there were my apparently spontaneous volumes.

It also explained why staring at FileCount:0 had not helped. The algorithm was not asking, “does this volume contain files?” It was maintaining a writable-volume pool.

The 0.9 threshold was not simply “grow when a volume is 90% full”, either. It was part of a heuristic involving the state of that writable pool.

That changed how I looked at the whole thing.

The defaults stopped appearing wasteful or irrational. SeaweedFS was preparing several writable destinations in advance, presumably favouring concurrency and avoiding allocation pauses for a busy collection. That is quite a reasonable thing for a distributed blob store to do.

My tiny lab was simply a terrible environment for those assumptions.

A production storage node might have terabytes of space and hundreds or thousands of useful volume slots. I had three 40 GB virtual disks, several S3 buckets, and the unreasonable expectation that a mostly empty disk should count as mostly empty.

Worse, S3 repeats the writable-pool policy for every bucket because every bucket receives its own collection.

That was the missing piece. I was no longer trying random values and hoping SeaweedFS would behave. I finally understood which parameters were controlling the behaviour, what units they were expressed in, and why changing one setting without considering the others had produced such confusing results.

So I went back to the lab one last time and reduced the global growth strategy to one logical volume at a time:

[master.volume_growth]
copy_1 = 1
copy_2 = 1

For the S3 bucket path, with replication 001, I also used:

fs.configure -locationPrefix=/buckets/ \
    -replication=001 \
    -volumeGrowthCount=2 \
    -apply

Those two settings look different because they use different units. copy_2=1 means one logical replicated volume set, while the path-specific growth count of two represents the two physical instances required by 001.

Then I destroyed the lab and rebuilt it from scratch.

This time a new bucket received one logical volume ID with its expected replica. When that volume reached its rollover point, SeaweedFS created the next one. I waited, and no mysterious flotilla of empty volumes arrived five minutes later.

The beast had not been defeated. It had simply been understood.

In the end, I settled on the parameters documented in my SeaweedFS on FreeBSD Codeberg repository.

Understanding the flight envelope

After spending rather too much time staring at volume IDs, source code and cups of tea, I think the most useful way to understand SeaweedFS is to look at where it came from.

SeaweedFS did not start life as an S3 server. It started as a blob store, originally inspired by Facebook’s Haystack design, with the rather specific goals of storing very large numbers of files and serving them quickly. The master manages volumes rather than every individual object, while the volume servers deal with the blobs themselves. The filer, filesystem semantics and S3 compatibility came on top of that architecture later. More recent additions, such as the Iceberg REST catalog and the developing message-queue work, make the project even broader.

That history matters because it also helped me make peace with the volume-growth behaviour that sent me down this particular rabbit hole. Those apparently strange defaults are not random numbers left behind by an absent-minded developer. Keeping several writable volumes ready makes a great deal of sense when the workload is a busy collection receiving many concurrent writes. The master is maintaining a pool of places where data can go without having to stop and allocate storage every few seconds.

That is comfortably inside the original flight envelope.

The problem appears when S3 semantics are placed on top of it and every bucket becomes its own collection. Suddenly a useful reserve for one busy namespace can become an expensive reserve repeated across dozens or hundreds of mostly empty ones.

SeaweedFS still works outside that original shape. My experiments eventually showed that it can be tuned to grow volumes much more conservatively. But I think the distinction is important: outside the natural flight envelope, you need to understand what the system is doing rather than simply accepting the defaults.

For S3 specifically, my own feeling after these experiments is that SeaweedFS looks particularly attractive where there are relatively few, substantial buckets and where the workload is reasonably well understood.

A Veeam or Proxmox backup target immediately comes to mind. So does a data lake, a large image or video repository, or something such as Mastodon media storage. These are workloads where a bucket can contain an enormous amount of data, and where the extremely small resource footprint of the volume servers becomes genuinely interesting.

I would be considerably more careful with an environment expected to accumulate a very large number of small buckets. Static websites are an obvious example. Terraform state is another. Modern CI/CD and infrastructure-as-code environments are remarkably good at turning one innocent resource definition into hundreds of little namespaces containing three files and a sense of purpose.

That matters because, even after tuning the growth policy, bucket cardinality is not free. Each S3 bucket maps to a collection, and collections have volumes and associated operational costs.

There is another consideration around multiple datacentres. SeaweedFS has useful placement rules, and replication values such as 100 or 101 can place replicas across datacentres. But data placement and service availability are not the same thing. The masters use Raft, so two active datacentres still need a third independent vote if the design is expected to survive losing either site.

The filer adds another layer to that discussion. Local filer stores and filer replication can be perfectly reasonable for some workloads, but a highly concurrent active-active design may push you towards PostgreSQL, MySQL or another external metadata store. At that point the failure model no longer belongs entirely to SeaweedFS.

That makes me slightly uncomfortable, not because I am afraid of complex systems. Quite the opposite. I like complex architectures. I simply prefer complexity that is internally coherent.

Ceph is an obvious comparison here, although it comes with a considerably heavier bill. OSDs consume substantially more resources, the minimum sensible hardware footprint is larger, and operating a Ceph cluster is not something I would casually inflict on somebody who only wanted somewhere to put a few backups.

But Ceph buys something with that cost. Its metadata ultimately lives inside RADOS, and its storage services share the same underlying failure model. It is a large, complicated machine, but it is one machine.

SeaweedFS has an enormous advantage when RAM and hardware footprint matter. Its volume servers are remarkably light, which is particularly attractive now that memory is once again something one notices on a quotation rather than an almost-free line item.

The trade-off is that I would want a clearer understanding of the workload before choosing it, and that may be the hardest requirement of all.

In my experience, customers surprisingly often do not know what their S3 service will eventually contain. Sometimes the requirement really is little more than: “I need an S3”.

No expected bucket count, no object distribution, no idea whether it will become a backup repository, an analytics platform, a collection of CI artefacts, or all three by next Thursday.

In that situation Ceph’s flexibility has value, even if the price tag is substantial. When the workload is known and fits SeaweedFS’s strengths, however, the much lighter architecture can be extremely compelling.

VersityGW and GarageHQ ended up belonging to somewhat different categories again, and I still think both deserve attention. VersityGW appeals to me particularly for the unfashionable but very understandable design of a large FreeBSD/ZFS storage system with a properly planned replicated system elsewhere. GarageHQ is a much more focused distributed S3 system.

There is no winner here, which is probably a healthy result. I started this experiment looking for relatively simple alternatives to MinIO. Instead I discovered that one of them was a complete software-defined storage platform hiding inside a surprisingly small binary.

I understand SeaweedFS much better now. More importantly, I think I understand where I would use it, where I would ask more questions, and where I would reach for something else.

For an architecture experiment carried out on three virtual machines, one laptop and an unreasonable quantity of biscuits, that feels like a useful outcome.

If you would like to continue the journey, the FreeBSD rc.d scripts, Vagrant lab, Ansible provisioning and the configuration I eventually settled on are all in the SeaweedFS on FreeBSD repository on Codeberg.

Biscuits, as previously established, are not included.