What a free plan actually costs when every customer has a database

Consonas gives every organisation its own database, for reasons set out elsewhere. One consequence is that a free account is not nearly free for us the way it is for a product built on a shared database. So before committing to a genuinely generous free plan, we worked out what it costs.

The estimate we started with

Our first plan assumed an active free organisation at its thousand contact limit would use roughly twenty megabytes. From that we concluded that at scale the storage bill would be significant enough to justify building hibernation: archiving a dormant organisation's database to cheaper storage, removing it, and restoring it when someone signed in again.

We designed it in full and got far enough to be confident of it. Then we measured the thing it was solving.

The measurement

Allowing for everything else a real organisation accumulates, a working figure is two to four megabytes rather than twenty. The estimate was out by roughly an order of magnitude.

How we measured it, so you can repeat it

Nothing here required tooling anyone else lacks, which is rather the point. The method was four steps.

Create an organisation and record the storage it occupies before anything is in it. That is the floor: the schema, the indexes, the journal, the handful of rows that exist in an account with one member and nothing else. It came to forty kibibytes.

Then insert contacts in known quantities, measuring after each batch, and take the gradient rather than dividing the total. Dividing the total by the count folds the floor into the per contact figure and flatters or punishes the number depending on how many you inserted. The gradient was three hundred and thirty six bytes per contact including the timeline entry that creation writes.

Then count rows written rather than statements executed, because that is the unit the bill is denominated in. This is where the surprise was.

Then project. A free organisation at its thousand contact limit is the floor plus a thousand gradients, which is 0.37 mebibytes. Allowing generously for everything a real organisation accumulates that a synthetic one does not, notes with actual text in them, attachments, a year of tasks and appointments, the working figure is two to four mebibytes.

The estimate we had been designing against was twenty. It was out by roughly an order of magnitude, and every hour spent on the mechanism to mitigate it was spent on the strength of a number nobody had checked.

What that means per organisation

At three mebibytes, with a realistic amount of activity, the monthly cost of an active free organisation is well under one penny. At a million such organisations, storage is roughly six hundred dollars a month. Real money, and not a threat to anybody's business.

That figure is worth holding against the thing it buys. A free plan with a thousand relationships, the full audit trail, consent records, subject access tooling, two factor sign in and complete export costs us, in storage, about the same per year as a single modest lunch. The reason free plans in this industry are usually stingy is almost never storage. It is that the free tier is designed as a funnel rather than as a product, and a funnel works better when it hurts.

The number that surprised us

Not storage. Rows written.

Creating one contact costs thirteen rows written against two logical inserts, because maintaining indexes more than doubles the count. Anyone estimating write costs from the number of inserts in their code is going to be light by a similar factor.

This is the general shape of the mistake and it is worth naming. Storage is intuitive because a person can picture a megabyte. Writes are not, because the unit billed is not the unit you wrote in your code, and the multiplier between them depends on how many indexes a table carries, which is a decision made months earlier by somebody thinking about read performance and not about the bill.

The bigger one is scheduled maintenance. A daily background task per organisation counts as a row written. A million organisations each running one daily task is thirty million rows a month, sixty per cent of the included allowance, spent achieving nothing in particular. Weekly is eight per cent.

What we built is stricter than either, and it is worth stating exactly because an earlier version of this post described a weekly band by plan that does not exist. There is no band. An organisation wakes only when it has earned a wake: a webhook delivery waiting to be retried, something in the recycle bin old enough to purge, or a retention rule somebody wrote, which earns one wake a day and only for as long as the rule stands. An organisation with none of those sets no alarm at all and therefore costs nothing to keep. That is nought per cent rather than eight, and it is the decision worth more than most of the optimisation we might have done elsewhere.

There is a lesson in the ratio. The expensive thing was not the work, it was the heartbeat. A million organisations doing nothing, but doing nothing on a schedule, cost more than a million organisations being used. Anything that runs per customer per interval is worth pricing before it is worth writing, and the interval is a bigger lever than the work inside it.

What the bill is denominated in, and what people assume it is

Half of the error in the original estimate was arithmetic. The other half was vocabulary. The words on a cloud invoice look like ordinary English and are not, and the distance between what they mean and what someone assumes they mean is where an estimate goes wrong by a factor rather than by a percentage. These are the terms that caught us, set out plainly enough that a reader can check their own assumptions before spending an afternoon producing a figure.

A row written is not an insert. An insert is something you wrote in your code. A row written is something the storage engine did, and one of the first is routinely several of the second, because every index on the table has to be brought up to date and each of those is itself a row. The multiplier is a property of the schema rather than of the code, which is why reading the code and counting the inserts produces an answer that feels careful and is not. An update is a write. So is a removal, which catches anyone who pictures writing as the act of adding something, and it means that tidying up after yourself has a price on it too.

A row read is not a query. A query is one thing you asked for. The rows read are what the engine had to look at in order to answer, which is the rows it touched rather than the rows it gave back. A query that returns one record out of a thousand has either read one or read a thousand, depending entirely on whether an index covered the question, and nothing about the shape of the query tells you which. We did not need this number to make the decision described here, and so we did not chase it, which is itself part of the method: a measurement that cannot change a decision is a hobby.

Storage is a rate instead of a quantity. It is billed as an amount of data held across a period, not as an amount of data. Half a megabyte kept for a year and half a megabyte kept for an afternoon are not the same purchase, and the second one barely exists. This is why the interesting question about a dormant organisation is how long it is going to sit there rather than how big it is, and it is why the dormancy clock does more for the bill than any amount of care over the schema. It is also the reason the phrase storage cost is misleading in a sentence that does not name a period.

A mebibyte is not a megabyte. The table above says KiB and MiB because those are the units the measurements arrived in, and converting them into round decimal figures for the sake of a nicer sentence would have made every number in this post slightly unreproducible. The difference is small enough that nobody would notice it in prose and large enough to matter when somebody is comparing our figure against one of their own. A measurement reported in the unit it was taken in can be checked. One converted for readability has to be trusted.

The floor is not the gradient. The floor is what an organisation costs for existing: the schema, the indexes, the journal, the few rows an empty account contains. The gradient is what one more record adds. These are separate numbers and they carry opposite implications. A product with a high floor and a shallow gradient is expensive to have many customers and cheap to have large ones, which is the arithmetic that pushes a company towards a stingy free plan and a high minimum. A product with a low floor and a steep gradient is the reverse, and it pushes towards free plans that are generous with accounts and mean with records. Knowing which one you are is more useful than knowing the total, and the total is what most people measure.

An allowance is not a free plan. There are two frees in this story and confusing them is easy. Ours is the plan you can sign up for. The other is the quantity included on our own bill before charging begins, and the trap in it is that spending an allowance feels like spending nothing, because nothing arrives to tell you. The daily maintenance task would have consumed a majority of the included write allowance and produced no invoice at all for as long as it stayed under the line. A cost you cannot see is not a cost you are not paying, it is a cost you will meet all at once on the day you cross a threshold, which is the worst possible day to discover the shape of it.

The rule we took out of this is smaller than it sounds. Before estimating anything, write down the units the invoice is denominated in, then describe the design again in those units rather than in the units you think in. Most of the surprise in this post disappears under that translation, because a design described in the units it is billed in tends to explain its own cost without anybody having to be clever about it.

What we did with hibernation

We withdrew it. Measured properly it saves about three hundred and sixty dollars a month at a million free organisations, which is the difference between holding that volume in the per organisation databases and holding it in object storage, not the several thousand the old estimate implied. Against that saving sits a mechanism whose job is to remove a customer's database, which also ends its point in time recovery, making it the only routine operation in the platform capable of destroying somebody's only copy of their data.

Put like that the decision makes itself. We were proposing to run, thousands of times a week, forever, an operation whose failure mode is a customer losing everything, in exchange for a saving that is a rounding error against the cost of the people who would maintain it.

Instead, a free organisation nobody signs into for six months is treated as dormant. We write to its administrators at six, seven and eight months, offering a complete export. At twelve months the organisation is closed rather than deleted: a final notice, then thirty days in which an owner can export or reopen it. That saves the same money using a mechanism we already have and already trust, and it is easier to defend under the storage limitation principle the Information Commissioner's Office publishes, because keeping personal data nobody has looked at in two years is harder to justify than deleting it with warning. What each of those three letters offers is the complete export, which is free on every plan and takes seconds.

There is no hibernation code in the product. An earlier version of this paragraph said the code was kept unused as evidence that it could be rebuilt quickly, and a reader who went looking for it would have found nothing, which is a worse outcome than the sentence was worth. What is kept is this post and the arithmetic in it, which is the part that would actually be needed again if measured storage per free organisation ever exceeded ten megabytes. A dormant intention with its working shown is more use than a dormant mechanism nobody has run.

Where these numbers could still be wrong

Publishing measurements invites the question of what they do not cover, so here is our own list.

The contact gradient was measured on records with realistic fields but short notes. An organisation whose people write two paragraphs on every interaction will sit above the working figure, though notes compress well and the effect is smaller than intuition suggests.

Files are excluded entirely, because they are not held in the database. They sit in object storage and are billed differently, at a rate that makes the calculation uninteresting until somebody stores video.

The million organisation projection assumes today's rates and today's shape of the product. Both will change. The reason to publish it anyway is that the decision it informed, weekly rather than daily maintenance, holds across a very wide range of rates, and a decision that is robust to the number being wrong by an order of magnitude is the kind worth making from a measurement.

Finally, the whole exercise measures a synthetic organisation. Real ones are messier in ways nobody predicts. We will measure again on real data once there is enough of it to be representative, and if the number moves we will say so here rather than leaving the optimistic version standing.

What we decided not to measure, and not to build on top of it

An afternoon of measurement produces two things: a number, and a list of adjacent projects that now look obvious. The second one is more dangerous than the first, because each item on it arrives wearing the authority of the measurement that suggested it. Here is what we declined, and why each refusal is a position rather than an item we have not got round to.

A ranking of customers by what they cost us. Being straight about this is worth more than the cleaner sentence we would rather write. Capacity planning has to know how large the largest organisations are, so a nightly job records a size and a few counts against each organisation, never any content, and sorting that table is one query. The operator interface does sort it, and pretending otherwise would be the cleaner sentence and the false one. There is a screen that answers where to look first this morning: the quarantined organisations, then the ones whose export failed its nightly read back, then the largest by stored bytes. It exists because a runbook was going to ask that question anyway and answering it before somebody asks is cheaper than answering it during an incident.

What we will not do is act on the sort. Nothing prices, throttles, deprioritises or degrades anything from that list, and there is no path in the code by which a position on it changes what an organisation gets. That is the guarantee worth making, because a company that can rank its customers by cost eventually acts on the ranking, and the action is always the same one, which is that the cheapest customers quietly get a worse product. The measurement exists for capacity and for triage, it carries counts rather than content, and this paragraph names it instead of a marketing page implying it does not exist.

A cost figure shown to you. There was a suggestion that the honest thing to do with all this arithmetic is to show each organisation what it costs to run. It sounds transparent and it is actually the opening line of a conversation no one benefits from. The figure would be a guess at the level of one organisation however carefully it was built, it would move for reasons the customer cannot control, and its only real use to a reader would be working out how much guilt to feel about being on the free plan. The whole argument of this post is that the answer is none. Publishing the aggregate arithmetic once, in public, where anybody can check it, does the useful part of that job without turning it into a monthly reminder.

The write multiplier itself. The natural reflex on discovering that one contact costs thirteen rows written is to attack the thirteen. Fewer indexes, a narrower timeline entry, a cleverer schema. We did not, and the reason is worth stating because it is the same reason the rest of the post exists: thirteen rows per contact created, multiplied by the number of contacts anybody actually creates in a day, costs nothing worth the effort. The daily maintenance task cost far more while doing far less. Having found a multiplier, the instinct is to fight the multiplier, and the arithmetic almost always says the interval is the better target. Indexes also make the product fast for the person using it, which is a benefit paid to the customer, whereas the heartbeat was a benefit paid to no one.

What to do to free organisations when the platform is busy. Measuring a free organisation invites a follow up project about slowing one down, deprioritising it, or giving it less when there is contention. We have not designed one. The objection is the one that ended hibernation: it is a permanent mechanism, running forever, whose failure mode lands on the people with the least ability to do anything about it, in exchange for a saving the measurement says is not there. It is also worth noticing that each organisation has its own database inside its own compute, so the contention that would justify such a mechanism is not the shape of problem this platform has.

What the mechanism would have cost in people. This is the one we deliberately left as a sentence rather than a figure. Hibernation would have needed maintenance, monitoring, an on call story, and an answer every time it misfired, and we could have produced an estimate of the salary attached to all of that. We did not, because a number in that position stops being an input and becomes a thing to argue with. Someone who wants the mechanism built halves the estimate, someone who does not doubles it, and the meeting is now about the estimate. The sentence, which is that the saving is smaller than the cost of the people who would look after the saving, is both true and unarguable, and it decided the question in less time than the spreadsheet would have taken to build.

What connects these is that measurement is not automatically virtuous. Every number you produce is an invitation to do something with it, and some of those somethings are worse than the ignorance they replace. The measurement in this post was worth taking because a specific decision was waiting on it. The projects it suggested afterwards were not waiting on anything, which is the test we now use.

What this means if you are the customer

Two things, and both are more useful than the arithmetic.

The free plan is not a trial with a countdown. It costs us a fraction of a penny a month to run, which is why it can be permanent, why it can carry the full audit trail and export, and why nothing about it is designed to become uncomfortable at a convenient moment. A free plan that costs the supplier almost nothing does not need a funnel to justify itself. What the plan actually allows is on the pricing page, stated as limits rather than as current values.

And the dormancy process is what it is because we chose the safer mechanism over the cheaper one. If you leave an organisation untouched for six months, you will hear from us three times with an offer of your data before anything happens to it. That is the customer facing shape of a decision that was made about database operations, which is usually how these things work and is rarely written down anywhere a customer can read it.

The free plans we designed first and threw away

The plan described on the pricing page went through several shapes before this one. The earlier ones are more interesting than the survivor, because each was killed by a specific objection and the objections are the argument. None of these exist. They were drawings on a wall, and the measurement above is what settled most of them.

Free organisations sharing one database. This was the cheapest idea by a very wide margin and it lasted about a day. Give paying customers their own database and put the free ones together in a shared one with a customer column, which is the arrangement most of this industry uses for everybody. It fails on three counts. The upgrade stops being a change to a row of allowances and becomes a migration, which means the moment a customer decides to pay us is the moment we are most likely to break something for them. The separation we describe elsewhere would need an asterisk, and a guarantee with an asterisk is a guarantee that will be explained badly by somebody in a hurry. And the free plan is where the audit trail matters most, because the organisations with the least money are the ones least able to survive a dispute they cannot answer. A free plan built on the weaker foundation is a free plan that gives the worse product to the people with the fewest alternatives.

A thirty day trial with a card at the front. This is the default shape in business software and it was in the first drawing because it is in everybody's first drawing. Two things killed it. The measurement says there is nothing at the end of thirty days that we need to stop paying for, so the countdown would exist purely to apply pressure, which makes it a sales mechanism wearing the costume of a technical limit. And a card taken at the start is a claim on somebody's attention for the whole of the trial: the question they are answering stops being whether the product suits them and becomes whether they remember to cancel in time. What survived is the half with neither of those properties: a new organisation may start on Professional for thirty days, without a card, and at the end of it drops to Free rather than to an invoice. Nobody has to remember anything.

A much smaller relationship limit. An early draft had a free plan of a hundred relationships, which sounds prudent and turns out to be indefensible once you know the gradient. At three hundred and thirty six bytes each, the difference between a hundred records and a thousand is a rounding error against a floor of forty kibibytes, and vanishes entirely against the cost of the request that reads them. So a limit of a hundred could not be justified by cost. It could only be justified as pressure, applied early, to somebody who has not yet finished importing. A thousand is where the limit stops being about our bill and starts being about the honest question of when an organisation has outgrown a three person tool.

Free for one person only. Tempting, and wrong for a reason that has nothing to do with money. A CRM used by exactly one person is a spreadsheet with worse ergonomics. The thing this product is for begins at the second person, when two people need to see the same record and know who touched it last, and the audit trail stops being paperwork and starts being the answer to a question somebody actually asked. A free plan of one would have driven small organisations to share a single account between three people, which is the arrangement that destroys attribution entirely. We would have created the problem the product exists to solve.

Obligations on the paid plans. Charging for the audit trail, consent records, subject access tooling or two factor sign in is the single most common way to make a free plan cheap to run, and it is dealt with at length on the pricing page, so one sentence here is enough: those are legal obligations that do not consult your plan, and the measurement removes the last excuse by showing that carrying them on the free plan costs a fraction of a penny a month.

Limits that tighten later. There was a version where the free limits were described as current, to be reviewed annually as costs changed. It reads as prudence and it is a price rise with a maintenance schedule. A limit that can move down is worth nothing to somebody planning around it, and the whole value of a published allowance is that it can be planned around. Since the measurement shows the cost is not going anywhere interesting, the review would have existed only to hold open a door nobody should walk through.

A short dormancy clock. The first dormancy proposal was far shorter than the six, seven, eight and twelve months we settled on. Once storage turns out to be the cheapest thing on the platform, a short clock cannot be justified by cost either, and it starts doing real damage to ordinary users: a club that runs a season and goes quiet for four months, a consultancy between contracts, a charity whose administrator is on leave. The clock we run is long enough that going quiet for a while is not a failure state, and the notices at six, seven and eight months exist so that nobody is told about the twelfth month for the first time in the twelfth month.

Every one of those rejected shapes saves money. What the measurement did was reveal that they save money we had just proved we did not need to save, which turns each of them into a cost imposed on a customer for no reason we could write down. That is the quiet function of a number: not to tell you what to build, but to remove the excuse from the things you were about to do anyway.

The general point

We built a whole mechanism to solve a problem that turned out to be a small fraction of the size we thought. The measurement took an afternoon. The mechanism took considerably longer, and had we shipped it, it would have carried the platform's single most dangerous operation forever, in exchange for a rounding error.

The rule we took from it is narrow enough to be usable: measure the problem before building the mitigation, and measure it in the unit you are billed in rather than the unit you think in. An afternoon of arithmetic is cheaper than any mechanism, and it is very much cheaper than a mechanism that can destroy data.

The uncomfortable half is that the original estimate was not careless. It was a reasonable guess by people who knew the system, and it was wrong by an order of magnitude. Reasonable guesses by informed people are exactly the ones that do not get checked, because nothing about them feels like it needs checking. That is the property that makes them expensive.

Questions this post keeps prompting

Which of these conclusions survive being wrong about the numbers

The rule this blog is held to is that a corrected post says what the correction did to its argument rather than quietly carrying on.

The decision survives, and that is the point

Refusing a heartbeat for free organisations is still worth about what it was worth when we worked it out: sixty per cent of the included write allowance for a daily task, eight per cent for a weekly one, and nought for waking only when something has earned a wake, which is what was actually built and what the correction above describes. That comparison is a ratio between schedules and does not care what a contact costs, which is exactly the property we claimed for it when we wrote that a decision robust to the number being wrong by an order of magnitude is the kind worth making from a measurement.

It has now been tested by the number being wrong by an order of magnitude, in the opposite direction, and it held.

Withdrawing hibernation survives, for a better reason than before

Storage per free organisation rose by a factor of about five, so the saving hibernation would have produced is larger than it looked. It is still a rounding error against maintaining a mechanism whose failure mode is a customer losing everything, and the trigger we published, ten megabytes of measured storage per free organisation, is a long way off at 1.88 MiB.

The working figure survives by luck rather than judgement

We wrote that a realistic organisation would use two to four megabytes instead of the 0.37 MiB measured, allowing for notes with actual text and a year of activity. The measured figure is now 1.88 MiB with none of that allowance spent, which lands inside the range we guessed.

That is a coincidence and we would rather say so than take credit for it. The allowance was meant to cover real usage and has instead been consumed by schema growth, so the headroom we thought we had is gone and two to four megabytes is now a floor instead of a ceiling.

What does not survive is the size of the original mistake

The post was built around an estimate being wrong by a factor of about fifty. Against the corrected measurement it was wrong by roughly an order of magnitude, which is still a large error and a smaller story. Every instance of the fifty has been changed.

The general lesson is unaffected and slightly reinforced. Measure the problem before building the mitigation, in the unit you are billed in. Then measure it again, because we did not, and a reader checking our arithmetic against a current build would have found four wrong numbers before we did.