Spacetime объясняет принципы масштабирования и отказ от ложной горизонтальной масштабируемости

This is probably the question I get asked most about Spacetime. It’s a simple enough question, and it certainly seems like it should have a simple answer. Scaling is a complex topic, and the devil is in the details, as it so often is. On the other hand, it’s also not so complex that we can’t understand scalability from first principles in a blog post. Let’s start by exploring scalability in general, and then let’s answer the question, “How does Spacetime scale?” If you want the TL;DR: There are three dimensions of scale: compute, storage, and networking. Horizontally scaling storage is relatively straightforward and shipping October 31st, 2026. However, not all networking and computation can be scaled horizontally. OLTP databases that claim general horizontal scalability often pay enormous overhead per transaction and perform extremely poorly when faced with contending transactions. Spacetime provides high performance under contention and provides tools to make it easy for you to scale your parallelizable OLTP workloads. Note NOTE: I talk about CockroachDB a lot in this article. CockroachDB is a rough stand-in for essentially all general purpose, horizontally scaling RDBMSs including Spanner and Aurora DSQL. Although I talk about some issues with these technologies, all of them are incredibly impressive feats of engineering. Scale Intuitively, everyone has an idea of what it means to “scale”. It means to be able to do more. It means to keep up with demand. It means to handle a billion requests, or “infinite” requests, or an infinite amount of data, or an infinite number of customers, or the ability to grow your app at 10x or 100x year over year without needing to rewrite your software. In particular, I think that when most people say a system is scalable, they’re talking about whether or not it’s specifically horizontally scalable. Whereas vertical scalability means to do more with a single computer, horizontal scalability is the ability to do more with more computers. If doubling the number of computers lets us do roughly twice as much work, the computation scales horizontally. After all, a single computer can only be so big and fast, but in theory there’s no limit to the number of computers you can buy. There’s something very satisfying about that idea, so that’s the property that everyone looks for. Note NOTE: The “doing more with more computers” definition implies that all horizontally scalable systems must be distributed systems. However, it does NOT imply that the only purpose of distributed systems is horizontal scalability. For example, distributed state machine replication is designed to redundantly do the same computation on many computers for the purpose of reliability, not scale. More on this below in the Spacetime section. The question “Does it scale horizontally?” is underspecified. A better question is, “In what ways does it scale horizontally?” This is because there are actually three pretty independent dimensions of scalability: - Compute: how many transactions you can process - Storage: how much data you can store - Networking: how many connections and how much bandwidth you can support Note Incidentally, they are the exact 3 “foundational ingredients” we referenced in the 1.0 announcement keynote. To see what I’m talking about, let’s look at a few database systems that all expose broadly PostgreSQL-compatible interfaces, but have radically different architectures. For example: - Postgres is for the most part a single-node database. Postgres does not scale compute, storage, or networking horizontally for you. You can of course scale Postgres horizontally by deploying many Postgres instances, but as far as the Postgres code is concerned, it’s largely unaware of those other instances. The one exception to this is read replicas which allow you to manually direct readers to a replica. This helps to scale both networking and compute, but it comes with caveats about read-after-write consistency and performance. Postgres itself has no notion of a cluster of primaries, cannot run transactions or queries across them, or route you to the appropriate one. You can of course write software to do this, and this is how people can and do scale Postgres horizontally, but how to do it is left as an exercise to the reader. - Neon, which is a modified variant of Postgres, scales storage horizontally. Neon tables are backed by object storage with a local page caching structure to make data access fast and efficient. Page access latency can be higher for cache misses, but this architecture gives your Neon databases effectively infinite storage. Neon does not automatically scale compute and networking horizontally, however. Like Postgres, all write transactions must go through a single primary. However, being a Postgres variant, Neon also supports read replicas to horizontally scale compute and networking for read workloads, although this comes with similar consistency caveats. Horizontally scaling storage is a big win, even without automatic compute scaling. Even small web apps with not so many users can in principle use a lot of storage. - CockroachDB (and Spanner), in principle, scales storage horizontally by spreading each table’s data around the cluster so that a portion of each table, called a “range”, is stored on each machine and typically replicated to two other machines. Provided that writers don’t all try to modify the same range, it can also scale networking horizontally. It has a symmetric architecture, meaning any node in the cluster can serve any SQL request, both reads and writes. Finally, for computation that parallelizes cleanly (e.g. analytics or writes to unrelated keys), it also scales horizontally. The node you connect to will compute the query plan and return the results, but both read and write operations will execute as part of a distributed transaction across the cluster based on the ranges. Can we find the Holy Grail? So if CockroachDB is able to scale in all three dimensions, it must be better than Postgres and Neon in all respects, right? Maybe it does something special with the CAP theorem[1] or atomic clocks? Unfortunately, there is no magic here. While CockroachDB is a modern marvel and does scale certain computations horizontally, not all computation can be scaled horizontally. Horizontal scalability is really a question of parallel computing: can we split the computation into pieces that different computers can do at the same time? The answer is often no, and the issue is with CockroachDB, you pay the enormous coordination cost of horizontal scalability even when data contention forces you to do updates one at a time. What CockroachDB gains in horizontal scalability, it loses in vertical scalability and then some. The ugly truth is that rather than doing more with more computers, horizontal scalability can often mean doing less with more computers: conceptually what one computer can do in 1 millisecond, 10 computers can do in 100 milliseconds. There are two main issues with CockroachDB’s approach to horizontal scalability: - The data involved in each transaction is rarely co-located on a single machine. - No transaction has exclusive access to a range, so every transaction pays distributed concurrency-control overhead, and conflicting transactions must wait, abort, or retry. The first problem is caused by spreading ownership of the data evenly around the cluster. If your data is spread around the cluster, you need to do network requests for essentially every transaction. Although sharding often sounds inconvenient, it can provide much better performance if most of your transactions operate within a single shard. Also note that Neon’s design does not suffer from this same issue for many workloads because it caches hot pages on the same machine. Note Spanner partially addresses this issue with “table-interleaving” which allows you to essentially tell Spanner to colocate related tables. This can dramatically improve performance in simple cases. CockroachDB supported table interleaving for several years but removed it in v21.2, judging the benefits too small to justify the complexity. The second problem is a general problem of parallelizing arbitrary computation: coordination under contention. Even vanilla Postgres runs into the same problem, just on a smaller timescale. Rather than coordinating transactions across a distributed system, it has to coordinate transactions across multiple cores. Postgres can run transactions on many CPU cores, but as soon as those transactions touch the same data, they need to spend time coordinating. The CPU itself has to coordinate raw memory access across L1, L2, and L3 cache. Another writer modifying the same cache line can invalidate your local copy and force the cores to synchronize. Postgres has to coordinate the transactions themselves: who owns a lock, which versions of rows are visible, what order transactions commit in, and whether conflicting work needs to wait, abort, or retry. For example, imagine a small OLTP transaction that involves a user row and a related metadata row. Firstly, whereas in single-node Postgres both rows will be on the same machine, in a CockroachDB cluster with N nodes and random primary keys there’s an approximately 1/N probability of them being on the same leaseholder/leader. What could have been a ~3 microsecond critical section on a single core is now a 1 millisecond distributed critical section with multiple network requests. Even in the 1/N case where the rows are colocated, the transaction still can't release its locks until the write has been replicated to a quorum, so it holds them for roughly the same round trip. Secondly, and even more importantly, a different transaction that wants to read or modify the same rows must now wait 1 millisecond for our transaction to commit or abort. These two problems compound multiplicatively. A hot key with a ~3 µs critical section admits ~300,000 contending transactions per second; at 1 ms it admits ~1,000 tps. Counterintuitively, the single-threaded solution is 300x more “scalable” in this scenario! Note Note that with 1 ms distributed commits, even just 1% of transactions contending on a single row makes every horizontally scalable cluster slower than a single core. This is just Amdahl's law applied to horizontal scalability. Contending transactions must be executed serially and total throughput can never exceed the serial rate divided by the fraction of serial transactions: (1 / 1 ms) / 1% = 100,000 TPS. No matter how many cores you add! There's essentially only one scenario where a cluster beats a single core on throughput: you have no contention AND you're paying more than $3,600 per month (based on the model's overhead assumptions). That is a scenario some companies find themselves in, but it's a niche scenario. Under contention, multiple writers make transactions substantially slower than just running them one at a time, because the cores spend more time agreeing on who gets to modify shared state than doing useful work. But, even for computations that can be executed in parallel, people often dramatically underestimate how much additional hardware can be needed just to overcome coordination overhead. As a rough illustration, L1 cache latency can be around 20x lower than L3 latency. If parallelizing a workload turns cheap, local cache accesses into frequent synchronization and cross-core communication, it’s entirely plausible that you could need 10+ cores just to match the performance of one carefully cache-optimized core. And that doesn’t even account for the much higher cost of going out to main memory when larger working sets and metadata, such as MVCC bookkeeping, push useful data out of cache. And we still haven’t considered network requests and serialization required for something like distributed MVCC bookkeeping and all the cache misses those will cause. Are you sure you want to pay for 10 cores when 1 core will do? Maybe for a subset of problems we end up achieving scalability, but at what cost? For inherently serial computation, you can move it to a faster machine, optimize the code, or replicate the results for fault tolerance, but you can’t make ten machines do it faster, no matter how much you’d like to. Every programmer who has read The Mythical Man-Month knows this intuitively. Adding programmers to a project doesn’t mean writing your software faster; it means writing it slower but with more meetings. 10 authors can’t write a novel faster than 1, especially if they’re coordinating by snail mail. The moral of the story is horizontal scalability isn't simply a property that a system has or doesn't have. It's primarily a property of the workload. Spacetime So let’s talk about Spacetime. First, it’s worth separating the API you use from the architecture that implements it. Nothing about the Spacetime programming model inherently requires a single-node or distributed implementation. There are many ways to implement: ctx.db.myTable.insert({ name: 'Tyler' }); or: SELECT * FROM my_table The API says nothing about where that data lives, which machine executes the transaction, or how many machines are involved behind the scenes. Convex, for example, built a transaction layer on top of existing storage engines like MySQL or Postgres. In principle, we could have implemented Spacetime with a fleet of Spacetime module servers and a giant CockroachDB cluster as the storage engine. It might surprise you to learn that this is pretty close to how we started. The first prototype of Spacetime was built on Postgres + Kafka! However, in situations with less than full parallelism or fewer than dozens or even hundreds of machines, this results in worse performance despite a much higher cost. This blog post explains in detail why that is. Spacetime began as the backend for our real-time MMORPG, so we simply had extreme transaction latency and throughput requirements from the very beginning. We could not afford to make the common case prohibitively expensive simply so that the uncommon case could span arbitrary numbers of machines. These requirements forced us to build our own custom storage and execution engine from scratch. Perhaps surprisingly, today the execution model for Spacetime databases is single-threaded by design. Intuitively, especially to engineers with limited experience optimizing cache performance, this could sound like “bad news”. Initially, we started with that assumption too: early versions of our custom database engine used parallel execution enabled by MVCC transactions. The issue is that when we actually measured, we found that single-threaded execution flat out outperformed parallel execution. In a sense, it probably cost us more than $1m just figuring out that we should just use a big ol’ lock. It’s not impossible that we could implement parallel execution within a single database in the future which achieves better performance in limited situations, but not without extremely careful engineering and performance measurement to ensure that it doesn’t introduce more complexity and overhead than it’s worth. So does this mean that Spacetime cannot scale horizontally? Not remotely. Scaling Spacetime Horizontally The foundation for Spacetime’s horizontal scaling solution is inspired by the actor model. The actor model is a general model of parallel computing, “motivated by the prospect of highly parallel computing machines consisting of dozens, hundreds, or even thousands of independent monoprocessors, each with its own local memory and communications processor, communicating via a high performance communications network.” Today in Spacetime, each database is a single-threaded actor and we have a six-step strategy to make horizontal scalability an increasingly friendlier dev experience. The next stage already has a date: asynchronous inter-database communication and tiered storage ship on October 31st, 2026 as part of our scalability launch: Spacetime Continuum. Before going into the details, there are two versions of Spacetime worth distinguishing: - SpacetimeDB Standalone, the single-node version available on GitHub. - SpacetimeDB Cloud, the proprietary, clusterized version. Replicated databases with world-class performance We’ve already talked at length about how each Spacetime database achieves world-class, single-threaded performance but it’s important to note that SpacetimeDB Cloud also supports distributed state machine replication for databases. This means that even though each database is single-threaded, it isn't necessarily running on only one machine. This sounds counterintuitive, but the general idea is that we can replicate what a single thread is doing onto multiple machines so that if machines fail, the database remains available. This means that each database in SpacetimeDB Cloud operates as a distributed system. It’s important to note that with our pipelined implementation, we've demonstrated in our benchmarking that replicated databases achieve the same throughput as unreplicated databases (roughly 300k TPS for the benchmark transactions), provided there is sufficient network bandwidth between nodes and memory for pipeline depth. Note that confirmedReads(true), the default setting for Spacetime, configures clients to read only after transactions reach durable status on the cluster. World-class sharding experience Sharding your database gives you the best performance for transactions that operate solely within a single shard. Distributed transactions give you the ability to run transactions which span shards. Why can’t we have the best of both worlds? What if most transactions operated within a single machine and we gave the programmer tools to organize their data so that most transactions occur within that machine? Only in rare instances should we need to do a distributed transaction. The key is to make the shard boundary explicit and ergonomic. In the actor model, each shard can behave like its own independently scheduled actor: transactions within a shard remain fast, local, and single-threaded, while only transactions that genuinely need to touch multiple shards pay the cost of coordination. This lets the architecture preserve the performance characteristics that make Spacetime fast today without pretending that distributed coordination is free. Each database is an independent actor with its own state and transaction log, so transactions against different databases can execute concurrently without coordination. Spacetime already provides tools for managing this architecture. Databases can be created as children of other databases. Procedures can publish databases and call functions on them. A root database can keep track of the identities and state of the databases beneath it. Several of our customers operate hundreds or thousands of databases in this way and BitCraft also scales this way. A single root database maintains global data, while region databases handle different parts of the world and different groups of players. Transactions within one region remain fast and local, while the regions themselves execute in parallel. We plan to make this model considerably easier through first-class inter-database communication (IDC). Asynchronous IDC will allow one database to call functions on another database in a type-safe way. At a high level, the TypeScript APIs will look something like this: // Async IDC: send a type-safe message to another database. // This will result in exactly one reducer call on the target DB. ctx.db.receivePlayer.insert({ msgId: 0n, target: regionDb, player, }); Scale each database’s storage horizontally As of today, Spacetime stores all table data in memory on the leader node. This means that in a single database, the amount of data you can put in your tables is constrained by the physical memory of the machine. However, this restriction is not fundamental to Spacetime or the key to its incredible performance. We view memory, disk, and object storage as a natural extension to the CPU cache model. Spiritually we treat memory as L4 cache, disk as L5 cache, and object storage as L6 cache. This caching model is a time-honored way to get maximum performance from a single writer. This is often referred to as “tiered storage”. Cache lines and prefetching allow you to preemptively pull in batches of data from cheaper, slower storage into more expensive, faster storage. Providing tiered storage to databases allows you to amortize the cost of cache misses, and achieve performance which is overall close to the faster, more expensive storage, while still retaining the scalability of bigger, cheaper storage. Won't paging stall the single thread? If every transaction executes on one thread, it might seem like a single cache miss to object storage would stall the whole database for 50 milliseconds. That would be like operating a library and any time a customer puts a book on order, you make the whole line of customers stand there and wait for two weeks for the book to come in before processing the next customer. Obviously silly. The rule that makes tiered storage compatible with single-threaded execution is simple: don't wait for a cache miss while holding the lock. Order books asynchronously and process the next customer while you're waiting! Fortunately, this is a well studied problem. H-Store called the technique anti-caching: execute the transaction normally, and the moment it touches data that isn't resident, abort it, fetch the data asynchronously, and run other transactions while you wait. When the data arrives, run the transaction again. Aborting is nearly free because nothing has committed, and each re-execution costs microseconds of CPU against the milliseconds of I/O it avoids serializing. Calvin used a similar trick it called reconnaissance queries: dry-run the transaction against a recent snapshot to discover what it reads, prefetch that, then execute for real. TigerBeetle runs an explicit prefetch phase before synchronously applying each batch of transactions. Spacetime is unusually well suited to these techniques because reducers are deterministic. A reducer cannot perform I/O, read clocks, or generate randomness, and every data access goes through the reducer context, so the engine observes every read a transaction makes. That means aborts have no visible effects, a re-execution against the same state touches exactly the same rows, and a dry run discovers the data the real run will need. This is a strategy that the extremely capable team at TigerBeetle refer to as “diagonal scaling”. Disk and object storage tables will be releasing on October 31st, 2026. We view this as a critical improvement to the scalability of Spacetime. With disk tables, we can greatly increase the storage limits and decrease the prices of data storage. With object storage tables, we can effectively remove storage limits and allow you to store a theoretically unlimited amount of data in a single database. Scale each database’s networking horizontally As alluded to previously, much like compute, scaling networking horizontally is not always possible because in some cases writers want to mutate the same state, and therefore need to connect to and send data to the same computer. This is one of the reasons CockroachDB recommends using random keys to avoid hotspots on the cluster. However, even in such instances we should strive to support maximum possible reader and writer throughput. Like CockroachDB, SpacetimeDB Cloud also has a symmetric architecture, meaning that you can connect to any node and SpacetimeDB Cloud will ensure your write request gets processed or proxied to the right place. From the outside it sort of looks like one giant computer. For writers, this allows us to “fan-in” the connections. Clients connect to any node, and that node will multiplex those connections over a single connection to the node that hosts the database. In principle, we can also move the processing of SQL subscriptions and read queries off the leader by introducing consistent read replicas. Consistent read replicas allow us to “fan-out” from the leader. Instead of connecting directly to the leader, subscription evaluation can be routed to a consistent read replica. In so doing, we can scale reader connections and bandwidth horizontally. “Consistent" here means the replica applies the leader's transaction log in the same total order, so a subscription sees exactly the same sequence of updates, just delayed. For point-in-time SQL queries, the leader will assign each query a position in the transaction log without executing it, and a replica won't answer the query until it has applied through that offset. Reads are therefore linearizable with respect to every write, and the leader's only cost is handing out sequence numbers. Inter-database Transactions Synchronous IDC provides the built-in two-phase commit machinery required to do true distributed transactions of the kind that CockroachDB provides. This will allow a transaction to span multiple databases within a cluster when true atomicity is required. The TypeScript API for this looks just like calling a regular reducer, except on a foreign database. // Sync IDC: call another database inside the current transaction. ctx.at(regionDb).reducers.receivePlayer(player); There is an obvious tension here. If each database is a single thread behind one big lock, then a synchronous call from database A into database B holds B's lock until A's transaction commits or aborts, which takes a network round trip. Our plan is to keep execution single-threaded but reintroduce MVCC for these distributed transactions to allow concurrent transactions to proceed while the round-trip is in-flight. We've designed a pipelined two-phase commit protocol which commits transactions in memory first and never holds the database lock during disk I/O, and we've model-checked its safety properties in TLA+. The details are worth a separate post. Together, these features will turn multiple databases into a cohesive distributed application without putting distributed coordination in the path of every transaction. Most work will remain local and asynchronous. Only operations that genuinely require atomicity across databases will pay the cost of a distributed transaction. Intra-database partitioning Multiple databases are a natural scaling boundary, but they also require you to manage multiple modules, update their schemas independently, and figure out how to rebalance load as it changes. The next step to address these UX issues is to introduce partitions within a single logical database. Partitions will allow one database module to contain multiple independently executing shards. This preserves a single deployment artifact and allows schema changes to be applied atomically across the entire database, while transactions against different partitions execute concurrently. A partition in Spacetime is a different thing from a range in CockroachDB. A range is a placement decision: it says which machine stores some slice of a table, but any transaction can still touch any range, and every write is replicated and coordinated the same way no matter where it lands. A Spacetime partition is an execution unit. It owns its data, it has its own serial transaction log, and the reducer code that operates on that data runs inside it. Developers choose partition boundaries to match the structure of their workload so that the common transaction never leaves its partition, and Spacetime can move whole partitions between machines to rebalance load without weakening that guarantee. A transaction that never leaves its partition pays nothing for the existence of the others. This is the zero-overhead principle: you should not have to pay for horizontal scalability until you use it.[2] So, does it scale? Yes, it does. Today, you can scale parallel workloads by sharding them across databases while keeping each shard fast and local. As we’ve seen, this architecture is as good as it gets if your workload has almost any contention, or if you'd rather not pay for a large cluster to get the performance of one core. Over time, tiered storage, inter-database communication, distributed transactions, and partitions will make that architecture increasingly transparent. The goal is not to make coordination free. It is to ensure you only pay for it when your workload actually requires it. Tyler Cloutier Cofounder, Clockwork Labs [1] A brief aside about the CAP theorem for the distributed systems nerds. The CAP theorem, also known as Brewer’s Theorem, simply states that no system can in general be all three of: consistent, available, and partition tolerant. A system can be 0, 1, or 2 of those things, but never 3. “Consistent” in this case means that every read receives the most recent write or an error. In practice, this property is achieved by correctly implementing distributed state machine replication (aka some variant of Paxos/Viewstamped Replication). “Available” means every request to a non-failing node receives a non-error response. “Partition tolerant” means that the system exists in a world where messages between nodes can be delayed or dropped entirely. If I hear another person vaguely gesture at the CAP theorem as “proof” that a database can or cannot be horizontally scalable, I may just have a mental breakdown. There is for some reason a common misconception that the CAP theorem, also known as Brewer’s theorem, somehow limits the scalability of OLTP database systems. For example, I have been asked by investors and engineers maybe half a dozen times how we “got around the CAP theorem” in order to get the Spacetime benchmark numbers. In reality, the CAP theorem makes no comment on it. My theory is that this misconception arose from this publication and this paper by Google’s Eric Brewer, the original formulator of the conjecture (now theorem: it was later proven in this paper by Seth Gilbert and Nancy Lynch), specifically because Spanner is known to be the famous horizontally scalable SQL database. Perhaps it got into the air that because this paper references both the CAP theorem and Spanner, CAP theorem must somehow be important for scalability. However, you will note that the original Spanner paper makes no mention of the CAP theorem at all. At any rate, the CAP theorem is only relevant to databases insofar as the best you can do is make a choice between designing a CP system or an AP system. In a CP system you’re always consistent, but sometimes unavailable due to network failure, and in an AP system you’re always available, but sometimes inconsistent due to network failure. So because we can’t make networks infallible, in a sense the CAP theorem is just asking the question: do you want to be correct or available? The CAP theorem says nothing at all about throughput, latency, or scalability. Regardless, in practice for OLTP databases, there isn’t even a real choice to make. OLTP databases with strong consistency guarantees, including Spanner, must be CP systems, because inconsistency means that your users see the wrong results (stale reads, non-monotonic reads, conflicting writes, which could in rare instances sink your whole e-commerce business). Through this lens, you can now clearly see that the CAP theorem is not interesting or relevant to scalability. The actually interesting question is: how can we build a horizontally scalable, consistent (i.e. correct) OLTP database system? The Consistency in CAP is closely related to Isolation in ACID (Confusingly, Consistency in ACID describes something else entirely). CAP's consistency is linearizability: operations appear to happen in a single order that respects real time. ACID's isolation, at its strongest, is serializability: transactions appear to happen in some serial order. A database that gives you both is “strictly serializable”, which is what Spanner offers. Linearizability requires that every transaction be assigned a position in one global order that respects real time. For transactions that touch the same data, that order is inherently serial, which is the contention problem discussed above. For transactions that don't, the order still has to be agreed on, and agreement would ordinarily mean coordination by way of message passing. Now we can also understand why Spanner uses atomic clocks. The atomic clocks allow Spanner to assign a linearizable order to transactions that have no mutual data dependencies (i.e. they are trivially parallelizable) and might be executing on different sides of the earth. The clocks allow Spanner to say which transaction happened first without needing to send messages around the world to decide the relative ordering of transactions which would NOT otherwise need to coordinate. [2] What about queries and transactions that span partitions? We certainly could eventually support them. Distributed SQL queries across partitions would recover the general horizontal scalability offered by systems such as CockroachDB while preserving Spacetime's fast local path for transactions that remain within one partition. However, we likely wouldn't allow reducers that could read and write across partitions because that's the same performance footgun we've discussed at length. Instead, partitions would still colocate related data and your module code would still explicitly respect those boundaries, but clients would be able to do SQL queries and subscriptions globally across all partitions. This could be operationally convenient or useful for specialized workloads, but it's unclear as of today whether this final stage is worth the complexity, or whether most of these queries should actually just target a read-only analytics database cluster instead. Either way, for performance's sake, cross-partition queries probably shouldn't be the mainstay of your application. For this one, I think it's best to take a listen-to-our-customers approach and see what they actually need.