PhaseLock

Introducing PhaseLockDB, part III

Slices solve event sourcing

PhaseLock is a sync engine built around a simple idea: save your events as a log, which is easy to replicate, and write a reducer that builds state from those events, which can be run anywhere. The result is one sync engine for your web app, your mobile app, and even your backend.

In case you haven't heard of event sourcing, it's simply the idea that you can store every event that happens in your system in a big log, and that becomes your source of truth. Your state is derived by looking at all past events.

And if you've read the last two blog posts, you probably already recognize that PhaseLock is a sync engine built on event sourcing.

You're probably asking, "Is that really a good idea though? Think of all the problems for your sync engine:"

Great questions, glad you asked. Slices provide an answer to each of these problems.

What are slices?

We introduced slices briefly last week, in the context of a discussion on permissions. Let's examine slices in more detail.

A "slice" is simply one event stream, fed to one reducer, to populate one key-prefix of state:

Why have slices at all?

Think of it this way: events, reducers, and state are the backbone of PhaseLock's sync system. Events make for efficient transmission, because they represent an ordered log of changes. State is what you actually need to populate your UI or make decisions. And reducers are how each client can receive events and calculate state deterministically.

But that determinism would break down if you only had one stream of events and one reducer to populate the entire state. Usually each client is only allowed to see a subset of total state, and each client's view is usually different. So each client would feed a different subset of events into its reducer, deriving a different state. That would be a very crappy sync engine.

PhaseLock solves this problem with slices. A slice serves as a boundary for visibility and determinism: a client can either see the entire slice or none of it. Every client that sees the slice feeds the same events through the same reducer and produces the same key-prefix of state.

The developer simply has to define which slices each client can see (the maySee function described last week), and the PhaseLock sync engine can handle the rest of the work automatically.

It turns out that individually deterministic slices buy you a lot.

How slices solve load times

Bootstrapping, in sync engine terms, is the process of populating a fresh client from nothing.

Without slices, even if you know which events a client is allowed to see, there's not a clear way to know which subset of key-value state it is allowed to see. So even though the server may hold the full state, you can't really calculate the keys that a client would have without a full replay.

With slices, each slice of state can be snapshotted independently (it's just a key-prefix of the total state). A client can bootstrap itself from one snapshot for each of its visible slices, skipping the full replay.

As a bonus, a client can tolerate stale snapshots by replaying the events that landed after the snapshot was taken. In fact, that can happen per-slice, so a client can bootstrap from many slice snapshots each taken at a different time; there's no practical need for coordinated snapshots across slices.

How slices solve retention

As a direct result of per-slice snapshots, deterministic replay no longer needs to start at the first event; it can start from a snapshot. This enables event retention policies with no cost to correctness.

Lots of event-sourced databases have retention policies, but it is up to the developer to ensure that discarding old events doesn't break the system. But with slices and per-slice snapshots, PhaseLockDB can ensure that replayability is possible across the entire retained log.

How slices solve deletions

GDPR deletion requirements are great for humans but a real problem for append-only event logs.

Without slices, deleting events breaks replay determinism without very careful reducer design.

With slices, a whole slice can be deleted and no other data in the system is affected. You still need to make sure your per-user PII lives in small per-user slices.

By combining a slice deletion with a short retention policy, you can delete the sensitive data promptly from both the key-value state and the underlying log. As a bonus, the PhaseLock sync engine running in each client will sync the deletion locally, so PII deletions propagate to user devices as well.

How slices solve evolving your schema

Anyone who builds an event-sourced system finds out that keeping all your events from the beginning of time has a fundamental cost: you also have to keep all your broken events from the beginning of time.

Slices solve this problem in a couple ways. The first is that slicing enables retention policies, so sufficiently old events can be simply forgotten without loss of correctness.

The second way is that the slice abstraction lets you build parallel slices when needed. You can create a v2 slice for your v2 app, writing new events to both v1 and v2 streams, until all your deployed v1 clients are upgraded.

Conclusion

PhaseLockDB is a sync-focused database built on event-sourcing ideas. In one database, it holds both the event log and the current state. Slices are the abstraction that makes that architecture work without unbounded log size and loading times, GDPR violations, or migration headaches.

Next week, we'll dig deeper into the client side: how the PhaseLock sync protocol works and what it promises for your frontend.