PhaseLock

One sync to rule them all

PhaseLock is a sync engine built around a simple idea: save your events as a log, which is easy to replicate, and write a reducer that builds state from those events, which can be run anywhere. The result is one sync engine for your web app, your mobile app, and even your backend.

The single best thing that Google Docs has done for the world is show people what live software is supposed to feel like.

You probably wish all of your applications were live and collaborative.

Ever sit in a meeting and hear, "Can you see the change to the Jira ticket yet? Try refreshing again..."

Or maybe you've heard, "Why did you land your GitHub PR without addressing my review?" and you had to say, "Sorry, I forgot to refresh so I didn't even see it."

Your user wants your app to be a live app too. But building live apps is a Hard Problem. Today's post discusses what makes sync hard and dives into how PhaseLock's sync protocol solves the hard parts for you.

Basic sync: all you need is append

Maybe you, like me, set out to build your first sync system as soon as you learned what Dropbox was. Maybe it sounded easy. You've got a file list from the server, and a file list from the client. The server has one extra file, so you send the file to the client. Bam! Sync solved.

Then you delete a file from the client, resync, and the file reappears. Oops, that sync protocol only works in one direction.

It turns out that directly comparing state on both sides is a pretty bad way to do sync. It can work in both directions if deletions are not allowed, or it can work in one direction with additions and deletions. But even then, it requires sending a lot of state just to calculate a small diff.

Among server-client systems, a better approach to sync is to split the problem in half. The client updates are usually few and are resent until the server acknowledges them. The server updates are the bulk of the sync work. The server keeps its updates in an ordered event log, so a client knows what to sync by the log position of the last event it received. The append-only nature of the log means that the client never has to download a full state in order to infer deletions. Instead, a deletion is recorded as an event and appended to the log.

Virtually every sync engine has this log-of-updates abstraction at some level. In PhaseLock, it's the primary storage mechanism.

Real-life sync: append is never enough

Maybe you, like me, built your first live app as soon as you learned about sync built around an event log. Maybe it sounded easy. The server appends events, then the client pulls them and updates its state. Sync solved, right?

Then that pesky product manager asks for something called "permissions" and insists that "users" can either be added to or removed from "projects".

So now your sync protocol meets real life, and you're up against some difficult challenges:

Slices to the rescue

Last week's post about slices provides PhaseLock's answer to these challenges: a slice is a boundary around a stream of events and a key prefix of state, granted or revoked as a unit. Each slice's reducer isn't allowed to interact with other slices.

For grants, that means that event ordering only matters within a slice, and a backfill containing old events derives a consistent result.

For revocations, that means deleting an entire slice's state is just dropping a key prefix from the client's store.

PhaseLock sync protocol

Now we're ready to sketch out the PhaseLock sync protocol.

In brief, the two important messages are ["events", ...], which carries events, and ["view", refs, hash], which carries the latest maySee result. refs is a list of slice names or slice prefixes the client can see, and hash is just a hash of that list.

A typical exchange might look like this:

Or, in a bit more detail:

So two messages solve sync, and the rest is plumbing: background downloads, keepalives, a termination message, and a "live" marker.

Conclusion

Backed by this protocol, PhaseLock's sync engine gives your app these features on day one:

Consistency: Every view of data your UI receives is a consistent view of all of its slices at the same moment in time. This is because events that land in a single transaction are delivered as one ["events", ...] message, and all slice updates come through a single ordered stream.

Autosubscribe: Your app doesn't have to do the dance of discovering a new grant, bootstrapping that download, and merging its updates with an existing stream of updates, because the ["view", ...] messages and background backfill handle that automatically.

Deletions and revocations: When data is deleted from the server, it propagates to the client as a deletion event, and the client deletes it locally. When data is revoked, the client sees a new ["view", ...] message and deletes it locally.

Client bootstrap and stale client reconnect: Fresh clients automatically backfill to bootstrap their state without a full log replay. Stale clients may hold slices so old that the events they need to catch up from the log have been pruned; they also use backfills to recover.

Your user wants these features in your app. But you don't need to build them yourself.