PhaseLock

One sync to rule them all

PhaseLock is an open-source backend for live apps that still work offline. You declare your types and write your business logic once, then PhaseLock's sync engine powers your web app, your mobile app, and even your backend workers.

The single best thing that Google Docs has done for the world is show people what live software is supposed to feel like.

You probably wish all of your applications were live and collaborative.

Ever sit in a meeting and hear, "Can you see the change to the Jira ticket yet? Try refreshing again..."

Or maybe you've heard, "Why did you land your GitHub PR without addressing my review?" and you had to say, "Sorry, I forgot to refresh so I didn't even see it."

Your user wants your app to be a live app too. But building live apps is a Hard Problem. Today's post discusses what makes sync hard and dives into how PhaseLock's sync protocol solves the hard parts for you.

Basic sync: all you need is append

Maybe you, like me, set out to build your first sync system as soon as you learned what Dropbox was. Maybe it sounded easy. You've got a file list from the server, and a file list from the client. The server has one extra file, so you send the file to the client. Bam! Sync solved.

Then you delete a file from the client, resync, and the file reappears. Oops, that sync protocol only works in one direction.

It turns out that directly comparing state on both sides is a pretty bad way to do sync. It can work in both directions if deletions are not allowed, or it can work in one direction with additions and deletions. But even then, it requires sending a lot of state just to calculate a small diff.

Among server-client systems, a better approach to sync is to split the problem in half. The client updates are usually few and are resent until the server acknowledges them. The server updates are the bulk of the sync work. The server keeps its updates in an ordered edit log, so a client knows what to sync by the log position of the last edit it received. The append-only nature of the log means that the client never has to download a full state in order to infer deletions. Instead, deletions are recorded as edits and appended to the log.

Virtually every sync engine has a client-focused log-of-updates abstraction. For many, it's an extra component that sits outside of your database. For PhaseLock, the edit log doubles as both our database's write-ahead log and the foundation for syncing clients.

Real-life sync: append is never enough

Maybe you, like me, built your first live app as soon as you learned about sync built around an edit log. Maybe it sounded easy. The server appends edits, then the client pulls them and updates its state. Sync solved, right?

Then that pesky product manager asks for something called "permissions" and insists that "users" can either be added to or removed from "projects".

So now your sync protocol meets real life, and you're up against some difficult challenges:

Slices to the rescue

Our last post about grantors and slices provides PhaseLock's answer to these challenges: a slice is a key prefix of state, granted or revoked as a unit. Edits are logged along slice lines, so no single edit applies to more than one slice.

For grants, that means that edit ordering only matters within a slice, and a backfill containing old edits derives a consistent result.

For revocations, that means deleting an entire slice's state is just dropping a key prefix from the client's store.

PhaseLock sync protocol

Now we're ready to sketch out the PhaseLock sync protocol.

In brief, the two important messages are ["edits", ...], which carries edits, and ["view", hash, position, selectors], which carries the latest selectors returned by your grantor function. selectors is a list of slice names or slice prefixes the client can see, hash is just a hash of that list, and position is the edit log position after which the view applies.

A typical exchange might look like this:

Or, in a bit more detail:

So two messages solve sync, and the rest is plumbing: backfill downloads, keepalives, a termination message, and a "live" marker.

Conclusion

Backed by this protocol, PhaseLock's sync engine gives your app these features on day one:

Consistency: Every view of data your UI receives is a consistent view of all of its slices at the same moment in time. This is because edits that land in a single transaction are delivered as one ["edits", ...] message, and all slice updates come through a single ordered stream. When new slices are granted, they land in a single transaction that is synchronized with existing edits in the client's view.

Autosubscribe: Your app doesn't have to do the dance of discovering a new grant, bootstrapping that download, and merging its updates with an existing stream of updates, because the ["view", ...] messages and backfills handle that automatically.

Deletions and revocations: When data is deleted from the server, it propagates to the client as a deletion edit, and the client deletes it locally. When data is revoked, the client sees a new ["view", ...] message and deletes it locally.

Client bootstrap and stale client reconnect: Fresh clients automatically backfill to bootstrap their state without a full log replay. Stale clients may hold slices so old that the edits they need to catch up from the log have been pruned; they also use backfills to recover.

Your user wants these features in your app. But you don't need to build them yourself.