One sync to rule them all
PhaseLock is an open-source backend for live apps that still work offline. You declare your types and write your business logic once, then PhaseLock's sync engine powers your web app, your mobile app, and even your backend workers.
The single best thing that Google Docs has done for the world is show people what live software is supposed to feel like.
You probably wish all of your applications were live and collaborative.
Ever sit in a meeting and hear, "Can you see the change to the Jira ticket yet? Try refreshing again..."
Or maybe you've heard, "Why did you land your GitHub PR without addressing my review?" and you had to say, "Sorry, I forgot to refresh so I didn't even see it."
Your user wants your app to be a live app too. But building live apps is a Hard Problem. Today's post discusses what makes sync hard and dives into how PhaseLock's sync protocol solves the hard parts for you.
Basic sync: all you need is append
Maybe you, like me, set out to build your first sync system as soon as you learned what Dropbox was. Maybe it sounded easy. You've got a file list from the server, and a file list from the client. The server has one extra file, so you send the file to the client. Bam! Sync solved.
Then you delete a file from the client, resync, and the file reappears. Oops, that sync protocol only works in one direction.
It turns out that directly comparing state on both sides is a pretty bad way to do sync. It can work in both directions if deletions are not allowed, or it can work in one direction with additions and deletions. But even then, it requires sending a lot of state just to calculate a small diff.
Among server-client systems, a better approach to sync is to split the problem in half. The client updates are usually few and are resent until the server acknowledges them. The server updates are the bulk of the sync work. The server keeps its updates in an ordered edit log, so a client knows what to sync by the log position of the last edit it received. The append-only nature of the log means that the client never has to download a full state in order to infer deletions. Instead, deletions are recorded as edits and appended to the log.
Virtually every sync engine has a client-focused log-of-updates abstraction. For many, it's an extra component that sits outside of your database. For PhaseLock, the edit log doubles as both our database's write-ahead log and the foundation for syncing clients.
Real-life sync: append is never enough
Maybe you, like me, built your first live app as soon as you learned about sync built around an edit log. Maybe it sounded easy. The server appends edits, then the client pulls them and updates its state. Sync solved, right?
Then that pesky product manager asks for something called "permissions" and insists that "users" can either be added to or removed from "projects".
So now your sync protocol meets real life, and you're up against some difficult challenges:
- Normally you process edits in log order, which ensures that every client derives the same state. But adding a user to a project means backfilling historical edits and processing them well after the fact. How can you be sure your state is correct when those old backfilled edits are processed out of order?
- Your client's state is the result of every edit it has applied. Removing a user from a project means undoing the effects of every edit that came from that project. How do you know what your client's state would look like if it had never seen those edits?
Slices to the rescue
Our last post about grantors and slices provides PhaseLock's answer to these challenges: a slice is a key prefix of state, granted or revoked as a unit. Edits are logged along slice lines, so no single edit applies to more than one slice.
For grants, that means that edit ordering only matters within a slice, and a backfill containing old edits derives a consistent result.
For revocations, that means deleting an entire slice's state is just dropping a key prefix from the client's store.
PhaseLock sync protocol
Now we're ready to sketch out the PhaseLock sync protocol.
In brief, the two important messages are ["edits", ...],
which carries edits, and ["view", hash, position,
selectors], which carries the latest selectors
returned by your grantor
function. selectors is a list of slice names or slice
prefixes the client can see, hash is just a hash of that
list, and position is the edit log position after which the
view applies.
A typical exchange might look like this:
Client sees Client does
-------------- ----------------------------
["edits", ...] apply some edits to state
["view", ...] delete revoked slices, backfill granted ones
["edits", ...] apply more edits to state
Or, in a bit more detail:
- A client opens a streaming HTTP request to the server with its authentication token, cursor position, and last-seen view hash.
- The server sends
["edits", ...]messages with each edit from just after the client's cursor position until the client is caught up, then streams live["edits", ...]messages as edits arrive. - Any time the server detects a difference between its own
grant()output and what the client is tracking, the server sends a new["view", ...]message. - The client compares its old (stored)
selectorsagainst the newselectors. Added selectors trigger a backfill; deleted selectors trigger a revocation. Revocations happen instantly (since deletion is easy). Backfills download a snapshot of a slice plus any edits between when the snapshot was taken and the view's position. When both the snapshot and edit tail are ready, the whole backfill lands in a single transaction, so the UI never sees a partial state. - Additional
["edits", ...]messages that arrive while the backfill is running are held in a queue. Edits resume processing in-order when the backfill lands.
So two messages solve sync, and the rest is plumbing: backfill downloads, keepalives, a termination message, and a "live" marker.
Conclusion
Backed by this protocol, PhaseLock's sync engine gives your app these features on day one:
Consistency: Every view of data your UI receives is a
consistent view of all of its slices at the same moment in time. This
is because edits that land in a single transaction are delivered as one
["edits", ...] message, and all slice updates come through
a single ordered stream. When new slices are granted, they land in a
single transaction that is synchronized with existing edits in the
client's view.
Autosubscribe: Your app doesn't have to do the dance of
discovering a new grant, bootstrapping that download, and merging its
updates with an existing stream of updates, because the ["view",
...] messages and backfills handle that automatically.
Deletions and revocations: When data is deleted from
the server, it propagates to the client as a deletion edit, and the
client deletes it locally. When data is revoked, the client sees a new
["view", ...] message and deletes it locally.
Client bootstrap and stale client reconnect: Fresh clients automatically backfill to bootstrap their state without a full log replay. Stale clients may hold slices so old that the edits they need to catch up from the log have been pruned; they also use backfills to recover.
Your user wants these features in your app. But you don't need to build them yourself.