Introducing PhaseLockDB, part II
Syncing as permissions change
PhaseLock is an open-source backend for live apps that still work offline. You declare your types and write your business logic once, then PhaseLock's sync engine powers your web app, your mobile app, and even your backend workers.
Last week we
introduced PhaseLockDB and discussed how state inside the database and a
user-defined decide() function can replace an entire
replicated service in front of your database.
Today we're going to talk about the most fundamental problem in sync: How does the server choose what updates to send to a client?
"The new ones," you say, "duh." Ok, but hold on... what if the client's permissions are also changing over time?
This problem comes in a couple of cases: the reconnecting case and the online case. Let's imagine a chat application to illustrate:
- The reconnecting case: a client needs the new messages in every channel it was already a part of, plus a backfill of full history for channels it has been added to, plus a revocation notice for every channel it was removed from.
- The online case: a client needs a subscription to every channel it is already a part of, plus updates to that subscription whenever the client's permissions change, plus the same backfill and revocation for permission changes as before, only now they need to be recalculated on the fly.
Once, while trying to bolt sync onto an existing CRUD app, we ran into this problem of dynamically recalculating permissions for long-lived connections. The only solution we trusted ourselves to build was what we called the "just boot'em strategy": close every connection on any change to the permissions tables (we had 5!) and recalculate client subscriptions when they reconnect.
But PhaseLockDB has one simple solution for both cases: your
data is organized into slices and permissions are one stateless
grant() function.
PhaseLockDB defines a "slice" of data one key prefix in your store. Your global data model is composed of one or more non-overlapping slices. For the example of a chat application, you might have one slice per channel, plus one user-private slice for each user.
Then any client's view of the global state is just a composition of one
or more slices, as defined by your grant() function.
The grant() function is written by you and runs in the
database. It has access to global state and it returns a list of slices
or slice prefixes that a given user is allowed to read. For our chat
example, it could look like this:
function* grant(txn: ChatTxn, userId: string) { const user = txn.get.user(userId); // admins can see anything (the bare "*" stream name glob) if (user.isAdmin) { return ["*"]; } // every user can see their own user slice // (the final "!" character indicates an exact stream name) const result = [`user:${userId}!`]; // every user can see the private channels they are members of for (const {channelId} of txn.scan.user_channels(userId, {keys: true})) { result.push(`channel:${channelId}!`); } return result; }
The result of that grant() function serves multiple
purposes:
- In the reconnect case, a hash of the result is sent to the client to confirm permissions haven't changed in the common case.
- If the client sees permissions have changed, the entire result is sent for the client to backfill or drop slices as needed.
- While streaming to online clients, the result defines the subscription.
Additionally, the keys read by grant() are tracked, and any
time a dependent key is modified, the function is rerun and the new
result takes effect. So tracking permissions changes on a long-lived
connection is cheap and correct.
If you've been reading our examples, you'll see that the
grant() function is actually just a normal PhaseLock query.
It's the exact same mechanism that frontend code uses to feed a reactive
UI, only this query runs in the server.
It's worth pointing out the drawback of slices too, which is that when
decide() processes a client mutation and needs to make that
mutation's effect known to multiple clients who aren't granted any
slices in common, the decider has to write the effect into multiple
slices so each client sees the result. All of those edits land in the
same transaction automatically, so everything stays consistent, but it
is a cost.
But what slices buy you is big: one stateless function fully defines your read permissions and every client view is a simple composition of slices. They add sync-friendly boundaries to your data, offering a foundation for PhaseLock's sync protocl between PhaseLockDB and our sync engine. More on that next time.