Introducing PhaseLockDB, part II
Syncing as permissions change
PhaseLock is a sync engine built around a simple idea: save your events as a log, which is easy to replicate, and write a reducer that builds state from those events, which can be run anywhere. The result is one sync engine for your web app, your mobile app, and even your backend.
Last week we
introduced PhaseLockDB and discussed how state inside the database and a
user-defined precommit() function can replace an entire
replicated service in front of your event log.
Today we're going to talk about the most fundamental problem in sync: How does the server choose what updates to send to a client?
"The new ones," you say, "duh." Ok, but hold on... what if the client's permissions are also changing over time?
This problem comes in a couple of cases: the reconnecting case and the online case. Let's imagine a chat application to illustrate:
- The reconnecting case: a client needs the new messages in every channel it was already a part of, plus a backfill of full history for channels it has been added to, plus a revocation notice for every channel it was removed from.
- The online case: a client needs a subscription to every channel it is already a part of, plus updates to that subscription whenever the client's permissions change, plus the same backfill and revocation for permission changes as before, only now they need to be recalculated on the fly.
Once, while trying to bolt sync onto an existing CRUD app, we ran into this problem of dynamically recalculating permissions for long-lived connections. The only solution we trusted ourselves to build was what we called the "just boot'em strategy": close every connection on any change to the permissions tables (we had 5!) and recalculate client subscriptions when they reconnect.
But PhaseLockDB has one simple solution for both cases: your
data is organized into slices and permissions are one stateless
maySee() function.
PhaseLockDB defines a "slice" of data as: one stream of events, one reducer for those events, and one key prefix populated by that reducer. Your global data model is composed of one or more non-overlapping slices. For the example of a chat application, you might have one slice per channel and a directory slice for all public channels, plus per-user data slices.
Then any client's view of the global state is just a composition of one
or more slices, as defined by your maySee() function.
The maySee() function is written by you and runs in the
database. It has access to global state and it returns a list of slices
or slice prefixes that a given user is allowed to read. For our chat
example, it could look like this:
function* maySee(qx: ChatQueryContext, userId: string, traits: string[]) { // admins can see anything if (traits.includes("admin")) { return ["*"]; } // every user can see their own user slice and the directory slice const result = [`user.${userId}/`, `directory/`]; // every user can see public channels listed in the directory for (const ch of (yield* qx.get.directory())) { result.push(`channel.${ch}/`); } // every user can see the private channels they are members of const user = yield* qx.get.users(userId); for (const ch of user.memberships) { result.push(`channel.${ch}/`); } return result; }
The result of that maySee() function serves multiple
purposes:
- In the reconnect case, a hash of the result is sent to the client to confirm permissions haven't changed in the common case.
- If the client sees permissions have changed, the entire result is sent for the client to backfill or drop slices as needed.
- While streaming to online clients, the result defines the subscription.
Additionally, the keys read by maySee() are tracked, and
any time a dependent key is modified, the function is rerun and the new
result takes effect. So tracking permissions changes on a long-lived
connection is cheap and correct.
If you've been reading our examples, you'll see that the
maySee() function is actually just a normal PhaseLock
query. It's the exact same mechanism that frontend code uses to feed a
reactive UI, only this query runs in the server.
It's worth pointing out the drawback of slices too, which is that when
precommit() processes a client command that affects
multiple slices, it must then emit separate events to each slice. They
all land in the same transaction, so everything stays consistent, but it
is a cost.
But what slices buy you is big: one stateless function fully defines your permissions, every client view is a simple composition of slices, and most importantly: slices answer all the common problems that event-log-based solutions face. More on that next week.