This article is the third and last part of a series on building data systems that prevent errors by design. Part 1 argued that data systems are in an urgent need of a discipline that prevents errors by design. Part 2 turned those steps of proving, checking and monitoring into a practical recipe, with a worked example.

This part will use the same example pipeline but will focus on the “proving” step. We will go through different levels of arguing about the systems we build, from least to most complex. And how to decide which level of argument a property deserves.

We saw how writing a simple argument can rule out errors earlier. This is the good news. In my experience, the less formal levels catch most of the errors. The more formal end of the spectrum is mainly for the few properties that are both expensive to get wrong, and easy to get wrong.

This is part one of the Towards a Data Discipline series:

• The first post explained the main principles.

• The second gives a step by step recipe to follow.

• And this last part focuses on one specific step: formal arguments to prove what we are building is correct.

Let’s agree on what we mean by “prove”

Recall the definition from Part 1 of this series: Given a function f over inputs I, a precondition P and a postcondition Q, a proof establishes that ∀ x ∈ I : P(x) ⇒ Q(x, f(x)).

In other words, that for every input: if this input satisfies the preconditions, the postcondition is satisfied for the input/output pair.

So a proof does not guarantee that the system will not fail. It relies on the precondition P, and it relies on how we model the system. Inputs in real life can violate P, and code can drift from the initial design. This is why we still use checking and monitoring. But a proof also forces us to reason about our design, which helps us define which assumptions we rely on and what errors we may run into and makes the checking and monitoring steps easier.

In the rest of this article I will actually use the word “Argument” more. This is because not all arguments are proofs in the strict sense, and when I said “prove what you can” what I really meant is to argue for these properties.

How hard should we prove it

As we mentioned in Part 2, we asked two questions about the properties we want to prove:

• What does it cost us if this property is false?

• How easy is it to get the argument wrong?

As we did in our example, the properties end up with a proof chosen from one of the following types:

Here is an example of ways to prove that go from easier to write to harder to write (which doesn’t mean better):

• A written argument: written in the design document or another type of text, a reviewer can spot wrong beliefs and reasoning errors. It doesn’t guarantee there are no mistakes.

• Hard constraints (Construction and types): simply making the invalid states impossible to represent, so that nothing is left to prove (for example, a string variable that contains ‘true’ or ‘false’ is obviously better off being a boolean). This includes database constraints, parsing, validated types. In machine learning: monotonic constraints, domain-constrained outputs, equivariant architectures and other types of constraints can be added depending on the use case.

• A structured argument: this is a step up from the written argument. The proof techniques you may have learned and stopped using (induction over days/partitions, contradiction, contraposition etc.). This is still written text, reviewed by people, but the structure makes it less likely to fail. It requires a bit more time than a written argument.

• Exhaustive checking: there are tools that help you write a small model of the design and let a tool explore every state (up to a bound). Alloy for relations/data models, TLA+ for concurrency and orchestration, P for distributed state machines. There are tools for the code itself, I personally have never used any of these but there are bounded model checkers such as CBMC for C and Kani for Rust. The reason for this type of checking, as Daniel Jackson puts it, is that most design issues show up in small examples (exploring all of them is worth more than sampling large ones).

• Automated proof with a solver: these are only possible in specific cases, you basically state the claim as logical formulas and let an SMT solver, such as Z3, prove it for every input or give a counterexample. You can call Z3 from Python in a dozen lines, as we will. For Python functions, we have CrossHair checking contracts. For SQL, some promising tools exist such as SQLSolver and VeriEQL but they are not yet everyday tools (solvers struggle with non-linear arithmetic and complex data structures, so this level is more for small but critical claims).

• Machine-checked deductive proof: in these types of proofs you write the code together with its specification and prove that they match. Tools include: Dafny (can compile verified code to Python, and other languages), Why3, F*. Proof assistants such as Lean, Rocq and Isabelle. TLA+ has TLAPS. This is a very strong guarantee, but it also implies the highest cost in time and expertise.

• “Statistical” arguments: this is not a proof and doesn’t really fit in this list. But some properties cannot be proven that way: how long a job takes to run, how quickly an API, how accurate a model is etc. For these cases, we can still make a good case using statistics (Evaluating a model on a separate validation set is an example). Kochenderfer’s group is one of those who have thought deeply about such arguments and how they apply for airplane collision avoidance as an example. They usually name three types of tasks: falsification (finding any input that causes failure), most-likely failure (finding errors that are more probable), and failure probability estimation (finding out how often something fails). The book Algorithms for Validation includes these along with formal methods and monitoring (and is part of the inspiration to this series as well).

Now let’s see where the properties we deduced in Part 2 fit here. We already argued for P1 and P2, so we will only focus on the rest. Here is the register with the type of proof we select in bold.

Proof register:

• P3. daily revenue matches the finance team definition. Type guarantees and a written argument. Depends on A2, A3 and A4.

• P4. daily revenue is the same whether the table was built daily or rebuilt once (incremental equals full): a structured argument, by induction. Depends on P1, P6 and P7.

• P5. the job is ready by 07:00: statistical argument by running multiple times (on the edge examples) and reporting the distribution of time it takes to run.

• P6. no on-time event is missed: if an event satisfies A1, it is accepted before the last run that recomputes the day of that event. Machine-checked, with SMT solver. Depends on A1 and on the scheduler.

• P7. isolation: an overlapping retry never overwrites newer results. Validated through an exhaustive check of possible interleavings.

This is just a way of seeing these properties and where each type of argument may be more appropriate. In our example, we already went through P1-P2 through a simple written argument. This post will focus on the remaining properties and how we can argue for them.

How to argue for a property

Hard constraints: make illegal states impossible

If an invalid state cannot be represented, it cannot be reached. For example, a string column holding “true” and “false” is better off being a boolean.

This level of argument should be used very often (and is used very often) in databases through constraints and type definitions. It also exists in machine learning in the form of monotonic constraints, outputs restricted to a valid domain, equivariant architectures etc.. A pricing model trained with a monotonic constraint on a variable does not need a proof that this variable will follow that rule. In training sets as well, point-in-time joins can be used to avoid leakage.

Now let’s look at our example P3. daily revenue matches the finance team definition. In the pseudocode below, the Event type can’t hold a negative amount, a naive timestamp or an event later than we allow, so nothing downstream needs to consider one (there are better types to write this code, but this is enough to show our point)

# imports and definitions constants here...
class ContractViolation(ValueError):
    """The record goes to quarantine, after this violation"""

@dataclass(frozen=True)
class Event:
    event_id: str
    kind: Literal["order", "refund"] # "order" or "refund"
    amount: Decimal # euros, whole cents, > 0 (A3)
    occurred_at: datetime  # timezone-aware
    received_at: datetime  # timezone-aware

    def __post_init__(self):
        if self.kind not in ("order", "refund"):
            raise ContractViolation(f"unknown kind: {self.kind!r}")
        if self.amount <= 0 or self.amount != self.amount.quantize(CENT):
            raise ContractViolation(f"not a positive whole-cent amount: {self.amount}")
        ...
        if self.received_at - self.occurred_at > MAX_DELAY:
            raise ContractViolation("arrived later than A1 allows")

    @property
    def day(self) -> date:  # A4: a calendar day in UTC
        return self.occurred_at.astimezone(timezone.utc).date()

def parse_event(raw: dict) -> Event:
    """Once we parse here, the assumptions will hold for the rest of the code"""
    return Event(
        event_id=str(raw["id"]),
        kind=raw["kind"],
        amount=Decimal(str(raw["amount"])),
        occurred_at=datetime.fromisoformat(raw["occurred_at"]),
        received_at=datetime.fromisoformat(raw["received_at"]),
    )

A type checker will reject code that tries to build an Event that doesn’t follow our assumptions. I talked about parsing as a “check” in part 2, and now I mentioned it as an argument. Well, it is both, it’s a check at runtime, but writing this earlier is a proof for what happens downstream. Now that we have this type guarantee, we can continue with the written argument.

Whichever the use case, this type of argument is a strong and cheap guarantee so check for what types of constraints are needed to guarantee your specific properties.

Structured arguments

Now let’s look at a more structured type of proof.

At the end of the day, this is still reviewed by people. But with a structure that makes it easy to spot any steps we skip, or any mistakes we make while thinking. This is where we can use arguments such as: induction, contradiction, contraposition, etc.

Here’s an example for our case: P4. incremental equals full: this property depends on a precondition: every event arrives in time for one of the runs that recompute the day of the event (either the run on the same day or in 3 days after). That precondition is P6, and we will prove P6 later so for now let’s assume it holds. It also depends on P7 because it assumes runs happen one after the other.

Proof by induction

Claim: after the run for day D, the result of every partition up to D equals the result of a full rebuild from all accepted events.
Base case: before the first run, the table is empty and there is nothing to compare.
Inductive step:
Assume the claim holds after the run for day D.
Let us consider accepted events between the run for day D and the next on D+1.
- These events belong either to a day from D-2 to D+1, or to a later day, not yet published.
- The run for D+1 recomputes exactly partitions from D-2 to D+1, each of them equals the full rebuild (by P1).
- Partitions older than D−2 are not touched by this run. By P6, none of the newly accepted events belongs to them. So they are unchanged, and by the induction hypothesis they equal the full rebuild.
So the claim holds after running D+1

Notice how this made use of: P1 and P6, while assuming P7. Trying to prove properties usually brings up what you depend on, and how confident you are in those dependencies.

Exhaustive Check

Testing tries a few cases, but an exhaustive checker tries all of them, up to a certain bound. You write a model of your design, and let a tool explore every state that can be reached. Alloy is such a tool for data models and relations. TLA+ works best for concurrency and orchestration. These tools will catch if something can go wrong within the bounds.

A small model is usually enough here, because most design errors are small. Daniel Jackson has called this the small scope hypothesis: if a design is flawed, a few small examples usually expose the flaw. This is also why testing as a first pass is encouraged, despite what I said earlier about testing. A test that finds an error early is good as long as it doesn’t make us overconfident.

The obvious, and heavily cited, example is that of AWS engineers who had reported that TLA+ caught design errors in their systems that they would not have found otherwise. And engineers became productive with TLA+ in two to three weeks (which is a small price to pay for a critical system, it seems that they have moved on from this recently, and use P language instead to model systems and PObserve to check the logs are following the model).

For our pipeline example, this type of argument would be overkill. But imagine a scenario where a job fails on day D, we keep retrying until D+1, while the job for D+1 itself already started. Two jobs are now writing the same three days at the same time. Which values end up in the table? Does idempotence still hold? Problems like this are what people tend to reason about worse. If you have taken a parallel programming course you may be better trained to reason in such a way. But even then, when the systems are complex enough, and critical enough, using exhaustive checks can let the machine examine what your mind is too small to hold.

So let us check it anyway for the sake of example:

P7. isolation: an overlapping retry never overwrites newer results

Once again, this exhaustive check will only check for the model we defined in this code. Our actual code can still be different or drift away from this model, what we basically do here is we take our data pipeline design → simplify into a model → enumerate and check interleavings.

Runs can overlap, but neither run can execute its own steps out of order.
We will add an input variable called fencing , when it is true older runs are not allowed to overwrite newer runs. We will see here that when this is set to false we end up with a lot of possibilities of interleavings (in our case, more than half of the time)

from itertools import combinations

def run(name, partitions):
    """This models the run as a sequence of steps, starting with 'start'
    followed by the partitions"""
    return [(name, "start")] + [(name, p) for p in partitions]

def interleavings(a, b):
    """Every order in which the steps of two runs can happen."""
    n = len(a) + len(b)
    for positions in combinations(range(n), len(a)):
        ia, ib = iter(a), iter(b)
        yield [next(ia) if i in positions else next(ib) for i in range(n)]

def execute(steps, fenced):
    """Executes the given steps with optional fencing.
    Fencing means that a partition will not be updated with an older snapshot."""
    clock, snapshot, table = 0, {}, {}
    for name, step in steps:
        # The "start" step records when the run begins
        # Each number represents a partition it writes
        if step == "start":
            clock += 1
            snapshot[name] = clock
        elif fenced and table.get(step, 0) > snapshot[name]:
            continue # here we do not overwrite a partition with older data
        else:
            # here we update the partition with the current snapshot of the run
            table[step] = snapshot[name]
    return table, snapshot

fenced = False
D=10
retry = run("retry of day D", range(D - 3, D + 1))
nxt = run("run for D+1", range(D - 2, D + 2))
total = bad = 0
for steps in interleavings(retry, nxt):
    table, snapshot = execute(steps, fenced)
    # each partition must end up with the newest snapshot that wrote it
    expected = {}
    for name, partitions in (("retry of D", retry), ("run for D+1", nxt)):
        for _, p in partitions[1:]:
            expected[p] = max(expected.get(p, 0), snapshot[name])
    total += 1
    bad += table != expected

print(bad, total)  # (143, 252)
# if you set fenced = True, we get no issues

What we have shown here is that “assuming this model holds, and fencing is enabled, we prevent older runs from overwriting newer data in all possible interleavings” This doesn’t show our pipelines truly follows this model, nor that fencing is enabled. And that if fencing is not enabled, this can definitely happen.

We can make this safe in practice for example by forcing the orchestrator to never run two of these jobs at once, or to have a version of the snapshot within the run.

Automated proof with a solver

Now let’s look at P6 which we promised to use an SMT solver for: P6. no on-time event is missed.
If an event satisfies the assumption (A1, arrives within 24h) the event is received before the last run that recomputes its day.

What a solver does: an SMT solver, such as Z3, takes a set of conditions on some unknowns, for example x > 3 and x + y = 10. It then does one of two things.
- Finds values that satisfy every condition at once
- Or it proves that no such values exist.
It does not try values one by one. It reasons about the conditions in a similar way that you would solve an equation instead of trying all numbers that may or may not solve it.

How that becomes a proof: in our case, we ask the solver to find a counterexample that satisfies A1 and is still missed. If it finds one, then we have an error. If it reports that no such event can exist, then P6 holds as long as the preconditions are valid.

A written argument for this simple case could be short and convincing enough. But a solver should be considered if we have the time to invest, and consider this property critical enough. Here’s an example using Z3

from z3 import Ints, Solver, sat, unsat
from revenue import DAYS_BACK, MAX_DELAY, RUN_START

DAY = 86_400 # in seconds

def seconds(t):
    return int(t.total_seconds())

def find_missed_event(days_back=DAYS_BACK, run_start=RUN_START, max_delay=MAX_DELAY):
    """Find event that:
      - satisfies A1
      - and is still missed.
      Missed here means: it is received only at or after the start of the
      last run that can recompute it"""
    occurred, received = Ints("occurred received")  # defined in seconds since 1970, UTC
    day = occurred / DAY # integer division: its UTC day (A4)
    # The run for day D starts on D+1 and recomputes D-days_back up to D,
    # so day d is last recomputed by the run for d + days_back.
    last_run = (day + days_back + 1) * DAY + seconds(run_start)
    solver = Solver()
    solver.add(occurred >= 0,
               received >= occurred,
               received - occurred <= seconds(max_delay),  # A1
               received >= last_run)                       # ...and yet missed
    result = solver.check()
    if result == unsat:
        return None # proved: no such event exists
    if result == sat:
        return solver.model() # a counterexample
    raise RuntimeError("The solver was unable to decide.")

In simple words, the function states three facts: when the event occurred, when it was received, and that the delay between the two respects A1. Then it adds the condition that the event arrived too late. If find_missed_event() returns None, no event can satisfy this. This also makes things more robust to changes in the code, if someone changes the DAYS_BACK in the code, this property will be validated again.

A little word of honesty: as you see in the end, a solver can answer sat, unsat or unknow. When I first wrote this, it worked such that unknown was also considered a proof. Errors like this can happen especially for someone like myself who is very new to z3 (I’m sure experienced people would have spotted this faster). But this is just a word of caution again, that the guarantees of these arguments are great, but they are not completely free of flaws.

You can also test the SMT solver by going into a case where you know it should fail (setting days back to 0 in our example) and see that it comes up with counterexamples.
The solver obviously proves this for a model of the design, not for what happens when we hand it over to the orchestrator.

Statistical arguments

I left P5 last as it is a different kind of argument. P5. ready by 07:00: assuming the run starts at 6:00 it must finish within the hour. A proof cannot really establish this. But we can use a statistical argument.

P5. ready by 07:00 : Before we start this argument let’s make an assumption, we can add this to our assumptions register and check for it when monitoring: A5. daily volume never exceeds twice the volume of the busiest day observed so far.

Let’s say we now run twice the maximum volume observed 300 times on our cluster. And say the slowest run takes 19 minutes. It feels like we have enough evidence, but the evidence is weaker than it feels. 300 runs without a single overrun allow us to say, with 95% confidence, that a run overruns less than one time in a hundred. But we don’t know the tail of our model.

Here are four directions we could go:

• Accept the risk: and consider this to be enough for our case. After all, the finance team wants to guarantee this just 90% of the time.

• Run it more: 3000 clean runs would bring those down to one in a thousand. But this is 10x more expensive.

• Make an assumption about the tail distribution: and add we add that distribution as an assumption to the register. We can use it to estimate how likely a run is to go beyond an hour. But because the distribution is an assumption we may want to verify it and monitor for it depending on how critical it is.

• Sample the harder cases: if we are interested in the rare event, this problem is what failure probability estimation deals with. Simulating ordinary days over and over is not an efficient way to learn about the rare events. One of the methods that can be used here is importance sampling, one of the families of methods Kochenderfer et al. survey in Algorithms for Validation. This means sampling from a distribution biased towards errors (more volume, more late events etc.). We can then give each run a weight: how likely the conditions are on an ordinary day, divided by how likely we made them be in our sampling distribution. The weights correct for the bias we introduced, so the estimate still describes ordinary days, but now most of your runs are spent where the errors are most likely. However, this requires a distribution of what a “normal” day looks like. We usually don’t have this, which means that if we take this route we need to model this distribution and add that as a new assumption, by fitting it to history and then monitoring like any other assumption. Once again, Algorithms for Validation is the place to read about such strategies.

Whichever way we go, what they all should have in common is a form of transparency: stating the confidence, and writing down the assumptions we are making so that we can monitor them later.

But the code will drift away

If you prove for your current design, the code can (will) drift away with time. So how can we keep it consistent with our proofs?

Testing can close some of the gap. For example, property-based testing states the property’s own claim, and can look for inputs that break it. This is what Hypothesis is for in Python as an example. This test is still only sampling, but it samples whether the code stays faithful to our argument.

from hypothesis import given, settings, strategies as st

# events(): a strategy for well-formed, on-time events (A1 to A3), duplicates included
@given(st.lists(events(), max_size=50))
@settings(max_examples=500)
def test_incremental_equals_full(evts):
    # replay_daily_runs: run the job every morning, on what had arrived by then
    # rebuild_from_scratch: one run at the end, over everything accepted
    assert replay_daily_runs(evts) == rebuild_from_scratch(evts)

On our design, this would pass 500 generated histories in about two seconds. We are still just sampling but we are checking if the code still behaves in a way that the proof expects.

It’s always good practice to make sure the test can fail. Try an example where it should fail and Hypothesis should find a counterexample in a few seconds.

And of course, this is not a guarantee that the code will not drift. There are other tools that can improve this. CrossHair for example can explore Python code symbolically, using Z3 in the background. At the far end of the arguments Dafny closes it entirely (the code and the proof are tied together and the program wouldn’t compile unless the proof holds). But let’s be honest, most data teams will not go that far. But in case of very critical systems, we have to know what exists and what tradeoffs we are making.

Conclusion

Most of the guarantees of correctness in this series came from the simple parts: a falsifiable statement, an assumptions register, a few written arguments, type constraints, and parsing. The expensive tools (solvers, model checkers, proof assistants) only become interesting on one or two properties at most (which we may have exaggerated in this example).

And if you use language models for assistance writing code (who doesn’t at this point), just make sure you do the reasoning yourself first. Language models can assist you in reviewing it, they can assist in spotting flaws, as well as assist in writing and reviewing Z3 or Hypothesis code or similar. But if we completely rely on them, we not only risk having them solve something different from what we define as correct. I’m writing a few skills that can help in this process that I will share both here and by email as soon as they’re ready.

This whole series can be put in three lines:

• Prove what you can, so that some errors are impossible by construction.

• Check at the boundaries, so that the preconditions of the proofs also hold.

• Monitor the rest, and give every alert an owner and a checklist.

None of this is new. What I hope this series adds is a practical way to apply it to data systems, without turning every engineer into a mathematician. If you build data or ML systems, I would genuinely like to hear if this is helpful at all.

Leave a comment

Subscribe now

My last post, Towards a data discipline, explained why we need a data discipline. A way of building data systems to prevent errors by design, instead of detecting them after the fact.

This post turns that into a practical step by step recipe. It follows the spirit of Karpathy’s recipe for training neural networks. His observation that badly trained networks rarely announce errors, and just quietly perform worse, is shared across data systems including data pipelines. The main reason for this article is to document a process for building data systems that are correct by construction. This post may be opinionated (read: biased), so any criticism is very welcome.

This is part one of the Towards a Data Discipline series:

• The first post explained the main principles.

• This post gives a step by step recipe to follow.

• And the last part will focus on writing more complex formal arguments.

The good news about proofs and formal methods

I want to make this clear. Formal methods have a reputation for being complex and expensive. And some of them are. But, in my experience, most errors are caught at the less formal end: when you try to write a falsifiable statement, when you list assumptions, and when you try to explain, in simple words, why a property holds. More complex solvers and proof assistants are for the few properties where a mistake is expensive and the argument is easy to get wrong. The next post will go into more detail on proofs and writing arguments.

The full recipe

We build things in the wrong order today:
write ad hoc code → write tests, maybe → ship and decorate with monitoring.

And we end up introducing a lot of new concepts such as data contracts, SLAs, SLOs, etc. etc. I would argue that skipping explicit reasoning about correctness is a big part of why we keep reinventing ideas that other engineering fields have always been using.

This recipe changes the order in which we do things:

For the sake of repetition: skipping a step is allowed but should be explicit.

#0 - Define as you go

As we mentioned already: you cannot define a data system you don’t understand, and you will not understand it unless you get your hands dirty with some experimentation first.

0.1. Experiment

Open a notebook, look at the events, do some exploratory data analysis, look for inconsistencies, duplicates, patterns, ask a domain expert etc. All of this can give you insights such as: all events “usually” arrive within a day, or that “daily” means a different thing for the finance team than it does for the logistics team etc. etc.

The goal of the experimentation step is not to be restrictive. But it has an exit condition. You are only done with experimentation when you can: state the problem properly, when you have an idea of how you can solve the problem (a rough sketch of a design you think will solve it), and you have a clear set of assumptions (such as what “daily” means etc.)

0.2. Define what correct means

After the experimentation, we may be able to have our definition of “correct” as something like this:

For 3 consecutive weeks,
rows for day D are successfully published by 07:00 UTC on D+1 at least 90% of the time. And by 08:00 UTC at least 99% of the time.
And once day D is closed, from D+4 onwards, its net revenue matches the finance ledger to the cent.

Now, if you want to evaluate your definition, here is a question to use as a test:
“What observation would imply your statement is incorrect?” If there is no observation that will show your statement is incorrect. Then your statement is not falsifiable.

I know that at times this can be hard to do for certain problems, but we have to start somewhere. When you approach a problem without having a proper definition, you are still having an implicit definition, so we must never use “this problem is hard to measure” as an excuse. The choice is between having this definition be implicit or having it explicitly and clearly stated.

As we previously mentioned measurements do mislead. Once a number becomes a target, people tend to optimise the number instead of what is meant. This is one more reason why we have to write down the assumptions.

0.3. Keep a list of assumptions

Write all the assumptions (and simplifications) you gather along the way in a register of assumptions. This can be a markdown file near the documentation.

Every pipeline rests on a set of small beliefs and decisions. Not writing them down will almost always come back to haunt you. Writing them down can be useful for us to decide: do we want to prove this, check for it, monitor it, or accept it if it’s trivial.

Here are examples of assumptions for our finance data pipeline above:

• A1. Every event arrives within 24 hours of when it happened. (Observed maximum: 21 hours.)

• A2. Every event has a unique id. Events may be delivered more than once, but a duplicate event is identical to the original.

• A3. Amounts are positive, in euros, in whole cents.

• A4. A “day” is a calendar day in UTC. (Confirmed with Anna from finance on the 17th of September 2025)

What matters here is that these assumptions can be cited later, they can be checked and updated, and they are open for review.

Step #0 can be considered done when: you have a clear statement of what it means for your system to be “correct”. You have a rough sketch of how to solve it and have verified that through experimentation. You have a register of assumptions about the world and what different definitions mean.

#1 Prove what you can

Because this step needs an article on its own, I will only introduce the simpler ideas here and a more detailed version is available in the next post here.

Now is the time to design and architect the data system. As we do so we will want to define “what properties must be true, and how can we convince someone else that they are?”.

This forces a modular design! Dijkstra puts this more eloquently when he said that the competent programmer is “fully aware of the strictly limited size of his own skull”. A proof of an entire pipeline cannot be reviewed, if at all can be written. A proof of one critical function is usually reviewable and relatively easy to write.

State the properties to prove

The entire goal of this step is that you have a list of properties to prove in a proof register. In this register we want to have:

• what we want to prove: properties that the data system should guarantee.

• how we want to prove it: we will do this in the next step, more critical properties may require more formal proofs, or other forms of validation.

The definition step before should have given us: 1. A statement about the whole data system. 2. A list of assumptions. 3. A rough design of the system that can be improved upon. The first thing I would do in this case is select from this what parts of the system are more critical in terms of correctness. Before we jump into generic properties like idempotence and determinism, we first want to show the system achieves its specific goals.

In our example, we can take part of the statement: “For 3 consecutive weeks, rows for day D are successfully published by 07:00 UTC on D+1 90% of the time” And see what it implies for each module in our design. What each module needs to verify to achieve this, and which modules are more critical to this statement. For example, the scheduler must start early enough, and the job must finish by 07:00 UTC. Add all those properties to your proof register.

On general properties to validate for data pipelines: After the above (specific) properties are added, in the case of a data pipeline we can add things that come up often. I have found the following list useful to go through, and I’m sure I have missed a few others. The reason they keep turning up is because retries, redeliveries, late data, etc. are usually part of every data pipeline. None of these are surprising to any data engineer, and they cover a big share of what goes wrong:

• Determinism: the same inputs give the same output, whatever the order in which they arrive, or how many workers process them.

• Idempotence: running the job twice for the same day leaves the table exactly as running it once. This is what makes retrying your job harmless.

• Incremental equals full: in other words that running the job every day for a month produces the same table as rebuilding it at the end of the month in one go. Think of when a backfill produces a different result from what you have been publishing daily, and now you have to explain which one is true.

• Conservation: nothing is created or lost in transit. In our pipeline example, every received event ends up in exactly one place: either accepted or quarantined. Keeping the raw history immutable makes this easy to verify.

• Invariants: what must be true of the output at rest (independent from how the pipeline ran or how it’s designed). For example in our pipeline this can mean: exactly one row per day, or the fact that net revenue equals the gross revenue minus refunds.

• Isolation between runs: two overlapping runs must not overwrite each other’s results. A retry still running when the next day’s run starts could cause issues.

I am not arguing that any of these properties must be guaranteed in every data system. These are properties I personally look at from time to time to check if they are needed in a specific scenario. Maxime Beauchemin’s article from 2018 on functional data engineering is still a read I would recommend for reasoning about some of these. Although it focuses on Airflow, much of it is generalisable.

Beyond data pipelines: the same principle applies to any data system: start from what the system is for, and ask what each module must guarantee for that to hold. The following are a few other properties from other kinds of system, each of which I have seen being repeated in enough cases to put as a list:

• For training sets used in an ML model: no feature value is used before the moment it would have been known. The absence of point in time correctness will lead to leakage.

• A pricing model: all else equal, the predicted price never decreases as floor area grows. XGBoost and LightGBM can guarantee this by construction, through monotonic constraints.

• An access layer: no combination of roles can read a column tagged as personal data (AWS for example uses an SMT solver to answer such questions, we will talk more about SMT solvers in the next post).

• Schema migration: queries run against the old schema return the same rows against the new schema.

Are there any similar things that you often see repeated? It would be very helpful if you let me know here in a comment.

Decide how hard you will prove each property

A proof can go from: a written argument in the design docs, to a structured argument, such as induction over partitions. Or even a machine-checked model or proof (such as using TLA+ for concurrency and orchestration)

More formal proofs are not always better. They are more expensive, but provide stronger guarantees to prevent errors. There are two questions I would ask to figure out what level to choose: “what does it cost us if this is false?”, and “how easy is it to get the argument wrong?”. I wrote more details on this in the next post.

From all the above we can complete our proof register, this list will now have what we want to prove and how we intend to prove it (including what we will not prove and why). Here’s an example for our daily revenue pipeline:

Proof register:

• P1. determinism: type guarantees and a written argument. Depends on assumption A2.

• P2. idempotence: written argument. Depends on P1.

• P3. daily revenue matches the finance team definition. Type guarantees and a written argument. Depends on A2, A3 and A4.

• P4. daily revenue is the same whether the table was built daily or rebuilt once (incremental equals full): a structured argument, by induction. Depends on P1 and P6.

• P5. the job is ready by 07:00: statistical argument by running multiple times (on the edge examples) and reporting the distribution of time it takes to run.

• P6. no on-time event is missed: if an event satisfies A1, it is accepted before the last run that recomputes the day of that event. Machine-checked, with SMT solver. Depends on A1 and on the scheduler.

• P7. isolation: an overlapping retry never overwrites newer results. Validated through an exhaustive check of possible interleavings.

• Not proven: agreement with the ledger depends on the finance system.

• Not proven: the scheduler runs the job at all, which is someone else’s software. Both go to monitoring.

The last lines matter just as much. A reviewer can now see what we skipped and argue about it.

As a summary, before you leave this step you should have a proof register including:

• what you will prove.

• how you will prove it.

• what you choose NOT to prove should be explicit.

The next article has exact steps to help you decide this with an example.

Write the arguments before writing code

Write the arguments before you write code. Quick falsification (a few tests, a small script) is still worth doing at any step, because it is the cheapest way to find out an argument is wrong, and perhaps you would have done much of this during the experimentation step but it is the argument that usually tells you what to test.

Now we have a list of things to prove and how we will prove each one.

Let’s only look at the simple arguments here, and keep the more complex ones for the next post.

P1. determinism: the function reads nothing about the environment (no clock, no environment variables, no randomness). It removes duplicates by id, which cannot depend on the order of arrival (A2). The order of summation does not matter, since amounts are Decimals, and Decimal addition is exact within the precision we need. The output therefore must depend only on the set of accepted events for that day.

Note that, for example, the last sentence could be false if we had floats in Python, (0.1 + 0.2) + 0.3 gives 0.6000….0000001 and 0.1 + (0.2 + 0.3) gives 0.6. Which means that the total could depend on the order, and produce different numbers for the same events.

P2. idempotence: running on day D replaces partitions from D-3 to D with values computed from the accepted events for those days. Replacing a value with the same value twice is the same as doing it once, and gives the same value.

That argument would fail, if the job appends rows instead of replacing partitions, and the system would fail on a retry. It also assumes P7, that there is no issue with overlapping jobs (we will argue for P7 in part 3).

These examples are perhaps too simple, but think of how many times we actually try to argue for idempotence and determinism when we build data pipelines. Very rarely.

For the more complex end of arguments check the next blog post on formal methods here.

Write the tests and the code

Write the tests for the arguments above. Tests now have a precise job, they do not ask about a few examples you can think of on the fly. They ask whether the code follows the properties you care about the most.

Property-based testing fits well here and the next post will show an example of it, with a library such as Hypothesis in Python. Each test states a property from the register, and Hypothesis generates hundreds of inputs that satisfy the preconditions and tries to break the test. Because they can find counterexamples early, it is justified to use property-based tests before investing time in writing an argument.

For writing good tests I would defer to: the testing section of Software Engineering at Google, How to Specify It! by John Hughes for property-based testing, and Google’s ML Test Score for data and ML systems.

Then you can write the code matching the design, and while doing so write the types and contracts that encode the assumptions, and the functions and jobs that attempt to guarantee them.

Definition of done: you are done with this step when the properties in your register are argued for, every argument names the assumptions it relies on, and every property has a test (or a machine-checked proof).

#2 Check at every boundary

Now is the time where we put those assumptions to a good use.

Check at the boundary: means simply execute the precondition.

In our example, we have four clear boundaries:

1. We receive raw events from the checkout service that become accepted events.

2. Those accepted events are handed over. No check is needed here: the types and parsing should already fulfill our needs.

3. When writing to the table that is handed to the finance team.

4. The orchestrator’s config, which must run as expected at the time expected.

When the events are received we can make use of the idea of parse don’t validate (recommended read by Alexis King) to parse the events in a way that holds the assumptions. If an event fails we can reject them and add them to a quarantine table instead of correcting them and letting an error leak into the results.

Isolating late events, for example, is not only good practice in this case, our proof for incremental-equals-full, which we will write in the next post, depends on it. Note that for our example, if we reject a late event and put it in quarantine, the results will probably disagree with the finance team’s calculations. But this will help us review our assumptions about late events, and will also be relatively easy to figure out.

When the events are written we need a contract with the reader (in our example the finance team). This should contain for example:

• Structure: one row per UTC day, including days without sales, with the columns day, gross, refunds, net and orders.

• Meaning: amounts in euros, net equals gross minus refunds, net may be negative.

• Delivery: provisional by 07:00 UTC on D+1, and final from D+4, and a final day never changes again without an announced reopening.

Two things about this contract. First, “net may be negative”: writing down what is allowed matters as much as writing down what is not: it is what stops a well-meaning colleague from trying to fix it. Second, “a final day never changes”: it follows from the design (each run only replaces partitions D-3 to D, see P2), since no run touches an older partition. We promise something we have argued for.

In Meyer’s terms, what we did here is: use the parser to state the preconditions, and the contract to state the postconditions to the finance team (and “one row per day” and “net equals gross minus refunds” are invariants that must hold)

Definition of done for step 2: every assumption about the world in the register is checked at the boundary where it enters, or there is a written note explaining why it cannot be; and every promise to downstream users is written down

#3 Monitor the rest

Whether the world keeps behaving the way the register says it does, whether the evidence behind our statistical arguments still applies, and whether the system as a whole meets the statement from step 0. Monitoring is now a short list, because we know exactly what to watch. Every item on it points to what it is for:

• A1. events arrive within 24 hours: the quarantine count and the distribution of delays. Any quarantined event alerts the pipeline owner, who reconciles with the source team and decides whether to reopen the day.

• P5’s evidence: the job’s runtime against the benchmark. A run still going at 06:45 is an early warning, not yet a breach.

• Not proven, the scheduler: if no run has started by 06:15, alert.

• Not proven, the ledger: from D+4, a daily reconciliation against the finance ledger. Any difference goes to the pipeline owner and to Anna.

• The definition itself: the weekly share of days published by 07:00, against the 90% in step 0, and by 08:00 against 99%.

As stated in the earlier post, every monitoring artefact should have: an owner, a publishing channel, and an action (this clarifies what to do in case of an alert and who does it).

Definition of done: assumptions that cannot be proven or that must be monitored anyway have a monitor, or an explicitly written acceptance of the risk. And every monitor has an owner, a channel and an action.

Optimise

Only now can you parallelise, pre-aggregate, change engines, and all that. But every change must preserve the properties. As Dijkstra wrote (in his essay How do we tell truths that might hurt?) “Simplicity is prerequisite for reliability,” it’s also a prerequisite for optimisation.

Conclusion

Is this too much to ask? This is a real question, not a statement. When I started this series, I worried that some of it may be too much. But look at the simplest version of this: a falsifiable statement, a few assumptions, a few lines of argument, a parser and a contract. That “should” be a few hours of writing, which would prevent days of fixing or worse. The expensive parts (like using a proof by induction, using solvers or statistics) are reserved for the properties that deserve that time.

Now that we use assistance from LLMs to write much of our code, this discipline matters even more. The definitions, the assumptions and arguments are good things to work on without help from any AI assistant. Writing code has become faster, let’s try to keep a certain base level of understanding.

So, honestly, if you are a data or ML engineer, do you think this is too much?

Leave a comment

Next post uses this same pipeline but will focus on the arguments themselves: how hard to argue for each property, with example proofs and using a solver and an exhaustive check.

Subscribe now

Scenario: You ask about the data quality and someone shows you a series of tests, monitoring dashboards, and metrics that are supposed to prove that everything works as it should.

Here’s the problem: testing, at best, can reveal the presence of errors, it can never show their absence. Dijkstra warned against this years ago, and this same intellectual shortcut has grown tremendously in the data profession. Especially where anything close to a formal proof is hard, and sometimes impossible.

Today, we build data pipelines and machine learning models in an ad hoc experimental way. We add some garnish (infrastructure as code, basic tests and checks) and ship them to production. Detection is cheaper and more convenient than prevention, and whenever we are given the choice and are under a hard deadline, it is easy to prioritise the instruments and skip the discipline. This is not an argument against testing but rather where it belongs.

What we are really lacking is a disciplined way to prevent errors from happening in the first place: a way of making certain errors impossible by design.
Ignoring this is similar to ignoring gravity when designing an airplane.

This series is an attempt at such a discipline, to bring back the engineering in data engineering, the science in data science, and to show that data systems can be made with a prevention-first approach.

What I propose here is an approach that aims to ensure a data system is correct by design. This is part one of the series:

• This part explains the main principles.

• The second part gives a step by step recipe to follow.

• Finally, the last part will focus more on: formal arguments to prove what we are building is correct.

I expect many people will disagree with this. Your comments and challenging arguments are welcome.

The why and how of correctness by design

If you are not convinced this problem is big enough, meditate on this quote:

“As long as there were no machines, programming was no problem at all;
when we had a few weak computers, programming became a mild problem,
and now we have gigantic computers,
programming has become an equally gigantic problem” The Humble Programmer, E. W. Dijkstra (1972)

Today this is orders of magnitude larger, and I suspect it is a matter of a few years (if not a few months) before this gets completely out of hand.

About seventeen years ago, when I took my first machine learning course, one of the notes I took reads something along the lines of: “deep learning is currently of little practical use, perhaps it may be useful in the future”. I mention this not to display my poor judgement, which speaks for itself, but as a reminder of the speed at which this field has grown. Problems that, back then, were not even considered possible to solve are today expected to reach production in one or two months. There is a gap that is growing even faster now with how we use language models to write code.

Our capacity to build has grown dramatically, our capacity to understand what we build has stayed about the same.

That is why we need a prevention-first discipline. The aim is not to guarantee every module is correct. It is to avoid as many errors as possible by construction.

The framework I will suggest here has three steps: 1 Prove → 2 Check → 3 Monitor

Skipping any step in this ladder is a choice that should be explicitly made rather than being implicit.

But before we explain this, it makes no sense to talk about “proving a system is correct” unless we define what it means for such a system to be “correct” in the first place.

So all of the steps above depend on having a proper definition. How can we define what correctness means for a data system?

First, what does “correct” mean

There is no reason to prove a system is correct if you cannot define what correct means. This section is about defining what the system does in a way that is falsifiable.

Most projects have a generalised statement of what they should achieve: “Make our data more up to date”, “Improve the data quality of X” or “Make our data products discoverable”. The issue with these generalised statements is that there is no way of ensuring we really achieved them other than a subjective feeling.

Just like saying “Cleanse the energy of the data” these generalised statements mean different things to different people. And, when we claim to have achieved the goal, what it meant for the person requesting it may be very different from what we have built. A falsifiable statement instead is clearly defined and avoids this problem. For example: at least 99% of daily datasets are available by 07:00, compared with the current 92%, for 8 consecutive weeks. Those proficient in the data jargon will state that this is just an SLO (service level objective), it does overlap with it but the goal here is to use this statement as the base that everything else is derived from.

One way to define this is to think about:

• What is measured

• What is the current state (the baseline we’re comparing with)

• What is a measurable improvement (for the project to be called a success)

• Time periods: how long until we know the measurable improvement

• Important constraints and limitations

We don’t always need all five, the main thing we need is for the statement to be, you guessed it, “falsifiable”. This gives us a clear definition of what it means for what we built to be “correct”.

Most of the time, teams resist writing such falsifiable statements because of two reasons. The first is that they don’t have time: this usually means that delivering fast is more important than delivering the right thing. Sometimes that trade-off is worth it. But it should just be clear that it is a choice, made explicitly and not just happening by default.

The second is that the problem is hard to understand and it is unclear what to measure: this just means the problem is not understood well enough, which only means the experimentation step is not finished.

You don’t need a falsifiable statement at the start. Often, you need to experiment with ad hoc code before you actually understand the problem, to go back and forth with the people involved. This step usually helps you get a true understanding of what the problem requires, how it can be solved and what can be measured. The experimentation step is necessary, because it helps us understand what is involved in the problem. The experimentation should not be restrictive, which means it allows for the kind of creativity that is needed to solve certain problems.

Sometimes measuring is hard, other times impossible, and for a few problems it can be misleading. And when a measure becomes the target, it ceases to be a good measure (Goodhart’s law, in Marilyn Strathern’s phrasing). None of it is a reason to skip the definition step. Because at the end of the day you will be making either implicit or explicit measures. It’s the choice between defining, and going with the vibe.

In summary,
Whether you like it or not, you are making a hypothesis. So make sure it is clear and explicit, and define your project’s goal as a falsifiable statement.

The prevention-first discipline: prove, check, monitor

Testing and monitoring are necessary. But given that these can only ever demonstrate the presence of errors, the better question is how can we make some errors structurally impossible before we even write code?

I will argue that we need to take a more formal approach: to borrow, as far as is practical, from formal mathematical reasoning. Explicit reasoning about correctness has been almost entirely absent from mainstream data engineering (despite the fact that this field rests on formal foundations such as relational algebra and type systems). Adjacent fields like compilers, databases, and distributed systems treat formal reasoning as a first-class tool. But data and machine learning pipelines largely don’t.

There are however a few reasonable objections to this. The first is that Data and ML pipelines cannot be fully proven, some parts depend on things we don’t control: upstream sources, the stochastic nature of models, data/concept shifts and drifts to name a few. And the second is that a proof is just as good as the assumptions we make at the start. And the assumptions we make in the data field are bound to change.

So I am not proposing we formally prove the entirety of a pipeline, what I’m suggesting here is a middle way.

This middle way looks at correctness as a ladder, where we are clear about which tradeoffs we are making when we skip an important step. As we’ve defined it above, this 3-step ladder depends on the problem at hand. And once again, skipping any one step in this ladder is a choice that should be explicitly made rather than being implicit:

Prove what is provable → Check at every boundary → Monitor the rest

Step 1 - Prove what is provable

A proof is an argument that a statement is true for every possible input (not just the inputs you tried). The next article here describes what to prove and how to prove it, for now let’s just define what we mean by “prove”.

In slightly more formal words:
Given a system defined by a function f, that accepts a defined set of inputs I
- under a defined precondition P over its inputs,
- and a postcondition Q over the input/output pairs.
Proving is establishing that

∀ x ∈ I; P(x) => Q(x, f(x))

In other words, that for every possible input, if the precondition is satisfied by the input, the postcondition is satisfied (by the input/output pair).

Testing instead of proving is equivalent to checking Q(x_i, f(x_i)) is true for a finite set of chosen inputs x_i that satisfy P.

The formal method crowd will recognise Hoare’s triple here, but the goal is not to get too formal. Of all the steps, this one invites the most criticism. Engineers will usually argue that proving would take too long, whereas mathematicians and theorists will usually argue that proving is worth the tradeoff. The reason we’re taking a middle way here is that we do not want to tradeoff time either.

Let me just explain the reasoning here:
We have to see this as a spectrum that goes from informal types of reasoning (like an intuitive explanation in a markdown documentation) to the more rigorous formal methods (such as using machine checkable proofs).

And the good news, in my experience, is that the cheaper methods find most of the errors, in other words: just spending time reasoning about the problem and writing an informal argument usually catches the majority of potential errors. Writing down in plain words why a job should be deterministic is often enough to find out it isn’t.

In between those ends, we find types that make invalid states unrepresentable, arguments by induction over partitions etc. (more details in the next article).

If we breakdown the system we are building into multiple components. We want anyone reading the design to see where every component is and why it is there.

In the next article, I will go into details about:

• What properties you should prove (such as idempotence, determinism etc.).

• What arguments you can use for different scenarios.

Where testing fits here: tests can help us make the argument stay more up to date with the implementation. The proof holds for the design, but the code may still drift from it. Property-based testing comes handy here, with libraries such as Hypothesis in Python. The property under test is the proof’s own statement: if P(x), then Q(x, f(x)), run on thousands of generated inputs that satisfy P. The test still only samples, but it samples something different: not the world, but whether the code faithfully follows an argument that already covers every input.

In summary,

Before writing code, and before writing tests, we should try to argue for the most important properties, the recipe in the next post gives an example of this, followed by the final post that goes into more details on using formal arguments.

Step 2 - Check at every boundary

None of the proving we did earlier ensures that the input satisfies the precondition. That is where checking comes. The moment an input reaches our system, we must check that it satisfies the preconditions we depend on, otherwise we reject it.

A boundary here is anywhere data is handed over: where it arrives from an upstream source we don’t own, where it passes from one module to another, where it is written to a table other people read, where it is fed to a model, served to a user etc.

A check is not a test: the two are easy to confuse. A test runs before deployment, on inputs we chose. A check runs in the deployment environment, and includes real inputs. A test asks: does my code behave on the examples I can think of? A test samples the domain. A check attempts to guard this domain instead.

This is in line with designing by contract,
a concept introduced by Bertrand Meyer in the mid-eighties. The current work on data contracts (popularised by Andrew Jones and Chad Sanderson) applies this idea to data, and is very useful. But a contract should say much more. A contract that says “quantity is an integer” is a very weak contract, it’s a simple type declaration.

A contract in Meyer’s terms binds two parties: the supplier guarantees the postcondition only if the client satisfies the precondition. A data system thus acts as a client of the upstream sources, and a supplier for the downstream users of the system. And as such it needs to define the contracts at both boundaries.

A useful contract has to define:

• Structure: fields, types, nullability, keys.

• Meaning: ranges, units, allowed values, uniqueness, and the relationships between fields and between tables (for example: every order refers to a customer that exists, a refund never exceeds the original payment).

• Delivery: when the data should arrive, how much of it we expect, and how complete it should be.

All the above should be applied to: the preconditions (write guarantees: rules that a data producer or client must satisfy before writing/streaming),
the postconditions (read guarantees: promises a data system or processing function makes upon completing a query or transformation, e.g., output dataset row counts match expected aggregates, null-free columns),
and the invariants (think here about the consistency of storage and states: rules at the level of the system itself, that must always remain true at rest or between mutations, these are things such as foreign key integrity, partition balance, chronological event ordering etc.)

Two principles complete the checking step:

1. Parse don’t validate: popularised by Alexis King in her blog post “parse, don’t validate” is about ensuring we cannot represent invalid states through parsing. Once parsed, the precondition is satisfied.

2. Reject rather than repair: as much as you can, if a check fails, reject the record. Fixing silently may end up hiding errors.

In summary,

A proof is only as good as its preconditions, a check ensures those preconditions hold in practice. Check every boundary, reject rather than repair (not always but as much as possible), parse don’t validate.

Step 3 - Monitor what is outside of your control

Some things will be outside of your control. Data drift, sources that change meaning with no notification, the stochastic nature of ML components whose behaviour is best described by black magic etc.

The proofs and checks rest on assumptions. Monitoring is simply about watching those assumptions. My criticism of monitoring is only because a lot of the time we make it the whole strategy. If you have gone through the previous steps, then you know by now what to monitor and alert for.

Monitoring what you cannot prevent is a necessity.
Monitoring what you could prevent is laziness, manifested as a dashboard.

On ownership in monitoring: monitoring and alerting should always have clear ownership. We have to be clear on who will take action, when they must take action and how they will know (for example: an alert email or so, or if they routinely check a dashboard every Tuesday morning), and what the action is. Setting a dashboard that nobody looks at is as good as doing nothing.

Working backwards from monitoring: when we find issues through monitoring, it is the opportunity to add checks and to revise the architecture/design of the system.

In summary,

Derive what you must monitor from your assumptions. Give every monitoring modality or alert an owner, and treat every recurring alert as an invitation to move that error back up the ladder, to prevent it from happening in the first place.

None of this is new

Nothing in this series is new. If anything, this argument has existed for decades, but the data professions seem to ignore most of this work. Proofs, contracts and checks are decades old.

So why has the data and machine learning world not taken them in (for the most part)? Not because they don’t apply, but because we are able to get away without them, until the day we’re not. Data errors rarely crash, they rather mislead, quietly. And when they do crash they create a new problem that we build another system to solve. But perhaps also because the data and machine learning fields have a lot of stochastic components built in that we use as an excuse. Add to that the speed at which we’re trying to build things today to “keep up”. That is exactly why we need the discipline more, not less. The theory is there, and so are the tools. What is missing is the habit.

I am in no way claiming:

• That every data pipeline should be formally proven.

• That tests and monitoring are useless. They are just not preventive, and proofs do not replace them.

• That this is faster. It costs some more design time up front. Which I believe is much cheaper in the long term, but I have no study to prove it. So do see this as a hypothesis, and if you have any evidence in either direction, I would love to hear from you.

In the next article we will go through a more detailed step by step recipe for building data systems that prevent errors, using a worked example.

Leave a comment

Subscribe now

How we use a central protobuf schema to make our data pipeline less chaotic

If I told you one of the most popular TV Shows in Sweden is about following the moose as they travel, for 21 days, you might say:
 — Stop it with the stereotypes!
or :
— Ismail, go home and stop experimenting with psychedelics

I promise, this TV show is the real deal and it’s only 21 days long.

The TV Show “den stora älg vandringen” (which translates to “the great moose walk”) is a type of TV Show known as slow TV. This trend started in the neighboring lovely country of Norway (at NRK).

This TV show is 21 days long and it was streamed live on SVTs (Swedish Television’s) streaming service SVT Play.

So why collect data then?
It turns out filming the moose is not the only thing that SVT does and SVT Play is just one of the services at SVT. These services range from streaming to delivering news in addition to games and services for children and more. The point is, those apps make different types of content available (news articles, movies, TV series etc.) on different platforms including web, iOS, Android and smart TVs.

So a little more than two years ago, a project for data collection at SVT got started. This was an effort to aid decision making in the multiple products of SVT (that used to rely on third-party services such as google analytics).

Of course, the data is not what makes or breaks the products, nor the shows, movies or news articles. It is there to verify assumptions, to understand the users better and to support the decision making.

As one can see, the diversity of SVTs services and the platforms on which they run adds multiple challenges to the collection of data.

Too long; didn’t read?

If you prefer watching this as a video feel free to head over here.

While there are many things to talk about when it comes to the collection and analysis of data, this post will focus on:
— A very brief intro to how we collect data at SVT.
— The chaos that can result from the changes in the business rules.
—How we use a central schema (with Protobuf) to define the data we collect and propagate this definition in the whole pipeline (from collection to storage).
 — Some of the issues we encountered with that choice.

A Brief on how we collect data at SVT

Let’s say you’re a news editor, and you want to see how your articles perform. How can we collect data about that?

For example, you would want to know: how many users read an article? We cannot know for sure if a user actually reads the article, but what we can do is estimate it from what we can collect such as: a mouse click on the article or a mouse scroll inside the article.

These things that we can collect are what we call “Events”. Other example events for other products can be:

• A video start or a video pause in the video player.

• A search query in a search bar.

• Or other events for collecting video player telemetry information.

An event is an action that happened on the client application side, for which the corresponding information can be collected (for example: a click or a video start)

These events are just observations, and from them we should be able to either calculate or approximate our metrics (an example of a metric that cannot be calculated exactly is the percentage watched of a video because we simply do not know if the user is actually watching)

So we send those events. We ingest them into our pipeline (more details later). We store them in a table. We process and analyze them and then we present them to the users of our solution.

So the event will go through:

• collection: on the user-facing application side. For this part, in our case, we built SDKs (mainly for iOS, Android, Web and smart TVs) that will collect events such as clicks and send them to an API, which brings us to the next part.

• ingestion: when the API gets the previous event it publishes it to a message queue. In our case, we are currently using Pub/Sub on the google cloud platform (GCP). We have creatively named the previous API: the event API.

• storage: a service gets the previous event from Pub/Sub, checks the validity of its content, then stores it in a table (a BigQuery table in our case). We have called this service judge-judi.

• analysis: mostly just a bunch of ETL

• presentation: at this stage, we can show a dashboard to the right person with the right metric.

For the purpose of this post, we will only focus on the collect — ingest — store parts of the previous pipeline.

To keep things simple, we can assume that we send one event at a time from the client application (In reality, those events are gathered by the client application and sent together in a sequence of events)

So, with all this in mind, where can problems happen?

The need for ONE (versioned) definition of the event data

From what we’ve talked about in the previous section, here’s what the life of an event looks like from collection to storage:
 — First, we send the event from an app (through our SDK) to the Event API.
 — The Event API then adds it to the queue for it to be processed later.
 — And then on the right, is an accurate picture of judge-judi (the service described in the ingestion part above) performing its duty of pulling and storing valid events in the table.

The first problem is that all parts of the pipeline, from collection to storage, have to agree on what the definition of an event is.

The first problem: consistency

Let’s assume that the event looks something like:

{
    'user_id':   '00000000',
    'content_type': 'video',
    'event_type': 'video_start',    
}

Given the event above. What if one of our products ends up sending an event by naming the type eventType instead of event_type?

{
    'user_id':   '00000000',
    'content_type': 'video',
    'eventType': 'video_start',    
}

When this event reaches the validation stage at judge-judi. The service is expecting a field event_type it will just conclude that the field is not there.

So the first problem is having all parts of the pipeline agree on the definition of the data.

We also have a multitude of products in SVT (svtplay, news etc.) and different platforms (web, iOS, Android etc.) it would be hard to make sure that the SDKs are always consistent.

The second problem: Murphy’s law of data engineering

Which states that: “If the data schema can change, the data schema will change.” Now, Murphy did not necessarily say that.
Well, he never actually did, but I’m sure he would agree.

The business rules and goals change more often than teenagers’ relationships. We should expect, as a result, that the definition of the event will change. And it will change often. New fields will be added, old fields will be removed, others merged together, and the list goes on and on.

These modifications put schemas, and therefore the definition of an event, in a state of perpetual change. Each change will require adjustment to the pipeline. We do not want these recurring changes in the definition of the data to break the whole (or part of) the pipeline. Ideally, we would want to preserve backward and forward compatibility whenever possible.

And then problems specific to JSON

• The type of a field in JSON is unclear and errors can happen. It would be preferable if we can provide guarantees of what the event structure to be expected is.

• Because we have different platforms, we are using different programming languages. This means that the JSON body can differ depending on the programming language and libraries used.

• Long serialization overhead on the Event API’s side with the use of JSON. Although I currently do not have the “data” to prove it, it is easy to see that the schema is sent with every event.

But since we have control over the applications and have control over the schema definition how can we make use of that?

The solution: centralizing the definition of an event with Protobuf

One way of tackling the issues above is to do the following:

• One repository where the definition of an event, which we will call the Event Model, is kept (we just keep it in a repository on Gitlab).

• This event model is versioned.

• Every version of the event model is then propagated across all the elements of the pipeline. For example, the web SDK will have access to an event model library and every new version in the event model repository will correspond to a new version in the library. This version should be the same for all platforms.

• We will preferably use a language-neutral definition for the event model’s definition that can enable us to serialize an event on the SDKs and then deserialize it on the side of judge-judi.

• Preferably adding or removing fields from the event model should not break backward/forward compatibility.

To make a choice we tried both Avro and Protobuf. While there are many choices when it comes to defining the schema, to have a definition that supports all of the languages and provides a “good enough” serialization: Protobuf was the tool of choice.

What is protobuf and how does it work?

Protocol buffers are a language-neutral, platform-neutral, extensible way of serializing structured data. At least that’s what they’re supposed to be.

We will not go too deeply in how protocol buffers work but here are the basics.

Let’s say Alice wants to send a letter message to Bob that would look like this in JSON:

{
  'firstName': 'Alice',
  'lastName': 'Alice',
  'email': 'alice@alice.alice',
  'content': 'Hi Bob, it's Alice'
}

To do so, in protobuf, Alice and Bob would have to agree on a message schema. This schema can look something like this:

message Letter{
  required string firstName = 1;
  required string lastName = 2;
  optional string email = 3;
  required string content = 4;
}

This letter schema will be stored in a .proto file.

Alice and Bob can then convert this message to C++ or Javascript code or whatever language they’re into using the proto compiler.

Using the generated code Alice will serialize the message. For example, if Alice is using C++ her code might look something like this:

Letter letter;
letter.set_first_name("Alice");

Alice can now send the letter to Bob. A series of bytes are sent over the network. Bob knows the Letter schema, Bob can deserialize the letter message.

So how can we use it for the solution above?

Step 1 — In our case, the event model repository hosts the .proto file.

Step 2 — This repository propagates code with the same for the rest of the pipeline through pushing libraries. This repository also creates the table on BigQuery (as we will mention in the next section, this part has changed).

Step 3 — The service judge-judi will also fetch the latest Go version of the event model because it will need it later. And, just like Alice, through the SDKs we can fetch the latest version of this event model on the client apps to serialize the events that we will send. The app will serialize the event and send it to the API.

Step 4 — We collect this event as it is, without deserializing it, and we publish it to the queue on Pub/Sub.

Step 5 — The service judge-judi pulls the events and deserializes them.

It’s all good now, you drop the mic, walk out of the room, head high…until…stuff happens. You realize that the reality is as far from your expectations as your tongue is from touching your nose… So close, and yet so far (don’t try that, you’ll look ridiculous)

Expectations vs. reality

The myth of language agnosticism

When the mechanism of protocol buffers is supposed to be language-agnostic and platform agnostic. In practice, this does not seem to be the case.

For example, some platforms had encoding that was not consistent with the deserializer. We use the gogofaster compiler for Go which is a third party implementation that leverages type-specific marshaling code for extra performance. But we use it mainly to provide more canonical Go structures, less typing and to not end up with the infamous xxx fields.

The myth of backward and forward compatibility

Let’s say a field gets added, and a field gets removed from the schema. If the service judge-judi is using the latest version of the event model, it will still be able to deserialize the data with no problem even if the client application does not update to the latest version. But this is only true for simple changes. For example, this would not hold if we change the nesting of some fields.

Moreover, although we avoided talking about the analysis that comes after the storage part earlier, a new event model would mean that all the jobs and tables that depend on it will change. For this reason, we separated the version of the event model from the version of the event table on BigQuery so that the table would change less often. We then provide a mapping on judge-judi from the event model to the table schema before storing the events.

Thus, we identify mainly three types of changes in the event model’s schema and act accordingly:
 — Non-breaking changes: that happen regularly. These are the changes that do not break backward compatibility. This can be the deletion or addition of fields in the protobuf schema.
 — Breaking changes that do not require a change of the BigQuery table: that rarely happen, and should be minimized. When these happen judge-judi just changes the mapping between the event and the table.
 — Breaking changes that require a change of the BigQuery table: very rare and have yet to happen but they could (thus they will). Since those are rare and we cannot know what these will be like we’ll tackle them on a case by case basis.

Conclusion

The reality of data collection and analysis is very different from the mathematical fantasies of well-defined axioms and controllable assertions.

Coming from a scientific background, we have a tendency to want to keep the pipeline clean. And that’s good. However, chaos in data engineering is not just real but probably unavoidable. And it is one of the reasons why we should try to iterate slowly, and add tests and data quality checks when possible. It is also why keeping a central definition of the data in one place can be very helpful.

Protocol buffers are one way of defining a data model that seems to improve serialization overhead and be almost language agnostic. If you already know what the data schema should look like, protobuf is good at managing additions and deletions of fields, although one has to be careful as always. More complex changes will require more thoughtfulness.

Hope you found something useful reading this… Have a beautiful day!

If you have any comments or suggestions would love to hear from you.

Leave a comment

Make your data pipeline less chaotic by one versioned definition of data was originally published in The SVT Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.