I've been sitting with a paper of ours, Measuring What the Crawler Sees (TPDL), and I want to write about the part that nobody puts in the abstract: the algebra is fun. Underneath the crawl statistics there's a two-parameter urn model that is almost embarrassingly well-behaved, and once you notice the pattern you start seeing the same little operator everywhere. This post is about that operator — the complement, the humble overbar — and the symmetry it enforces.

The urn, in one breath

Picture the web the crawler can reach as an urn of N slots. Each round, two things happen:

  • Sampling. You draw a fraction c of the slots and write down what you saw. c is coverage — how much of the reachable web this crawl actually touches.
  • Churn. Of everything in the urn, a fraction α survives to the next round; the remaining α is replaced by fresh URLs. α is survival — how sticky the web is.

That's it. Two numbers, c and α. Everything the paper measures — pairwise overlap between crawls, the discovery curve, core-vs-shell structure — falls out of these two.

The first nice thing: c and α are genuinely independent knobs. Coverage is about your instrument (how wide you cast the net); survival is about the world (how fast pages die). Nothing in the model couples them. So any slot in any round meets exactly one of three fates, and they multiply out cleanly:

  • replaced — probability α
  • survives and is sampled — probability α·c
  • survives but is missed — probability α·c

Add them up:

α + αc + αc = α + α(c + c) = α + α = 1

The bookkeeping closes because the two knobs never interfere.

Meet the bar

That overbar is the whole story. Define, for any probability x, its complement:

x ≡ 1 − x

So c is the miss rate and α is the churn rate. The bar is an involution — bar it twice and you're back where you started, 1 − (1 − x) = x — and it's the only nonlinear thing in the entire model. Every quantity of interest is built by (a) multiplying cs and αs together, and (b) barring the result.

Here's where it gets pleasant. Apply the bar to a construction, not just to a symbol. Take the "persists-unseen" rate αc — a page survived but you didn't see it. Its complement is the resolution rate: the chance that this round settles the page one way or another (either it's gone, or you finally saw it).

1 − αc = α + αc

Read the right-hand side out loud: churned out (α) or survived and sampled (αc). The bar didn't just flip a number; it reorganized the three fates into "resolved / unresolved." And of course the dual holds by the same move:

1 − αc = α + αc

Same three atoms — α, αc, αc — regrouped along a different seam. The bar is a lens: point it at αc and you partition the world into seen-or-dead vs. hiding; point it at αc and you get missed-or-dead vs. captured. Two readings of one urn.

The symmetry between the complements

The reason the algebra feels symmetric is that c and α enter the survival/sampling story in mirror-image roles, and the bar respects the mirror. Pairwise containment — the fraction of a crawl's URLs still present Δt rounds later — is

g(Δt) = c · αΔt

c sets the height (how much you saw to begin with), α sets the decay (how fast it slips away). Coverage is a scale, survival is a rate, and they sit in the same product without ever mixing. The discovery curve — the cumulative footprint over a window — is a messier-looking but still closed form,

Û(t)/M = (t+1)·α / (α+αc) + αc·(1 − (αc)t+1) / (α+αc)2

but stare at it and you see the same cast of characters: a linear term governed by churn α, a geometric transient governed by the persists-unseen rate αc, all normalized by the resolution rate α+αc = 1−αc. Nothing new is introduced. The whole apparatus is {c, α} and the bar.

Characteristic times are odds

Now the part I actually wanted to write down, because it reframes how I think about time constants in general.

Ask: how many rounds does a page live? Survival each round is α, so lifetime is geometric, and its expectation is

ℓ = α / α

Look at the shape of that: α over α is α/(1−α) — the odds of survival. A life expectancy is not really a "time"; it's the odds of the thing that keeps you alive. High-survival worlds (α → 1) send α → 0 and the odds blow up, exactly as a lifetime should.

Once you've seen it as an odds, every characteristic time in the paper turns out to have the same skeleton — something divided by bar-something:

  • Life expectancy — survival odds: ℓ = α / α
  • Sampled time — expected re-fetches in a lifetime: η = c·ℓ = αc / α
  • Hidden time — expected rounds alive but unseen: η = c·ℓ = αc / α
  • Hiding time per resolution — average length of an unseen spell: τh = αc / αc = αc / (1 − αc)

That last one wears the pattern on its sleeve: the hiding time is the persists-unseen rate αc divided by its own bar, αc = 1 − αc. Every one of them is X over X for some construction X. That's the abstract punchline: a characteristic time is a rate divided by its own complement. The bar isn't decoration — it's the machine that manufactures the denominator. Time constants are odds, and odds are exactly "the thing / the bar of the thing."

And because sampled and hidden time share the same α / α factor, it cancels when you take their ratio, leaving something almost startlingly clean:

η / η = c / c

The ratio of time-seen to time-hidden is just the coverage odds — c/(1−c) — with survival gone entirely. Take a page's whole life, and the fraction of it you witness depends only on your instrument, not on how long the page lives. The world's stickiness sets the timescale; your coverage sets what fraction of that timescale is visible. α and c divide the labor perfectly, and the bar is the seam along which they separate.

Why I find this satisfying

It's the same reason odds are nicer than probabilities for a lot of reasoning: the bar turns addition-constrained quantities (probabilities that must sum to one) into free multiplicative ones (odds that just multiply). The urn model is a little worked example of that philosophy. Two independent knobs, one involution, and out fall containment, discovery, life, and hiding — each one either a product of {c, α} or an odds X over X.

You could measure a crawl a hundred ways. The paper's quiet claim is that if the urn is homogeneous, all those ways agree, because they're all the same two numbers wearing different bars. And when they disagree — when the containment fit and the discovery fit hand you different (c, α) — that disagreement is itself the measurement: it's the world telling you the urn was never homogeneous to begin with.

And our results do say that. But there's a constructive flip side worth hinting at, because the disagreement isn't a dead end — it's a partition waiting to happen. Imagine you could tag each domain with its own lifetime and sort the urn into shells of equal life expectancy: all the URLs with the same ℓ = α/α in one sub-urn, the shorter-lived in the next. Each shell is homogeneous by construction, so its containment collapses back to a single clean c·αΔt — with a sharp horizon around its own Δt_ℓ, a complete cut-off past the point where that shell's pages simply don't live long enough to be seen again. The messy aggregate curve is just a sum over shells: a set with one lifetime, plus the set of all URLs living shorter, and so on down. Because a homogeneous urn sits in that tidy linear regime, the clearest way to cut the urn is precisely the one that makes each sub-urn homogeneous with respect to Δt — partition by lifetime, and the algebra of the single bar comes back, one shell at a time. That's a post for another day.

Small algebra, big leverage. That's my favorite kind.

The paper is arXiv:2607.13636. If you want the actual crawl data and the fits, that's where to look — this post is just me enjoying the machinery. There's also a Common Crawl write-up putting the same ideas to work on real coverage numbers.