Designing Hot-Reloadable Per-Tenant Routing

TL;DR. MEV-Boost, the middleware validators use to source blocks from a builder market, applied one global relay list to every validator. An operator whose clients trust different relays had to run a full duplicate stack (consensus client and MEV-Boost) per relay policy: infrastructure and operational cost that scaled with every client.

I designed it proposal-first (acceptance criteria and failure semantics settled before any code), then shipped per-tenant relay routing that folds those N stacks into one instance and reconfigures live, with no restart, so relays rotate without dropping a validator or missing a block. The core is an immutable routing table swapped atomically: lock-free reads, live configuration updates, and a failed reload that leaves the last known-good config serving. It sits behind a two-method interface, with pluggable config sources and a test suite built from the design’s decision table.

It ran in institutional production. The benchmarks show a -26% geomean latency change, reported in full: a large GetPayload win and a smaller register-path cost. Upstream it was praised but never merged, so it runs as a production fork.

Builders submit blocks through relays, trusted intermediaries, and each validator picks the most profitable one. Stock MEV-Boost takes one global list of relays at startup and applies it to every validator behind it.

At institutional scale that single list becomes the problem.

Although this lives inside Ethereum infrastructure, the problem is generic, and you do not need to know Ethereum to follow the rest: a latency-sensitive, multi-tenant router whose configuration changes while the service runs, the same shape as a feature-flag service, a rate limiter, or an upstream pool.

The missing capability

A large operator runs validators for many clients, and the clients don’t trust the same relays. The person who filed the request put it plainly:

One customer “trusts” relays 3,4,5 as sources of MEV. Another customer “trusts” relays 1,2,3. Right now, the only way to handle this is to have different consensus clients for the two customers with differently associated MEV boost sidecars.

That workaround is the whole cost. One relay policy meant one full stack: a consensus client plus its own MEV-Boost. Two policies meant two stacks; ten meant ten. Double the infrastructure and double the operational surface to express what amounts to a lookup.

This was not a speculative refactoring. Institutional customers needed different trust policies without multiplying operational infrastructure.

Before any code, I wrote the design proposal in issue #455 that the implementation would follow. It captured the customer requirements, acceptance criteria, architecture, failure semantics, interoperability strategy, test matrix, and benchmark plan. The key architectural decisions were already made there; the code became the exercise of realizing them. One MEV-Boost, each validator routed to its own relays.

The constraints were the design

The proposal didn’t start from code. It started from acceptance criteria the operator signed off on, and those criteria forced every later decision:

• Different keys on the same client can point to different relay sets.
• No new attack surface: security can’t regress.
• Minimal impact on the latency-sensitive path: block-building has a deadline.
• Optional: anyone can keep running MEV-Boost exactly as before (backward compatible).
• Relay changes adopted quickly, with no disruption: no restart.

Read those together and the design is almost forced. “No restart” plus “minimal latency impact” means reads dominate and config has to swap live, in place, without a lock the readers contend on. “No regression, adopted quickly” means a bad config must never take down a running service. The interesting problem was never the implementation; it was finding one design that satisfied all five at once: per-tenant lookup, zero read-path locking, zero-downtime reload, and a bad update that can’t hurt you.

That last one the proposal made an explicit rule, not an implementation detail. On a malformed or unreachable config the service holds the last good one instead of going dark:

The Relay Registry won’t be updated if the configuration is malformed or invalid, in which case the previous valid configuration is used.

Availability over freshness: a stale-but-valid route beats no route. At startup the choice inverts: with no valid config at all, fail fast and refuse to start.

From there everything flows one way:

acceptance criteria
   ↓
decision table
   ↓
architecture
   ↓
tests
   ↓
benchmarks

The shape of it

Three packages carry the design. rcm is the manager and the reload loop; rcp is the config providers; relay is the domain (entries, sets, the registry). The base server reaches all of it through one read-only interface.

(The design sketch models the config provider as an interface; the shipped code realizes it as an equivalent func type, covered in the providers section below.)

A big feature behind a two-method seam

The whole subsystem is roughly 4,600 lines. The base server sees almost none of it. Its request handlers depend on a small read-only interface (server/service.go):

// RelayConfigurator provides relays for a given proposer.
type RelayConfigurator interface {
	RelaysForProposer(publicKey relay.ValidatorPublicKey) relay.List
	AllRelays() relay.List
}

Two methods: “relays for this validator” and “every relay I might talk to.” The request handlers depend on those and nothing else. Configuration loading, validation, hot reload, storage: all of it hides behind this seam, so the feature lands without changing the latency-sensitive request path.

Where config comes from: pluggable providers

Config has to arrive from somewhere, and “somewhere” varies: a static list in the simplest deployment, a file on disk, an HTTP API in production. Rather than an interface with implementations, the provider is just a function type (config/rcm/rcm.go):

// ConfigProvider provides relay configuration.
type ConfigProvider func() (*relay.Config, error)

Anything that can produce a config satisfies it. Three do, wired by a single CLI flag:

• rcp.NewDefault(relays) wraps a static set and preserves stock single-list behaviour.
• rcp.NewFile(path) reads a proposer-config JSON file.
• rcp.NewJSONAPI(client, url) pulls proposer config from an HTTP endpoint.

Each is a struct with a single FetchConfig method, wired in by one CLI flag. Because ConfigProvider is a func type, a new source is just a new function, with no interface to implement.

The providers don’t invent a config format either. They parse the proposer-config JSON that Teku already uses. Reusing an existing schema instead of minting one is the difference between an operator wiring in a file they already have and yet another format to learn. That kind of interop decision lives in a design doc, not a diff.

The core: lock-free copy-on-write reload

The mechanism is small. The manager holds the live registry in a single atomic.Value (config/rcm/configurator.go):

type Configurator struct {
	registryCreator *RegistryCreator
	relayRegistry   atomic.Value
}

A reload never mutates the registry in place. It builds a whole new, immutable registry off to the side, then swaps one pointer:

func (m *Configurator) SyncConfig() error {
	// fetch + validate + build a fresh registry
	r, err := m.registryCreator.Create()
	if err != nil {
		return fmt.Errorf("cannot syncronise configuration: %w", err)
	}

	m.relayRegistry.Store(r) // atomically replace registry

	return nil
}

Reads load the current pointer and use it. No mutex, no read-side contention:

func (m *Configurator) RelaysForProposer(
	publicKey relay.ValidatorPublicKey) relay.List {
	return m.loadRegistry().RelaysForProposer(publicKey)
}

func (m *Configurator) loadRegistry() RelayRegistry {
	r, ok := m.relayRegistry.Load().(RelayRegistry)
	if !ok {
		panic("unexpected relay registry type")
	}
	return r
}

(RelayRegistry is the same two-method contract as the RelayConfigurator seam from earlier, here on the registry’s own side of it.)

That is all there is to it, and it discharges the design’s criteria directly:

• No lock on the read path. Readers do an atomic load, not a mutex acquire; under a storm of block requests there is nothing to contend on.
• Zero downtime. A reload is one Store; requests in flight finish on the old registry, and the GC reclaims it once the last reader lets go. No stop-the-world, no window where lookups fail.
• Correct by construction. The registry is immutable after Create, so there is no torn read to reason about; the only shared write is the pointer swap, and atomic.Value makes that safe.
• The failure semantics fall out of where Store sits. It runs only after Create fetches and validates a config, so a bad fetch never reaches the swap and the last good registry stays live: the availability-over-freshness rule from the design, now just a consequence of ordering.

The obvious alternative, an RWMutex around a mutable registry, puts every read on a contended lock. Reads are constant and reloads are rare, so the better trade is to make reads free and pay on the write: rebuild, then swap. Copy-on-write plus an atomic swap is the pattern for read-heavy, rarely-written state on a hot path, not specific to relays.

Immutability also carries the “no new attack surface” criterion. Routing is a local lookup into a read-only registry, and config is validated before it is published, so nothing mutable is shared across tenants and no request can steer another validator’s relays.

Driving it is a small syncer on a ticker (config/rcm/sync.go), default interval half an epoch, built with functional options and a clean context-cancellation shutdown.

The tests are the spec, executed

The interesting part starts before any test code. The proposal ended with a decision table: for every combination of proposer config, proposer relays, default config, and default relays, use the proposer’s relays, fall back to the default set, build a local block with no MEV, or reject the config as invalid. Eight valid rows, four invalid ones. A slice of it:

Proposer relays	Default relays	Result
enabled, filled	enabled, filled	use proposer relays
absent	enabled, filled	fall back to default
absent	disabled / empty	no MEV, build locally
enabled, empty	any	reject: invalid config

The implementation never invented new behaviour. Every routing test is a direct translation of one row, so the subtest names read back as the table itself:

# routing
proposer with relays is enabled & default with relays is enabled -> proposer
proposer is not found & default with relays is enabled -> default
proposer is not found & default with no relays is disabled -> empty

# invalid config, rejected up front
it returns an error if a proposer builder is enabled but has no relays
it returns an error if a default builder is enabled but has no relays

# reliability
it uses the previously stored relays if synchronisation error occurs
it is thread-safe

You can reconstruct the design from the test output, including the two reliability rows: keep the last good config when a sync fails, and stay correct under concurrent reads.

The tests run in parallel, and reconfiguration is tested without the usual sleep trap. A mock provider scripts a sequence of configs, so one test drives “config A, then B, then C” across successive syncs with no wall-clock waiting.

The load-bearing claim is the swap’s: reads take no lock. A test has to earn it. runConcurrently manufactures the contention, one goroutine per core pounding the same Configurator at once, each with a seeded RNG so a failing interleaving reproduces instead of vanishing on the next run:

func runConcurrently(
	numOfWorkers int, num int64, fn func(*rand.Rand, int64)) {
	var wg sync.WaitGroup
	wg.Add(numOfWorkers)

	for g := numOfWorkers; g > 0; g-- {
		// seeded per worker, so failures reproduce
		r := rand.New(rand.NewSource(int64(g)))
		go func(r *rand.Rand) {
			defer wg.Done()
			for n := int64(1); n <= num; n++ {
				fn(r, n)
			}
		}(r)
	}
	wg.Wait()
}

The workload each goroutine runs ten thousand times is a random mix of the single writer and the two readers, all against that one shared Configurator:

runConcurrently(runtime.NumCPU(), 10_000, func(r *rand.Rand, n int64) {
	switch {
	case r.Int63n(n)%2 == 0:
		require.NoError(t, sut.SyncConfig()) // writer: build + swap
	case r.Int63n(n)%3 == 0:
		sut.RelaysForProposer(reltest.RandomBLSPublicKey(t).String())
	default:
		sut.AllRelays()
	}
})

Every core swaps the registry and loads it at once, tens of thousands of times, under -race: if a reader could ever see a half-updated registry, the detector says so. goleak.VerifyTestMain fails the run if the sync goroutine outlives its test. Background reconfiguration usually breaks two ways: a data race, or a leaked goroutine. Both fail loudly here.

More than half of the implementation is tests and scaffolding, none of it there before. A hot-reloadable, concurrent subsystem is hard to reason about after the fact, so it is built to prove the intended behaviour, not assume it.

Did it pay off?

Yes, and the numbers are public, but the honest reading matters more than the headline. The benchmarks in issue #455 compare the new path against the base with benchstat (ten runs, significance tested):

operation	before	after	delta
RegisterValidator-8	108.0µs	134.6µs	+24.63% (p=0.000)
GetHeader-8	1.762ms	1.792ms	~ (p=0.579, no change)
GetPayload-8	1651.4µs	515.4µs	-68.79% (p=0.000)
geomean	679.8µs	499.1µs	-26.59%

The geomean is carried by GetPayload (-69%). GetHeader didn’t move. And RegisterValidator got about 25% slower, which is not a mystery: routing is not free, and that path is where it gets paid for. The old handler blasted the whole registration payload to every relay. The new one demultiplexes first, grouping each validator’s registration by its own relay set before fanning out, so a relay only hears about the validators that route to it.

That extra bookkeeping is the +25% time and the +60% allocations on the register path, and it buys the per-tenant routing the feature is for. It also lands where it is affordable. registerValidator runs ahead of proposal duties, off the block-building deadline; the two calls that actually race that deadline in-slot, getHeader and getPayload, stayed flat and dropped sharply. The one acceptance criterion a routing layer could have threatened, minimal impact on the latency-sensitive path, holds exactly where it counts.

Quoting the -26% geomean on its own would hide the register-path regression, which is cherry-picking. That is why the whole table is here, not just the number. The regression was an acceptable trade for the win on the payload path, and the number is only worth stating because it is measured with significance and reproducible from the repo’s own benchmarks. The maintainer’s reaction on the thread:

Btw, love the speed improvements of 25% on the side 🔥

Where it landed

The design ran in institutional production on a live validator fleet. The stock client shipped no metrics, so the fork added the Prometheus instrumentation it lacked, with Grafana dashboards on top that made the routing and proposal paths observable under load.

Upstream, the pull request never merged, and not for lack of enthusiasm. The maintainers were positive throughout (“I like it a lot,” on the record); a change this deep was simply more than the project chose to take onto the critical path of block production. That is theirs to decide, and it changed nothing for us. The capability was already live in production, so we kept it on a fork and kept operating. Delivery never depended on the merge, which on infrastructure that must not fail is the entire point. The design and the review are public, on #455 and #470.

The takeaway travels

Rename the nouns and none of this is about Ethereum. What generalizes is the approach: start from operational constraints, not code; make the shared state immutable and swap it atomically; keep the request path behind the smallest interface that works; translate the decision table straight into tests; and report the trade-offs, regressions included.

That is the kind of infrastructure work I enjoy most: reducing operational constraints to a small set of invariants, then building systems whose behaviour stays easy to reason about under failure. The relays were just one instance; the design approach is the reusable part.