Endorser blocks: forging, propagation, acquiring, certifying

2026-10-22-23-20-e651d-w42notxsub-value-leios-workload-all-n2n-40k260k-1i1o1w-nomadperf-dira-nom
maxEndorserBlockReferencesSize limited · 99.9% full, against 85.4% of maxEndorserBlockTxsSize

Setup

Topology

51 pools · 3 regions: AP 17, EU 17, US 17 · torus-dense · 306 producer entries = 204 links (102 reciprocated) · 153 intra-region links, 51 inter-region · every pool has exactly 8 link partners · diameter 4, radius 4, mean shortest path 2.48 hops · peer sharing off (PeerSharing false in the node's configuration), so a node talks only and always to its listed producers

Full graph

EUUSAP

Intra-region links only

EUUSAP

Inter-region links only

EUUSAP

Node positions are identical in all three views; hover a node to highlight its links everywhere and read its degrees. Region colours: EU green, US red, AP blue. Links are drawn undirected; direction and each pool's producer list are in the adjacency table.

Adjacency table
NodeRegionOut intraOut interIn intraIn interProducers (outbound)
node-0EU4242node-1, node-2, node-3, node-48, node-15, node-33
node-1US4242node-2, node-0, node-4, node-49, node-16, node-34
node-2AP4242node-0, node-1, node-5, node-50, node-17, node-35
node-3EU4242node-4, node-5, node-6, node-0, node-18, node-36
node-4US4242node-5, node-3, node-7, node-1, node-19, node-37
node-5AP4242node-3, node-4, node-8, node-2, node-20, node-38
node-6EU4242node-7, node-8, node-9, node-3, node-21, node-39
node-7US4242node-8, node-6, node-10, node-4, node-22, node-40
node-8AP4242node-6, node-7, node-11, node-5, node-23, node-41
node-9EU4242node-10, node-11, node-12, node-6, node-24, node-42
node-10US4242node-11, node-9, node-13, node-7, node-25, node-43
node-11AP4242node-9, node-10, node-14, node-8, node-26, node-44
node-12EU4242node-13, node-14, node-15, node-9, node-27, node-45
node-13US4242node-14, node-12, node-16, node-10, node-28, node-46
node-14AP4242node-12, node-13, node-17, node-11, node-29, node-47
node-15EU4242node-16, node-17, node-18, node-12, node-30, node-48
node-16US4242node-17, node-15, node-19, node-13, node-31, node-49
node-17AP4242node-15, node-16, node-20, node-14, node-32, node-50
node-18EU4242node-19, node-20, node-21, node-15, node-33, node-0
node-19US4242node-20, node-18, node-22, node-16, node-34, node-1
node-20AP4242node-18, node-19, node-23, node-17, node-35, node-2
node-21EU4242node-22, node-23, node-24, node-18, node-36, node-3
node-22US4242node-23, node-21, node-25, node-19, node-37, node-4
node-23AP4242node-21, node-22, node-26, node-20, node-38, node-5
node-24EU4242node-25, node-26, node-27, node-21, node-39, node-6
node-25US4242node-26, node-24, node-28, node-22, node-40, node-7
node-26AP4242node-24, node-25, node-29, node-23, node-41, node-8
node-27EU4242node-28, node-29, node-30, node-24, node-42, node-9
node-28US4242node-29, node-27, node-31, node-25, node-43, node-10
node-29AP4242node-27, node-28, node-32, node-26, node-44, node-11
node-30EU4242node-31, node-32, node-33, node-27, node-45, node-12
node-31US4242node-32, node-30, node-34, node-28, node-46, node-13
node-32AP4242node-30, node-31, node-35, node-29, node-47, node-14
node-33EU4242node-34, node-35, node-36, node-30, node-48, node-15
node-34US4242node-35, node-33, node-37, node-31, node-49, node-16
node-35AP4242node-33, node-34, node-38, node-32, node-50, node-17
node-36EU4242node-37, node-38, node-39, node-33, node-0, node-18
node-37US4242node-38, node-36, node-40, node-34, node-1, node-19
node-38AP4242node-36, node-37, node-41, node-35, node-2, node-20
node-39EU4242node-40, node-41, node-42, node-36, node-3, node-21
node-40US4242node-41, node-39, node-43, node-37, node-4, node-22
node-41AP4242node-39, node-40, node-44, node-38, node-5, node-23
node-42EU4242node-43, node-44, node-45, node-39, node-6, node-24
node-43US4242node-44, node-42, node-46, node-40, node-7, node-25
node-44AP4242node-42, node-43, node-47, node-41, node-8, node-26
node-45EU4242node-46, node-47, node-48, node-42, node-9, node-27
node-46US4242node-47, node-45, node-49, node-43, node-10, node-28
node-47AP4242node-45, node-46, node-50, node-44, node-11, node-29
node-48EU4242node-49, node-50, node-0, node-45, node-12, node-30
node-49US4242node-50, node-48, node-1, node-46, node-13, node-31
node-50AP4242node-48, node-49, node-2, node-47, node-14, node-32

Latency

cardano-cli ping round trips over the node-to-node handshake, measured at cluster start by each pool's healthcheck against its producers: 918 samples over 6 region pairs. Milliseconds.

LinknminmedianmeanmaxSDSEMTCP connect
within AP (Sydney)2040.360.500.531.820.1550.0110.25
within EU (Frankfurt)2040.420.620.681.170.1760.0120.42
within US (N. Virginia)2040.410.620.641.670.1750.0120.36
AP ↔ EU102284.36289.00291.17300.044.8720.482290.84
AP ↔ US102202.02209.39210.33220.025.0560.501210.07
EU ↔ US10290.0790.8490.8491.760.3430.03490.54

Not an ICMP ping: cardano-cli ping opens a TCP connection and runs the node-to-node handshake, so each sample is an application-level round trip through both nodes' network stacks and mux, which is what the Cardano protocols actually experience. The TCP connect column is the bare TCP round trip from the same tool, the closest stand-in for a raw ping. SEM = SD / sqrt(n). The spread is 581x, from 0.50 ms within AP to 289.00 ms between AP and EU, and it is the floor under every interval in this report.

tx-centrifuge

The load generator, a closed-loop driver. A workload is a self-contained circuit: its own disjoint partition of funds, single-use UTxOs, its own builder turning funds into transactions, its own token-bucket rate limit, its own injection targets and its own observer following the chain. Every transaction spends funds and creates outputs, and an output becomes spendable again only once that workload's observer has seen its transaction reach the confirmation depth, so the transactions form chains gated by the chain itself and the funds budget is the buffer bridging confirmation latency. With on_exhaustion = error, a workload that cannot recycle funds as fast as it spends them runs that buffer to zero and fails the run rather than quietly slowing down.

The generator runs on the explorer machine in eu-central-1, which forges nothing, so every transaction crosses the network before the pool that must place it in a block ever sees it: the round trip a workload pays is the EU row for its target's region in the table above. Each workload holds two buffers of its own, funds waiting to be built and built transactions waiting to be delivered.

1. Forging, generated load

Every EB forging found, nothing filtered. A block that lost its height battle, that was rolled away from, or that no other node ever adopted still counts.

Totals

3,402RBs forged
3,372includes EB hash
99.1%of all RBs
30no EB hash
3,369distinct EB hashes
3repeated EBs forged

By region

regionRBs forgedshare of all RBsincludes EB hashof its own RBs
AP1,15133.8%1,14299.2%
EU1,11232.7%1,10299.1%
US1,13933.5%1,12899.0%

Endorser block contents

What an endorser block holds. An endorser block is a list of (tx hash, size) references and nothing else, so its size is that list's size.

minp50p90maxmean
EB size (bytes)7,20239,96339,96339,96339,900
EB full, of the 40,000 B reference cap18.0%99.9%99.9%99.9%99.8%
tx references per EB2001,1101,1101,1101,108
bytes per reference36.0036.0036.0036.0136.00
derived closure bytes, references × 200 B40,000222,000222,000222,000221,651
derived closure full, of the 260,000 B transaction cap15.4%85.4%85.4%85.4%85.3%

3,360 of 3,372 EB forgings sit exactly at the run's largest EB, 1,110 references.

The reference list is what binds this run. Its transactions are 200 B each, so its largest endorser block holds 1,110 of them: 39,963 B of references against the 40,000 B cap (99.9%), and 222,000 B of transactions against the 260,000 B the closure allows (85.4%). The one that reaches 100% is the one that stopped the endorser block from growing.

The other run of this pair, 2026-10-23-18-49-e651d-w42notxsub-value-leios-workload-all-n2n-40k260k-2i2o2w-nomadperf-dira-nom, is the control for exactly this: its transactions are 374 B, its largest endorser block holds 695 references for 25,718 B (64.3% of the reference cap), and their closure is 259,930 B (100.0% of the closure cap). With 200 B transactions the reference list fills first, with 374 B transactions the closure does, off the same two caps. So the binding cap moved with the transaction size, and nothing else changed.

2. Propagation, fetching the EB references

Three instants are used:

Endorser blocks whose hash was forged more than once, and so referenced by more than one ranking block, are left out of this section.

Announcing

The forger alone: from forging the endorser block to sending its first announcement of it.

samplesminp50p75p90p99p99.9maxmean
Forger: forged to its first announcement3,3661 ms5 ms7 ms9 ms15 ms27 ms68 ms6 ms

Fetching

From the forger's first announcement of the endorser block to that endorser block first arriving at a peer, one sample per node and endorser block, 168,300 of them. An endorser block is its list of transaction references, and that list is what arrives; the transactions themselves are fetched separately and are not in this section. Repeated deliveries of the same endorser block are counted under Activity, never timed here.

samplesminp50p75p90p99p99.9maxmean
announced, to the EB arriving at a node168,30012 ms1.08 s1.51 s1.67 s2.22 s3.17 s4.91 s1.14 s

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first received it, measured from the forger's first announcement as above. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers282 ms1.04 s1.07 s1.14 s1.52 s2.36 s2.61 s914 ms
to 50% of its peers619 ms1.08 s1.52 s1.56 s1.63 s2.69 s3.00 s1.14 s
to 75% of its peers702 ms1.36 s1.66 s1.69 s2.15 s3.07 s3.20 s1.40 s
to 90% of its peers762 ms1.55 s1.79 s2.17 s2.57 s3.34 s4.51 s1.56 s
to 100% of its peers798 ms1.70 s2.20 s2.49 s3.54 s4.28 s4.91 s1.81 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers168,30012 ms1.08 s1.51 s1.67 s2.22 s3.17 s4.91 s1.14 s
Peers in AP56,08217 ms1.15 s1.56 s1.71 s2.30 s3.23 s4.60 s1.23 s
Peers in EU56,12412 ms1.08 s1.45 s1.68 s2.21 s3.20 s4.91 s1.13 s
Peers in US56,09417 ms1.04 s1.24 s1.58 s2.11 s3.01 s4.28 s1.06 s

Activity

The same events counted instead of timed, and counted per node and endorser block: for one endorser block, how many announcements a node sent out, and how many times it fetched the endorser block. Anything above one is a repeat the protocol paid for.

306.0announcements per EB, all nodes
6.00announcements per EB, per node
82.4fetches per EB, all nodes
1.65fetches per EB, per node

Times a node fetched the same endorser block

Inbound: the node pulling the endorser block from a peer. One means it fetched it once, more than one means it fetched the same endorser block again.

times it fetched itcasesshare of cases
1 time110,88965.9%
2 times27,52216.4%
3 times16,3029.7%
4 times or more13,5878.1%

277,334 fetches in all, of which 109,034 brought an endorser block the node already had. The most one node fetched a single endorser block was 8 times.

By region of the receiving node

regioncasesannouncements sentper casefetchesper caserepeats
AP56,082336,4926.00103,0501.8445.6%
EU56,124336,7446.0092,6631.6539.4%
US56,094336,5646.0081,6211.4631.3%

3. Acquiring, the EB written down

A delivered endorser block is not yet held: until its write is durable the body still reads as not-held, which is why the same endorser block can be fetched again meanwhile. Consensus.LeiosKernel.BlockAcquired is the moment that ends: the node has the reference list stored and claims to hold it, registering which referenced transactions it still needs. This is the step from arrival to that claim, which includes waiting for the single writer, and how often the same endorser block was stored again.

Writing

samplesminp50p75p90p99p99.9maxmean
the EB arriving, to the node writing it168,3002 ms22 ms30 ms42 ms105 ms421 ms2.68 s27 ms

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first stored it, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers3 ms19 ms23 ms26 ms36 ms78 ms102 ms19 ms
to 50% of its peers3 ms22 ms26 ms31 ms46 ms129 ms167 ms23 ms
to 75% of its peers7 ms27 ms35 ms42 ms85 ms289 ms341 ms30 ms
to 90% of its peers9 ms35 ms46 ms60 ms185 ms488 ms613 ms42 ms
to 100% of its peers15 ms60 ms92 ms135 ms1.58 s2.27 s2.68 s99 ms

By region

samplesminp50p75p90p99p99.9maxmean
All peers168,3002 ms22 ms30 ms42 ms105 ms421 ms2.68 s27 ms
Peers in AP56,0822 ms22 ms30 ms41 ms97 ms467 ms2.68 s27 ms
Peers in EU56,1243 ms22 ms30 ms44 ms120 ms430 ms2.27 s28 ms
Peers in US56,0942 ms21 ms29 ms41 ms100 ms374 ms2.45 s27 ms

Activity

Counted per node and endorser block, like section 2: how many times a node wrote the same endorser block down. This one is local, nothing crosses the network.

64.1writes per EB, all nodes
1.28writes per EB, per node
times it was writtencasesshare of cases
1 time136,27581.0%
2 times19,43811.5%
3 times10,0426.0%
4 times or more2,5451.5%

215,653 writes in all, of which 47,353 were of an endorser block the node had already written. The most one node wrote a single endorser block was 7 times.

By region of the writing node

regioncaseswritesper case
AP56,08274,1921.32
EU56,12473,0221.30
US56,09468,4391.22

4. Closures, the transactions an EB references

An endorser block is a list of references. The transactions themselves are a separate transfer, and a node has the closure once it holds all of them: Consensus.LeiosKernel.BlockTxsAcquired. Counted per node and endorser block, on the same endorser blocks as sections 2 and 3.

Obtaining

samplesminp50p75p90p99p99.9maxmean
EB fetched to the closure obtained168,3004 ms510 ms561 ms809 ms1.23 s2.26 s4.24 s505 ms

A voter has until 7 s after the announcing block's slot onset to hold and validate the transactions: lHdrWait 3 s then lVoteWindow 4 s, both fixed in LeiosVoting.hs rather than read from genesis. 0 of 168,300 closures were obtained later than that, counted from the endorser block's forging.

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first obtained the closure, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers7 ms498 ms509 ms515 ms537 ms822 ms2.01 s340 ms
to 50% of its peers8 ms512 ms528 ms601 ms1.01 s1.07 s2.03 s533 ms
to 75% of its peers11 ms525 ms712 ms941 ms1.09 s1.86 s2.06 s626 ms
to 90% of its peers13 ms605 ms869 ms1.02 s1.39 s2.38 s2.68 s713 ms
to 100% of its peers26 ms927 ms1.05 s1.51 s2.64 s3.29 s4.24 s962 ms

By region

samplesminp50p75p90p99p99.9maxmean
All peers168,3004 ms510 ms561 ms809 ms1.23 s2.26 s4.24 s505 ms
Peers in AP56,0826 ms508 ms605 ms926 ms1.36 s2.10 s3.29 s435 ms
Peers in EU56,1247 ms512 ms560 ms826 ms1.22 s2.31 s3.93 s569 ms
Peers in US56,0944 ms508 ms532 ms725 ms1.08 s2.31 s4.24 s511 ms

Activity

51.0closures per EB, all nodes
1.02closures per EB, per node
2,962obtained more than once
0never obtained

Times a node obtained the same closure

times it obtained itcasesshare of cases
1 time165,33898.2%
2 times2,6181.6%
3 times3250.2%
4 times or more190.0%

171,627 closures obtained in all, of which 3,327 were of one the node already had. The most one node obtained a single closure was 5 times.

By region of the obtaining node

regioncasesclosures obtainedper case
AP56,08258,7601.05
EU56,12456,2071.00
US56,09456,6591.01

5. Certifying

Counted on the blocks the whole cluster took. A block qualifies when all 50 nodes other than its forger adopted it, every peer it has of the 51 in the cluster: 3,271 of the run's 3,402 walkable blocks. Both sides of the ratio are held to it, so an announcement made by a block only part of the cluster held, and a certificate carried by one, are both out. No battle or fork filter is applied: nothing here asks what happened at a contested height, and the walk the blocks come from keeps every side of every fork.

Totals

3,239EBs announced
1,613certificate carried
49.8%of announcements
1,626never carried

By region

region of the announcerEBs announcedcertifiedshare
AP1,10254349.3%
EU1,05154752.0%
US1,08652348.2%

What happened to every endorser block

stagecount% of announced% of its groupgroupvotes received
Endorser blocks announced3,369100.0%···
    A voting window opened3,369100.0%···
Reached quorum2,95287.6%··39–51 of 51, median 51
    Certificate carried by a later block1,61547.9%54.7%reached quorum·
    Quorum reached, never carried1,33739.7%45.3%reached quorum·
Never reached quorum41712.4%··0–37 of 51, median 0
    voters declined: chainTipDoesNotAnnounce17,387·81.8%no-quorum voters·
    voters declined: tooLate64·0.3%no-quorum voters·
    voters voted for it anyway3,816·17.9%no-quorum voters·

Hover a stage for what it counts and which trace says so. The stages down to Never reached quorum count endorser blocks; the rows under it count voter decisions, one per trace event, which is why they sum past 100%. Nothing is gated here, not even the run's opening epochs, so these counts are wider than section 2's.

When the transactions reach a node's own chain

The end of the line for an endorser block: a node adopts a ranking block that carries its certificate, and only then are those transactions on that node's chain. Measured from the endorser block's forging, over the 80,601 cases, one node and one endorser block each, where it happened.

samplesminp50p75p90p99p99.9maxmean
forged, to a node adopting a block that certifies it80,60113.92 s27.25 s40.20 s58.52 s101.68 s127.29 s167.68 s33.10 s
AP26,86313.94 s27.19 s40.23 s58.49 s103.68 s127.49 s167.26 s33.16 s
EU26,85713.92 s27.25 s40.21 s58.63 s101.23 s127.30 s167.68 s33.10 s
US26,88113.99 s27.31 s40.13 s58.52 s101.38 s127.16 s167.59 s33.05 s

6. Forks

Every switch a node made off a chain it already had, one row per node and switch (ChainDB.AddBlockEvent.SwitchedToAFork). A contest is one pair of tips, so a contest the whole cluster lived through counts once. Both rates below are contests per 1,000 blocks on the chain, so runs of different length compare directly.

24.8 expected from activeSlotsCoeff 0.0522-23 e651d 40k260k 1i1oslot25.3height11.223-18 e651d 40k260k 2i2oslot20.9height6.30102030

The slot rate is the control and the height rate is the signal: two leaders in one slot is fixed by activeSlotsCoeff and cannot be engineered away, while a height battle is a node forging on a parent a sibling already had, which is propagation losing a race and is free to move. Reading the height rate without the control beside it cannot tell a propagation regression from a run that simply drew more forks.

What the nodes paid

84.4switches per node
70 to 95fewest to most, one node
25.3slot battles per 1,000 chain blocks
11.2height battles per 1,000 chain blocks