Endorser blocks: forging, propagation, acquiring, certifying

2026-10-20-20-42-81787-w42notxsub-value-leios-workload-all-n2n-100k1m-1i1o1w-nomadperf-dira-nom
maxEndorserBlockReferencesSize limited · 100.0% full, against 55.5% of maxEndorserBlockTxsSize

Setup

Topology

51 pools · 3 regions: AP 17, EU 17, US 17 · torus-dense · 306 producer entries = 204 links (102 reciprocated) · 153 intra-region links, 51 inter-region · every pool has exactly 8 link partners · diameter 4, radius 4, mean shortest path 2.48 hops · peer sharing off (PeerSharing false in the node's configuration), so a node talks only and always to its listed producers

Full graph

EUUSAP

Intra-region links only

EUUSAP

Inter-region links only

EUUSAP

Node positions are identical in all three views; hover a node to highlight its links everywhere and read its degrees. Region colours: EU green, US red, AP blue. Links are drawn undirected; direction and each pool's producer list are in the adjacency table.

Adjacency table
NodeRegionOut intraOut interIn intraIn interProducers (outbound)
node-0EU4242node-1, node-2, node-3, node-48, node-15, node-33
node-1US4242node-2, node-0, node-4, node-49, node-16, node-34
node-2AP4242node-0, node-1, node-5, node-50, node-17, node-35
node-3EU4242node-4, node-5, node-6, node-0, node-18, node-36
node-4US4242node-5, node-3, node-7, node-1, node-19, node-37
node-5AP4242node-3, node-4, node-8, node-2, node-20, node-38
node-6EU4242node-7, node-8, node-9, node-3, node-21, node-39
node-7US4242node-8, node-6, node-10, node-4, node-22, node-40
node-8AP4242node-6, node-7, node-11, node-5, node-23, node-41
node-9EU4242node-10, node-11, node-12, node-6, node-24, node-42
node-10US4242node-11, node-9, node-13, node-7, node-25, node-43
node-11AP4242node-9, node-10, node-14, node-8, node-26, node-44
node-12EU4242node-13, node-14, node-15, node-9, node-27, node-45
node-13US4242node-14, node-12, node-16, node-10, node-28, node-46
node-14AP4242node-12, node-13, node-17, node-11, node-29, node-47
node-15EU4242node-16, node-17, node-18, node-12, node-30, node-48
node-16US4242node-17, node-15, node-19, node-13, node-31, node-49
node-17AP4242node-15, node-16, node-20, node-14, node-32, node-50
node-18EU4242node-19, node-20, node-21, node-15, node-33, node-0
node-19US4242node-20, node-18, node-22, node-16, node-34, node-1
node-20AP4242node-18, node-19, node-23, node-17, node-35, node-2
node-21EU4242node-22, node-23, node-24, node-18, node-36, node-3
node-22US4242node-23, node-21, node-25, node-19, node-37, node-4
node-23AP4242node-21, node-22, node-26, node-20, node-38, node-5
node-24EU4242node-25, node-26, node-27, node-21, node-39, node-6
node-25US4242node-26, node-24, node-28, node-22, node-40, node-7
node-26AP4242node-24, node-25, node-29, node-23, node-41, node-8
node-27EU4242node-28, node-29, node-30, node-24, node-42, node-9
node-28US4242node-29, node-27, node-31, node-25, node-43, node-10
node-29AP4242node-27, node-28, node-32, node-26, node-44, node-11
node-30EU4242node-31, node-32, node-33, node-27, node-45, node-12
node-31US4242node-32, node-30, node-34, node-28, node-46, node-13
node-32AP4242node-30, node-31, node-35, node-29, node-47, node-14
node-33EU4242node-34, node-35, node-36, node-30, node-48, node-15
node-34US4242node-35, node-33, node-37, node-31, node-49, node-16
node-35AP4242node-33, node-34, node-38, node-32, node-50, node-17
node-36EU4242node-37, node-38, node-39, node-33, node-0, node-18
node-37US4242node-38, node-36, node-40, node-34, node-1, node-19
node-38AP4242node-36, node-37, node-41, node-35, node-2, node-20
node-39EU4242node-40, node-41, node-42, node-36, node-3, node-21
node-40US4242node-41, node-39, node-43, node-37, node-4, node-22
node-41AP4242node-39, node-40, node-44, node-38, node-5, node-23
node-42EU4242node-43, node-44, node-45, node-39, node-6, node-24
node-43US4242node-44, node-42, node-46, node-40, node-7, node-25
node-44AP4242node-42, node-43, node-47, node-41, node-8, node-26
node-45EU4242node-46, node-47, node-48, node-42, node-9, node-27
node-46US4242node-47, node-45, node-49, node-43, node-10, node-28
node-47AP4242node-45, node-46, node-50, node-44, node-11, node-29
node-48EU4242node-49, node-50, node-0, node-45, node-12, node-30
node-49US4242node-50, node-48, node-1, node-46, node-13, node-31
node-50AP4242node-48, node-49, node-2, node-47, node-14, node-32

Latency

cardano-cli ping round trips over the node-to-node handshake, measured at cluster start by each pool's healthcheck against its producers: 918 samples over 6 region pairs. Milliseconds.

LinknminmedianmeanmaxSDSEMTCP connect
within AP (Sydney)2040.370.510.541.590.1480.0100.26
within EU (Frankfurt)2040.410.680.801.990.3120.0220.51
within US (N. Virginia)2040.420.620.661.810.1900.0130.37
AP ↔ EU102304.02308.40307.82310.111.5340.152307.52
AP ↔ US102214.49217.22217.36220.021.5860.157217.08
EU ↔ US10289.9890.8090.8391.680.3890.03890.56

Not an ICMP ping: cardano-cli ping opens a TCP connection and runs the node-to-node handshake, so each sample is an application-level round trip through both nodes' network stacks and mux, which is what the Cardano protocols actually experience. The TCP connect column is the bare TCP round trip from the same tool, the closest stand-in for a raw ping. SEM = SD / sqrt(n). The spread is 607x, from 0.51 ms within AP to 308.40 ms between AP and EU, and it is the floor under every interval in this report.

tx-centrifuge

The load generator, a closed-loop driver. A workload is a self-contained circuit: its own disjoint partition of funds, single-use UTxOs, its own builder turning funds into transactions, its own token-bucket rate limit, its own injection targets and its own observer following the chain. Every transaction spends funds and creates outputs, and an output becomes spendable again only once that workload's observer has seen its transaction reach the confirmation depth, so the transactions form chains gated by the chain itself and the funds budget is the buffer bridging confirmation latency. With on_exhaustion = error, a workload that cannot recycle funds as fast as it spends them runs that buffer to zero and fails the run rather than quietly slowing down.

The generator runs on the explorer machine in eu-central-1, which forges nothing, so every transaction crosses the network before the pool that must place it in a block ever sees it: the round trip a workload pays is the EU row for its target's region in the table above. Each workload holds two buffers of its own, funds waiting to be built and built transactions waiting to be delivered.

1. Forging, generated load

Every EB forging found, nothing filtered. A block that lost its height battle, that was rolled away from, or that no other node ever adopted still counts.

Totals

3,291RBs forged
3,254includes EB hash
98.9%of all RBs
37no EB hash
3,242distinct EB hashes
12repeated EBs forged

By region

regionRBs forgedshare of all RBsincludes EB hashof its own RBs
AP1,08733.0%1,07899.2%
EU1,09533.3%1,08298.8%
US1,10933.7%1,09498.6%

Endorser block contents

What an endorser block holds. An endorser block is a list of (tx hash, size) references and nothing else, so its size is that list's size.

minp50p90maxmean
EB size (bytes)9,18299,97599,97599,97597,142
EB full, of the 100,000 B reference cap9.2%100.0%100.0%100.0%97.1%
tx references per EB2552,7772,7772,7772,698
bytes per reference36.0036.0036.0036.0136.00
derived closure bytes, references × 200 B51,000555,400555,400555,400539,666
derived closure full, of the 1,000,000 B transaction cap5.1%55.5%55.5%55.5%54.0%

2,617 of 3,254 EB forgings sit exactly at the run's largest EB, 2,777 references.

The reference list is what binds this run. Its transactions are 200 B each, so its largest endorser block holds 2,777 of them: 99,975 B of references against the 100,000 B cap (100.0%), and 555,400 B of transactions against the 1,000,000 B the closure allows (55.5%). The one that reaches 100% is the one that stopped the endorser block from growing.

The other run of this pair, 2026-10-21-16-22-81787-w42notxsub-value-leios-workload-all-n2n-100k1m-2i2o2w-nomadperf-dira-nom, is the control for exactly this: its transactions are 374 B, its largest endorser block holds 2,673 references for 98,904 B (98.9% of the reference cap), and their closure is 999,702 B (100.0% of the closure cap). With 200 B transactions the reference list fills first, with 374 B transactions the closure does, off the same two caps. So the binding cap moved with the transaction size, and nothing else changed.

2. Propagation, fetching the EB references

Three instants are used:

Endorser blocks whose hash was forged more than once, and so referenced by more than one ranking block, are left out of this section.

Announcing

The forger alone: from forging the endorser block to sending its first announcement of it.

samplesminp50p75p90p99p99.9maxmean
Forger: forged to its first announcement3,2302 ms11 ms14 ms19 ms28 ms48 ms58 ms13 ms

Fetching

From the forger's first announcement of the endorser block to that endorser block first arriving at a peer, one sample per node and endorser block, 161,500 of them. An endorser block is its list of transaction references, and that list is what arrives; the transactions themselves are fetched separately and are not in this section. Repeated deliveries of the same endorser block are counted under Activity, never timed here.

samplesminp50p75p90p99p99.9maxmean
announced, to the EB arriving at a node161,50028 ms1.18 s1.67 s2.19 s2.98 s5.00 s14.86 s1.33 s

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first received it, measured from the forger's first announcement as above. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers327 ms1.02 s1.13 s1.53 s2.21 s3.23 s5.57 s1.02 s
to 50% of its peers714 ms1.26 s1.59 s1.75 s2.56 s3.68 s5.71 s1.31 s
to 75% of its peers892 ms1.63 s2.14 s2.21 s2.97 s4.71 s7.23 s1.70 s
to 90% of its peers1.03 s1.83 s2.26 s2.41 s3.35 s6.14 s7.92 s1.87 s
to 100% of its peers1.10 s2.28 s2.76 s3.16 s4.69 s9.26 s14.86 s2.28 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers161,50028 ms1.18 s1.67 s2.19 s2.98 s5.00 s14.86 s1.33 s
Peers in AP53,83263 ms1.46 s1.78 s2.25 s3.09 s5.06 s9.26 s1.44 s
Peers in EU53,85028 ms1.25 s1.63 s2.18 s2.99 s5.00 s14.86 s1.32 s
Peers in US53,81850 ms1.09 s1.58 s1.95 s2.84 s4.72 s10.99 s1.22 s

Activity

The same events counted instead of timed, and counted per node and endorser block: for one endorser block, how many announcements a node sent out, and how many times it fetched the endorser block. Anything above one is a repeat the protocol paid for.

306.0announcements per EB, all nodes
6.00announcements per EB, per node
90.9fetches per EB, all nodes
1.82fetches per EB, per node

Times a node fetched the same endorser block

Inbound: the node pulling the endorser block from a peer. One means it fetched it once, more than one means it fetched the same endorser block again.

times it fetched itcasesshare of cases
1 time93,71458.0%
2 times33,34220.6%
3 times18,50611.5%
4 times or more15,9389.9%

293,447 fetches in all, of which 131,947 brought an endorser block the node already had. The most one node fetched a single endorser block was 21 times.

By region of the receiving node

regioncasesannouncements sentper casefetchesper caserepeats
AP53,832322,9926.00109,2272.0350.7%
EU53,850323,0996.0095,3811.7743.5%
US53,818322,9086.0088,8391.6539.4%

3. Acquiring, the EB written down

A delivered endorser block is not yet held: until its write is durable the body still reads as not-held, which is why the same endorser block can be fetched again meanwhile. Consensus.LeiosKernel.BlockAcquired is the moment that ends: the node has the reference list stored and claims to hold it, registering which referenced transactions it still needs. This is the step from arrival to that claim, which includes waiting for the single writer, and how often the same endorser block was stored again.

Writing

samplesminp50p75p90p99p99.9maxmean
the EB arriving, to the node writing it161,5003 ms64 ms102 ms191 ms1.34 s2.39 s4.34 s127 ms

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first stored it, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers7 ms55 ms63 ms77 ms136 ms429 ms600 ms57 ms
to 50% of its peers11 ms65 ms80 ms114 ms721 ms1.21 s1.86 s90 ms
to 75% of its peers17 ms95 ms130 ms298 ms1.26 s2.02 s2.62 s168 ms
to 90% of its peers30 ms136 ms188 ms615 ms1.65 s2.58 s3.03 s244 ms
to 100% of its peers35 ms261 ms465 ms1.62 s3.07 s4.03 s4.34 s532 ms

By region

samplesminp50p75p90p99p99.9maxmean
All peers161,5003 ms64 ms102 ms191 ms1.34 s2.39 s4.34 s127 ms
Peers in AP53,8323 ms59 ms84 ms145 ms1.16 s2.21 s3.62 s106 ms
Peers in EU53,8508 ms72 ms125 ms248 ms1.46 s2.67 s4.16 s153 ms
Peers in US53,8184 ms63 ms102 ms182 ms1.34 s2.31 s4.34 s121 ms

Activity

Counted per node and endorser block, like section 2: how many times a node wrote the same endorser block down. This one is local, nothing crosses the network.

69.2writes per EB, all nodes
1.38writes per EB, per node
times it was writtencasesshare of cases
1 time124,09676.8%
2 times22,11613.7%
3 times10,1966.3%
4 times or more5,0923.2%

223,477 writes in all, of which 61,977 were of an endorser block the node had already written. The most one node wrote a single endorser block was 19 times.

By region of the writing node

regioncaseswritesper case
AP53,83274,2811.38
EU53,85076,6221.42
US53,81872,5741.35

4. Closures, the transactions an EB references

An endorser block is a list of references. The transactions themselves are a separate transfer, and a node has the closure once it holds all of them: Consensus.LeiosKernel.BlockTxsAcquired. Counted per node and endorser block, on the same endorser blocks as sections 2 and 3.

Obtaining

samplesminp50p75p90p99p99.9maxmean
EB fetched to the closure obtained161,50010 ms532 ms771 ms1.05 s1.93 s3.37 s6.80 s614 ms

A voter has until 7 s after the announcing block's slot onset to hold and validate the transactions: lHdrWait 3 s then lVoteWindow 4 s, both fixed in LeiosVoting.hs rather than read from genesis. 104 of 161,500 closures were obtained later than that, counted from the endorser block's forging.

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first obtained the closure, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers21 ms502 ms529 ms586 ms983 ms1.12 s2.29 s393 ms
to 50% of its peers26 ms539 ms711 ms980 ms1.25 s2.04 s3.19 s627 ms
to 75% of its peers39 ms743 ms1.02 s1.10 s1.72 s2.61 s4.19 s786 ms
to 90% of its peers52 ms834 ms1.08 s1.39 s2.24 s3.49 s4.79 s903 ms
to 100% of its peers134 ms1.11 s1.62 s2.51 s3.94 s5.74 s6.80 s1.38 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers161,50010 ms532 ms771 ms1.05 s1.93 s3.37 s6.80 s614 ms
Peers in AP53,83211 ms532 ms840 ms1.06 s1.82 s3.27 s6.16 s560 ms
Peers in EU53,85018 ms546 ms733 ms1.07 s2.02 s3.64 s6.80 s672 ms
Peers in US53,81810 ms523 ms718 ms1.03 s1.92 s3.10 s5.59 s609 ms

Activity

51.8closures per EB, all nodes
1.04closures per EB, per node
3,696obtained more than once
0never obtained

Times a node obtained the same closure

times it obtained itcasesshare of cases
1 time157,80497.7%
2 times2,7701.7%
3 times5800.4%
4 times or more3460.2%

167,189 closures obtained in all, of which 5,689 were of one the node already had. The most one node obtained a single closure was 17 times.

By region of the obtaining node

regioncasesclosures obtainedper case
AP53,83256,6871.05
EU53,85055,0701.02
US53,81855,4301.03

5. Certifying

Counted on the blocks the whole cluster took. A block qualifies when all 50 nodes other than its forger adopted it, every peer it has of the 51 in the cluster: 3,128 of the run's 3,291 walkable blocks. Both sides of the ratio are held to it, so an announcement made by a block only part of the cluster held, and a certificate carried by one, are both out. No battle or fork filter is applied: nothing here asks what happened at a contested height, and the walk the blocks come from keeps every side of every fork.

Totals

3,081EBs announced
1,128certificate carried
36.6%of announcements
1,953never carried

By region

region of the announcerEBs announcedcertifiedshare
AP1,02038637.8%
EU1,01729729.2%
US1,04444542.6%

What happened to every endorser block

stagecount% of announced% of its groupgroupvotes received
Endorser blocks announced3,242100.0%···
    A voting window opened3,242100.0%···
Reached quorum1,96660.6%··39–51 of 51, median 50
    Certificate carried by a later block1,12934.8%57.4%reached quorum·
    Quorum reached, never carried83725.8%42.6%reached quorum·
Never reached quorum1,27639.4%··0–50 of 51, median 18
    voters declined: chainTipDoesNotAnnounce14,930·22.9%no-quorum voters·
    voters declined: tooLate27,521·42.2%no-quorum voters·
    voters declined: voteRejected: AlreadyKnown31·0.0%no-quorum voters·
    voters declined: voteRejected: all 1 votes rejected31·0.0%no-quorum voters·
    voters voted for it anyway22,680·34.8%no-quorum voters·

Hover a stage for what it counts and which trace says so. The stages down to Never reached quorum count endorser blocks; the rows under it count voter decisions, one per trace event, which is why they sum past 100%. Nothing is gated here, not even the run's opening epochs, so these counts are wider than section 2's.

When the transactions reach a node's own chain

The end of the line for an endorser block: a node adopts a ranking block that carries its certificate, and only then are those transactions on that node's chain. Measured from the endorser block's forging, over the 56,100 cases, one node and one endorser block each, where it happened.

samplesminp50p75p90p99p99.9maxmean
forged, to a node adopting a block that certifies it56,10014.20 s27.33 s40.51 s58.04 s97.64 s150.84 s217.98 s33.13 s
AP18,68814.22 s27.37 s40.53 s58.18 s97.71 s150.86 s217.98 s33.23 s
EU18,78314.20 s27.33 s40.49 s58.04 s97.46 s150.66 s217.50 s33.12 s
US18,62914.23 s27.28 s40.48 s57.88 s97.31 s150.52 s217.63 s33.04 s

6. Forks

Every switch a node made off a chain it already had, one row per node and switch (ChainDB.AddBlockEvent.SwitchedToAFork). A contest is one pair of tips, so a contest the whole cluster lived through counts once. Both rates below are contests per 1,000 blocks on the chain, so runs of different length compare directly.

24.8 expected from activeSlotsCoeff 0.0520-20 81787 100k1m 1i1oslot24.9height24.321-16 81787 100k1m 2i2oslot23.3height23.00102030

The slot rate is the control and the height rate is the signal: two leaders in one slot is fixed by activeSlotsCoeff and cannot be engineered away, while a height battle is a node forging on a parent a sibling already had, which is propagation losing a race and is free to move. Reading the height rate without the control beside it cannot tell a propagation regression from a run that simply drew more forks.

What the nodes paid

112.7switches per node
104 to 122fewest to most, one node
24.9slot battles per 1,000 chain blocks
24.3height battles per 1,000 chain blocks