Endorser blocks: forging, propagation, acquiring, certifying

2026-10-21-16-22-81787-w42notxsub-value-leios-workload-all-n2n-100k1m-2i2o2w-nomadperf-dira-nom
maxEndorserBlockTxsSize limited · 100.0% full, against 98.9% of maxEndorserBlockReferencesSize

Setup

Topology

51 pools · 3 regions: AP 17, EU 17, US 17 · torus-dense · 306 producer entries = 204 links (102 reciprocated) · 153 intra-region links, 51 inter-region · every pool has exactly 8 link partners · diameter 4, radius 4, mean shortest path 2.48 hops · peer sharing off (PeerSharing false in the node's configuration), so a node talks only and always to its listed producers

Full graph

EUUSAP

Intra-region links only

EUUSAP

Inter-region links only

EUUSAP

Node positions are identical in all three views; hover a node to highlight its links everywhere and read its degrees. Region colours: EU green, US red, AP blue. Links are drawn undirected; direction and each pool's producer list are in the adjacency table.

Adjacency table
NodeRegionOut intraOut interIn intraIn interProducers (outbound)
node-0EU4242node-1, node-2, node-3, node-48, node-15, node-33
node-1US4242node-2, node-0, node-4, node-49, node-16, node-34
node-2AP4242node-0, node-1, node-5, node-50, node-17, node-35
node-3EU4242node-4, node-5, node-6, node-0, node-18, node-36
node-4US4242node-5, node-3, node-7, node-1, node-19, node-37
node-5AP4242node-3, node-4, node-8, node-2, node-20, node-38
node-6EU4242node-7, node-8, node-9, node-3, node-21, node-39
node-7US4242node-8, node-6, node-10, node-4, node-22, node-40
node-8AP4242node-6, node-7, node-11, node-5, node-23, node-41
node-9EU4242node-10, node-11, node-12, node-6, node-24, node-42
node-10US4242node-11, node-9, node-13, node-7, node-25, node-43
node-11AP4242node-9, node-10, node-14, node-8, node-26, node-44
node-12EU4242node-13, node-14, node-15, node-9, node-27, node-45
node-13US4242node-14, node-12, node-16, node-10, node-28, node-46
node-14AP4242node-12, node-13, node-17, node-11, node-29, node-47
node-15EU4242node-16, node-17, node-18, node-12, node-30, node-48
node-16US4242node-17, node-15, node-19, node-13, node-31, node-49
node-17AP4242node-15, node-16, node-20, node-14, node-32, node-50
node-18EU4242node-19, node-20, node-21, node-15, node-33, node-0
node-19US4242node-20, node-18, node-22, node-16, node-34, node-1
node-20AP4242node-18, node-19, node-23, node-17, node-35, node-2
node-21EU4242node-22, node-23, node-24, node-18, node-36, node-3
node-22US4242node-23, node-21, node-25, node-19, node-37, node-4
node-23AP4242node-21, node-22, node-26, node-20, node-38, node-5
node-24EU4242node-25, node-26, node-27, node-21, node-39, node-6
node-25US4242node-26, node-24, node-28, node-22, node-40, node-7
node-26AP4242node-24, node-25, node-29, node-23, node-41, node-8
node-27EU4242node-28, node-29, node-30, node-24, node-42, node-9
node-28US4242node-29, node-27, node-31, node-25, node-43, node-10
node-29AP4242node-27, node-28, node-32, node-26, node-44, node-11
node-30EU4242node-31, node-32, node-33, node-27, node-45, node-12
node-31US4242node-32, node-30, node-34, node-28, node-46, node-13
node-32AP4242node-30, node-31, node-35, node-29, node-47, node-14
node-33EU4242node-34, node-35, node-36, node-30, node-48, node-15
node-34US4242node-35, node-33, node-37, node-31, node-49, node-16
node-35AP4242node-33, node-34, node-38, node-32, node-50, node-17
node-36EU4242node-37, node-38, node-39, node-33, node-0, node-18
node-37US4242node-38, node-36, node-40, node-34, node-1, node-19
node-38AP4242node-36, node-37, node-41, node-35, node-2, node-20
node-39EU4242node-40, node-41, node-42, node-36, node-3, node-21
node-40US4242node-41, node-39, node-43, node-37, node-4, node-22
node-41AP4242node-39, node-40, node-44, node-38, node-5, node-23
node-42EU4242node-43, node-44, node-45, node-39, node-6, node-24
node-43US4242node-44, node-42, node-46, node-40, node-7, node-25
node-44AP4242node-42, node-43, node-47, node-41, node-8, node-26
node-45EU4242node-46, node-47, node-48, node-42, node-9, node-27
node-46US4242node-47, node-45, node-49, node-43, node-10, node-28
node-47AP4242node-45, node-46, node-50, node-44, node-11, node-29
node-48EU4242node-49, node-50, node-0, node-45, node-12, node-30
node-49US4242node-50, node-48, node-1, node-46, node-13, node-31
node-50AP4242node-48, node-49, node-2, node-47, node-14, node-32

Latency

cardano-cli ping round trips over the node-to-node handshake, measured at cluster start by each pool's healthcheck against its producers: 918 samples over 6 region pairs. Milliseconds.

LinknminmedianmeanmaxSDSEMTCP connect
within AP (Sydney)2040.360.520.551.980.2160.0150.25
within EU (Frankfurt)2040.410.690.772.010.2780.0190.50
within US (N. Virginia)2040.430.620.652.020.1810.0130.36
AP ↔ EU102304.29308.82308.36309.901.2350.122308.08
AP ↔ US102208.21219.23218.18220.072.6770.265217.89
EU ↔ US10290.0490.8690.8691.870.3520.03590.56

Not an ICMP ping: cardano-cli ping opens a TCP connection and runs the node-to-node handshake, so each sample is an application-level round trip through both nodes' network stacks and mux, which is what the Cardano protocols actually experience. The TCP connect column is the bare TCP round trip from the same tool, the closest stand-in for a raw ping. SEM = SD / sqrt(n). The spread is 599x, from 0.52 ms within AP to 308.82 ms between AP and EU, and it is the floor under every interval in this report.

tx-centrifuge

The load generator, a closed-loop driver. A workload is a self-contained circuit: its own disjoint partition of funds, single-use UTxOs, its own builder turning funds into transactions, its own token-bucket rate limit, its own injection targets and its own observer following the chain. Every transaction spends funds and creates outputs, and an output becomes spendable again only once that workload's observer has seen its transaction reach the confirmation depth, so the transactions form chains gated by the chain itself and the funds budget is the buffer bridging confirmation latency. With on_exhaustion = error, a workload that cannot recycle funds as fast as it spends them runs that buffer to zero and fails the run rather than quietly slowing down.

The generator runs on the explorer machine in eu-central-1, which forges nothing, so every transaction crosses the network before the pool that must place it in a block ever sees it: the round trip a workload pays is the EU row for its target's region in the table above. Each workload holds two buffers of its own, funds waiting to be built and built transactions waiting to be delivered.

1. Forging, generated load

Every EB forging found, nothing filtered. A block that lost its height battle, that was rolled away from, or that no other node ever adopted still counts.

Totals

3,175RBs forged
3,151includes EB hash
99.2%of all RBs
24no EB hash
3,151distinct EB hashes
0repeated EBs forged

By region

regionRBs forgedshare of all RBsincludes EB hashof its own RBs
AP1,07133.7%1,06399.3%
EU1,07333.8%1,06499.2%
US1,03132.5%1,02499.3%

Endorser block contents

What an endorser block holds. An endorser block is a list of (tx hash, size) references and nothing else, so its size is that list's size.

minp50p90maxmean
EB size (bytes)2,29698,90498,90498,90492,677
EB full, of the 100,000 B reference cap2.3%98.9%98.9%98.9%92.7%
tx references per EB622,6732,6732,6732,504
bytes per reference37.0037.0037.0037.0337.00
derived closure bytes, references × 374 B23,188999,702999,702999,702936,762
derived closure full, of the 1,000,000 B transaction cap2.3%100.0%100.0%100.0%93.7%

1,674 of 3,151 EB forgings sit exactly at the run's largest EB, 2,673 references.

The transaction closure is what binds this run. Its transactions are 374 B each, so its largest endorser block holds 2,673 of them: 98,904 B of references against the 100,000 B cap (98.9%), and 999,702 B of transactions against the 1,000,000 B the closure allows (100.0%). The one that reaches 100% is the one that stopped the endorser block from growing.

The other run of this pair, 2026-10-20-20-42-81787-w42notxsub-value-leios-workload-all-n2n-100k1m-1i1o1w-nomadperf-dira-nom, is the control for exactly this: its transactions are 200 B, its largest endorser block holds 2,777 references for 99,975 B (100.0% of the reference cap), and their closure is 555,400 B (55.5% of the closure cap). With 374 B transactions the closure fills first, with 200 B transactions the reference list does, off the same two caps. So the binding cap moved with the transaction size, and nothing else changed.

2. Propagation, fetching the EB references

Three instants are used:

Endorser blocks whose hash was forged more than once, and so referenced by more than one ranking block, are left out of this section.

Announcing

The forger alone: from forging the endorser block to sending its first announcement of it.

samplesminp50p75p90p99p99.9maxmean
Forger: forged to its first announcement3,1512 ms11 ms13 ms18 ms27 ms43 ms101 ms12 ms

Fetching

From the forger's first announcement of the endorser block to that endorser block first arriving at a peer, one sample per node and endorser block, 157,546 of them. An endorser block is its list of transaction references, and that list is what arrives; the transactions themselves are fetched separately and are not in this section. Repeated deliveries of the same endorser block are counted under Activity, never timed here.

samplesminp50p75p90p99p99.9maxmean
announced, to the EB arriving at a node157,54647 ms1.13 s1.58 s2.01 s2.84 s4.88 s10.59 s1.26 s

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first received it, measured from the forger's first announcement as above. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers519 ms825 ms1.11 s1.52 s2.22 s3.06 s6.71 s987 ms
to 50% of its peers670 ms1.12 s1.57 s1.67 s2.55 s3.71 s7.42 s1.25 s
to 75% of its peers867 ms1.50 s1.82 s2.12 s2.87 s4.70 s8.46 s1.59 s
to 90% of its peers897 ms1.63 s2.17 s2.23 s3.23 s6.08 s8.88 s1.75 s
to 100% of its peers923 ms2.00 s2.54 s3.07 s4.23 s9.71 s10.59 s2.11 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers157,54647 ms1.13 s1.58 s2.01 s2.84 s4.88 s10.59 s1.26 s
Peers in AP52,50449 ms1.35 s1.68 s2.14 s2.90 s5.59 s10.49 s1.36 s
Peers in EU52,50153 ms1.19 s1.53 s1.95 s2.87 s4.80 s10.59 s1.25 s
Peers in US52,54147 ms1.07 s1.53 s1.88 s2.78 s4.66 s9.41 s1.18 s

Activity

The same events counted instead of timed, and counted per node and endorser block: for one endorser block, how many announcements a node sent out, and how many times it fetched the endorser block. Anything above one is a repeat the protocol paid for.

306.0announcements per EB, all nodes
6.00announcements per EB, per node
85.4fetches per EB, all nodes
1.71fetches per EB, per node

Times a node fetched the same endorser block

Inbound: the node pulling the endorser block from a peer. One means it fetched it once, more than one means it fetched the same endorser block again.

times it fetched itcasesshare of cases
never20.0%
1 time100,92764.1%
2 times26,98617.1%
3 times15,5629.9%
4 times or more14,0718.9%

269,100 fetches in all, of which 111,554 brought an endorser block the node already had. The most one node fetched a single endorser block was 21 times.

By region of the receiving node

regioncasesannouncements sentper casefetchesper caserepeats
AP52,504315,0216.00101,8281.9448.4%
EU52,502315,0056.0086,6761.6539.4%
US52,542315,2466.0080,5961.5334.8%

3. Acquiring, the EB written down

A delivered endorser block is not yet held: until its write is durable the body still reads as not-held, which is why the same endorser block can be fetched again meanwhile. Consensus.LeiosKernel.BlockAcquired is the moment that ends: the node has the reference list stored and claims to hold it, registering which referenced transactions it still needs. This is the step from arrival to that claim, which includes waiting for the single writer, and how often the same endorser block was stored again.

Writing

samplesminp50p75p90p99p99.9maxmean
the EB arriving, to the node writing it157,5462 ms67 ms108 ms204 ms962 ms2.10 s5.37 s120 ms

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first stored it, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers5 ms57 ms65 ms76 ms120 ms301 ms602 ms58 ms
to 50% of its peers5 ms69 ms82 ms109 ms487 ms682 ms985 ms84 ms
to 75% of its peers7 ms102 ms138 ms290 ms823 ms1.21 s1.51 s151 ms
to 90% of its peers8 ms144 ms216 ms622 ms1.16 s1.90 s2.23 s234 ms
to 100% of its peers11 ms279 ms672 ms1.22 s2.73 s3.72 s5.37 s527 ms

By region

samplesminp50p75p90p99p99.9maxmean
All peers157,5462 ms67 ms108 ms204 ms962 ms2.10 s5.37 s120 ms
Peers in AP52,5042 ms62 ms92 ms157 ms911 ms2.12 s3.42 s104 ms
Peers in EU52,5014 ms75 ms129 ms267 ms1.03 s2.20 s3.98 s141 ms
Peers in US52,5413 ms67 ms108 ms196 ms901 ms2.01 s5.37 s115 ms

Activity

Counted per node and endorser block, like section 2: how many times a node wrote the same endorser block down. This one is local, nothing crosses the network.

67.0writes per EB, all nodes
1.34writes per EB, per node
times it was writtencasesshare of cases
never20.0%
1 time125,20579.5%
2 times18,50711.7%
3 times9,3415.9%
4 times or more4,4932.9%

211,111 writes in all, of which 53,565 were of an endorser block the node had already written. The most one node wrote a single endorser block was 18 times.

By region of the writing node

regioncaseswritesper case
AP52,50470,9801.35
EU52,50271,9591.37
US52,54268,1721.30

4. Closures, the transactions an EB references

An endorser block is a list of references. The transactions themselves are a separate transfer, and a node has the closure once it holds all of them: Consensus.LeiosKernel.BlockTxsAcquired. Counted per node and endorser block, on the same endorser blocks as sections 2 and 3.

Obtaining

samplesminp50p75p90p99p99.9maxmean
EB fetched to the closure obtained157,5464 ms528 ms780 ms1.05 s1.59 s3.06 s6.28 s617 ms

A voter has until 7 s after the announcing block's slot onset to hold and validate the transactions: lHdrWait 3 s then lVoteWindow 4 s, both fixed in LeiosVoting.hs rather than read from genesis. 120 of 157,546 closures were obtained later than that, counted from the endorser block's forging.

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first obtained the closure, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers7 ms500 ms523 ms561 ms933 ms1.07 s3.04 s395 ms
to 50% of its peers11 ms542 ms705 ms944 ms1.09 s1.39 s3.24 s622 ms
to 75% of its peers19 ms751 ms1.01 s1.09 s1.44 s2.41 s3.53 s788 ms
to 90% of its peers27 ms889 ms1.08 s1.29 s1.75 s3.20 s3.70 s899 ms
to 100% of its peers122 ms1.17 s1.55 s2.44 s3.50 s4.93 s6.28 s1.35 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers157,5464 ms528 ms780 ms1.05 s1.59 s3.06 s6.28 s617 ms
Peers in AP52,5044 ms538 ms902 ms1.08 s1.62 s3.07 s4.49 s578 ms
Peers in EU52,5015 ms535 ms754 ms1.05 s1.62 s3.05 s5.63 s660 ms
Peers in US52,5414 ms521 ms720 ms998 ms1.52 s3.09 s6.28 s613 ms

Activity

51.2closures per EB, all nodes
1.02closures per EB, per node
2,719obtained more than once
2never obtained

Times a node obtained the same closure

times it obtained itcasesshare of cases
never20.0%
1 time154,82798.3%
2 times2,0881.3%
3 times4210.3%
4 times or more2100.1%

161,446 closures obtained in all, of which 3,900 were of one the node already had. The most one node obtained a single closure was 18 times.

By region of the obtaining node

regioncasesclosures obtainedper case
AP52,50454,7591.04
EU52,50253,2521.01
US52,54253,4351.02

5. Certifying

Counted on the blocks the whole cluster took. A block qualifies when all 50 nodes other than its forger adopted it, every peer it has of the 51 in the cluster: 3,019 of the run's 3,175 walkable blocks. Both sides of the ratio are held to it, so an announcement made by a block only part of the cluster held, and a certificate carried by one, are both out. No battle or fork filter is applied: nothing here asks what happened at a contested height, and the walk the blocks come from keeps every side of every fork.

Totals

2,996EBs announced
820certificate carried
27.4%of announcements
2,176never carried

By region

region of the announcerEBs announcedcertifiedshare
AP1,01135635.2%
EU1,01318818.6%
US97227628.4%

What happened to every endorser block

stagecount% of announced% of its groupgroupvotes received
Endorser blocks announced3,151100.0%···
    A voting window opened3,151100.0%···
Reached quorum1,45446.1%··39–51 of 51, median 50
    Certificate carried by a later block82126.1%56.5%reached quorum·
    Quorum reached, never carried63320.1%43.5%reached quorum·
Never reached quorum1,69753.9%··0–38 of 51, median 14
    voters declined: chainTipDoesNotAnnounce14,800·17.1%no-quorum voters·
    voters declined: tooLate48,622·56.2%no-quorum voters·
    voters declined: voteRejected: AlreadyKnown32·0.0%no-quorum voters·
    voters declined: voteRejected: all 1 votes rejected32·0.0%no-quorum voters·
    voters voted for it anyway23,094·26.7%no-quorum voters·

Hover a stage for what it counts and which trace says so. The stages down to Never reached quorum count endorser blocks; the rows under it count voter decisions, one per trace event, which is why they sum past 100%. Nothing is gated here, not even the run's opening epochs, so these counts are wider than section 2's.

When the transactions reach a node's own chain

The end of the line for an endorser block: a node adopts a ranking block that carries its certificate, and only then are those transactions on that node's chain. Measured from the endorser block's forging, over the 41,000 cases, one node and one endorser block each, where it happened.

samplesminp50p75p90p99p99.9maxmean
forged, to a node adopting a block that certifies it41,00013.74 s27.74 s41.75 s61.07 s114.00 s184.51 s184.86 s34.31 s
AP13,58414.17 s27.88 s41.86 s61.58 s113.60 s184.82 s184.86 s34.36 s
EU13,75213.74 s27.70 s41.73 s61.04 s114.02 s184.67 s184.70 s34.30 s
US13,66413.89 s27.73 s41.72 s61.37 s113.92 s184.49 s184.53 s34.27 s

6. Forks

Every switch a node made off a chain it already had, one row per node and switch (ChainDB.AddBlockEvent.SwitchedToAFork). A contest is one pair of tips, so a contest the whole cluster lived through counts once. Both rates below are contests per 1,000 blocks on the chain, so runs of different length compare directly.

24.8 expected from activeSlotsCoeff 0.0521-16 81787 100k1m 2i2oslot23.3height23.020-20 81787 100k1m 1i1oslot24.9height24.30102030

The slot rate is the control and the height rate is the signal: two leaders in one slot is fixed by activeSlotsCoeff and cannot be engineered away, while a height battle is a node forging on a parent a sibling already had, which is propagation losing a race and is free to move. Reading the height rate without the control beside it cannot tell a propagation regression from a run that simply drew more forks.

What the nodes paid

102.7switches per node
97 to 107fewest to most, one node
23.3slot battles per 1,000 chain blocks
23.0height battles per 1,000 chain blocks