Endorser blocks: forging, propagation, acquiring, certifying

2026-10-23-18-49-e651d-w42notxsub-value-leios-workload-all-n2n-40k260k-2i2o2w-nomadperf-dira-nom
maxEndorserBlockTxsSize limited · 100.0% full, against 64.3% of maxEndorserBlockReferencesSize

Setup

Topology

51 pools · 3 regions: AP 17, EU 17, US 17 · torus-dense · 306 producer entries = 204 links (102 reciprocated) · 153 intra-region links, 51 inter-region · every pool has exactly 8 link partners · diameter 4, radius 4, mean shortest path 2.48 hops · peer sharing off (PeerSharing false in the node's configuration), so a node talks only and always to its listed producers

Full graph

EUUSAP

Intra-region links only

EUUSAP

Inter-region links only

EUUSAP

Node positions are identical in all three views; hover a node to highlight its links everywhere and read its degrees. Region colours: EU green, US red, AP blue. Links are drawn undirected; direction and each pool's producer list are in the adjacency table.

Adjacency table
NodeRegionOut intraOut interIn intraIn interProducers (outbound)
node-0EU4242node-1, node-2, node-3, node-48, node-15, node-33
node-1US4242node-2, node-0, node-4, node-49, node-16, node-34
node-2AP4242node-0, node-1, node-5, node-50, node-17, node-35
node-3EU4242node-4, node-5, node-6, node-0, node-18, node-36
node-4US4242node-5, node-3, node-7, node-1, node-19, node-37
node-5AP4242node-3, node-4, node-8, node-2, node-20, node-38
node-6EU4242node-7, node-8, node-9, node-3, node-21, node-39
node-7US4242node-8, node-6, node-10, node-4, node-22, node-40
node-8AP4242node-6, node-7, node-11, node-5, node-23, node-41
node-9EU4242node-10, node-11, node-12, node-6, node-24, node-42
node-10US4242node-11, node-9, node-13, node-7, node-25, node-43
node-11AP4242node-9, node-10, node-14, node-8, node-26, node-44
node-12EU4242node-13, node-14, node-15, node-9, node-27, node-45
node-13US4242node-14, node-12, node-16, node-10, node-28, node-46
node-14AP4242node-12, node-13, node-17, node-11, node-29, node-47
node-15EU4242node-16, node-17, node-18, node-12, node-30, node-48
node-16US4242node-17, node-15, node-19, node-13, node-31, node-49
node-17AP4242node-15, node-16, node-20, node-14, node-32, node-50
node-18EU4242node-19, node-20, node-21, node-15, node-33, node-0
node-19US4242node-20, node-18, node-22, node-16, node-34, node-1
node-20AP4242node-18, node-19, node-23, node-17, node-35, node-2
node-21EU4242node-22, node-23, node-24, node-18, node-36, node-3
node-22US4242node-23, node-21, node-25, node-19, node-37, node-4
node-23AP4242node-21, node-22, node-26, node-20, node-38, node-5
node-24EU4242node-25, node-26, node-27, node-21, node-39, node-6
node-25US4242node-26, node-24, node-28, node-22, node-40, node-7
node-26AP4242node-24, node-25, node-29, node-23, node-41, node-8
node-27EU4242node-28, node-29, node-30, node-24, node-42, node-9
node-28US4242node-29, node-27, node-31, node-25, node-43, node-10
node-29AP4242node-27, node-28, node-32, node-26, node-44, node-11
node-30EU4242node-31, node-32, node-33, node-27, node-45, node-12
node-31US4242node-32, node-30, node-34, node-28, node-46, node-13
node-32AP4242node-30, node-31, node-35, node-29, node-47, node-14
node-33EU4242node-34, node-35, node-36, node-30, node-48, node-15
node-34US4242node-35, node-33, node-37, node-31, node-49, node-16
node-35AP4242node-33, node-34, node-38, node-32, node-50, node-17
node-36EU4242node-37, node-38, node-39, node-33, node-0, node-18
node-37US4242node-38, node-36, node-40, node-34, node-1, node-19
node-38AP4242node-36, node-37, node-41, node-35, node-2, node-20
node-39EU4242node-40, node-41, node-42, node-36, node-3, node-21
node-40US4242node-41, node-39, node-43, node-37, node-4, node-22
node-41AP4242node-39, node-40, node-44, node-38, node-5, node-23
node-42EU4242node-43, node-44, node-45, node-39, node-6, node-24
node-43US4242node-44, node-42, node-46, node-40, node-7, node-25
node-44AP4242node-42, node-43, node-47, node-41, node-8, node-26
node-45EU4242node-46, node-47, node-48, node-42, node-9, node-27
node-46US4242node-47, node-45, node-49, node-43, node-10, node-28
node-47AP4242node-45, node-46, node-50, node-44, node-11, node-29
node-48EU4242node-49, node-50, node-0, node-45, node-12, node-30
node-49US4242node-50, node-48, node-1, node-46, node-13, node-31
node-50AP4242node-48, node-49, node-2, node-47, node-14, node-32

Latency

cardano-cli ping round trips over the node-to-node handshake, measured at cluster start by each pool's healthcheck against its producers: 918 samples over 6 region pairs. Milliseconds.

LinknminmedianmeanmaxSDSEMTCP connect
within AP (Sydney)2040.370.510.541.870.1500.0110.25
within EU (Frankfurt)2040.400.650.741.800.2710.0190.46
within US (N. Virginia)2040.370.620.641.710.1400.0100.37
AP ↔ EU102284.44288.36290.71299.315.7260.567290.44
AP ↔ US102204.76217.01214.23219.745.3890.534213.93
EU ↔ US10290.1890.7690.8798.240.7890.07890.53

Not an ICMP ping: cardano-cli ping opens a TCP connection and runs the node-to-node handshake, so each sample is an application-level round trip through both nodes' network stacks and mux, which is what the Cardano protocols actually experience. The TCP connect column is the bare TCP round trip from the same tool, the closest stand-in for a raw ping. SEM = SD / sqrt(n). The spread is 568x, from 0.51 ms within AP to 288.36 ms between AP and EU, and it is the floor under every interval in this report.

tx-centrifuge

The load generator, a closed-loop driver. A workload is a self-contained circuit: its own disjoint partition of funds, single-use UTxOs, its own builder turning funds into transactions, its own token-bucket rate limit, its own injection targets and its own observer following the chain. Every transaction spends funds and creates outputs, and an output becomes spendable again only once that workload's observer has seen its transaction reach the confirmation depth, so the transactions form chains gated by the chain itself and the funds budget is the buffer bridging confirmation latency. With on_exhaustion = error, a workload that cannot recycle funds as fast as it spends them runs that buffer to zero and fails the run rather than quietly slowing down.

The generator runs on the explorer machine in eu-central-1, which forges nothing, so every transaction crosses the network before the pool that must place it in a block ever sees it: the round trip a workload pays is the EU row for its target's region in the table above. Each workload holds two buffers of its own, funds waiting to be built and built transactions waiting to be delivered.

1. Forging, generated load

Every EB forging found, nothing filtered. A block that lost its height battle, that was rolled away from, or that no other node ever adopted still counts.

Totals

3,309RBs forged
3,289includes EB hash
99.4%of all RBs
20no EB hash
3,286distinct EB hashes
3repeated EBs forged

By region

regionRBs forgedshare of all RBsincludes EB hashof its own RBs
AP1,09533.1%1,09199.6%
EU1,14734.7%1,13799.1%
US1,06732.2%1,06199.4%

Endorser block contents

What an endorser block holds. An endorser block is a list of (tx hash, size) references and nothing else, so its size is that list's size.

minp50p90maxmean
EB size (bytes)3,70225,71825,71825,71825,674
EB full, of the 40,000 B reference cap9.3%64.3%64.3%64.3%64.2%
tx references per EB100695695695693
bytes per reference37.0037.0037.0037.0237.00
derived closure bytes, references × 374 B37,400259,930259,930259,930259,491
derived closure full, of the 260,000 B transaction cap14.4%100.0%100.0%100.0%99.8%

3,279 of 3,289 EB forgings sit exactly at the run's largest EB, 695 references.

The transaction closure is what binds this run. Its transactions are 374 B each, so its largest endorser block holds 695 of them: 25,718 B of references against the 40,000 B cap (64.3%), and 259,930 B of transactions against the 260,000 B the closure allows (100.0%). The one that reaches 100% is the one that stopped the endorser block from growing.

The other run of this pair, 2026-10-22-23-20-e651d-w42notxsub-value-leios-workload-all-n2n-40k260k-1i1o1w-nomadperf-dira-nom, is the control for exactly this: its transactions are 200 B, its largest endorser block holds 1,110 references for 39,963 B (99.9% of the reference cap), and their closure is 222,000 B (85.4% of the closure cap). With 374 B transactions the closure fills first, with 200 B transactions the reference list does, off the same two caps. So the binding cap moved with the transaction size, and nothing else changed.

2. Propagation, fetching the EB references

Three instants are used:

Endorser blocks whose hash was forged more than once, and so referenced by more than one ranking block, are left out of this section.

Announcing

The forger alone: from forging the endorser block to sending its first announcement of it.

samplesminp50p75p90p99p99.9maxmean
Forger: forged to its first announcement3,2831 ms3 ms4 ms7 ms11 ms16 ms26 ms4 ms

Fetching

From the forger's first announcement of the endorser block to that endorser block first arriving at a peer, one sample per node and endorser block, 164,150 of them. An endorser block is its list of transaction references, and that list is what arrives; the transactions themselves are fetched separately and are not in this section. Repeated deliveries of the same endorser block are counted under Activity, never timed here.

samplesminp50p75p90p99p99.9maxmean
announced, to the EB arriving at a node164,15011 ms1.07 s1.39 s1.64 s2.17 s2.83 s5.50 s1.09 s

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first received it, measured from the forger's first announcement as above. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers364 ms1.03 s1.06 s1.13 s1.22 s1.51 s1.95 s866 ms
to 50% of its peers579 ms1.07 s1.33 s1.56 s1.58 s1.71 s2.17 s1.09 s
to 75% of its peers696 ms1.26 s1.56 s1.67 s2.01 s2.48 s2.72 s1.33 s
to 90% of its peers775 ms1.42 s1.67 s1.97 s2.18 s3.36 s3.70 s1.47 s
to 100% of its peers797 ms1.62 s1.96 s2.20 s3.12 s3.97 s5.50 s1.71 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers164,15011 ms1.07 s1.39 s1.64 s2.17 s2.83 s5.50 s1.09 s
Peers in AP54,72212 ms1.13 s1.51 s1.68 s2.19 s2.93 s5.50 s1.18 s
Peers in EU54,67811 ms1.07 s1.39 s1.64 s2.04 s2.83 s4.34 s1.07 s
Peers in US54,75013 ms1.03 s1.15 s1.57 s2.03 s2.78 s3.85 s1.02 s

Activity

The same events counted instead of timed, and counted per node and endorser block: for one endorser block, how many announcements a node sent out, and how many times it fetched the endorser block. Anything above one is a repeat the protocol paid for.

306.0announcements per EB, all nodes
6.00announcements per EB, per node
80.2fetches per EB, all nodes
1.60fetches per EB, per node

Times a node fetched the same endorser block

Inbound: the node pulling the endorser block from a peer. One means it fetched it once, more than one means it fetched the same endorser block again.

times it fetched itcasesshare of cases
1 time111,12867.7%
2 times27,12516.5%
3 times13,3618.1%
4 times or more12,5367.6%

263,423 fetches in all, of which 99,273 brought an endorser block the node already had. The most one node fetched a single endorser block was 7 times.

By region of the receiving node

regioncasesannouncements sentper casefetchesper caserepeats
AP54,722328,3326.0095,9191.7542.9%
EU54,678328,0686.0088,4731.6238.2%
US54,750328,5006.0079,0311.4430.7%

3. Acquiring, the EB written down

A delivered endorser block is not yet held: until its write is durable the body still reads as not-held, which is why the same endorser block can be fetched again meanwhile. Consensus.LeiosKernel.BlockAcquired is the moment that ends: the node has the reference list stored and claims to hold it, registering which referenced transactions it still needs. This is the step from arrival to that claim, which includes waiting for the single writer, and how often the same endorser block was stored again.

Writing

samplesminp50p75p90p99p99.9maxmean
the EB arriving, to the node writing it164,1501 ms14 ms19 ms26 ms60 ms178 ms2.35 s17 ms

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first stored it, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers2 ms11 ms14 ms17 ms22 ms40 ms43 ms12 ms
to 50% of its peers3 ms14 ms17 ms20 ms27 ms76 ms102 ms15 ms
to 75% of its peers4 ms17 ms22 ms27 ms40 ms118 ms135 ms19 ms
to 90% of its peers6 ms21 ms28 ms36 ms76 ms180 ms235 ms24 ms
to 100% of its peers9 ms34 ms52 ms83 ms319 ms2.16 s2.35 s57 ms

By region

samplesminp50p75p90p99p99.9maxmean
All peers164,1501 ms14 ms19 ms26 ms60 ms178 ms2.35 s17 ms
Peers in AP54,7221 ms14 ms19 ms25 ms56 ms162 ms2.33 s16 ms
Peers in EU54,6782 ms14 ms18 ms26 ms67 ms197 ms2.14 s17 ms
Peers in US54,7501 ms14 ms18 ms26 ms58 ms159 ms2.35 s16 ms

Activity

Counted per node and endorser block, like section 2: how many times a node wrote the same endorser block down. This one is local, nothing crosses the network.

63.6writes per EB, all nodes
1.27writes per EB, per node
times it was writtencasesshare of cases
1 time134,19281.7%
2 times17,82810.9%
3 times9,5755.8%
4 times or more2,5551.6%

208,814 writes in all, of which 44,664 were of an endorser block the node had already written. The most one node wrote a single endorser block was 5 times.

By region of the writing node

regioncaseswritesper case
AP54,72271,8051.31
EU54,67871,2891.30
US54,75065,7201.20

4. Closures, the transactions an EB references

An endorser block is a list of references. The transactions themselves are a separate transfer, and a node has the closure once it holds all of them: Consensus.LeiosKernel.BlockTxsAcquired. Counted per node and endorser block, on the same endorser blocks as sections 2 and 3.

Obtaining

samplesminp50p75p90p99p99.9maxmean
EB fetched to the closure obtained164,1502 ms513 ms724 ms1.02 s1.54 s2.64 s4.50 s603 ms

A voter has until 7 s after the announcing block's slot onset to hold and validate the transactions: lHdrWait 3 s then lVoteWindow 4 s, both fixed in LeiosVoting.hs rather than read from genesis. 0 of 164,150 closures were obtained later than that, counted from the endorser block's forging.

Spread over the cluster

One value per endorser block: the delay by which that share of the other nodes had first obtained the closure, each measured from that node's own fetch of the endorser block. The 100% row holds each endorser block's slowest node, so it sits above the table of every node and endorser block.

minp50p75p90p99p99.9maxmean
to 30% of its peers3 ms504 ms511 ms519 ms675 ms1.04 s2.00 s405 ms
to 50% of its peers4 ms518 ms631 ms966 ms1.03 s1.74 s2.59 s605 ms
to 75% of its peers12 ms715 ms1.01 s1.05 s1.42 s2.87 s3.19 s770 ms
to 90% of its peers14 ms874 ms1.08 s1.50 s1.56 s3.18 s3.63 s904 ms
to 100% of its peers22 ms1.02 s1.52 s1.62 s2.62 s3.77 s4.50 s1.14 s

By region

samplesminp50p75p90p99p99.9maxmean
All peers164,1502 ms513 ms724 ms1.02 s1.54 s2.64 s4.50 s603 ms
Peers in AP54,7222 ms513 ms809 ms1.08 s1.57 s2.56 s4.50 s563 ms
Peers in EU54,6783 ms515 ms722 ms1.02 s1.52 s2.61 s3.86 s647 ms
Peers in US54,7502 ms511 ms683 ms1.01 s1.52 s2.76 s3.75 s599 ms

Activity

50.9closures per EB, all nodes
1.02closures per EB, per node
2,667obtained more than once
0never obtained

Times a node obtained the same closure

times it obtained itcasesshare of cases
1 time161,48398.4%
2 times2,3891.5%
3 times2640.2%
4 times or more140.0%

167,109 closures obtained in all, of which 2,959 were of one the node already had. The most one node obtained a single closure was 4 times.

By region of the obtaining node

regioncasesclosures obtainedper case
AP54,72257,1231.04
EU54,67854,7551.00
US54,75055,2311.01

5. Certifying

Counted on the blocks the whole cluster took. A block qualifies when all 50 nodes other than its forger adopted it, every peer it has of the 51 in the cluster: 3,217 of the run's 3,309 walkable blocks. Both sides of the ratio are held to it, so an announcement made by a block only part of the cluster held, and a certificate carried by one, are both out. No battle or fork filter is applied: nothing here asks what happened at a contested height, and the walk the blocks come from keeps every side of every fork.

Totals

3,194EBs announced
1,654certificate carried
51.8%of announcements
1,540never carried

By region

region of the announcerEBs announcedcertifiedshare
AP1,05053350.8%
EU1,10757752.1%
US1,03754452.5%

What happened to every endorser block

stagecount% of announced% of its groupgroupvotes received
Endorser blocks announced3,286100.0%···
    A voting window opened3,286100.0%···
Reached quorum2,90688.4%··39–51 of 51, median 51
    Certificate carried by a later block1,65550.4%57.0%reached quorum·
    Quorum reached, never carried1,25138.1%43.0%reached quorum·
Never reached quorum38011.6%··0–34 of 51, median 0
    voters declined: chainTipDoesNotAnnounce17,099·88.2%no-quorum voters·
    voters declined: tooLate100·0.5%no-quorum voters·
    voters voted for it anyway2,181·11.3%no-quorum voters·

Hover a stage for what it counts and which trace says so. The stages down to Never reached quorum count endorser blocks; the rows under it count voter decisions, one per trace event, which is why they sum past 100%. Nothing is gated here, not even the run's opening epochs, so these counts are wider than section 2's.

When the transactions reach a node's own chain

The end of the line for an endorser block: a node adopts a ranking block that carries its certificate, and only then are those transactions on that node's chain. Measured from the endorser block's forging, over the 82,600 cases, one node and one endorser block each, where it happened.

samplesminp50p75p90p99p99.9maxmean
forged, to a node adopting a block that certifies it82,60012.46 s26.52 s40.15 s57.25 s95.50 s157.48 s196.57 s32.64 s
AP27,55212.90 s26.63 s40.18 s57.14 s95.56 s157.16 s196.12 s32.70 s
EU27,50812.46 s26.37 s40.14 s57.18 s95.38 s157.61 s196.57 s32.62 s
US27,54012.60 s26.40 s40.14 s57.26 s95.24 s157.49 s196.46 s32.59 s

6. Forks

Every switch a node made off a chain it already had, one row per node and switch (ChainDB.AddBlockEvent.SwitchedToAFork). A contest is one pair of tips, so a contest the whole cluster lived through counts once. Both rates below are contests per 1,000 blocks on the chain, so runs of different length compare directly.

24.8 expected from activeSlotsCoeff 0.0523-18 e651d 40k260k 2i2oslot20.9height6.322-23 e651d 40k260k 1i1oslot25.3height11.20102030

The slot rate is the control and the height rate is the signal: two leaders in one slot is fixed by activeSlotsCoeff and cannot be engineered away, while a height battle is a node forging on a parent a sibling already had, which is propagation losing a race and is free to move. Reading the height rate without the control beside it cannot tell a propagation regression from a run that simply drew more forks.

What the nodes paid

48.9switches per node
44 to 54fewest to most, one node
20.9slot battles per 1,000 chain blocks
6.3height battles per 1,000 chain blocks