2 posts tagged with "validation-analysis"

This week, the Leios team made significant progress on CIP development, refined validation timing analysis with improved methodologies, and achieved high-throughput validation milestones. The team completed major components of the CIP specification, proposed improved throughput metrics for better comparability, and demonstrated a 1,000 TPS capability with specific protocol variants.

CIP development progress​

Protocol specification completion​

• Completed a comprehensive review of the protocol overview, component flow, and parameters
• Integrated vote and certificate specifications into the CIP documentation
• Drafted node behavior and network specifications, including mini-protocol definitions
• Progressed the CIP towards completion, with all core protocol components now fully specified.

Throughput metrics standardization​

Improved measurement methodology​

• Proposed transition from transaction-per-second (Tx/s) to transaction-bytes-per-second (TxB/s) metrics for enhanced comparability
• Recommended using Tx/s only in introductory statements with transaction size context (eg, '100 Tx/s with 1,400 B transactions')
• Established TxkB/s and TxMB/s as primary throughput metrics for analysis
• Benefits include:
  • Direct comparability across different transaction sizes
  • Clear nominal storage and network demand calculations
  • Example: 100 Tx/s with 1,400 B transactions = 140 TxkB/s = ~12 GB/day storage
  • Network overhead calculation: 140 TxkB/s × 10 peers = 11.2 Mb/s.

Enhanced validation analysis​

Revised Cardano validation timing study​

• Completed a refined analysis of Cardano mainnet validation times using a clean dataset on an idle machine
• Significantly improved accuracy over preliminary results by eliminating CPU contention effects
• Updated findings for transaction signature verification and Plutus script execution:
  • Median times: 428.4 μs/tx and 211.5 μs/kB
  • Linear model: 148.1 μs/tx plus 114.1 μs/kB
  • Enhanced model: 137.5 μs/tx plus 60.2 μs/kB plus 585.2 μs/Gstep with Laplace error distribution
• Results are suitable for bulk block estimates despite individual transaction prediction limitations
• Findings support reducing CPU-timing parameters in default Leios simulation configurations
• A comprehensive analysis is available in the validation timing documentation.

High-throughput protocol validation​

1,000 TPS Linear Leios demonstration​

• Successfully demonstrated Linear Leios with transaction references supporting 1,000 tx/s at 300 B/tx
• Validated Stracciatella variant capability at 1,000 TPS throughput levels
• Confirmed that Linear Leios with embedded transactions cannot sustain such throughput
• Results provide clear protocol variant performance boundaries for high-throughput scenarios
• Detailed evidence and analysis are available in the 1,000 TPS study notebook.

Simulation infrastructure improvements​

Rust simulation documentation​

• Enhanced the documentation of the current Rust simulation implementation
• Documented available protocol variants and their implementation status
• Improved accessibility and usability of the simulation framework for protocol development.

Next steps​

• Finalize CIP documentation for community review and feedback
• Implement standardized throughput metrics across analysis frameworks
• Apply revised validation timing parameters to simulation configurations
• Expand high-throughput testing to additional protocol variants and scenarios.

This week, the Leios team focused on improving simulation analysis, conducting validation time studies, and working on comprehensive protocol variant experiments. The team developed new analysis tools, conducted fundamental performance studies of Cardano validation times, and completed extensive mapping of Linear Leios protocol performance under various conditions.

Analysis tools and infrastructure​

Generic trace analysis framework​

• Developed a generic analysis script for processing Leios simulator output
• Created a comprehensive R-based analysis pipeline generating diagnostic plots from leios-simulation-trace-processor output
• Enhanced analysis capabilities for systematic evaluation of simulation results
• Documentation and usage instructions are available in the trace processor README.

Validation performance analysis​

Cardano mainnet validation timing study​

• Completed a preliminary analysis of block and transaction validation times for Cardano mainnet since epoch 350
• Key findings using the db-analyser tool include:
  • Median transaction signature verification: 0.53 ms/tx
  • Median validation time per kilobyte: 0.29 ms/kB
  • Joint linear model estimate: 0.066 ms/tx plus 0.221 ms/kB
  • Data suitable for bulk estimates but too noisy for individual transaction predictions
• Identified missing explanatory variables (UTXO set size, input/output counts) extractable from the ledger or cardano-db-sync
• Results provide a foundation for more accurate simulator validation time modeling
• Detailed analysis is available in the validation timing notebook.

Simulation optimization studies​

Timestep resolution analysis​

• Conducted a comparative study of simulation timestep effects at 1,000 TPS Full Leios scenarios
• Compared 0.100 ms and 0.025 ms time resolutions with no significant differences in results
• Validated the use of coarser timesteps for improved parallelism and reduced simulation runtime
• Supporting analysis and evidence are available in the timestep study notebook.

Protocol variant experiments​

Mid-throughput protocol validation​

• Completed 100 TPS experiments for Stracciatella and Linear Leios variants using 1,400 B/tx over 900 seconds
• Key findings include:
  • 5 slots/stages are insufficient for Linear Leios at 100 tx/s
  • Including transactions in EBs causes congestion compared to transaction references
  • 10 MB/EB is required for 100 tx/s performance (5 MB/EB is insufficient)
  • EB-sortition unluckiness in Stracciatella extends the transaction lifecycle but can be mitigated
  • CPU and network peaks occur when transactions are embedded in EBs
• Analysis artifacts are available in 100 TPS experiment documentation and the analysis notebook.

Comprehensive Linear Leios performance mapping​

• Completed an extensive simulation study mapping Linear Leios performance under various loads and configurations
• Compared multiple Linear Leios variants against the Stracciatella baseline
• Generated comprehensive performance characterizations across different throughput scenarios
• Results are documented in the Linear Leios summary with variant-specific analysis:
  • Linear Leios with embedded transactions
  • Linear Leios with transaction references
  • Linear Leios without transactions
  • Stracciatella baseline comparison.

Simulation model refinements​

• Revised the Linear Leios model based on analysis findings, particularly regarding partial EB validation before peer propagation
• Reimplemented Stracciatella as a separate simulation to identify specification deviations
• Identified and resolved multiple implementation inconsistencies during the specification verification process.

Next steps​

• Continue developing analysis infrastructure for systematic protocol evaluation
• Integrate improved validation timing models into simulation configurations
• Expand protocol variant testing based on performance mapping results
• Refine simulation models for enhanced accuracy and specification compliance.