bench result summary
This commit is contained in:
@@ -0,0 +1,5 @@
|
||||
target
|
||||
Cargo.lock
|
||||
smarm_trace.json
|
||||
.direnv/
|
||||
.envrc
|
||||
@@ -0,0 +1,514 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Urus Benchmark Report · 2026-05-26</title>
|
||||
<style>
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
|
||||
background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
|
||||
min-height: 100vh;
|
||||
padding: 40px 20px;
|
||||
}
|
||||
|
||||
.container {
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
border-radius: 12px;
|
||||
box-shadow: 0 20px 60px rgba(0,0,0,0.15);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.header {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
color: white;
|
||||
padding: 60px 40px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.header h1 {
|
||||
font-size: 2.5em;
|
||||
margin-bottom: 10px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.header p {
|
||||
font-size: 1.1em;
|
||||
opacity: 0.95;
|
||||
margin: 5px 0;
|
||||
}
|
||||
|
||||
.meta {
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
gap: 30px;
|
||||
margin-top: 20px;
|
||||
font-size: 0.95em;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.meta-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.meta-label {
|
||||
opacity: 0.8;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.content {
|
||||
padding: 40px;
|
||||
}
|
||||
|
||||
.section {
|
||||
margin-bottom: 50px;
|
||||
}
|
||||
|
||||
.section h2 {
|
||||
color: #333;
|
||||
font-size: 1.6em;
|
||||
margin-bottom: 25px;
|
||||
padding-bottom: 12px;
|
||||
border-bottom: 3px solid #667eea;
|
||||
display: inline-block;
|
||||
}
|
||||
|
||||
.summary-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
|
||||
gap: 20px;
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.summary-card {
|
||||
background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
|
||||
padding: 25px;
|
||||
border-radius: 8px;
|
||||
border-left: 4px solid #667eea;
|
||||
}
|
||||
|
||||
.summary-card h3 {
|
||||
color: #667eea;
|
||||
font-size: 0.9em;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 1px;
|
||||
margin-bottom: 10px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.summary-card .value {
|
||||
font-size: 1.8em;
|
||||
color: #333;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.results-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 30px 0;
|
||||
font-size: 0.95em;
|
||||
}
|
||||
|
||||
.results-table thead {
|
||||
background: #f8f9fa;
|
||||
border-bottom: 2px solid #667eea;
|
||||
}
|
||||
|
||||
.results-table th {
|
||||
padding: 15px;
|
||||
text-align: left;
|
||||
font-weight: 600;
|
||||
color: #333;
|
||||
}
|
||||
|
||||
.results-table td {
|
||||
padding: 15px;
|
||||
border-bottom: 1px solid #e0e0e0;
|
||||
}
|
||||
|
||||
.results-table tbody tr:hover {
|
||||
background: #f8f9fa;
|
||||
}
|
||||
|
||||
.result-row-urus {
|
||||
background: rgba(102, 126, 234, 0.05);
|
||||
}
|
||||
|
||||
.result-row-axum {
|
||||
background: rgba(220, 38, 38, 0.05);
|
||||
}
|
||||
|
||||
.metric-rps {
|
||||
font-weight: 600;
|
||||
color: #333;
|
||||
}
|
||||
|
||||
.metric-latency {
|
||||
color: #666;
|
||||
}
|
||||
|
||||
.framework-label {
|
||||
display: inline-block;
|
||||
padding: 4px 12px;
|
||||
border-radius: 20px;
|
||||
font-size: 0.85em;
|
||||
font-weight: 600;
|
||||
margin-right: 8px;
|
||||
}
|
||||
|
||||
.framework-urus {
|
||||
background: rgba(102, 126, 234, 0.2);
|
||||
color: #667eea;
|
||||
}
|
||||
|
||||
.framework-axum {
|
||||
background: rgba(220, 38, 38, 0.2);
|
||||
color: #dc2626;
|
||||
}
|
||||
|
||||
.findings {
|
||||
background: #f0f7ff;
|
||||
border-left: 4px solid #667eea;
|
||||
padding: 25px;
|
||||
border-radius: 8px;
|
||||
margin: 30px 0;
|
||||
}
|
||||
|
||||
.findings h3 {
|
||||
color: #667eea;
|
||||
margin-bottom: 15px;
|
||||
font-size: 1.1em;
|
||||
}
|
||||
|
||||
.findings ul {
|
||||
list-style: none;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.findings li {
|
||||
padding: 10px 0;
|
||||
border-bottom: 1px solid rgba(102, 126, 234, 0.2);
|
||||
color: #333;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.findings li:last-child {
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
.findings li:before {
|
||||
content: "→ ";
|
||||
color: #667eea;
|
||||
font-weight: 600;
|
||||
margin-right: 8px;
|
||||
}
|
||||
|
||||
.ratio-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.ratio-table th,
|
||||
.ratio-table td {
|
||||
padding: 12px 15px;
|
||||
text-align: center;
|
||||
border: 1px solid #e0e0e0;
|
||||
}
|
||||
|
||||
.ratio-table th {
|
||||
background: #f8f9fa;
|
||||
font-weight: 600;
|
||||
color: #333;
|
||||
}
|
||||
|
||||
.ratio-fail {
|
||||
background: rgba(220, 38, 38, 0.1);
|
||||
color: #dc2626;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.caution {
|
||||
background: #fff7ed;
|
||||
border-left: 4px solid #f97316;
|
||||
padding: 20px;
|
||||
border-radius: 8px;
|
||||
margin: 30px 0;
|
||||
font-size: 0.95em;
|
||||
color: #333;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.footer {
|
||||
background: #f8f9fa;
|
||||
padding: 30px 40px;
|
||||
border-top: 1px solid #e0e0e0;
|
||||
text-align: center;
|
||||
font-size: 0.85em;
|
||||
color: #666;
|
||||
}
|
||||
|
||||
.run-id {
|
||||
font-family: "Monaco", "Courier New", monospace;
|
||||
font-size: 0.85em;
|
||||
color: #667eea;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.header {
|
||||
padding: 40px 20px;
|
||||
}
|
||||
|
||||
.header h1 {
|
||||
font-size: 1.8em;
|
||||
}
|
||||
|
||||
.content {
|
||||
padding: 20px;
|
||||
}
|
||||
|
||||
.results-table {
|
||||
font-size: 0.85em;
|
||||
}
|
||||
|
||||
.results-table th,
|
||||
.results-table td {
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.meta {
|
||||
gap: 15px;
|
||||
}
|
||||
|
||||
.summary-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<div class="header">
|
||||
<h1>Urus Benchmark Report</h1>
|
||||
<p>HTTP/1.1 Performance: Urus vs. Axum</p>
|
||||
<div class="meta">
|
||||
<div class="meta-item">
|
||||
<span class="meta-label">Run Date:</span>
|
||||
<span>May 26, 2026</span>
|
||||
</div>
|
||||
<div class="meta-item">
|
||||
<span class="meta-label">Run ID:</span>
|
||||
<span class="run-id">20260526-220625</span>
|
||||
</div>
|
||||
<div class="meta-item">
|
||||
<span class="meta-label">Scenarios:</span>
|
||||
<span>S1–S4</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="content">
|
||||
<!-- Summary -->
|
||||
<div class="section">
|
||||
<h2>Summary</h2>
|
||||
<div class="summary-grid">
|
||||
<div class="summary-card">
|
||||
<h3>Urus RPS</h3>
|
||||
<div class="value">53–54k</div>
|
||||
<p style="font-size: 0.85em; color: #666; margin-top: 5px;">Consistent s1–s4</p>
|
||||
</div>
|
||||
<div class="summary-card">
|
||||
<h3>Axum RPS</h3>
|
||||
<div class="value">288–320k</div>
|
||||
<p style="font-size: 0.85em; color: #666; margin-top: 5px;">Consistent s1–s4</p>
|
||||
</div>
|
||||
<div class="summary-card">
|
||||
<h3>Performance Gap</h3>
|
||||
<div class="value" style="color: #dc2626;">5.7×</div>
|
||||
<p style="font-size: 0.85em; color: #666; margin-top: 5px;">Axum faster (RPS)</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Results -->
|
||||
<div class="section">
|
||||
<h2>Results</h2>
|
||||
|
||||
<table class="results-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="width: 15%;">Framework</th>
|
||||
<th style="width: 15%;">Scenario</th>
|
||||
<th style="width: 20%;">Sustained RPS</th>
|
||||
<th style="width: 15%;">p99 Latency</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr class="result-row-urus">
|
||||
<td><span class="framework-label framework-urus">Urus</span></td>
|
||||
<td><strong>S1</strong></td>
|
||||
<td class="metric-rps">54,640</td>
|
||||
<td class="metric-latency">2.70ms</td>
|
||||
<td>Bare HTTP</td>
|
||||
</tr>
|
||||
<tr class="result-row-urus">
|
||||
<td><span class="framework-label framework-urus">Urus</span></td>
|
||||
<td><strong>S2</strong></td>
|
||||
<td class="metric-rps">53,139</td>
|
||||
<td class="metric-latency">2.62ms</td>
|
||||
<td>+ Middleware (logger, auth, router)</td>
|
||||
</tr>
|
||||
<tr class="result-row-urus">
|
||||
<td><span class="framework-label framework-urus">Urus</span></td>
|
||||
<td><strong>S3</strong></td>
|
||||
<td class="metric-rps">53,616</td>
|
||||
<td class="metric-latency">2.67ms</td>
|
||||
<td>+ In-memory store (mixed workload)</td>
|
||||
</tr>
|
||||
<tr class="result-row-urus">
|
||||
<td><span class="framework-label framework-urus">Urus</span></td>
|
||||
<td><strong>S4</strong></td>
|
||||
<td class="metric-rps">52,590</td>
|
||||
<td class="metric-latency">2.53ms</td>
|
||||
<td>+ SQLite (1 writer, 4 readers)</td>
|
||||
</tr>
|
||||
<tr class="result-row-axum">
|
||||
<td><span class="framework-label framework-axum">Axum</span></td>
|
||||
<td><strong>S1</strong></td>
|
||||
<td class="metric-rps">288,330</td>
|
||||
<td class="metric-latency">11.64ms</td>
|
||||
<td>Bare HTTP</td>
|
||||
</tr>
|
||||
<tr class="result-row-axum">
|
||||
<td><span class="framework-label framework-axum">Axum</span></td>
|
||||
<td><strong>S2</strong></td>
|
||||
<td class="metric-rps">315,596</td>
|
||||
<td class="metric-latency">49.66ms</td>
|
||||
<td>+ Middleware (logger, auth, router)</td>
|
||||
</tr>
|
||||
<tr class="result-row-axum">
|
||||
<td><span class="framework-label framework-axum">Axum</span></td>
|
||||
<td><strong>S3</strong></td>
|
||||
<td class="metric-rps">316,342</td>
|
||||
<td class="metric-latency">60.45ms</td>
|
||||
<td>+ In-memory store (mixed workload)</td>
|
||||
</tr>
|
||||
<tr class="result-row-axum">
|
||||
<td><span class="framework-label framework-axum">Axum</span></td>
|
||||
<td><strong>S4</strong></td>
|
||||
<td class="metric-rps">320,150</td>
|
||||
<td class="metric-latency">56.13ms</td>
|
||||
<td>+ SQLite (sqlx::SqlitePool)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Spec Compliance -->
|
||||
<div class="section">
|
||||
<h2>Spec Compliance</h2>
|
||||
<p style="margin-bottom: 20px; color: #666;">
|
||||
<strong>Target:</strong> Urus RPS ≥ 50% of Axum ("within 2×") on S1 and S2.
|
||||
</p>
|
||||
|
||||
<table class="ratio-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Scenario</th>
|
||||
<th>Urus RPS</th>
|
||||
<th>Axum RPS</th>
|
||||
<th>Ratio</th>
|
||||
<th>Status</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>S1</strong></td>
|
||||
<td>54,640</td>
|
||||
<td>288,330</td>
|
||||
<td>0.19 (5.3×)</td>
|
||||
<td class="ratio-fail">FAIL</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>S2</strong></td>
|
||||
<td>53,139</td>
|
||||
<td>315,596</td>
|
||||
<td>0.17 (5.9×)</td>
|
||||
<td class="ratio-fail">FAIL</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>S3</strong></td>
|
||||
<td>53,616</td>
|
||||
<td>316,342</td>
|
||||
<td>0.17 (5.9×)</td>
|
||||
<td class="ratio-fail">FAIL</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<div class="findings">
|
||||
<h3>Observations</h3>
|
||||
<ul>
|
||||
<li><strong>Consistent Gap:</strong> Axum is 5.3–5.9× faster across all scenarios. The gap does not narrow with additional complexity.</li>
|
||||
<li><strong>No Performance Collapse:</strong> Both frameworks maintain stable RPS from S1 to S4. Urus holds 52.6–54.6k RPS; Axum holds 288–320k RPS. SQLite does not degrade either one.</li>
|
||||
<li><strong>Tail Latency Pattern:</strong> Axum's p99 rises with load (11.6ms → 60.5ms). Urus remains sub-3ms across all scenarios. This is typical of async Rust under concurrent load vs. actor-based systems (BEAM behavior).</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Analysis -->
|
||||
<div class="section">
|
||||
<h2>Analysis</h2>
|
||||
|
||||
<div class="findings">
|
||||
<h3>RPS Shortfall</h3>
|
||||
<ul>
|
||||
<li>Urus fails the 2× target by 3.3–5.9×. This is the primary result.</li>
|
||||
<li>The gap stems from Urus's message-passing model: each request is routed through actor channels, adding per-request overhead vs. Axum's direct memory access and async scheduling.</li>
|
||||
<li>This is a fundamental architectural difference, not an optimization opportunity.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<div class="findings">
|
||||
<h3>Tail Latency Observation</h3>
|
||||
<ul>
|
||||
<li>Urus exhibits sub-3ms p99 under all loads. Axum's p99 grows linearly with concurrency (11–60ms). This mirrors known BEAM vs. async Rust behavior: actor systems maintain consistent tail latency, async/await shows load-dependent tail growth.</li>
|
||||
<li>This is notable but not a spec redemption. Both metrics matter. Urus's low tail latency is a side effect of lower throughput, not architectural superiority—at identical offered load, the comparison would differ.</li>
|
||||
<li>For workloads with strict tail-latency SLAs and modest throughput needs, Urus's profile could be preferable. For high-throughput services, Axum's throughput advantage outweighs tail-latency concerns.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Conclusion -->
|
||||
<div class="section" style="border-top: 2px solid #e0e0e0; padding-top: 30px;">
|
||||
<h2>Conclusion</h2>
|
||||
|
||||
<p style="line-height: 1.7; margin-bottom: 20px; color: #333;">
|
||||
<strong>Urus does not meet the spec target of "within 2× of Axum RPS."</strong> It is 5.3–5.9× slower. This gap is consistent across all scenarios.
|
||||
</p>
|
||||
|
||||
<p style="line-height: 1.7; color: #333;">
|
||||
<strong>Verdict:</strong> FAIL on throughput (primary benchmark metric). Tail latency is a secondary characteristic; do not use it to offset the RPS result.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="footer">
|
||||
<p><strong>Report:</strong> May 26, 2026 · <strong>Batch:</strong> 20260526-220625</p>
|
||||
<p style="margin-top: 8px; opacity: 0.7;">Urus HTTP/1.1 Benchmark Suite</p>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user