From d1207f0d2f5855821e1ea341d602b48325bef0d1 Mon Sep 17 00:00:00 2001 From: Mark Kalsbeek Date: Tue, 26 May 2026 23:21:13 +0200 Subject: [PATCH] bench result summary --- .gitignore | 5 + benchmark-report.html | 514 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 519 insertions(+) create mode 100644 .gitignore create mode 100644 benchmark-report.html diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e1d1de0 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +target +Cargo.lock +smarm_trace.json +.direnv/ +.envrc diff --git a/benchmark-report.html b/benchmark-report.html new file mode 100644 index 0000000..82e98a8 --- /dev/null +++ b/benchmark-report.html @@ -0,0 +1,514 @@ + + + + + + Urus Benchmark Report · 2026-05-26 + + + +
+
+

Urus Benchmark Report

+

HTTP/1.1 Performance: Urus vs. Axum

+
+
+ Run Date: + May 26, 2026 +
+
+ Run ID: + 20260526-220625 +
+
+ Scenarios: + S1–S4 +
+
+
+ +
+ +
+

Summary

+
+
+

Urus RPS

+
53–54k
+

Consistent s1–s4

+
+
+

Axum RPS

+
288–320k
+

Consistent s1–s4

+
+
+

Performance Gap

+
5.7×
+

Axum faster (RPS)

+
+
+
+ + +
+

Results

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FrameworkScenarioSustained RPSp99 LatencyDescription
UrusS154,6402.70msBare HTTP
UrusS253,1392.62ms+ Middleware (logger, auth, router)
UrusS353,6162.67ms+ In-memory store (mixed workload)
UrusS452,5902.53ms+ SQLite (1 writer, 4 readers)
AxumS1288,33011.64msBare HTTP
AxumS2315,59649.66ms+ Middleware (logger, auth, router)
AxumS3316,34260.45ms+ In-memory store (mixed workload)
AxumS4320,15056.13ms+ SQLite (sqlx::SqlitePool)
+
+ + +
+

Spec Compliance

+

+ Target: Urus RPS ≥ 50% of Axum ("within 2×") on S1 and S2. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ScenarioUrus RPSAxum RPSRatioStatus
S154,640288,3300.19 (5.3×)FAIL
S253,139315,5960.17 (5.9×)FAIL
S353,616316,3420.17 (5.9×)FAIL
+ +
+

Observations

+
    +
  • Consistent Gap: Axum is 5.3–5.9× faster across all scenarios. The gap does not narrow with additional complexity.
    • +
  • No Performance Collapse: Both frameworks maintain stable RPS from S1 to S4. Urus holds 52.6–54.6k RPS; Axum holds 288–320k RPS. SQLite does not degrade either one.
    • +
  • Tail Latency Pattern: Axum's p99 rises with load (11.6ms → 60.5ms). Urus remains sub-3ms across all scenarios. This is typical of async Rust under concurrent load vs. actor-based systems (BEAM behavior).
    • +
+
+
+ + +
+

Analysis

+ +
+

RPS Shortfall

+
    +
  • Urus fails the 2× target by 3.3–5.9×. This is the primary result.
    • +
  • The gap stems from Urus's message-passing model: each request is routed through actor channels, adding per-request overhead vs. Axum's direct memory access and async scheduling.
    • +
  • This is a fundamental architectural difference, not an optimization opportunity.
    • +
+
+ +
+

Tail Latency Observation

+
    +
  • Urus exhibits sub-3ms p99 under all loads. Axum's p99 grows linearly with concurrency (11–60ms). This mirrors known BEAM vs. async Rust behavior: actor systems maintain consistent tail latency, async/await shows load-dependent tail growth.
    • +
  • This is notable but not a spec redemption. Both metrics matter. Urus's low tail latency is a side effect of lower throughput, not architectural superiority—at identical offered load, the comparison would differ.
    • +
  • For workloads with strict tail-latency SLAs and modest throughput needs, Urus's profile could be preferable. For high-throughput services, Axum's throughput advantage outweighs tail-latency concerns.
    • +
+
+
+ + +
+

Conclusion

+ +

+ Urus does not meet the spec target of "within 2× of Axum RPS." It is 5.3–5.9× slower. This gap is consistent across all scenarios. +

+ +

+ Verdict: FAIL on throughput (primary benchmark metric). Tail latency is a secondary characteristic; do not use it to offset the RPS result. +

+
+
+ +
+

Report: May 26, 2026 · Batch: 20260526-220625

+

Urus HTTP/1.1 Benchmark Suite

+
+
+ +