Add small in-actor LRU cache for hot assets
AssetStoreServer runs on a single dedicated smarm actor thread, so a plain HashMap+VecDeque LRU in front of the SQLite lookup needs no locking. Capacity configurable via CCC_CACHE_CAPACITY (default 256). Bench (bench/RPS): +42-47% RPS under a cache-sized/hot-set workload, but a small net loss under adversarial uniform-random access with a cache smaller than the catalog. Real traffic is hot-set skewed, so net win in practice; capacity should be tuned to the expected hot set.
This commit is contained in:
@@ -0,0 +1,78 @@
|
||||
# CCC bench: RPS ceiling + LRU cache impact
|
||||
|
||||
Quick and dirty throughput bench, run locally on a 24-core box. Not
|
||||
scientific, just enough to sanity-check the actor-based SQLite store and
|
||||
the small LRU cache in front of it.
|
||||
|
||||
## Harness
|
||||
|
||||
- `bench/seed.py` fills a fresh `cdn.db` with random packages/versions/
|
||||
assets (default: 500 packages x 5 versions = 2500 assets, 512B-8KB
|
||||
gzipped JS each) and writes `bench/urls.txt` (one `/assets/...` path per
|
||||
line, for every seeded asset).
|
||||
- Server pinned to 2 CPUs, load generator (`oha`, via
|
||||
`nix-shell -p oha`) pinned to 6 CPUs, both via `taskset`, so client
|
||||
capacity is never the bottleneck.
|
||||
|
||||
```
|
||||
nix-shell -p python3 --run "python3 bench/seed.py cdn.db"
|
||||
awk '{print "http://127.0.0.1:8333"$0}' bench/urls.txt > /tmp/full_urls.txt
|
||||
|
||||
CCC_DB_PATH=$(pwd)/cdn.db taskset -c 0,1 ./target/release/CCC serve --port 8333 &
|
||||
|
||||
nix-shell -p oha --run \
|
||||
"taskset -c 2-7 oha -z 8s -c 200 --no-tui --urls-from-file /tmp/full_urls.txt"
|
||||
```
|
||||
|
||||
Skewed/hot-set workload (80% of requests hit the top 50 of 2500 assets,
|
||||
i.e. a realistic CDN access pattern) was generated with a short Python
|
||||
snippet sampling from `bench/urls.txt` with `random.random() < 0.8` picking
|
||||
from the first 50 lines, else uniformly from the rest, written to a
|
||||
`urls_hot80_20.txt` file and expanded the same way as above.
|
||||
|
||||
## Results: no cache (baseline)
|
||||
|
||||
Single-threaded `smarm` actor (`AssetStoreServer::loop_runner`) serializes
|
||||
every asset lookup onto one thread/one SQLite connection, so this is
|
||||
inherently CPU-bound on the 2 pinned cores regardless of client
|
||||
concurrency.
|
||||
|
||||
| Server CPUs | Client concurrency | RPS |
|
||||
|---|---|---|
|
||||
| 2 | 50 | ~56.7k |
|
||||
| 2 | 100 | ~60.1k |
|
||||
| 2 | 200 | ~62.1k (peak, server at ~176-200% CPU, saturated) |
|
||||
| 2 | 400 | ~60.0k (plateaued) |
|
||||
| 2 | 800 | ~55.9k (queueing overhead) |
|
||||
| 2 (client sends `Accept-Encoding: gzip`, server skips decompression) | 200 | ~60.4k |
|
||||
| 1 | 200 | ~30.8k (confirms CPU-bound, scales with cores) |
|
||||
|
||||
Client (6 CPUs) stayed at ~4% usr / 8% sys throughout - never the
|
||||
bottleneck.
|
||||
|
||||
## Results: with the LRU cache (`CCC_CACHE_CAPACITY`, default 256)
|
||||
|
||||
The cache lives inside `AssetStoreServer` itself (see `src/main.rs`), so
|
||||
it needs no locking - the actor thread is already strictly sequential.
|
||||
|
||||
| Scenario | Cache capacity | RPS | vs. no-cache baseline (62.1k) |
|
||||
|---|---|---|---|
|
||||
| Uniform-random over all 2500 assets | 256 (default) | ~55.3k | **-11%** |
|
||||
| Uniform-random over all 2500 assets | 3000 (covers full catalog) | ~88.1k | **+42%** |
|
||||
| 80/20 hot-set (50 hot assets get 80% of traffic) | 256 (default) | ~91.1k | **+47%** |
|
||||
|
||||
### Caveat
|
||||
|
||||
Under a purely uniform-random access pattern with a cache smaller than
|
||||
the catalog (low hit rate), the cache is a net loss: every request now
|
||||
pays HashMap lookup + insert + eviction bookkeeping on top of the SQLite
|
||||
query, for a hit rate too low to earn it back. The `touch()` on hit is
|
||||
also an O(n) scan of the recency queue, which doesn't help at low
|
||||
capacities.
|
||||
|
||||
Real CDN traffic is essentially never uniform-random (it's hot-set/
|
||||
power-law skewed), so in practice this is a clear win, but `CCC_CACHE_CAPACITY`
|
||||
should be sized to the actual hot set rather than left at the arbitrary
|
||||
default of 256. A follow-up would swap the O(n) recency scan for a proper
|
||||
O(1) LRU (e.g. an intrusive linked-hashmap) to remove the downside case
|
||||
entirely.
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seed cdn.db with random packages/versions/files for benchmarking."""
|
||||
import gzip
|
||||
import os
|
||||
import random
|
||||
import sqlite3
|
||||
import string
|
||||
import sys
|
||||
|
||||
DB_PATH = sys.argv[1] if len(sys.argv) > 1 else "cdn.db"
|
||||
N_PACKAGES = int(os.environ.get("N_PACKAGES", 500))
|
||||
VERSIONS_PER_PKG = int(os.environ.get("VERSIONS_PER_PKG", 5))
|
||||
MIN_SIZE = int(os.environ.get("MIN_SIZE", 512))
|
||||
MAX_SIZE = int(os.environ.get("MAX_SIZE", 8192))
|
||||
|
||||
random.seed(42)
|
||||
|
||||
if os.path.exists(DB_PATH):
|
||||
os.remove(DB_PATH)
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.execute("""CREATE TABLE IF NOT EXISTS packages (
|
||||
name TEXT PRIMARY KEY,
|
||||
archived INTEGER NOT NULL DEFAULT 0
|
||||
)""")
|
||||
conn.execute("""CREATE TABLE IF NOT EXISTS versions (
|
||||
package TEXT NOT NULL REFERENCES packages(name),
|
||||
version TEXT NOT NULL,
|
||||
filename TEXT NOT NULL,
|
||||
mime_type TEXT NOT NULL,
|
||||
gzipped_bytes BLOB NOT NULL,
|
||||
archived INTEGER NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (package, version)
|
||||
)""")
|
||||
|
||||
def rand_name(n=10):
|
||||
return "".join(random.choices(string.ascii_lowercase, k=n))
|
||||
|
||||
def rand_body(size):
|
||||
chars = string.ascii_letters + string.digits + " \n"
|
||||
return "".join(random.choices(chars, k=size)).encode()
|
||||
|
||||
manifest = [] # (package, version, filename) for the load generator
|
||||
|
||||
pkg_names = [f"pkg-{rand_name(8)}-{i}" for i in range(N_PACKAGES)]
|
||||
|
||||
for name in pkg_names:
|
||||
conn.execute("INSERT INTO packages (name, archived) VALUES (?, 0)", (name,))
|
||||
for v in range(VERSIONS_PER_PKG):
|
||||
version = f"{v+1}.0.0"
|
||||
filename = f"{rand_name(6)}.js"
|
||||
size = random.randint(MIN_SIZE, MAX_SIZE)
|
||||
raw = rand_body(size)
|
||||
gz = gzip.compress(raw, compresslevel=6)
|
||||
conn.execute(
|
||||
"INSERT INTO versions (package, version, filename, mime_type, gzipped_bytes, archived) "
|
||||
"VALUES (?, ?, ?, 'application/javascript', ?, 0)",
|
||||
(name, version, filename, gz),
|
||||
)
|
||||
manifest.append((name, version, filename))
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
with open("bench/urls.txt", "w") as f:
|
||||
for pkg, ver, fn in manifest:
|
||||
f.write(f"/assets/{pkg}/{ver}/{fn}\n")
|
||||
|
||||
print(f"seeded {len(pkg_names)} packages, {len(manifest)} versions -> {DB_PATH}")
|
||||
print(f"wrote {len(manifest)} urls -> bench/urls.txt")
|
||||
Reference in New Issue
Block a user