cproof/tests/bench/README.md

Flat-file backend bench harness

Synthetic load tests for the flat-file database backend. Not part of make check — must be invoked explicitly. See REVIEW.txt Phase-9 plan for the design and scenario list.

Quick start

# small (~40 MB corpus due to mixed length profile, ~10 seconds total)
make bench-quick

# medium (~2 GB corpus, ~1 minute)
make bench BENCH_VOLUME=medium

# max (~20 GB corpus, ~5–10 minutes, requires NVMe + plenty of free disk)
make bench BENCH_VOLUME=max

# Everything: P1 + long messages + failure-injection + S9/S10/S11 variants
make bench-full

# Compare against committed baseline (see Baseline section below)
make bench-compare

# Just the long-message battery (L1–L14, ~5 seconds, self-contained)
make bench-longmsg

# Just failure-injection tests (F1–F15, ~10 ms total)
make bench-failure

# S7/S8 export/import pipeline at default 100k rows
make bench-pipeline

# Same pipeline at 1M rows (~5 min, ~3 GB disk)
make bench-pipeline-max

After a run, results land in tests/bench/current.csv:

scenario,volume,bytes,lines,wall_ms,peak_rss_kb,note
S1_cold_tail,small,1453782,10000,4.512,12340,tail page=100
S2_warm_tail,small,1453782,10000,1.823,12340,tail page=100
...

Volume profiles

Profile	Lines	Years	Length profile	Disk	Time on NVMe
small	10 000	1	mixed	~5 MB	~10 s
medium	500 000	5	mixed	~100 MB	~1 min
max	5 000 000	10	long	~1 GB	~5–10 min

Scenarios in P1 (make bench)

ID	What	Bench function
S1	Cold tail-access (drop cache, fetch last 100 lines via sparse-index seek)	run_tail_access(drop=1)
S2	Warm tail-access (cache hot)	run_tail_access(drop=0)
S3	Deep pagination (1 000 binary-search lookups across the index)	run_deep_pagination
S4	First-time index build (cold file → ff_state_ensure_fresh)	run_first_build
S5	Incremental extend (append N lines, ensure no full rebuild)	run_incremental_extend
S6	Verify-equivalent full parse pass	run_verify

P2 corpus variants (separate make targets)

Each variant generates its own corpus and reuses bench_runner's scenarios.

Target	Corpus	Purpose
make bench-multicontact	200 contacts × 5k lines × 3 years	S9 — directory discovery & multi-state cost
make bench-lmc	100k lines, 30 % LMC corrections	S10 — correction-heavy LMC chain throughput
make bench-ooo	100k lines, 20 % MAM out-of-order	S11 — verify must surface timestamp warnings
make bench-longmsg	(no corpus, self-contained tests)	L1–L14 — long-message stress
make bench-full	runs all of the above sequentially	reporting baseline

Long-message tests (L1–L14)

Run via make bench-longmsg. Each test generates N messages with a specific body size and content pattern, writes via ff_write_line, reads via ff_readline + ff_parse_line, asserts body length match, emits a CSV row.

ID	Body	Pattern	Count	What it verifies
L1	1 KB	filler	100	sanity / fast path
L2	10 KB	filler	100	escape pipeline at moderate size
L3	100 KB	filler	100	paste-bomb territory
L4	1 MB	filler	50	50 MB pipe — measures throughput
L5	5 MB	filler	10	50 MB pipe
L6	9.9 MB	filler	4	just under FF_MAX_LINE_LEN (10 MB)
L7	10 MB+1	filler	1+1	rejection path: ff_readline returns "" + skips
L8	100 KB	embedded \n	50	escape stress (each \n doubles to \\n)
L9	100 KB	embedded |	50	body pipes (not metadata) shouldn't choke parser
L10	100 KB	UTF-8 emoji	50	4-byte codepoints / g_utf8_validate path
L11	1 KB	filler	100	sanity baseline (re-run, expect L1-equivalent)
L12	mixed	5 × 1 MB at end	1000	pagination memory: last-100 with paste-bombs
L13	1 MB	filler	100	full parse pass on 100 MB file
L14	100 KB	filler	1000	sustained append throughput

L7 specifically asserts ff_readline rejects oversized lines (returns "", logs "line too long", continues to next line — the very-next-line is a normal record, must parse OK).

Export/import pipeline (S7/S8)

Run via make bench-pipeline (default 100k rows) or make bench-pipeline-max (1M rows). Each subcommand emits a CSV row; full content diff is enforced on every roundtrip.

Target	Sub-scenarios	What
bench-export	S7a_export_cold, S7b_export_dedup	Seed SQLite N rows → real log_database_export_to_flatfile; second pass measures dedup hot path
bench-import	S7_seed_export, S8a_import_cold, S8b_import_idempotent	Seed → export → wipe DB → real log_database_import_from_flatfile; second import asserts dedup (rows added = 0)
bench-roundtrip	S8e_roundtrip	Seed DB-A → export to flatfile → import to fresh DB-B → full byte-by-byte content diff of every row in (from_jid, to_jid, message, timestamp, type, stanza_id, archive_id, encryption, replace_id)
bench-pipeline	all of the above	combined run
bench-pipeline-max	same, at 1M rows	heavy: ~3 GB disk, ~5 min

The bench links the real database_export.c + database_sqlite.c + database.c so timings reflect actual production code paths (merge sort, GHashTable dedup, transactional INSERT, etc.).

BENCH_PIPE_ROWS overrides the default row count; BENCH_PIPE_ROWS_MAX overrides the max-volume default.

The roundtrip CSV note records each phase wall-time: rows=N seed=Xms export=Yms import=Zms diff=Wms exported=… imported=… mismatches=….

The bench also exposes bench_export_import directly with five subcommands (seed, export, import, roundtrip, verify) for ad-hoc use:

./tests/bench/bench_export_import seed --rows=1000000 --account=foo@bar
./tests/bench/bench_export_import export --account=foo@bar --csv=out.csv
./tests/bench/bench_export_import roundtrip --rows=100000 --csv=out.csv --full-diff
./tests/bench/bench_export_import verify --db-a=A.db --db-b=B.db

Failure-injection (F1–F15)

Run via make bench-failure. Each test crafts a deliberately corrupt log file and asserts how the backend handles it. Exits non-zero on any failure.

ID	Injection	Expected behaviour
F1	Last line truncated mid-write + partial appended line	Verify completes; line counted as a parse error or skipped without crash
F2	CRLF line endings on three lines mid-corpus	Verify reports CRLF warning
F3	UTF-8 BOM bytes appearing mid-file (not at start)	Verify reports unparsable line ERROR for that record
F7	LMC cycle: A→B references and B→A references	Verify completes without infinite loop (cycle handled at read-time, not verify)
F8	LMC chain depth 200 (over FF_MAX_LMC_DEPTH=100)	Verify reports no broken refs (depth-truncation lives at read-time)
F9	Unescaped : inside resource (manual edit)	Parser splits at first : — body truncated. Not flagged by verify (known parser quirk; documented)
F10	RTL override + zero-width space in resource	Bytes preserved literally, parses fine
F11	Latin-1 byte (0xC9) followed by ASCII	Verify reports invalid-UTF-8 ERROR (parse-line has Latin-1 fallback, but verify doesn't apply it — intentional)
F12	Empty body line	Parses OK, no error
F14	File replaced under us (mtime + inode change)	ff_state_ensure_fresh triggers full rebuild; total_lines updates
F15	Empty file (0 bytes)	Verify reports INFO "empty"

Latent gaps surfaced:

• F1: ff_readline sets *truncated=TRUE but verify doesn't surface this — partial writes go unflagged.
• F9: parser silently truncates body at first unescaped : — valid manual edits with embedded : are silently corrupted at read-time.

Baseline & regression checking (P4)

# Capture a fresh CSV
make bench-full BENCH_CSV=tests/bench/current.csv

# Compare against baseline.csv (committed)
make bench-compare BENCH_CSV=tests/bench/current.csv

# Once you've reviewed the numbers and they're healthy, snapshot:
make bench-update-baseline BENCH_CSV=tests/bench/current.csv

compare_baseline.py aggregates by (scenario, volume) and uses median across duplicate rows. Regression threshold defaults to ±25 %; override with BENCH_THRESHOLD=15. Exit code is 0 on no regressions, 1 if any scenario exceeds the threshold.

The committed baseline.csv was captured on this hardware (Debian-bookworm container, NVMe). It's a reference, not absolute — run make bench-full and compare on your own hardware to track regressions over time.

Generator (gen_history) options

--lines=N                          total lines (default 10000)
--contacts=K                       distinct contact JIDs (default 1)
--years=Y                          year span (default 1)
--seed=S                           RNG seed (default 42)
--stanza-id={uuid|libpurple|conversations|mixed}
--lmc-rate=PCT                     0..50, default 3
--mam-ooo-rate=PCT                 0..50, default 0
--resources-per-contact=R          default 3
--msg-len-profile={short|mixed|long|extreme}
--output=DIR                       default /tmp/cproof-bench-corpus
--quiet

Layout produced (matches files_get_data_path("flatlog") + ff_jid_to_dir, so ff_verify_integrity can walk the tree without changes):

$BENCH_DATA_DIR/
  flatlog/
    bench_at_bench.example/             # ff_jid_to_dir(--account)
      buddy000_at_bench.example/history.log
      buddy001_at_bench.example/history.log
      ...
  manifest.txt

Determinism

Same --seed and same --lines etc. → byte-identical corpus. Stanza-ids, timestamps, body content all derive from xorshift64(seed).

The bench runner does not re-generate the corpus — make bench runs gen_history once into $BENCH_DATA_DIR, then runs bench_runner on it. Re-runs reuse the corpus unless you make bench-clean or change BENCH_VOLUME.

Tuning

• BENCH_DATA_DIR=/path — override corpus location (default /tmp/cproof-bench-corpus)
• BENCH_CSV=/path/out.csv — override results CSV
• BENCH_LOG=1 — surface the flatfile backend's log_* output to stderr
• BENCH_VOLUME=small|medium|max

Known limitations

• Tail-access is simulated by sparse-index lookup + read-forward, not the full _flatfile_get_previous_chat (which is unreachable here without pulling in profanity's xmpp/connection layer). This still measures the expensive parts (state build, index seek, line parse, LMC application).
• S6 verify now calls the real ff_verify_integrity and walks the canonical layout. CSV note format: total=N err=N warn=N info=N. S11 OOO flags ~17 500 timestamp-out-of-order warnings on the 100k/20% corpus. (Resolved as of P2.5.)
• S7/S8 export/import wired up in P5 — the bench links real database_export.c + database_sqlite.c + database.c and drives log_database_export_to_flatfile / log_database_import_from_flatfile end-to-end. See "Export/import pipeline (S7/S8)" above.
• No baseline-comparison script yet (P4).
• F1–F15 failure injection not yet implemented (P3).

Disk-usage warning

Variant corpora can be large because the default mixed length profile includes a small fraction of paste-bomb (5–100 KB) and extreme (100 KB–1 MB) messages. Examples:

Target	Lines	On-disk size	Note
bench-quick	10 000	~40 MB	due to ~50 paste-bombs in mixed profile
bench BENCH_VOLUME=medium	500 000	~2 GB	check df first
bench BENCH_VOLUME=max	5 000 000	~20 GB	NVMe + plenty of free disk
bench-multicontact	1 000 000 (200×5k)	~4 GB	200 separate files
bench-lmc	100 000	~400 MB	single-file
bench-ooo	100 000	~400 MB	single-file
bench-pipeline	100 000	~80 MB SQLite + ~50 MB flatfile	export/import
bench-pipeline-max	1 000 000	~800 MB SQLite + ~500 MB flatfile + ~800 MB DB-B	heavy

Set BENCH_DATA_DIR if /tmp is too small.

Adding new scenarios

1. Add a run_* function in bench_runner.c.
2. Wire it into main() with a scenario_enabled("Sx") check.
3. Document it in this README + REVIEW.txt Phase 9 plan.
4. Run make bench-full BENCH_CSV=tests/bench/current.csv && make bench-compare — verify no unexpected regressions surface elsewhere.

File map

tests/bench/
├── README.md                  # this file
├── baseline.csv               # committed reference numbers
├── compare_baseline.py        # diff current.csv vs baseline.csv
├── gen_history.c              # corpus generator (P1 + S9/S10/S11)
├── bench_runner.c             # S1–S6 driver
├── bench_long_messages.c      # L1–L14 driver
├── bench_failure_modes.c      # F1–F15 driver
├── bench_export_import.c      # S7/S8 driver (seed/export/import/roundtrip/verify)
├── bench_stubs.c              # link-time stubs (log_*, prefs_*, etc.)
├── bench_common.c/.h          # timing, RSS, formatting helpers
└── bench_csv.c/.h             # CSV-row writer