555faaa2328a3fc607b2cafa2ba0860a2ffac875
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
555faaa232
|
perf+harden(export): 25x faster export at 1M, mkstemp, full-body dedup
Some checks failed
CI Code / Check spelling (pull_request) Failing after 21s
CI Code / Linux (arch) (pull_request) Failing after 3m6s
CI Code / Check coding style (pull_request) Failing after 3m10s
CI Code / Code Coverage (pull_request) Successful in 2m49s
CI Code / Linux (debian) (pull_request) Successful in 4m26s
CI Code / Linux (ubuntu) (pull_request) Successful in 4m38s
Address 6 of 7 high/medium audit findings on database_export.c.
End-to-end migration of 1M rows: ~5 min -> ~1 min.
H1 cache g_slist_length(merged) out of the write loop — was O(n²)
via 2000 list walks at 500-row progress interval, now O(1).
S7a_export_cold @ 1M: 309 s -> 11.2 s (-96 %)
L1 splice existing flatfile lines into merged via ownership
transfer instead of 12-strdup deep-copy + g_date_time_ref;
free list nodes only, NULL out existing.
S7b_export_dedup @ 1M: 304 s -> 13.3 s (-96 %)
RSS @ S7b 1M: 2.4 GB -> 1.9 GB
H2 mkstemp() with random suffix instead of fixed
"{log_path}.export.tmp". Eliminates the unlink-and-retry race
where two concurrent exports could clobber each other's
in-flight tmp files.
M4 SHA-256 over full body (incremental g_checksum_update) in
the dedup-key fallback. Previous body[:256] hash collided
on common preambles (signatures, code blocks) — second
occurrence silently dropped on import.
M2 Refuse _ff_read_all_lines on files > 2 GB. Each line expands
~10x in heap on parse; without a cap we OOM-kill prof on
large flatfiles. Surface a clear error instead.
M5 Break the outer import loop on system-level INSERT failure.
Was cascading "Import of X failed" through every remaining
contact when the real issue was disk-full / sqlite locked.
Verified on bench-pipeline-max: 1M rows, full content diff,
mismatches=0. baseline.csv updated.
Deferred: M3 O_TMPFILE (platform-conditional + linkat dance;
mkstemp covers most of the same window) and M6 SIGINT cancellation
(needs event-loop architecture changes).
|
|||
|
b1d820462a
|
feat(bench): synthetic load harness for flat-file backend (P1-P5)
End-to-end performance and correctness harness for the flat-file +
SQLite database backends. Lives in tests/bench/, built only on
demand (`make bench`); not part of `make check`.
Components
gen_history (P1)
Deterministic corpus generator. Knobs: lines, contacts, years,
seed, stanza-id mode (uuid/libpurple/conversations/mixed), LMC
rate, MAM-OOO rate, resources/contact, length profile
(short/mixed/long/extreme). Emits the canonical
flatlog/<account>/<contact>/history.log layout used by
ff_verify_integrity. ~340 LOC.
bench_runner (P1, P2.5)
S1 cold tail-access via sparse index
S2 warm tail-access (page cache hot)
S3 deep pagination (1000 binary-search lookups)
S4 first-time index build (cold file -> ff_state_ensure_fresh)
S5 incremental extend (asserts no full rebuild path)
S6 real ff_verify_integrity over the contact tree
Reports total/err/warn/info issue counts in the CSV note.
bench_long_messages (P2)
L1-L14 long-message stress: 1KB up to 9.9MB bodies, plus
oversized line rejection (10MB+1 -> ff_readline returns ""),
embedded-newline / pipe / emoji body patterns, full parse on
100x1MB, 1000x100KB sustained append.
bench_failure_modes (P3)
F1-F15 failure-injection: truncated last line, mid-file CRLF,
mid-file BOM, LMC cycle, LMC depth>FF_MAX_LMC_DEPTH, manual
': ' in resource, RTL/ZWSP, Latin-1 byte, empty body,
mtime/inode flip, empty file. Each test asserts expected issue
levels and reports PASS/FAIL.
bench_export_import (P5)
Links real database_export.c + database_sqlite.c + database.c
and drives log_database_export_to_flatfile /
log_database_import_from_flatfile under load.
Subcommands: seed, export, import, roundtrip, verify.
S7a/b export, S8a/b import, S8e roundtrip with full byte-by-byte
content diff of every row in (from_jid, to_jid, message,
timestamp, type, stanza_id, archive_id, encryption, replace_id).
Make targets
bench-quick / bench / bench-full
bench-longmsg, bench-failure
bench-multicontact, bench-lmc, bench-ooo
bench-export, bench-import, bench-roundtrip
bench-pipeline, bench-pipeline-max (1M rows)
bench-compare, bench-update-baseline
Volume controls: BENCH_VOLUME (small/medium/max), BENCH_PIPE_ROWS,
BENCH_PIPE_ROWS_MAX, BENCH_DATA_DIR, BENCH_CSV.
Baseline + regression checking (P4)
tests/bench/baseline.csv 51 rows: S1-S6 x {small,lmc,ooo}
+ L1-L14 + F1-F15 (11 of 15) +
S7/S8 x {pipe100k, pipe1M}.
compare_baseline.py median over duplicate rows;
exits 1 on any (scenario, volume)
slowdown >= threshold (default 25%).
Verified at scale
bench-pipeline-max: 1,000,000 rows, full content diff
seed 17 s, export 304 s, import 31 s, idempotent re-import 10 s,
diff 2.7 s -- mismatches=0.
Findings surfaced by the harness
Export scales super-linearly: 4 s @ 100k -> 304 s @ 1M (76x for
10x rows). Cause: g_slist_sort on the merged list + per-row
ProfMessage/ff_parsed_line_t allocations. RSS peaks at 1.4 GB
on 1M. Worth a follow-up.
Export progress reporting only fires during the write phase --
the merge+sort phase (~95% of wall time at 1M) is silent.
/history export and /history import are blocking on the main
UI thread; profanity is frozen for the duration (~5 min @ 1M).
ff_readline sets *truncated=TRUE on partial-write tail but
ff_verify_integrity does not surface this -- partial writes
go unflagged (failure-injection F1).
Parser silently truncates body at first unescaped ': ' if a
resource was manually edited to contain it (F9).
In gen_history (caught by the bench's own real-verify pass):
g_strndup mid-codepoint truncation on UTF-8 bank strings -- fixed.
Linkage strategy
database_flatfile.c + parser + verify + common.c are linked
unconditionally. The export/import bench additionally links
database.c + database_sqlite.c + database_export.c. bench_stubs.c
provides minimal stubs for log_*, prefs_*, connection_get_jid,
jid_create, files_*, message_*, ui hooks. integrity_issue_free
is a weak symbol so it falls back to the real database.c
implementation when that file is linked.
|
