feat(history): flat-file backend with bidirectional SQLite migration

A flat-file alternative to the SQLite chatlog backend with runtime switching, full migration tooling, integrity verification, and a synthetic load harness. SQLite remains the default; both backends share one dispatch layer (db_backend_t vtable) so callers don't change. Storage layout - Per-contact append-only `flatlog/<account>/<contact>/history.log` under XDG_DATA_HOME, one line per message - Single-line file header with embedded format-version marker (FLATFILE_FORMAT_VERSION); reader warns on missing or mismatched marker, writer and checker stay in sync via preprocessor stringification - Deterministic key=value metadata (`id`, `aid`, `corrects`, `to`, `to_res`, `read`) plus escaped body \u2014 `\|`, `\]`, `\\`, `\n`, `\r` literals prevent log injection - Sparse byte-offset index (FF_INDEX_STEP=500) per contact for O(log n) time-range lookups; rebuilt on inode / size / mtime change, extended in-place when the file just grew - Per-contact GHashTable caches for archive_id presence and stanza_id \u2192 from_jid mapping (O(1) MAM dedup, O(1) LMC sender validation) Hardening - Path-traversal protection: JID directory name normalisation (`@` \u2192 `_at_`, slashes and `..` rejected at construction); every per-contact path is anchored under the account's flatlog/ directory and validated before open - Symlink-attack protection: every fopen / open uses O_NOFOLLOW; on ELOOP the operation aborts with an error rather than following - Filesystem permissions: log files created with mode 0600, directories with mode 0700; both enforced at creation, verified on each open and reported on drift by `/history verify` - Atomic crash-safe export: write to a temp file via mkstemp (mode 0600, random suffix, no name collisions between concurrent exports), fsync, then rename \u2014 partial state never replaces the live file - Concurrency: advisory flock(LOCK_EX) held for the duration of every write, including append from live messages and full rewrite from export, so two profanity processes can't interleave bytes on the same log - DoS / abuse guards: * FF_MAX_LINE_LEN = 10 MB \u2014 lines longer than this are rejected at read with a warning; the parser will not allocate unbounded memory for a single record * FF_MAX_LMC_DEPTH = 100 \u2014 `corrects:` chain walk stops at this depth and emits a warning, preventing a malicious correction cycle from spinning the apply pass * FF_VERSION_SCAN_MAX = 16 \u2014 header version probe never reads past 16 leading comment lines, even on garbage input * Empty / inverted byte-range early-return in page-up read path so a malformed time filter cannot cause an unbounded scan * Zero-entry index guard so a file whose every line failed to parse cannot cause a NULL deref on later page-up - LMC sender validation: an incoming correction whose sender does not match the original message's sender is rejected at write time and surfaced via cons_show_error; a cycle in the apply pass is broken via a visited-set - jid_create_from_bare_and_resource treats NULL, empty string, and the literal "(null)" as no resource and returns a bare jid; similar normalisation for barejid eliminates the legacy "user@host/(null)" artefact that leaked into stored fulljids whenever g_strdup_printf("%s", NULL) ran inside create_fulljid Commands - `/history switch sqlite|flatfile` \u2014 runtime backend swap, closes the old backend and opens the new one without reconnecting - `/history export [<jid>]` \u2014 SQLite -> flat-file, merging with any existing flatlog (dedup keyed on a SHA-256 hash mixing stanza_id, timestamp, from_jid, body \u2014 robust against id reuse by older clients) - `/history import [<jid>]` \u2014 flat-file -> SQLite, same merge semantics, runs inside a single SQLite transaction with rollback on per-contact failure - `/history verify [<jid>]` \u2014 integrity check; emits a structured list of issues (ERROR / WARNING / INFO) per file: * file-level: missing log, wrong permissions (\u2260 0600), UTF-8 BOM present, CRLF line endings, empty file * line-level: invalid UTF-8 (with byte offset), embedded control characters, unparsable lines, timestamps out of order, duplicate `id:` and `aid:` (tracked separately so a stanza/archive id collision isn't double-reported) * cross-line: broken `corrects:` references whose target id is not present in the file - `/history backend` \u2014 show currently active backend - Active backend indicator `[sqlite]` / `[flatfile]` in the status bar next to the JID - Roster-JID autocomplete for verify / export / import - export and import open a SQLite handle on demand when the flatfile backend is currently active, so migration works regardless of which backend is live Tests - Unit: database_export (parser round-trip, escape/unescape, dedup key stability, JID normalisation), database_stress (14 cases exercising rapid writes, large messages, deep LMC chains, MAM dedup, concurrent contacts) - Functional: history persistence across reconnects, export / import round-trip with content equality, MUC migration, timestamp normalisation across timezones - Bench harness P1\u2013P5 (synthetic load: bulk insert, time-range read, page-up scroll, MAM ingest, mixed workload) and failure modes F1\u2013F17 (page-up cursor and forward-iteration symmetry, oversized lines, MAM dedup, LMC depth and cycles, BOM/CRLF, missing log, empty file, mtime+inode flip, broken corrects, etc.) - All bench tests integrate with the existing make targets and emit CSV rows for baseline comparison Author: jabber.developer2 <jabber.developer2@jabber.space> Reviewed-by: jabber.developer <jabber.developer@jabber.space>
2026-05-05 19:26:07 +00:00
parent 0feacbc9da
commit 3f36c303c2
50 changed files with 13811 additions and 793 deletions
--- a/tests/bench/bench_runner.c
+++ b/tests/bench/bench_runner.c
@@ -0,0 +1,558 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+//
+// This file is part of CProof.
+// See LICENSE for the full GPLv3 text and the special OpenSSL linking exception.
+
+/*
+ * bench_runner.c
+ * vim: expandtab:ts=4:sts=4:sw=4
+ *
+ * Bench harness driver for the flat-file backend. Measures the underlying
+ * mechanics (state-build, sparse-index lookup, line throughput, verify).
+ *
+ * Pipeline:
+ *   1. Run gen_history first to populate $BENCH_DATA_DIR/contacts/<jid>/history.log.
+ *   2. ./bench_runner --data=DIR --csv=PATH [--scenarios=S1,S2,...]
+ *   3. Each scenario writes a row to the CSV.
+ *
+ * The harness does NOT invoke gen_history itself — that's the Makefile's
+ * job. This keeps the binary's deps minimal.
+ *
+ * Scenarios (S1–S6 in P1):
+ *   S1  cold tail-access    open contact, drop cache, fetch last 100 lines
+ *   S2  warm tail-access    same as S1 but page-cache hot
+ *   S3  deep pagination     scroll back N=100 pages × 100 lines via index
+ *   S4  first index build   ff_state_new + ff_state_ensure_fresh on cold file
+ *   S5  incremental extend  append M lines, ensure_fresh -> measure (no rebuild)
+ *   S6  verify integrity    ff_verify_integrity over full file
+ */
+
+#include "config.h"
+
+#include <errno.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <time.h>
+#include <unistd.h>
+
+#include <glib.h>
+#include <glib/gstdio.h>
+
+#include "bench_common.h"
+#include "bench_csv.h"
+#include "database_flatfile.h"
+
+// g_flatfile_account_jid is declared in database_flatfile.h. We assign to
+// it directly (skip _flatfile_init which pulls in xmpp / connection deps).
+
+// ---------------------------------------------------------------------------
+// CLI
+
+typedef struct
+{
+    const char* data_dir;
+    const char* csv_path;
+    const char* scenarios; // comma-separated list, or "all"
+    const char* account_jid;
+    int extend_lines; // for S5
+} cli_t;
+
+static void
+cli_default(cli_t* c)
+{
+    c->data_dir = NULL;
+    c->csv_path = NULL;
+    c->scenarios = "all";
+    c->account_jid = "bench@bench.example";
+    c->extend_lines = 1000;
+}
+
+static int
+parse_args(int argc, char** argv, cli_t* c)
+{
+    cli_default(c);
+    for (int i = 1; i < argc; i++) {
+        const char* a = argv[i];
+        if (strncmp(a, "--data=", 7) == 0)
+            c->data_dir = a + 7;
+        else if (strncmp(a, "--csv=", 6) == 0)
+            c->csv_path = a + 6;
+        else if (strncmp(a, "--scenarios=", 12) == 0)
+            c->scenarios = a + 12;
+        else if (strncmp(a, "--account=", 10) == 0)
+            c->account_jid = a + 10;
+        else if (strncmp(a, "--extend-lines=", 15) == 0)
+            c->extend_lines = atoi(a + 15);
+        else {
+            fprintf(stderr, "unknown option: %s\n", a);
+            return -1;
+        }
+    }
+    if (!c->data_dir) {
+        fprintf(stderr, "--data=DIR required\n");
+        return -1;
+    }
+    return 0;
+}
+
+static int
+scenario_enabled(const char* list, const char* name)
+{
+    if (!list || g_strcmp0(list, "all") == 0)
+        return 1;
+    auto_gcharv gchar** parts = g_strsplit(list, ",", -1);
+    for (int i = 0; parts[i]; i++) {
+        if (g_strcmp0(g_strstrip(parts[i]), name) == 0)
+            return 1;
+    }
+    return 0;
+}
+
+// ---------------------------------------------------------------------------
+// Pick the largest history.log in the corpus — that's our "primary" contact.
+
+typedef struct
+{
+    char* path;
+    int64_t size;
+    int64_t lines; // best-effort
+} corpus_pick_t;
+
+static int64_t
+count_lines(const char* path)
+{
+    FILE* fp = fopen(path, "r");
+    if (!fp)
+        return -1;
+    int64_t n = 0;
+    int c;
+    while ((c = fgetc(fp)) != EOF) {
+        if (c == '\n')
+            n++;
+    }
+    fclose(fp);
+    return n;
+}
+
+static gboolean
+pick_primary(const char* data_dir, const char* account_dir, corpus_pick_t* out)
+{
+    auto_gchar gchar* contacts_root = g_strdup_printf("%s/flatlog/%s",
+                                                      data_dir, account_dir);
+    GDir* d = g_dir_open(contacts_root, 0, NULL);
+    if (!d) {
+        fprintf(stderr, "cannot open %s\n", contacts_root);
+        return FALSE;
+    }
+    char* best_path = NULL;
+    int64_t best_size = 0;
+    const char* name;
+    while ((name = g_dir_read_name(d)) != NULL) {
+        auto_gchar gchar* path = g_strdup_printf("%s/%s/history.log", contacts_root, name);
+        struct stat st;
+        if (stat(path, &st) == 0 && S_ISREG(st.st_mode) && st.st_size > best_size) {
+            best_size = st.st_size;
+            g_free(best_path);
+            best_path = g_strdup(path);
+        }
+    }
+    g_dir_close(d);
+    if (!best_path) {
+        fprintf(stderr, "no history.log under %s\n", contacts_root);
+        return FALSE;
+    }
+    out->path = best_path;
+    out->size = best_size;
+    fprintf(stderr, "  primary contact: %s (%" PRId64 " bytes)\n", best_path, best_size);
+    out->lines = count_lines(best_path);
+    fprintf(stderr, "  primary lines:   %" PRId64 "\n", out->lines);
+    return TRUE;
+}
+
+// ---------------------------------------------------------------------------
+// Scenario S1/S2: tail-access via state index
+//
+// Build state for the file (fresh first time, cached subsequent), then for
+// the last 100 lines:
+//   - find the byte offset of the (n_entries-1)-th sparse-index entry
+//   - seek there, read forward to EOF with ff_readline + ff_parse_line,
+//     keep the last 100 in a ring buffer.
+// This mirrors the read path used by _flatfile_get_previous_chat without
+// pulling in xmpp/connection deps.
+
+#define TAIL_PAGE 100
+
+static double
+run_tail_access(const char* path, int drop_cache, long* peak_rss_kb, int64_t* lines_read)
+{
+    if (drop_cache)
+        bench_drop_page_cache(path);
+
+    double t0 = bench_now_ms();
+
+    ff_contact_state_t* state = ff_state_new(path);
+    if (!ff_state_ensure_fresh(state)) {
+        ff_state_free(state);
+        if (lines_read)
+            *lines_read = 0;
+        return -1.0;
+    }
+
+    off_t start = 0;
+    if (state->n_entries > 1) {
+        start = state->entries[state->n_entries - 1].byte_offset;
+    } else {
+        start = state->bom_len; // tiny file
+    }
+
+    FILE* fp = fopen(path, "r");
+    if (!fp) {
+        ff_state_free(state);
+        return -1.0;
+    }
+    if (fseeko(fp, start, SEEK_SET) != 0) {
+        fclose(fp);
+        ff_state_free(state);
+        return -1.0;
+    }
+
+    char* ring[TAIL_PAGE];
+    memset(ring, 0, sizeof(ring));
+    int ring_pos = 0;
+    int64_t parsed = 0;
+
+    char* buf;
+    while ((buf = ff_readline(fp, NULL)) != NULL) {
+        if (buf[0] == '\0' || buf[0] == '#') {
+            free(buf);
+            continue;
+        }
+        ff_parsed_line_t* pl = ff_parse_line(buf);
+        free(buf);
+        if (!pl)
+            continue;
+        parsed++;
+        if (ring[ring_pos])
+            g_free(ring[ring_pos]);
+        ring[ring_pos] = g_strdup(pl->message ? pl->message : "");
+        ring_pos = (ring_pos + 1) % TAIL_PAGE;
+        ff_parsed_line_free(pl);
+    }
+    fclose(fp);
+    ff_state_free(state);
+
+    for (int i = 0; i < TAIL_PAGE; i++)
+        g_free(ring[i]);
+
+    double dt = bench_now_ms() - t0;
+    if (peak_rss_kb)
+        *peak_rss_kb = bench_peak_rss_kb();
+    if (lines_read)
+        *lines_read = parsed;
+    return dt;
+}
+
+// ---------------------------------------------------------------------------
+// S3: deep pagination — for each of N pages, pick a synthetic ts hint and
+// call ff_state_offset_for_time to locate the page start. Measures binary
+// search throughput.
+
+static double
+run_deep_pagination(const char* path, int n_pages, long* peak_rss_kb)
+{
+    ff_contact_state_t* state = ff_state_new(path);
+    if (!ff_state_ensure_fresh(state)) {
+        ff_state_free(state);
+        return -1.0;
+    }
+    if (state->n_entries < 2) {
+        ff_state_free(state);
+        return 0.0;
+    }
+
+    double t0 = bench_now_ms();
+    // Walk through index entries, calling ff_state_offset_for_time with their
+    // recorded epochs (cheaper than parsing, still exercises binary search).
+    for (int i = 0; i < n_pages; i++) {
+        size_t idx = (size_t)i % state->n_entries;
+        gint64 epoch = state->entries[idx].timestamp_epoch;
+        // Convert epoch to iso roughly — ff_state_offset_for_time parses ISO.
+        time_t t = (time_t)epoch;
+        GDateTime* dt = g_date_time_new_from_unix_utc((gint64)t);
+        auto_gchar gchar* iso = g_date_time_format_iso8601(dt);
+        g_date_time_unref(dt);
+        (void)ff_state_offset_for_time(state, iso);
+    }
+    double dt = bench_now_ms() - t0;
+    if (peak_rss_kb)
+        *peak_rss_kb = bench_peak_rss_kb();
+    ff_state_free(state);
+    return dt;
+}
+
+// ---------------------------------------------------------------------------
+// S4: first-time build — drop cache, ff_state_new + ensure_fresh
+
+static double
+run_first_build(const char* path, long* peak_rss_kb, size_t* idx_entries)
+{
+    bench_drop_page_cache(path);
+    double t0 = bench_now_ms();
+    ff_contact_state_t* state = ff_state_new(path);
+    int ok = ff_state_ensure_fresh(state);
+    double dt = bench_now_ms() - t0;
+    if (idx_entries)
+        *idx_entries = ok ? state->n_entries : 0;
+    if (peak_rss_kb)
+        *peak_rss_kb = bench_peak_rss_kb();
+    ff_state_free(state);
+    return ok ? dt : -1.0;
+}
+
+// ---------------------------------------------------------------------------
+// S5: incremental extend — already-built state, then append N synthetic lines
+// (writing them via ff_write_line directly) and call ensure_fresh to verify
+// it takes the extend path, not full rebuild.
+
+static double
+run_incremental_extend(const char* path, int n_lines, long* peak_rss_kb,
+                       int64_t* added_bytes)
+{
+    ff_contact_state_t* state = ff_state_new(path);
+    if (!ff_state_ensure_fresh(state)) {
+        ff_state_free(state);
+        return -1.0;
+    }
+    size_t before_entries = state->n_entries;
+    size_t before_lines = state->total_lines;
+    (void)before_entries;
+
+    // Append n_lines lines directly to the file
+    FILE* fp = fopen(path, "a");
+    if (!fp) {
+        ff_state_free(state);
+        return -1.0;
+    }
+    int64_t before_size = -1;
+    {
+        struct stat st;
+        if (stat(path, &st) == 0)
+            before_size = st.st_size;
+    }
+    GDateTime* now = g_date_time_new_now_utc();
+    for (int i = 0; i < n_lines; i++) {
+        GDateTime* t = g_date_time_add_seconds(now, i);
+        auto_gchar gchar* iso = g_date_time_format_iso8601(t);
+        g_date_time_unref(t);
+        auto_gchar gchar* sid = g_strdup_printf("ext-%d-%ld", i, (long)random());
+        ff_write_line(fp, iso, "chat", "none",
+                      sid, NULL, NULL,
+                      "buddy000@bench.example", "ext",
+                      NULL, NULL, -1,
+                      "extend payload");
+    }
+    g_date_time_unref(now);
+    fclose(fp);
+
+    // Now measure: ensure_fresh should hit the extend path.
+    double t0 = bench_now_ms();
+    int ok = ff_state_ensure_fresh(state);
+    double dt = bench_now_ms() - t0;
+
+    if (added_bytes) {
+        struct stat st;
+        if (stat(path, &st) == 0 && before_size >= 0)
+            *added_bytes = (int64_t)st.st_size - before_size;
+        else
+            *added_bytes = -1;
+    }
+    if (peak_rss_kb)
+        *peak_rss_kb = bench_peak_rss_kb();
+
+    fprintf(stderr, "  S5: lines before=%zu after=%zu (delta=%zu)\n",
+            before_lines, state->total_lines,
+            state->total_lines - before_lines);
+    ff_state_free(state);
+    return ok ? dt : -1.0;
+}
+
+// ---------------------------------------------------------------------------
+// S6: verify integrity — calls the real ff_verify_integrity, walking the
+// canonical $data/flatlog/$account/$contact/history.log layout that
+// gen_history produces. We set g_flatfile_account_jid in main() and
+// BENCH_DATA_DIR + the stubbed files_get_data_path() resolves the prefix.
+
+static int
+_count_level(GSList* issues, integrity_level_t level)
+{
+    int n = 0;
+    for (GSList* l = issues; l; l = l->next) {
+        integrity_issue_t* i = (integrity_issue_t*)l->data;
+        if (i && i->level == level)
+            n++;
+    }
+    return n;
+}
+
+static double
+run_verify(long* peak_rss_kb, int64_t* total_issues, int* errors, int* warnings, int* infos)
+{
+    // Drop cache for everything under the flatlog tree we'll walk. Cheap
+    // compared to the verify itself, and keeps the cold-start signal clean.
+    auto_gchar gchar* data_path = g_strdup_printf("%s/flatlog", getenv("BENCH_DATA_DIR"));
+    (void)data_path;
+
+    double t0 = bench_now_ms();
+    GSList* issues = ff_verify_integrity(NULL); // NULL = all contacts
+    double dt = bench_now_ms() - t0;
+
+    int e = _count_level(issues, INTEGRITY_ERROR);
+    int w = _count_level(issues, INTEGRITY_WARNING);
+    int i = _count_level(issues, INTEGRITY_INFO);
+    int total = g_slist_length(issues);
+
+    // Surface up to 5 errors / 3 warnings — helps debug bench-vs-real mismatches.
+    int shown_err = 0, shown_warn = 0;
+    for (GSList* l = issues; l; l = l->next) {
+        integrity_issue_t* iss = (integrity_issue_t*)l->data;
+        if (!iss)
+            continue;
+        if (iss->level == INTEGRITY_ERROR && shown_err < 5) {
+            fprintf(stderr, "    ERR  %s:%d  %s\n",
+                    iss->file ? iss->file : "?", iss->line, iss->message ? iss->message : "");
+            shown_err++;
+        } else if (iss->level == INTEGRITY_WARNING && shown_warn < 3) {
+            fprintf(stderr, "    WARN %s:%d  %s\n",
+                    iss->file ? iss->file : "?", iss->line, iss->message ? iss->message : "");
+            shown_warn++;
+        }
+    }
+    g_slist_free_full(issues, (GDestroyNotify)integrity_issue_free);
+
+    if (peak_rss_kb)
+        *peak_rss_kb = bench_peak_rss_kb();
+    if (total_issues)
+        *total_issues = total;
+    if (errors)
+        *errors = e;
+    if (warnings)
+        *warnings = w;
+    if (infos)
+        *infos = i;
+    return dt;
+}
+
+// ---------------------------------------------------------------------------
+// Driver
+
+int
+main(int argc, char** argv)
+{
+    cli_t cli;
+    if (parse_args(argc, argv, &cli) != 0)
+        return 2;
+
+    g_flatfile_account_jid = g_strdup(cli.account_jid);
+    auto_gchar gchar* account_dir = ff_jid_to_dir(cli.account_jid);
+
+    // Sanity: data_dir/flatlog/<account_dir>/ must exist
+    auto_gchar gchar* contacts_root = g_strdup_printf("%s/flatlog/%s",
+                                                      cli.data_dir, account_dir);
+    if (!g_file_test(contacts_root, G_FILE_TEST_IS_DIR)) {
+        fprintf(stderr, "ERROR: %s does not exist. Run gen_history first "
+                        "(make sure --account matches).\n",
+                contacts_root);
+        return 2;
+    }
+
+    corpus_pick_t pick = { 0 };
+    if (!pick_primary(cli.data_dir, account_dir, &pick))
+        return 2;
+
+    const char* volume_name = getenv("BENCH_VOLUME");
+    if (!volume_name)
+        volume_name = "medium";
+
+    fprintf(stderr, "===== bench_runner: data=%s volume=%s =====\n", cli.data_dir, volume_name);
+
+    // ---- S1: cold tail access
+    if (scenario_enabled(cli.scenarios, "S1")) {
+        long rss = 0;
+        int64_t lr = 0;
+        double dt = run_tail_access(pick.path, /*drop=*/1, &rss, &lr);
+        fprintf(stderr, "  S1 cold tail-access: %.2f ms (read %" PRId64 " lines)\n", dt, lr);
+        bench_csv_append(cli.csv_path, "S1_cold_tail", volume_name,
+                         (uint64_t)pick.size, (uint64_t)pick.lines, dt, rss,
+                         "tail page=100");
+    }
+
+    // ---- S2: warm tail access (run twice, take the warm)
+    if (scenario_enabled(cli.scenarios, "S2")) {
+        long rss = 0;
+        int64_t lr = 0;
+        run_tail_access(pick.path, /*drop=*/0, NULL, NULL);
+        double dt = run_tail_access(pick.path, /*drop=*/0, &rss, &lr);
+        fprintf(stderr, "  S2 warm tail-access: %.2f ms (read %" PRId64 " lines)\n", dt, lr);
+        bench_csv_append(cli.csv_path, "S2_warm_tail", volume_name,
+                         (uint64_t)pick.size, (uint64_t)pick.lines, dt, rss,
+                         "tail page=100");
+    }
+
+    // ---- S3: deep pagination
+    if (scenario_enabled(cli.scenarios, "S3")) {
+        long rss = 0;
+        double dt = run_deep_pagination(pick.path, /*n_pages=*/1000, &rss);
+        fprintf(stderr, "  S3 deep pagination (1000 lookups): %.2f ms\n", dt);
+        bench_csv_append(cli.csv_path, "S3_deep_pagination", volume_name,
+                         (uint64_t)pick.size, (uint64_t)pick.lines, dt, rss,
+                         "n_pages=1000");
+    }
+
+    // ---- S4: first-time index build
+    if (scenario_enabled(cli.scenarios, "S4")) {
+        long rss = 0;
+        size_t entries = 0;
+        double dt = run_first_build(pick.path, &rss, &entries);
+        fprintf(stderr, "  S4 first build: %.2f ms (idx entries=%zu)\n", dt, entries);
+        auto_gchar gchar* note = g_strdup_printf("idx_entries=%zu", entries);
+        bench_csv_append(cli.csv_path, "S4_first_build", volume_name,
+                         (uint64_t)pick.size, (uint64_t)pick.lines, dt, rss, note);
+    }
+
+    // ---- S5: incremental extend
+    if (scenario_enabled(cli.scenarios, "S5")) {
+        long rss = 0;
+        int64_t added = 0;
+        double dt = run_incremental_extend(pick.path, cli.extend_lines, &rss, &added);
+        fprintf(stderr, "  S5 incremental extend (%d lines, %" PRId64 " bytes): %.2f ms\n",
+                cli.extend_lines, added, dt);
+        auto_gchar gchar* note = g_strdup_printf("appended=%d_lines", cli.extend_lines);
+        bench_csv_append(cli.csv_path, "S5_incremental_extend", volume_name,
+                         (uint64_t)added, (uint64_t)cli.extend_lines, dt, rss, note);
+    }
+
+    // ---- S6: verify integrity (real ff_verify_integrity over the tree)
+    if (scenario_enabled(cli.scenarios, "S6")) {
+        long rss = 0;
+        int64_t total = 0;
+        int errors = 0, warnings = 0, infos = 0;
+        double dt = run_verify(&rss, &total, &errors, &warnings, &infos);
+        fprintf(stderr,
+                "  S6 verify_integrity: %.2f ms (total=%" PRId64
+                " err=%d warn=%d info=%d)\n",
+                dt, total, errors, warnings, infos);
+        auto_gchar gchar* note = g_strdup_printf(
+            "total=%" PRId64 " err=%d warn=%d info=%d", total, errors, warnings, infos);
+        bench_csv_append(cli.csv_path, "S6_verify", volume_name,
+                         (uint64_t)pick.size, (uint64_t)pick.lines, dt, rss, note);
+    }
+
+    g_free(pick.path);
+    g_free(g_flatfile_account_jid);
+    g_flatfile_account_jid = NULL;
+    return 0;
+}