feat(history): flat-file backend with bidirectional SQLite migration

A flat-file alternative to the SQLite chatlog backend with runtime
switching, full migration tooling, integrity verification, and a
synthetic load harness. SQLite remains the default; both backends share
one dispatch layer (db_backend_t vtable) so callers don't change.

Storage layout
- Per-contact append-only `flatlog/<account>/<contact>/history.log`
  under XDG_DATA_HOME, one line per message
- Single-line file header with embedded format-version marker
  (FLATFILE_FORMAT_VERSION); reader warns on missing or mismatched
  marker, writer and checker stay in sync via preprocessor
  stringification
- Deterministic key=value metadata (`id`, `aid`, `corrects`, `to`,
  `to_res`, `read`) plus escaped body \u2014 `\|`, `\]`, `\\`, `\n`, `\r`
  literals prevent log injection
- Sparse byte-offset index (FF_INDEX_STEP=500) per contact for
  O(log n) time-range lookups; rebuilt on inode / size / mtime
  change, extended in-place when the file just grew
- Per-contact GHashTable caches for archive_id presence and
  stanza_id \u2192 from_jid mapping (O(1) MAM dedup, O(1) LMC sender
  validation)

Hardening
- Path-traversal protection: JID directory name normalisation
  (`@` \u2192 `_at_`, slashes and `..` rejected at construction); every
  per-contact path is anchored under the account's flatlog/
  directory and validated before open
- Symlink-attack protection: every fopen / open uses O_NOFOLLOW; on
  ELOOP the operation aborts with an error rather than following
- Filesystem permissions: log files created with mode 0600,
  directories with mode 0700; both enforced at creation, verified
  on each open and reported on drift by `/history verify`
- Atomic crash-safe export: write to a temp file via mkstemp (mode
  0600, random suffix, no name collisions between concurrent
  exports), fsync, then rename \u2014 partial state never replaces the
  live file
- Concurrency: advisory flock(LOCK_EX) held for the duration of
  every write, including append from live messages and full rewrite
  from export, so two profanity processes can't interleave bytes
  on the same log
- DoS / abuse guards:
    * FF_MAX_LINE_LEN = 10 MB \u2014 lines longer than this are rejected
      at read with a warning; the parser will not allocate
      unbounded memory for a single record
    * FF_MAX_LMC_DEPTH = 100 \u2014 `corrects:` chain walk stops at this
      depth and emits a warning, preventing a malicious correction
      cycle from spinning the apply pass
    * FF_VERSION_SCAN_MAX = 16 \u2014 header version probe never reads
      past 16 leading comment lines, even on garbage input
    * Empty / inverted byte-range early-return in page-up read path
      so a malformed time filter cannot cause an unbounded scan
    * Zero-entry index guard so a file whose every line failed to
      parse cannot cause a NULL deref on later page-up
- LMC sender validation: an incoming correction whose sender does
  not match the original message's sender is rejected at write
  time and surfaced via cons_show_error; a cycle in the apply pass
  is broken via a visited-set
- jid_create_from_bare_and_resource treats NULL, empty string, and
  the literal "(null)" as no resource and returns a bare jid;
  similar normalisation for barejid eliminates the legacy
  "user@host/(null)" artefact that leaked into stored fulljids
  whenever g_strdup_printf("%s", NULL) ran inside create_fulljid

Commands
- `/history switch sqlite|flatfile` \u2014 runtime backend swap, closes
  the old backend and opens the new one without reconnecting
- `/history export [<jid>]` \u2014 SQLite -> flat-file, merging with any
  existing flatlog (dedup keyed on a SHA-256 hash mixing stanza_id,
  timestamp, from_jid, body \u2014 robust against id reuse by older
  clients)
- `/history import [<jid>]` \u2014 flat-file -> SQLite, same merge
  semantics, runs inside a single SQLite transaction with rollback
  on per-contact failure
- `/history verify [<jid>]` \u2014 integrity check; emits a structured
  list of issues (ERROR / WARNING / INFO) per file:
    * file-level: missing log, wrong permissions (\u2260 0600), UTF-8
      BOM present, CRLF line endings, empty file
    * line-level: invalid UTF-8 (with byte offset), embedded
      control characters, unparsable lines, timestamps out of
      order, duplicate `id:` and `aid:` (tracked separately so a
      stanza/archive id collision isn't double-reported)
    * cross-line: broken `corrects:` references whose target id is
      not present in the file
- `/history backend` \u2014 show currently active backend
- Active backend indicator `[sqlite]` / `[flatfile]` in the status
  bar next to the JID
- Roster-JID autocomplete for verify / export / import
- export and import open a SQLite handle on demand when the
  flatfile backend is currently active, so migration works
  regardless of which backend is live

Tests
- Unit: database_export (parser round-trip, escape/unescape, dedup
  key stability, JID normalisation), database_stress (14 cases
  exercising rapid writes, large messages, deep LMC chains, MAM
  dedup, concurrent contacts)
- Functional: history persistence across reconnects, export /
  import round-trip with content equality, MUC migration,
  timestamp normalisation across timezones
- Bench harness P1\u2013P5 (synthetic load: bulk insert, time-range
  read, page-up scroll, MAM ingest, mixed workload) and failure
  modes F1\u2013F17 (page-up cursor and forward-iteration symmetry,
  oversized lines, MAM dedup, LMC depth and cycles, BOM/CRLF,
  missing log, empty file, mtime+inode flip, broken corrects, etc.)
- All bench tests integrate with the existing make targets and
  emit CSV rows for baseline comparison

Author: jabber.developer2 <jabber.developer2@jabber.space>
Reviewed-by: jabber.developer <jabber.developer@jabber.space>

src/command/cmd_ac.c

@@ -275,6 +275,8 @@ static Autocomplete logging_ac;
 static Autocomplete logging_group_ac;
 static Autocomplete privacy_ac;
 static Autocomplete privacy_log_ac;
+static Autocomplete history_ac;
+static Autocomplete history_switch_ac;
 static Autocomplete color_ac;
 static Autocomplete correction_ac;
 static Autocomplete avatar_ac;
@@ -429,6 +431,8 @@ static Autocomplete* all_acs[] = {
     &logging_group_ac,
     &privacy_ac,
     &privacy_log_ac,
+    &history_ac,
+    &history_switch_ac,
     &color_ac,
     &correction_ac,
     &avatar_ac,
@@ -1138,6 +1142,18 @@ cmd_ac_init(void)
     autocomplete_add(privacy_log_ac, "on");
     autocomplete_add(privacy_log_ac, "off");
     autocomplete_add(privacy_log_ac, "redact");
+    autocomplete_add(privacy_log_ac, "flatfile");
+
+    autocomplete_add(history_ac, "on");
+    autocomplete_add(history_ac, "off");
+    autocomplete_add(history_ac, "backend");
+    autocomplete_add(history_ac, "switch");
+    autocomplete_add(history_ac, "verify");
+    autocomplete_add(history_ac, "export");
+    autocomplete_add(history_ac, "import");
+
+    autocomplete_add(history_switch_ac, "sqlite");
+    autocomplete_add(history_switch_ac, "flatfile");
 
     autocomplete_add(logging_group_ac, "on");
     autocomplete_add(logging_group_ac, "off");
@@ -1776,7 +1792,7 @@ _cmd_ac_complete_params(ProfWin* window, const char* const input, gboolean previ
 
     // autocomplete boolean settings
     gchar* boolean_choices[] = { "/beep", "/states", "/outtype", "/flash", "/splash",
-                                 "/history", "/vercheck", "/privileges", "/wrap",
+                                 "/vercheck", "/privileges", "/wrap",
                                  "/carbons", "/slashguard", "/mam", "/silence" };
 
     for (int i = 0; i < ARRAY_SIZE(boolean_choices); i++) {
@@ -1786,6 +1802,24 @@ _cmd_ac_complete_params(ProfWin* window, const char* const input, gboolean previ
         }
     }
 
+    result = autocomplete_param_with_ac(input, "/history switch", history_switch_ac, TRUE, previous);
+    if (result) {
+        return result;
+    }
+
+    gchar* history_jid_subcmds[] = { "/history verify", "/history export", "/history import" };
+    for (int i = 0; i < ARRAY_SIZE(history_jid_subcmds); i++) {
+        result = autocomplete_param_with_func(input, history_jid_subcmds[i], roster_barejid_autocomplete, previous, NULL);
+        if (result) {
+            return result;
+        }
+    }
+
+    result = autocomplete_param_with_ac(input, "/history", history_ac, TRUE, previous);
+    if (result) {
+        return result;
+    }
+
     // autocomplete nickname in chat rooms
     if (window->type == WIN_MUC) {
         ProfMucWin* mucwin = (ProfMucWin*)window;

src/command/cmd_defs.c

@@ -1875,18 +1875,33 @@ static const struct cmd_t command_defs[] = {
     },
 
     { CMD_PREAMBLE("/history",
-                   parse_args, 1, 1, &cons_history_setting)
+                   parse_args, 1, 2, &cons_history_setting)
       CMD_MAINFUNC(cmd_history)
       CMD_TAGS(
               CMD_TAG_UI,
               CMD_TAG_CHAT)
       CMD_SYN(
-              "/history on|off")
+              "/history on|off",
+              "/history backend",
+              "/history switch sqlite|flatfile",
+              "/history verify [<jid>]",
+              "/history export [<jid>]",
+              "/history import [<jid>]")
       CMD_DESC(
               "Switch chat history on or off, /logging chat will automatically be enabled when this setting is on. "
-              "When history is enabled, previous messages are shown in chat windows.")
+              "When history is enabled, previous messages are shown in chat windows. "
+              "Use 'backend' to show the active database backend. "
+              "Use 'switch' to change the active database backend at runtime without reconnecting. "
+              "Use 'verify' to check integrity of stored message history. "
+              "Use 'export' to copy messages from SQLite to flat-file format, or 'import' to copy from flat-file to SQLite. "
+              "Both export and import merge with existing data (duplicates are skipped).")
       CMD_ARGS(
-              { "on|off", "Enable or disable showing chat history." })
+              { "on|off", "Enable or disable showing chat history." },
+              { "backend", "Show the name of the active database backend." },
+              { "switch sqlite|flatfile", "Switch the active database backend at runtime." },
+              { "verify [<jid>]", "Verify integrity of message history. Optionally specify a JID to check only one contact." },
+              { "export [<jid>]", "Export SQLite history to flat-file format. Optionally specify a JID to export only one contact." },
+              { "import [<jid>]", "Import flat-file history into SQLite. Optionally specify a JID to import only one contact." })
     },
 
     { CMD_PREAMBLE("/log",
@@ -2718,7 +2733,7 @@ static const struct cmd_t command_defs[] = {
               CMD_TAG_CHAT,
               CMD_TAG_DISCOVERY)
       CMD_SYN(
-              "/privacy logging on|redact|off",
+              "/privacy logging on|redact|off|flatfile",
               "/privacy os on|off")
       CMD_DESC(
               "Configure privacy settings. "
@@ -2726,12 +2741,13 @@ static const struct cmd_t command_defs[] = {
               "clientid to set the client identification name "
               "session_alarm to configure an alarm when more clients log in.")
       CMD_ARGS(
-              { "logging on|redact|off", "Switch chat logging. This will also disable logging in the internally used SQL database. Your messages will not be saved anywhere locally. This might have unintended consequences, such as not being able to decrypt OMEMO encrypted messages received later via MAM, and should be used with caution." },
+              { "logging on|redact|off|flatfile", "Switch chat logging. 'on' uses SQLite database (default). 'flatfile' stores messages as plain text files in ~/.local/share/profanity/flatlog/ that can be manually edited with any text editor. 'off' disables logging entirely. 'redact' stores messages with content replaced by '[redacted]'. Note: 'off' might have unintended consequences, such as not being able to decrypt OMEMO encrypted messages received later via MAM. The 'flatfile' setting takes effect on next connection." },
               { "os on|off", "Choose whether to include the OS name if a user asks for software information (XEP-0092)." }
               )
       CMD_EXAMPLES(
               "/privacy",
               "/privacy logging off",
+              "/privacy logging flatfile",
               "/privacy os off")
     },
 

src/command/cmd_funcs.c

@@ -6686,6 +6686,13 @@ cmd_privacy(ProfWin* window, const char* const command, gchar** args)
         } else if (g_strcmp0(arg, "redact") == 0) {
             cons_show("Messages are going to be redacted.");
             prefs_set_string(PREF_DBLOG, arg);
+        } else if (g_strcmp0(arg, "flatfile") == 0) {
+            auto_gchar gchar* ff_path = files_get_data_path(DIR_FLATLOG);
+            cons_show("Using flat-file backend for message logging. Takes effect on next connection.");
+            cons_show("Flatfile directory: %s", ff_path);
+            prefs_set_string(PREF_DBLOG, arg);
+            prefs_set_boolean(PREF_CHLOG, TRUE);
+            prefs_set_boolean(PREF_HISTORY, TRUE);
         } else {
             cons_bad_cmd_usage(command);
             return TRUE;
@@ -6722,6 +6729,54 @@ cmd_logging(ProfWin* window, const char* const command, gchar** args)
     return TRUE;
 }
 
+static void
+_show_integrity_issues(GSList* issues)
+{
+    int errors = 0, warnings = 0, infos = 0;
+    for (GSList* l = issues; l; l = l->next) {
+        integrity_issue_t* issue = l->data;
+        const char* level_str;
+        switch (issue->level) {
+        case INTEGRITY_ERROR:
+            level_str = "ERROR";
+            errors++;
+            break;
+        case INTEGRITY_WARNING:
+            level_str = "WARN";
+            warnings++;
+            break;
+        case INTEGRITY_INFO:
+            level_str = "INFO";
+            infos++;
+            break;
+        default:
+            level_str = "???";
+            break;
+        }
+        if (issue->line > 0) {
+            if (issue->level == INTEGRITY_ERROR) {
+                cons_show_error("[%s] %s:%d — %s", level_str, issue->file, issue->line, issue->message);
+            } else {
+                cons_show("[%s] %s:%d — %s", level_str, issue->file, issue->line, issue->message);
+            }
+        } else {
+            if (issue->level == INTEGRITY_ERROR) {
+                cons_show_error("[%s] %s — %s", level_str, issue->file, issue->message);
+            } else {
+                cons_show("[%s] %s — %s", level_str, issue->file, issue->message);
+            }
+        }
+    }
+
+    if (!issues) {
+        cons_show("Verification complete: no issues found.");
+    } else {
+        cons_show("Verification complete: %d error(s), %d warning(s), %d info(s).",
+                  errors, warnings, infos);
+        g_slist_free_full(issues, (GDestroyNotify)integrity_issue_free);
+    }
+}
+
 gboolean
 cmd_history(ProfWin* window, const char* const command, gchar** args)
 {
@@ -6730,6 +6785,91 @@ cmd_history(ProfWin* window, const char* const command, gchar** args)
         return TRUE;
     }
 
+    if (g_strcmp0(args[0], "backend") == 0) {
+        if (active_db_backend && active_db_backend->name) {
+            cons_show("Active database backend: %s", active_db_backend->name);
+        } else {
+            cons_show("No database backend is currently active.");
+        }
+        return TRUE;
+    }
+
+    if (g_strcmp0(args[0], "switch") == 0) {
+        jabber_conn_status_t conn_status = connection_get_status();
+        if (conn_status != JABBER_CONNECTED) {
+            cons_show_error("You must be connected to switch database backend.");
+            return TRUE;
+        }
+
+        if (args[1] == NULL) {
+            cons_bad_cmd_usage(command);
+            return TRUE;
+        }
+
+        if (g_strcmp0(args[1], "sqlite") != 0 && g_strcmp0(args[1], "flatfile") != 0) {
+            cons_show_error("Backend must be 'sqlite' or 'flatfile'.");
+            return TRUE;
+        }
+
+        if (active_db_backend && g_strcmp0(active_db_backend->name, args[1]) == 0) {
+            cons_show("Already using '%s' backend.", args[1]);
+            return TRUE;
+        }
+
+        cons_show("Switching database backend to '%s'...", args[1]);
+        if (log_database_switch_backend(args[1])) {
+            cons_show("Database backend switched to '%s'.", args[1]);
+        } else {
+            cons_show_error("Failed to switch database backend.");
+        }
+        return TRUE;
+    }
+
+    if (g_strcmp0(args[0], "verify") == 0) {
+        const gchar* contact_jid = args[1]; // may be NULL (verify all)
+        cons_show("Verifying history integrity...");
+
+        GSList* issues = log_database_verify_integrity(contact_jid);
+        _show_integrity_issues(issues);
+        return TRUE;
+    }
+
+    if (g_strcmp0(args[0], "export") == 0) {
+#ifdef HAVE_SQLITE
+        const gchar* contact_jid = args[1]; // may be NULL (export all)
+        if (contact_jid) {
+            cons_show("Exporting SQLite history for %s to flat-file...", contact_jid);
+        } else {
+            cons_show("Exporting all SQLite history to flat-file...");
+        }
+        int n = log_database_export_to_flatfile(contact_jid);
+        if (n >= 0) {
+            cons_show("Export complete: %d message(s) exported.", n);
+        }
+#else
+        cons_show_error("Export requires SQLite support. Rebuild with --with-sqlite.");
+#endif
+        return TRUE;
+    }
+
+    if (g_strcmp0(args[0], "import") == 0) {
+#ifdef HAVE_SQLITE
+        const gchar* contact_jid = args[1]; // may be NULL (import all)
+        if (contact_jid) {
+            cons_show("Importing flat-file history for %s into SQLite...", contact_jid);
+        } else {
+            cons_show("Importing all flat-file history into SQLite...");
+        }
+        int n = log_database_import_from_flatfile(contact_jid);
+        if (n >= 0) {
+            cons_show("Import complete: %d message(s) imported.", n);
+        }
+#else
+        cons_show_error("Import requires SQLite support. Rebuild with --with-sqlite.");
+#endif
+        return TRUE;
+    }
+
     _cmd_set_boolean_preference(args[0], "Chat history", PREF_HISTORY);
 
     // if set to on, set chlog (/logging chat on)