Export/Import improvements:
- Replace pagination with direct SQL query (db_sqlite_get_all_chat)
- Wrap import in SQL transaction with rollback on error
- Add fsync before fclose in export for data safety
- Sort merged output by timestamp with secondary key (stanza_id, from_jid)
- Export archive_id and marked_read from SQLite (lossless migration)
- Add progress indication every 500 messages during write/import
- Expand dedup key body prefix from 64 to 256 chars
- Fix g_slist_append O(n²) → g_slist_prepend + g_slist_reverse O(n)
Field parity (to_jid, to_resource, marked_read):
- Add fields to ff_parsed_line_t struct
- Write/parse to:|to_res:|read: metadata tags in flatfile format
- Pass to_resource through _ff_add_message and all callers
- Add marked_read to ProfMessage struct with -1 default (unset)
- Preserve fields across export/import round-trips
Tests (19 new: 11 unit + 8 functional):
- Unit: to_jid_and_marked_read, bracket_in_stanza_id, backslash_in_resource,
mucpm_type, all_enc_types, crlf_handling, to_jid_special_chars,
multiple_lines, parsed_line_free_null_safe, no_space_rejected,
unclosed_bracket
- Functional: export_idempotent_no_duplicates, export_lmc_correction_survives,
switch_preserves_old_backend_data, export_all_contacts,
import_double_dedup, verify_after_export,
switch_backends_independent_messages, export_empty_contact
- Rebalance test groups: move Chat Session from Group 3 to Group 4
(25/33/30/27 instead of 25/33/36/21)
- Remove hardcoded test counts from group comments
Man page:
- Document /history switch sqlite|flatfile
Add log_database_switch_backend() that closes the current backend,
updates PREF_DBLOG preference, and reinitializes with the new backend
without requiring a reconnect or restart.
New command: /history switch sqlite|flatfile
- Validates connection state and backend name
- Skips if already using the requested backend
- Autocomplete support for the subcommand
DRAFT — not yet tested end-to-end with a live XMPP session.
New commands:
/history export [<jid>] — copy messages from SQLite to flat-file
/history import [<jid>] — copy messages from flat-file to SQLite
Both merge with existing data; duplicates are skipped using stanza-id
or a SHA-256 fallback key (timestamp + sender + body prefix).
Implementation (src/database_export.c):
- Export: paginate SQLite via get_previous_chat, read existing flatfile
for dedup, write merged result via ff_write_line + atomic rename
- Import: parse flatfile lines via ff_parse_line, build dedup set from
SQLite, insert new messages via add_incoming (preserves original
timestamps for both directions)
- List all contacts: db_sqlite_list_contacts() queries UNION of
DISTINCT from_jid/to_jid; flatfile enumerates flatlog directories
Wiring:
- database.h: declare export/import functions + db_sqlite_list_contacts
- database_sqlite.c: add db_sqlite_list_contacts()
- cmd_defs.c: add export/import to /history synopsis and args
- cmd_funcs.c: add export/import handlers in cmd_history()
- cmd_ac.c: add 'export' and 'import' to history_ac
- Makefile.am: add database_export.c to core_sources
- stub_database.c: add stubs for test linking
- profanity.1: document export/import in man page
All code guarded with #ifdef HAVE_SQLITE — builds cleanly without it.
Build system:
- Add --without-sqlite configure flag (AM_CONDITIONAL BUILD_SQLITE)
- Guard database_sqlite.c with HAVE_SQLITE in Makefile.am, database.c,
database.h and stub_database.c
- Fall back to flatfile when SQLite not compiled in
Security (database_flatfile.c):
- MAM dedup: skip incoming messages with duplicate stanza-id (archive_id)
- LMC sender validation: reject corrections from mismatched JIDs
- Add flock() advisory locking to prevent interleaved writes from
concurrent instances
- Check fprintf return when writing file header
Autocomplete (cmd_ac.c):
- Add 'flatfile' to /privacy logging autocomplete
- Add dedicated /history autocomplete with on/off/verify (was boolean-only)
Code quality:
- Fix fwrite return type: size_t not ssize_t (database_flatfile_parser.c)
- Fix mixed allocators in ff_readline: use malloc() consistently for
overlength-line fallback instead of g_strdup()
- Fix index alignment in _ff_state_extend: use local counter so index
step doesn't depend on total_lines from initial build
Documentation:
- Fix page: correct directory structure (history.log not per-day files)
- Fix page: add aid:{archive_id} to line format example
- Fix page: missing newline before .SH BUGS
Add pluggable storage backend abstraction (vtable) to the database layer,
allowing selection between SQLite (default) and a new flat-file backend
that stores messages as human-readable plain text files.
New files:
- database_sqlite.c: extracted SQLite backend from database.c
- database_flatfile.c: plain text backend with tolerant parser,
LMC correction chains, UTF-8/BOM/CRLF handling, integrity checks
Commands:
- /privacy logging flatfile — switch to flat-file backend
- /history verify [<jid>] — check integrity of stored history
Fixes:
- Add missing 'off' guard in flatfile backend
- Enable CHLOG/HISTORY prefs when switching to flatfile mode
Logs stored in ~/.local/share/profanity/flatlog/{account}/{contact}/{date}.log
Format: {ISO8601} [{type}|{enc}|id:{id}|corrects:{id}] {sender}: {message}
Updated: CHANGELOG, CONTRIBUTING.md, profrc.example, man page, cmd_defs,
Makefile.am, test stubs, functional test support (PROF_FLATFILE=1)
As 9f2abc75 accidentally got the ordering of some of the includes wrong,
I decided to propose my initial solution again.
Additional to that, I've opened a MR against CMocka to solve this on
their side, since I believe that the current way this is done is not
sustainable [0].
[0] https://gitlab.com/cmocka/cmocka/-/merge_requests/91
Fixes: 9f2abc75 ("Fix tests with gcc15 (uintptr_t)")
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
CProof note: our new tests need to also be updated.
