Address 6 of 7 high/medium audit findings on database_export.c.
End-to-end migration of 1M rows: ~5 min -> ~1 min.
H1 cache g_slist_length(merged) out of the write loop — was O(n²)
via 2000 list walks at 500-row progress interval, now O(1).
S7a_export_cold @ 1M: 309 s -> 11.2 s (-96 %)
L1 splice existing flatfile lines into merged via ownership
transfer instead of 12-strdup deep-copy + g_date_time_ref;
free list nodes only, NULL out existing.
S7b_export_dedup @ 1M: 304 s -> 13.3 s (-96 %)
RSS @ S7b 1M: 2.4 GB -> 1.9 GB
H2 mkstemp() with random suffix instead of fixed
"{log_path}.export.tmp". Eliminates the unlink-and-retry race
where two concurrent exports could clobber each other's
in-flight tmp files.
M4 SHA-256 over full body (incremental g_checksum_update) in
the dedup-key fallback. Previous body[:256] hash collided
on common preambles (signatures, code blocks) — second
occurrence silently dropped on import.
M2 Refuse _ff_read_all_lines on files > 2 GB. Each line expands
~10x in heap on parse; without a cap we OOM-kill prof on
large flatfiles. Surface a clear error instead.
M5 Break the outer import loop on system-level INSERT failure.
Was cascading "Import of X failed" through every remaining
contact when the real issue was disk-full / sqlite locked.
Verified on bench-pipeline-max: 1M rows, full content diff,
mismatches=0. baseline.csv updated.
Deferred: M3 O_TMPFILE (platform-conditional + linkat dance;
mkstemp covers most of the same window) and M6 SIGINT cancellation
(needs event-loop architecture changes).
- Add archive_ids hash set and stanza_senders hash map to
ff_contact_state_t, populated during index build/extend via
lightweight _ff_cache_line_ids() (no full parse overhead).
- Replace O(n) file scans in _ff_has_archive_id() and
_ff_find_original_sender() with O(1) hash table lookups.
- Replace g_slist_append with g_slist_prepend + g_slist_reverse
in _ff_apply_lmc, _flatfile_get_previous_chat, and
ff_verify_integrity to avoid O(n²) list building.
- Add db_sqlite_last_changes() declaration to database.h.
Export/Import improvements:
- Replace pagination with direct SQL query (db_sqlite_get_all_chat)
- Wrap import in SQL transaction with rollback on error
- Add fsync before fclose in export for data safety
- Sort merged output by timestamp with secondary key (stanza_id, from_jid)
- Export archive_id and marked_read from SQLite (lossless migration)
- Add progress indication every 500 messages during write/import
- Expand dedup key body prefix from 64 to 256 chars
- Fix g_slist_append O(n²) → g_slist_prepend + g_slist_reverse O(n)
Field parity (to_jid, to_resource, marked_read):
- Add fields to ff_parsed_line_t struct
- Write/parse to:|to_res:|read: metadata tags in flatfile format
- Pass to_resource through _ff_add_message and all callers
- Add marked_read to ProfMessage struct with -1 default (unset)
- Preserve fields across export/import round-trips
Tests (19 new: 11 unit + 8 functional):
- Unit: to_jid_and_marked_read, bracket_in_stanza_id, backslash_in_resource,
mucpm_type, all_enc_types, crlf_handling, to_jid_special_chars,
multiple_lines, parsed_line_free_null_safe, no_space_rejected,
unclosed_bracket
- Functional: export_idempotent_no_duplicates, export_lmc_correction_survives,
switch_preserves_old_backend_data, export_all_contacts,
import_double_dedup, verify_after_export,
switch_backends_independent_messages, export_empty_contact
- Rebalance test groups: move Chat Session from Group 3 to Group 4
(25/33/30/27 instead of 25/33/36/21)
- Remove hardcoded test counts from group comments
Man page:
- Document /history switch sqlite|flatfile
DRAFT — not yet tested end-to-end with a live XMPP session.
New commands:
/history export [<jid>] — copy messages from SQLite to flat-file
/history import [<jid>] — copy messages from flat-file to SQLite
Both merge with existing data; duplicates are skipped using stanza-id
or a SHA-256 fallback key (timestamp + sender + body prefix).
Implementation (src/database_export.c):
- Export: paginate SQLite via get_previous_chat, read existing flatfile
for dedup, write merged result via ff_write_line + atomic rename
- Import: parse flatfile lines via ff_parse_line, build dedup set from
SQLite, insert new messages via add_incoming (preserves original
timestamps for both directions)
- List all contacts: db_sqlite_list_contacts() queries UNION of
DISTINCT from_jid/to_jid; flatfile enumerates flatlog directories
Wiring:
- database.h: declare export/import functions + db_sqlite_list_contacts
- database_sqlite.c: add db_sqlite_list_contacts()
- cmd_defs.c: add export/import to /history synopsis and args
- cmd_funcs.c: add export/import handlers in cmd_history()
- cmd_ac.c: add 'export' and 'import' to history_ac
- Makefile.am: add database_export.c to core_sources
- stub_database.c: add stubs for test linking
- profanity.1: document export/import in man page
All code guarded with #ifdef HAVE_SQLITE — builds cleanly without it.
