Cursor was set to oldest of read window, not oldest of returned
batch — each page-up jumped past entire ranges (F16: 600/2000 msgs).
Always bisect now; correct + simpler. Cost: +5-10 ms per page-up at
1M msgs, well below UI threshold.
Also guard state->entries NULL when total_lines>0 && n_entries==0
(every line failed _ff_maybe_index_line). Backup would segfault.
F16 tightened to received == total_msgs. Catches both stuck-on-
entries[0] and cursor-jumping regressions.
When the cursor lands on entries[0].byte_offset, the !from_start
backup logic produces an empty [X, X) range, get_previous_chat
returns DB_RESPONSE_EMPTY, the UI sets WIN_SCROLL_REACHED_TOP and
never tries again — page-up is permanently stuck.
Guard the cursor optimisation with two conditions: index must be
non-empty AND cursor must not equal entries[0].byte_offset. When
either fails, fall through to the end_time bisect path which
correctly locates the byte range with older messages.
F16 regression test in bench_failure_modes:
without guard: received=200 in 3 iters (premature EMPTY)
with guard: received=600 in 7 iters (truly reaches top)
Address 6 of 7 high/medium audit findings on database_export.c.
End-to-end migration of 1M rows: ~5 min -> ~1 min.
H1 cache g_slist_length(merged) out of the write loop — was O(n²)
via 2000 list walks at 500-row progress interval, now O(1).
S7a_export_cold @ 1M: 309 s -> 11.2 s (-96 %)
L1 splice existing flatfile lines into merged via ownership
transfer instead of 12-strdup deep-copy + g_date_time_ref;
free list nodes only, NULL out existing.
S7b_export_dedup @ 1M: 304 s -> 13.3 s (-96 %)
RSS @ S7b 1M: 2.4 GB -> 1.9 GB
H2 mkstemp() with random suffix instead of fixed
"{log_path}.export.tmp". Eliminates the unlink-and-retry race
where two concurrent exports could clobber each other's
in-flight tmp files.
M4 SHA-256 over full body (incremental g_checksum_update) in
the dedup-key fallback. Previous body[:256] hash collided
on common preambles (signatures, code blocks) — second
occurrence silently dropped on import.
M2 Refuse _ff_read_all_lines on files > 2 GB. Each line expands
~10x in heap on parse; without a cap we OOM-kill prof on
large flatfiles. Surface a clear error instead.
M5 Break the outer import loop on system-level INSERT failure.
Was cascading "Import of X failed" through every remaining
contact when the real issue was disk-full / sqlite locked.
Verified on bench-pipeline-max: 1M rows, full content diff,
mismatches=0. baseline.csv updated.
Deferred: M3 O_TMPFILE (platform-conditional + linkat dance;
mkstemp covers most of the same window) and M6 SIGINT cancellation
(needs event-loop architecture changes).
End-to-end performance and correctness harness for the flat-file +
SQLite database backends. Lives in tests/bench/, built only on
demand (`make bench`); not part of `make check`.
Components
gen_history (P1)
Deterministic corpus generator. Knobs: lines, contacts, years,
seed, stanza-id mode (uuid/libpurple/conversations/mixed), LMC
rate, MAM-OOO rate, resources/contact, length profile
(short/mixed/long/extreme). Emits the canonical
flatlog/<account>/<contact>/history.log layout used by
ff_verify_integrity. ~340 LOC.
bench_runner (P1, P2.5)
S1 cold tail-access via sparse index
S2 warm tail-access (page cache hot)
S3 deep pagination (1000 binary-search lookups)
S4 first-time index build (cold file -> ff_state_ensure_fresh)
S5 incremental extend (asserts no full rebuild path)
S6 real ff_verify_integrity over the contact tree
Reports total/err/warn/info issue counts in the CSV note.
bench_long_messages (P2)
L1-L14 long-message stress: 1KB up to 9.9MB bodies, plus
oversized line rejection (10MB+1 -> ff_readline returns ""),
embedded-newline / pipe / emoji body patterns, full parse on
100x1MB, 1000x100KB sustained append.
bench_failure_modes (P3)
F1-F15 failure-injection: truncated last line, mid-file CRLF,
mid-file BOM, LMC cycle, LMC depth>FF_MAX_LMC_DEPTH, manual
': ' in resource, RTL/ZWSP, Latin-1 byte, empty body,
mtime/inode flip, empty file. Each test asserts expected issue
levels and reports PASS/FAIL.
bench_export_import (P5)
Links real database_export.c + database_sqlite.c + database.c
and drives log_database_export_to_flatfile /
log_database_import_from_flatfile under load.
Subcommands: seed, export, import, roundtrip, verify.
S7a/b export, S8a/b import, S8e roundtrip with full byte-by-byte
content diff of every row in (from_jid, to_jid, message,
timestamp, type, stanza_id, archive_id, encryption, replace_id).
Make targets
bench-quick / bench / bench-full
bench-longmsg, bench-failure
bench-multicontact, bench-lmc, bench-ooo
bench-export, bench-import, bench-roundtrip
bench-pipeline, bench-pipeline-max (1M rows)
bench-compare, bench-update-baseline
Volume controls: BENCH_VOLUME (small/medium/max), BENCH_PIPE_ROWS,
BENCH_PIPE_ROWS_MAX, BENCH_DATA_DIR, BENCH_CSV.
Baseline + regression checking (P4)
tests/bench/baseline.csv 51 rows: S1-S6 x {small,lmc,ooo}
+ L1-L14 + F1-F15 (11 of 15) +
S7/S8 x {pipe100k, pipe1M}.
compare_baseline.py median over duplicate rows;
exits 1 on any (scenario, volume)
slowdown >= threshold (default 25%).
Verified at scale
bench-pipeline-max: 1,000,000 rows, full content diff
seed 17 s, export 304 s, import 31 s, idempotent re-import 10 s,
diff 2.7 s -- mismatches=0.
Findings surfaced by the harness
Export scales super-linearly: 4 s @ 100k -> 304 s @ 1M (76x for
10x rows). Cause: g_slist_sort on the merged list + per-row
ProfMessage/ff_parsed_line_t allocations. RSS peaks at 1.4 GB
on 1M. Worth a follow-up.
Export progress reporting only fires during the write phase --
the merge+sort phase (~95% of wall time at 1M) is silent.
/history export and /history import are blocking on the main
UI thread; profanity is frozen for the duration (~5 min @ 1M).
ff_readline sets *truncated=TRUE on partial-write tail but
ff_verify_integrity does not surface this -- partial writes
go unflagged (failure-injection F1).
Parser silently truncates body at first unescaped ': ' if a
resource was manually edited to contain it (F9).
In gen_history (caught by the bench's own real-verify pass):
g_strndup mid-codepoint truncation on UTF-8 bank strings -- fixed.
Linkage strategy
database_flatfile.c + parser + verify + common.c are linked
unconditionally. The export/import bench additionally links
database.c + database_sqlite.c + database_export.c. bench_stubs.c
provides minimal stubs for log_*, prefs_*, connection_get_jid,
jid_create, files_*, message_*, ui hooks. integrity_issue_free
is a weak symbol so it falls back to the real database.c
implementation when that file is linked.
Fix send_receipt_request test that wasted 60s on every run:
- Change caps hash from sha-256 to sha-1 (profanity only supports sha-1
for standard caps path, sha-256 goes to unsupported-hash fallback)
- Fix expected output: 'Buddy1 is online' → 'Buddy1 (laptop) is online'
(resource is always shown for full JID presence)
- Wrap prof_output_exact in assert_true so mismatches fail fast instead
of silently waiting the full expect_timeout
Rebalance test groups for parallel execution:
- Move Disco tests (14) from Group 3 → Group 1
- Move DB History tests (12) from Group 2 → Group 3
- Before: G1=31s G2=63s G3=30s G4=37s (bottleneck 63s)
- After: G1=46s G2=38s G3=40s G4=36s (bottleneck 46s, -27%)
Add per-test wall-clock timing via clock_gettime(CLOCK_MONOTONIC)
in init_prof_test/close_prof_test for profiling individual tests.
Export/Import improvements:
- Replace pagination with direct SQL query (db_sqlite_get_all_chat)
- Wrap import in SQL transaction with rollback on error
- Add fsync before fclose in export for data safety
- Sort merged output by timestamp with secondary key (stanza_id, from_jid)
- Export archive_id and marked_read from SQLite (lossless migration)
- Add progress indication every 500 messages during write/import
- Expand dedup key body prefix from 64 to 256 chars
- Fix g_slist_append O(n²) → g_slist_prepend + g_slist_reverse O(n)
Field parity (to_jid, to_resource, marked_read):
- Add fields to ff_parsed_line_t struct
- Write/parse to:|to_res:|read: metadata tags in flatfile format
- Pass to_resource through _ff_add_message and all callers
- Add marked_read to ProfMessage struct with -1 default (unset)
- Preserve fields across export/import round-trips
Tests (19 new: 11 unit + 8 functional):
- Unit: to_jid_and_marked_read, bracket_in_stanza_id, backslash_in_resource,
mucpm_type, all_enc_types, crlf_handling, to_jid_special_chars,
multiple_lines, parsed_line_free_null_safe, no_space_rejected,
unclosed_bracket
- Functional: export_idempotent_no_duplicates, export_lmc_correction_survives,
switch_preserves_old_backend_data, export_all_contacts,
import_double_dedup, verify_after_export,
switch_backends_independent_messages, export_empty_contact
- Rebalance test groups: move Chat Session from Group 3 to Group 4
(25/33/30/27 instead of 25/33/36/21)
- Remove hardcoded test counts from group comments
Man page:
- Document /history switch sqlite|flatfile
Add backend-agnostic functional tests for database message persistence.
All tests work with both SQLite and flat-file backends.
New tests in test_history.c:
- message_db_history_on_reopen: basic incoming write+read round-trip
- message_db_history_multiple: 3 messages from different resources
- message_db_history_contact_isolation: buddy1 msg absent from buddy2
- message_db_history_special_chars: XML entities survive decode+DB
- message_db_history_outgoing: sent message persists across reopen
- message_db_history_dialog: outgoing + incoming both in history
- message_db_history_empty: no crash on contact with no history
- message_db_history_long_message: 1000+ char body not truncated
- message_db_history_newline: embedded LF stored correctly
- message_db_history_service_chars: backslash pipe percent braces etc
- message_db_history_verify: /history verify (ifdef HAVE_HISTORY_VERIFY)
- message_db_history_lmc: XEP-0308 correction replaces original in DB
Rebalance test groups for parallel execution (19/22/22/21):
- Group 1: Connect, Ping, Rooms, Software, LastActivity
- Group 2: Message, Receipts, Roster, DB History
- Group 3: Chat Session, Presence, Disconnect
- Group 4: MUC, Carbons
Add log_database_switch_backend() that closes the current backend,
updates PREF_DBLOG preference, and reinitializes with the new backend
without requiring a reconnect or restart.
New command: /history switch sqlite|flatfile
- Validates connection state and backend name
- Skips if already using the requested backend
- Autocomplete support for the subcommand
DRAFT — not yet tested end-to-end with a live XMPP session.
New commands:
/history export [<jid>] — copy messages from SQLite to flat-file
/history import [<jid>] — copy messages from flat-file to SQLite
Both merge with existing data; duplicates are skipped using stanza-id
or a SHA-256 fallback key (timestamp + sender + body prefix).
Implementation (src/database_export.c):
- Export: paginate SQLite via get_previous_chat, read existing flatfile
for dedup, write merged result via ff_write_line + atomic rename
- Import: parse flatfile lines via ff_parse_line, build dedup set from
SQLite, insert new messages via add_incoming (preserves original
timestamps for both directions)
- List all contacts: db_sqlite_list_contacts() queries UNION of
DISTINCT from_jid/to_jid; flatfile enumerates flatlog directories
Wiring:
- database.h: declare export/import functions + db_sqlite_list_contacts
- database_sqlite.c: add db_sqlite_list_contacts()
- cmd_defs.c: add export/import to /history synopsis and args
- cmd_funcs.c: add export/import handlers in cmd_history()
- cmd_ac.c: add 'export' and 'import' to history_ac
- Makefile.am: add database_export.c to core_sources
- stub_database.c: add stubs for test linking
- profanity.1: document export/import in man page
All code guarded with #ifdef HAVE_SQLITE — builds cleanly without it.
Build system:
- Add --without-sqlite configure flag (AM_CONDITIONAL BUILD_SQLITE)
- Guard database_sqlite.c with HAVE_SQLITE in Makefile.am, database.c,
database.h and stub_database.c
- Fall back to flatfile when SQLite not compiled in
Security (database_flatfile.c):
- MAM dedup: skip incoming messages with duplicate stanza-id (archive_id)
- LMC sender validation: reject corrections from mismatched JIDs
- Add flock() advisory locking to prevent interleaved writes from
concurrent instances
- Check fprintf return when writing file header
Autocomplete (cmd_ac.c):
- Add 'flatfile' to /privacy logging autocomplete
- Add dedicated /history autocomplete with on/off/verify (was boolean-only)
Code quality:
- Fix fwrite return type: size_t not ssize_t (database_flatfile_parser.c)
- Fix mixed allocators in ff_readline: use malloc() consistently for
overlength-line fallback instead of g_strdup()
- Fix index alignment in _ff_state_extend: use local counter so index
step doesn't depend on total_lines from initial build
Documentation:
- Fix page: correct directory structure (history.log not per-day files)
- Fix page: add aid:{archive_id} to line format example
- Fix page: missing newline before .SH BUGS
Add pluggable storage backend abstraction (vtable) to the database layer,
allowing selection between SQLite (default) and a new flat-file backend
that stores messages as human-readable plain text files.
New files:
- database_sqlite.c: extracted SQLite backend from database.c
- database_flatfile.c: plain text backend with tolerant parser,
LMC correction chains, UTF-8/BOM/CRLF handling, integrity checks
Commands:
- /privacy logging flatfile — switch to flat-file backend
- /history verify [<jid>] — check integrity of stored history
Fixes:
- Add missing 'off' guard in flatfile backend
- Enable CHLOG/HISTORY prefs when switching to flatfile mode
Logs stored in ~/.local/share/profanity/flatlog/{account}/{contact}/{date}.log
Format: {ISO8601} [{type}|{enc}|id:{id}|corrects:{id}] {sender}: {message}
Updated: CHANGELOG, CONTRIBUTING.md, profrc.example, man page, cmd_defs,
Makefile.am, test stubs, functional test support (PROF_FLATFILE=1)
Replace 16 interactive UI setup commands with pre-written profrc file
containing [ui] and [notifications] sections (~1800ms saved per test).
Replace sleep(1) + blocking waitpid with close(pty fd) → SIGHUP →
polling waitpid(WNOHANG) → SIGTERM/SIGKILL fallback chain (~4900ms
saved per test).
Remove post-stbbr_start() and post-stbbr_stop() sleeps — bind+listen
completes synchronously before stbbr_start() returns, and
pthread_join() in stbbr_stop() guarantees socket cleanup (~200ms saved).
Add PORTS_PER_GROUP=50 isolated port ranges per test group to enable
safe parallel execution of 4 groups without port conflicts.
Add 7 new tests for /disco command:
- disco_items_to_jid: query items to specific JID
- disco_info_empty_result: handle empty disco#info response
- disco_info_multiple_identities: multiple identity elements
- disco_info_without_name: identity without optional name attr
- disco_items_without_name: items without optional name attr
- disco_info_service_unavailable: error handling for info
- disco_items_error_handling: error handling for items (XEP-0030 §7)
The disco_items_error_handling test documents a bug where disco#items
errors are silently ignored (unlike disco#info which handles them).
This violates XEP-0030 Section 7 which requires error feedback to user.
- Add 8 tests for disco info and disco items commands
- Fix XEP-0030 compliance bug: show message for empty disco#items results
- Tests cover: identity display, features, server/jid queries, error handling,
items display, empty results, and connection requirement
Major changes:
Run 4 build configurations in parallel with Valgrind on Linux
Add test failure detection verification (meta-test)
Port allocation per build to prevent conflicts in parallel runs
Add --coverage-only flag for dedicated coverage builds
Code quality:
Add TEST_GROUPS constant, CMOCKA patterns, helper functions
Organize ci-build.sh into sections
Split functional tests into 4 parallel groups and add check-functional-parallel target (~3x faster CI runs).
Add branch-aware LCOV coverage reporting with new --enable-coverage option and lcov summary in CI pipeline.
Enable ccache via -C configure flag for faster recompilations.
Install lcov in all Docker images and use --depth 1 git clones + parallel make -j$(nproc) for quicker container builds.
Update CONTRIBUTING.md with instructions for parallel test groups and adding new ones.
All changes are tightly related CI/performance improvements developed in sequence. No external service uploads (e.g. Codecov skipped due to Gitea incompatibility).
- Replace stbbr_for_id() with stbbr_for_query()/stbbr_send()
- Content-based stubbing matches stanzas by namespace instead of ID
- Use regex assertions for flexible output matching
- Fix timing issues in chat_session and presence tests
Remove autodisable logic (prefs_set_autoping(0)) and early return in autoping
to treat intermittent false negatives from connection_supports(XMPP_FEATURE_PING).
Add one-time error display with debug features print for monitoring; warn on first failing
check without aborting to maintain functionality while investigating root cause.
Includes were updated in 988d366, but the change did not propagate to forced_encryption.c due to the gap between the upstream Profanity and the CProof.
As 9f2abc75 accidentally got the ordering of some of the includes wrong,
I decided to propose my initial solution again.
Additional to that, I've opened a MR against CMocka to solve this on
their side, since I believe that the current way this is done is not
sustainable [0].
[0] https://gitlab.com/cmocka/cmocka/-/merge_requests/91
Fixes: 9f2abc75 ("Fix tests with gcc15 (uintptr_t)")
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
CProof note: our new tests need to also be updated.
fixes: error: ‘uintptr_t’ undeclared, defined in header ‘<stdint.h>
(cherry picked from commit 9f2abc75ad)
Author: Andreas Stieger <Andreas.Stieger@gmx.de>
- Modified chatwin_outgoing_msg to call _chatwin_set_last_message for corrected messages, using replace_id per XEP-0308.
- Updated _chatwin_set_last_message to skip redundant and invalid free/strdup for same pointers.
- Added null checks in _chatwin_set_last_message for safety.
- Fixes autocompletion suggesting original message content instead of corrected content.
Fixes#24
Added null check for jidp in cmd_sub to handle jid_create returning NULL.
Crash occurred when processing malformed JID inputs like @example.com.
Ensures robust handling of invalid JIDs.
Fixes#22
Implement /force-encryption command to configure forced encryption
settings with on|off and policy resend-to-confirm|block options.
The changes enhance user experience by clarifying encryption requirements
and providing actionable commands. Users can now press Enter again to
confirm unencrypted messages in resend-to-confirm mode.
- Update formatting for stub files
- Update docs
While original project, Profanity, provides valuable basis for this
fork, we are not affiliated with the upstream. This commit
is updating documentation to reflect accurate information.
The docs are going to be updated further once the website is created.
Reviewed-on: devs/profanity#5
Co-authored-by: Jabber Developer <jabber.developer@jabber.space>
Co-committed-by: Jabber Developer <jabber.developer@jabber.space>
* Also pass `$*` to `configure` when invoking `ci-build.sh`, so one can
e.g. run `./ci-build.sh --without-xscreensaver`
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
While trying to get the unit tests working again I stumbled over all those
things that I thought could be better^TM.
Now we also know "TODO: why does this make the test fail?" - because
the unit tests are brittle AF ... and we have to init the subsystems
we use in the test, otherwise the cleanup will fail...
BTW. you can now also only run a single test ... or a pattern or so ...
you'd have to read how `cmocka_set_test_filter()` works exactly.
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
* destroy/free/shutdown/close in reverse order of allocation
* more static/auto_Xfree
* less variables/strlen/GString
* properly `\0`-terminate string of testcase
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
Instead of adding stuff to `_shutdown()`, we can now register a shutdown
routine from the respective `init()` function of our modules.
This also has the advantage, that we're sure they're called in reverse
order from how the initialization happened.
I didn't simply use `atexit()` because POSIX says that the number of
possible callbacks is limited (min 32) and I was not sure whether
we will maybe extend this number at one point.
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
