- Add archive_ids hash set and stanza_senders hash map to
ff_contact_state_t, populated during index build/extend via
lightweight _ff_cache_line_ids() (no full parse overhead).
- Replace O(n) file scans in _ff_has_archive_id() and
_ff_find_original_sender() with O(1) hash table lookups.
- Replace g_slist_append with g_slist_prepend + g_slist_reverse
in _ff_apply_lmc, _flatfile_get_previous_chat, and
ff_verify_integrity to avoid O(n²) list building.
- Add db_sqlite_last_changes() declaration to database.h.
Export/Import improvements:
- Replace pagination with direct SQL query (db_sqlite_get_all_chat)
- Wrap import in SQL transaction with rollback on error
- Add fsync before fclose in export for data safety
- Sort merged output by timestamp with secondary key (stanza_id, from_jid)
- Export archive_id and marked_read from SQLite (lossless migration)
- Add progress indication every 500 messages during write/import
- Expand dedup key body prefix from 64 to 256 chars
- Fix g_slist_append O(n²) → g_slist_prepend + g_slist_reverse O(n)
Field parity (to_jid, to_resource, marked_read):
- Add fields to ff_parsed_line_t struct
- Write/parse to:|to_res:|read: metadata tags in flatfile format
- Pass to_resource through _ff_add_message and all callers
- Add marked_read to ProfMessage struct with -1 default (unset)
- Preserve fields across export/import round-trips
Tests (19 new: 11 unit + 8 functional):
- Unit: to_jid_and_marked_read, bracket_in_stanza_id, backslash_in_resource,
mucpm_type, all_enc_types, crlf_handling, to_jid_special_chars,
multiple_lines, parsed_line_free_null_safe, no_space_rejected,
unclosed_bracket
- Functional: export_idempotent_no_duplicates, export_lmc_correction_survives,
switch_preserves_old_backend_data, export_all_contacts,
import_double_dedup, verify_after_export,
switch_backends_independent_messages, export_empty_contact
- Rebalance test groups: move Chat Session from Group 3 to Group 4
(25/33/30/27 instead of 25/33/36/21)
- Remove hardcoded test counts from group comments
Man page:
- Document /history switch sqlite|flatfile
Add log_database_switch_backend() that closes the current backend,
updates PREF_DBLOG preference, and reinitializes with the new backend
without requiring a reconnect or restart.
New command: /history switch sqlite|flatfile
- Validates connection state and backend name
- Skips if already using the requested backend
- Autocomplete support for the subcommand
DRAFT — not yet tested end-to-end with a live XMPP session.
New commands:
/history export [<jid>] — copy messages from SQLite to flat-file
/history import [<jid>] — copy messages from flat-file to SQLite
Both merge with existing data; duplicates are skipped using stanza-id
or a SHA-256 fallback key (timestamp + sender + body prefix).
Implementation (src/database_export.c):
- Export: paginate SQLite via get_previous_chat, read existing flatfile
for dedup, write merged result via ff_write_line + atomic rename
- Import: parse flatfile lines via ff_parse_line, build dedup set from
SQLite, insert new messages via add_incoming (preserves original
timestamps for both directions)
- List all contacts: db_sqlite_list_contacts() queries UNION of
DISTINCT from_jid/to_jid; flatfile enumerates flatlog directories
Wiring:
- database.h: declare export/import functions + db_sqlite_list_contacts
- database_sqlite.c: add db_sqlite_list_contacts()
- cmd_defs.c: add export/import to /history synopsis and args
- cmd_funcs.c: add export/import handlers in cmd_history()
- cmd_ac.c: add 'export' and 'import' to history_ac
- Makefile.am: add database_export.c to core_sources
- stub_database.c: add stubs for test linking
- profanity.1: document export/import in man page
All code guarded with #ifdef HAVE_SQLITE — builds cleanly without it.
Build system:
- Add --without-sqlite configure flag (AM_CONDITIONAL BUILD_SQLITE)
- Guard database_sqlite.c with HAVE_SQLITE in Makefile.am, database.c,
database.h and stub_database.c
- Fall back to flatfile when SQLite not compiled in
Security (database_flatfile.c):
- MAM dedup: skip incoming messages with duplicate stanza-id (archive_id)
- LMC sender validation: reject corrections from mismatched JIDs
- Add flock() advisory locking to prevent interleaved writes from
concurrent instances
- Check fprintf return when writing file header
Autocomplete (cmd_ac.c):
- Add 'flatfile' to /privacy logging autocomplete
- Add dedicated /history autocomplete with on/off/verify (was boolean-only)
Code quality:
- Fix fwrite return type: size_t not ssize_t (database_flatfile_parser.c)
- Fix mixed allocators in ff_readline: use malloc() consistently for
overlength-line fallback instead of g_strdup()
- Fix index alignment in _ff_state_extend: use local counter so index
step doesn't depend on total_lines from initial build
Documentation:
- Fix page: correct directory structure (history.log not per-day files)
- Fix page: add aid:{archive_id} to line format example
- Fix page: missing newline before .SH BUGS
Replace per-contact directory structure with a single flat file per contact.
Each file uses pipe-delimited records with a sparse byte-offset index
that is rebuilt on demand and cached in memory.
- Rewrite _ff_write_record / _ff_read_records for single-file format
- Add sparse index (every Nth record) for O(log N) seek on history read
- Cursor-based pagination: _ff_get_previous_chat uses saved byte offset
- LMC corrections applied in-place during read (_ff_apply_lmc)
- Newline escaping (\n) in message bodies for line-based storage
- Simplify verify: parse + timestamp order + LMC reference check
- Update parser helpers for new field layout
Add pluggable storage backend abstraction (vtable) to the database layer,
allowing selection between SQLite (default) and a new flat-file backend
that stores messages as human-readable plain text files.
New files:
- database_sqlite.c: extracted SQLite backend from database.c
- database_flatfile.c: plain text backend with tolerant parser,
LMC correction chains, UTF-8/BOM/CRLF handling, integrity checks
Commands:
- /privacy logging flatfile — switch to flat-file backend
- /history verify [<jid>] — check integrity of stored history
Fixes:
- Add missing 'off' guard in flatfile backend
- Enable CHLOG/HISTORY prefs when switching to flatfile mode
Logs stored in ~/.local/share/profanity/flatlog/{account}/{contact}/{date}.log
Format: {ISO8601} [{type}|{enc}|id:{id}|corrects:{id}] {sender}: {message}
Updated: CHANGELOG, CONTRIBUTING.md, profrc.example, man page, cmd_defs,
Makefile.am, test stubs, functional test support (PROF_FLATFILE=1)
Per XEP-0030 section 3.1: 'if an entity has no associated items,
it MUST return an empty <query/> element.'
The client should display 'No service discovery items for X' when
receiving an empty result, not silently ignore it.
Previous version introduced in commit f28655c5c (2016-05-08) which added an early
return when child == NULL, preventing the message from being shown.
The /connect command can take: account + server <s> + port <p> + tls <t> + auth <a>
which totals 9 arguments, not 7. This fixes argument parsing for full command usage.
- Track unread message indicators and paging state in the window component
- Clear and restore unread markers correctly when navigating between conversations
- Suppress buffer updates and message printing while viewing history to prevent
unwanted scroll jumps and viewport lock
Also includes minor code-style, whitespace, and comment cleanups as well as
updated documentation for metrics and buffer handling.
Add debug output tracking connection lifecycle.
- Track disconnects and reconnects
- Record session login, logout, and reconnection attempts
- Use [CONNDBG] tag for easy log filtering
Add prof_get_current_window API to retrieve the title of the currently active window, matching the titlebar display.
The feature is especially useful to track currently used plugin window.
Remove autodisable logic (prefs_set_autoping(0)) and early return in autoping
to treat intermittent false negatives from connection_supports(XMPP_FEATURE_PING).
Add one-time error display with debug features print for monitoring; warn on first failing
check without aborting to maintain functionality while investigating root cause.
Prior commit (6ad8a190) did not properly handle overflow by long (9000 lines) message,
to address this issue, multiple changes were made:
- Add lines recalculation on win_redraw
- Move `prof_buff_t` struct to header file so its lines could be externally changed
- Call win_redraw once overflow is detected: it allows to recalculate sizes in lines and apply changes to the buffer
Refactor buffer trimming logic to use dynamic line counts (_lines per entry)
against PAD_SIZE (10k) instead of fixed entry count (MAX_BUFFER_SIZE=200). This
prevents premature cleanups for short messages, reduces scrolling disruptions
during history loads, and scales better with rendered content size.
Delete oldest/newest entry from opposite end only when total lines approach
ncurses pad limit, minimizing unnecessary deletions.
Simplify overflow warning log by removing unused MAX_BUFFER_SIZE reference.
Update buffer.c header for improved DX:
- Add detailed module overview explaining role (in-memory UI buffer vs. DB
persistence), key features (trimming, metadata), and integration (ncurses).
- Shorten copyright/license boilerplate to concise pointer (LICENSE ref).
- Add fork notice; preserve original copyrights per GPLv3.
Partially addresses #36 (stuck scroll mitigation via fewer cleanups)
Related to #36
When running profanity under Valgrind with slashguard enabled, the
following error could occur:
```
[...]
==4021347== Invalid read of size 1
==4021347== at 0x4851F49: memchr (vg_replace_strmem.c:986)
==4021347== by 0x45CEAD: _inp_slashguard_check (inputwin.c:183)
==4021347== by 0x45CEAD: inp_readline (inputwin.c:225)
==4021347== by 0x431184: prof_run (profanity.c:121)
==4021347== by 0x42C609: main (main.c:176)
==4021347== Address 0xe850883 is 0 bytes after a block of size 3 alloc'd
==4021347== at 0x48477C4: malloc (vg_replace_malloc.c:446)
[...]
```
`memchr()` requires the complete memory that shall be searched to be
accessible. Using `strchr()` could work for shorter strings, but we only
want to search in the first 4 chars.
Instead of somehow working around those limitations, simply search manually
in the first 4 bytes.
Fixes: 3c56b289 ("Add slashguard feature")
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
(cherry picked from commit 7f48452d84)
In the past `Py_XDECREF()` was a macro. Preserve compat to ancient Python
versions by having a trampoline which calls `Py_XDECREF()`.
Fixes: #2043
Fixes: c0da36c4 ("Rage-cleanup.")
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
(cherry picked from commit 3299dd8fc6)
- Modified chatwin_outgoing_msg to call _chatwin_set_last_message for corrected messages, using replace_id per XEP-0308.
- Updated _chatwin_set_last_message to skip redundant and invalid free/strdup for same pointers.
- Added null checks in _chatwin_set_last_message for safety.
- Fixes autocompletion suggesting original message content instead of corrected content.
Fixes#24
Added null check for jidp in cmd_sub to handle jid_create returning NULL.
Crash occurred when processing malformed JID inputs like @example.com.
Ensures robust handling of invalid JIDs.
Fixes#22
- Added null check for `jid` in `cmd_caps` to handle `jid_create` returning NULL.
- Crash occurred when processing malformed inputs like `@example.com` in `/caps`.
- Ensures robust handling of invalid JID strings in `WIN_CHAT` and `WIN_CONSOLE` cases.
Fixes#20
Implement /force-encryption command to configure forced encryption
settings with on|off and policy resend-to-confirm|block options.
The changes enhance user experience by clarifying encryption requirements
and providing actionable commands. Users can now press Enter again to
confirm unencrypted messages in resend-to-confirm mode.
The synchronous `get_message_from_editor` blocked the main loop (`prof_run`) while launching an external editor like `vim`, halting network I/O in `session_process_events` and preventing incoming message reception.
Introduced `get_message_from_editor_async` for `cmd_editor` to run the editor asynchronously. It forks and execs the editor in a thread (`editor_thread`), suspending NCurses to free the terminal. The main loop skips `inp_readline` and `ui_update` via a new `background_mode` flag while the editor runs, allowing `session_process_events` to keep the connection alive.
On editor completion, `editor_process` (called per loop iteration) resumes NCurses with `ui_resize`, inserts the result into the readline buffer and clears `background_mode`.
Retained synchronous `get_message_from_editor` for ~20 existing code paths (e.g., `vcard_nickname`) to avoid breaking them.
Tested with `vim` and `nano`: confirms no rendering conflicts, messages received during editing, and seamless resume. Edge cases like editor crashes handled via error logging and seamless resume.
Previously, features like `XMPP_FEATURE_OMEMO_DEVICELIST_NOTIFY` were
added in separate init methods (e.g., `omemo_devicelist_subscribe`) via
`caps_add_feature`, triggering `cl_ev_presence_send` and sending multiple
"online" XMPP presence stanzas. This caused other clients to see users
as appearing "online" multiple times.
This change adds `STANZA_NS_MOOD_NOTIFY` and
`XMPP_FEATURE_OMEMO_DEVICELIST_NOTIFY` in `caps_init`, ensuring they are
in `prof_features` early. This allows `caps_add_feature` to exit early
when called for these features, avoiding redundant presence updates.
When paging down beyond the visible chat buffer, the scroll position could
jump unexpectedly. This occurred because the visual distance from the bottom
(e.g. 6 lines until the last message) wasn’t preserved after loading new
messages from the DB.
This patch fixes that by capturing the visual offset (`current_offset`) before
the DB fetch and reapplying it after, based on the y-position of the final
buffer entry. This ensures consistent and smooth scroll transitions, even
as message heights vary.
In addition to fixing the visual glitch, this commit introduces a targeted
performance optimization: once the bottom of the chat has been reached
(`WIN_SCROLL_REACHED_BOTTOM`), we skip expensive operations entirely —
including datetime lookups and entry formatting — during further page_down
events. This reduces performance stalls
during rapid paging (e.g. holding the Page Down key at the bottom of the scrolling area).
This update also adapts to the recent change in `chatwin_db_history()`,
which now returns a `db_history_result_t` instead of `gboolean`, allowing
correct differentiation between an empty result and an actual DB error.
Together, these changes:
- Fix a scroll-jump bug in `page_down`
- Improve performance and responsiveness during rapid navigation
- Align with improved DB result handling
The function did not decref the Python object on type error paths,
resulting in a reference count leak. This caused memory to accumulate
over time. Added Py_XDECREF on all early return paths to properly free
the Python object reference and prevent leaks.
- Introduce static helper `_truncate_datetime_suffix()` to safely trim datetime strings, removing unwanted suffixes like timezone offsets
- Replace manual string management with auto_gchar and g_strdup for safer, clearer ownership and to prevent leaks
- Add safety checks and logging warnings for unexpected datetime string lengths or null pointers
- Refactor _mam_rsm_id_handler to use the helper function and updated string handling
- Change log_database_get_previous_chat parameters for consistent ownership semantics, avoiding double frees and mem leaks
- Overall improve stability and prevent memory leaks during log database queries
This commit refines the existing logic in win_page_up() by:
- Improving comments to clearly explain the rationale behind adjusting
the scroll offset relative to the first buffer entry’s visual position,
helping future maintainers understand why this is necessary.
- Fixing offset recalculation to better handle cases where older messages
with variable heights are loaded from the DB, improving scroll smoothness.
- Changing the logging of negative *page_start values from warning to debug,
recognizing that this can be a normal scenario when insufficient history is loaded.
- Simplifying some conditionals and renaming variables for clearer intent.
No changes yet applied to win_page_down(), but similar improvements could
be considered in the future.
Overall, this enhances the robustness and user experience of scrolling up
in chat windows, while preserving existing functional logic.
