A flat-file alternative to the SQLite chatlog backend with runtime
switching, full migration tooling, integrity verification, and a
synthetic load harness. SQLite remains the default; both backends share
one dispatch layer (db_backend_t vtable) so callers don't change.
Storage layout
- Per-contact append-only `flatlog/<account>/<contact>/history.log`
under XDG_DATA_HOME, one line per message
- Single-line file header with embedded format-version marker
(FLATFILE_FORMAT_VERSION); reader warns on missing or mismatched
marker, writer and checker stay in sync via preprocessor
stringification
- Deterministic key=value metadata (`id`, `aid`, `corrects`, `to`,
`to_res`, `read`) plus escaped body \u2014 `\|`, `\]`, `\\`, `\n`, `\r`
literals prevent log injection
- Sparse byte-offset index (FF_INDEX_STEP=500) per contact for
O(log n) time-range lookups; rebuilt on inode / size / mtime
change, extended in-place when the file just grew
- Per-contact GHashTable caches for archive_id presence and
stanza_id \u2192 from_jid mapping (O(1) MAM dedup, O(1) LMC sender
validation)
Hardening
- Path-traversal protection: JID directory name normalisation
(`@` \u2192 `_at_`, slashes and `..` rejected at construction); every
per-contact path is anchored under the account's flatlog/
directory and validated before open
- Symlink-attack protection: every fopen / open uses O_NOFOLLOW; on
ELOOP the operation aborts with an error rather than following
- Filesystem permissions: log files created with mode 0600,
directories with mode 0700; both enforced at creation, verified
on each open and reported on drift by `/history verify`
- Atomic crash-safe export: write to a temp file via mkstemp (mode
0600, random suffix, no name collisions between concurrent
exports), fsync, then rename \u2014 partial state never replaces the
live file
- Concurrency: advisory flock(LOCK_EX) held for the duration of
every write, including append from live messages and full rewrite
from export, so two profanity processes can't interleave bytes
on the same log
- DoS / abuse guards:
* FF_MAX_LINE_LEN = 10 MB \u2014 lines longer than this are rejected
at read with a warning; the parser will not allocate
unbounded memory for a single record
* FF_MAX_LMC_DEPTH = 100 \u2014 `corrects:` chain walk stops at this
depth and emits a warning, preventing a malicious correction
cycle from spinning the apply pass
* FF_VERSION_SCAN_MAX = 16 \u2014 header version probe never reads
past 16 leading comment lines, even on garbage input
* Empty / inverted byte-range early-return in page-up read path
so a malformed time filter cannot cause an unbounded scan
* Zero-entry index guard so a file whose every line failed to
parse cannot cause a NULL deref on later page-up
- LMC sender validation: an incoming correction whose sender does
not match the original message's sender is rejected at write
time and surfaced via cons_show_error; a cycle in the apply pass
is broken via a visited-set
- jid_create_from_bare_and_resource treats NULL, empty string, and
the literal "(null)" as no resource and returns a bare jid;
similar normalisation for barejid eliminates the legacy
"user@host/(null)" artefact that leaked into stored fulljids
whenever g_strdup_printf("%s", NULL) ran inside create_fulljid
Commands
- `/history switch sqlite|flatfile` \u2014 runtime backend swap, closes
the old backend and opens the new one without reconnecting
- `/history export [<jid>]` \u2014 SQLite -> flat-file, merging with any
existing flatlog (dedup keyed on a SHA-256 hash mixing stanza_id,
timestamp, from_jid, body \u2014 robust against id reuse by older
clients)
- `/history import [<jid>]` \u2014 flat-file -> SQLite, same merge
semantics, runs inside a single SQLite transaction with rollback
on per-contact failure
- `/history verify [<jid>]` \u2014 integrity check; emits a structured
list of issues (ERROR / WARNING / INFO) per file:
* file-level: missing log, wrong permissions (\u2260 0600), UTF-8
BOM present, CRLF line endings, empty file
* line-level: invalid UTF-8 (with byte offset), embedded
control characters, unparsable lines, timestamps out of
order, duplicate `id:` and `aid:` (tracked separately so a
stanza/archive id collision isn't double-reported)
* cross-line: broken `corrects:` references whose target id is
not present in the file
- `/history backend` \u2014 show currently active backend
- Active backend indicator `[sqlite]` / `[flatfile]` in the status
bar next to the JID
- Roster-JID autocomplete for verify / export / import
- export and import open a SQLite handle on demand when the
flatfile backend is currently active, so migration works
regardless of which backend is live
Tests
- Unit: database_export (parser round-trip, escape/unescape, dedup
key stability, JID normalisation), database_stress (14 cases
exercising rapid writes, large messages, deep LMC chains, MAM
dedup, concurrent contacts)
- Functional: history persistence across reconnects, export /
import round-trip with content equality, MUC migration,
timestamp normalisation across timezones
- Bench harness P1\u2013P5 (synthetic load: bulk insert, time-range
read, page-up scroll, MAM ingest, mixed workload) and failure
modes F1\u2013F17 (page-up cursor and forward-iteration symmetry,
oversized lines, MAM dedup, LMC depth and cycles, BOM/CRLF,
missing log, empty file, mtime+inode flip, broken corrects, etc.)
- All bench tests integrate with the existing make targets and
emit CSV rows for baseline comparison
Author: jabber.developer2 <jabber.developer2@jabber.space>
Reviewed-by: jabber.developer <jabber.developer@jabber.space>
- Introduce static helper `_truncate_datetime_suffix()` to safely trim datetime strings, removing unwanted suffixes like timezone offsets
- Replace manual string management with auto_gchar and g_strdup for safer, clearer ownership and to prevent leaks
- Add safety checks and logging warnings for unexpected datetime string lengths or null pointers
- Refactor _mam_rsm_id_handler to use the helper function and updated string handling
- Change log_database_get_previous_chat parameters for consistent ownership semantics, avoiding double frees and mem leaks
- Overall improve stability and prevent memory leaks during log database queries
This commit refines the existing logic in win_page_up() by:
- Improving comments to clearly explain the rationale behind adjusting
the scroll offset relative to the first buffer entry’s visual position,
helping future maintainers understand why this is necessary.
- Fixing offset recalculation to better handle cases where older messages
with variable heights are loaded from the DB, improving scroll smoothness.
- Changing the logging of negative *page_start values from warning to debug,
recognizing that this can be a normal scenario when insufficient history is loaded.
- Simplifying some conditionals and renaming variables for clearer intent.
No changes yet applied to win_page_down(), but similar improvements could
be considered in the future.
Overall, this enhances the robustness and user experience of scrolling up
in chat windows, while preserving existing functional logic.
Before there was a problem of overscrolling:
when messages longer than y axis of the terminal are fetched from the DB,
profanity scroll "jumps" to the top, skipping some messages.
It's resolved by keeping messages' starting and ending line in the
internal profanity buffer, which allows to track proper message positions
and to adjust window position accordingly.
Message size is now tracked as part of the buffer's record in `_line`
variable, which allows calculation of the total buffer size, which
might be a part of the improved solution for the "underscrolling" problem,
if we are going to limit profanity's buffer size by amount of lines as
opposed to the limitation based on the amount of message which is currently
used.
Before adding a limitation by amount of lines, careful consideration is
required, as some users don't use history and their temporary message
history can be cut to minimal limit because of 1 long received/sent message.
Underscrolling problem was fixed in a previous commit
d7e46d64fe
Short recap of the problem:
Despite user scrolling to top/bottom of history,
factual position is offset from the intended location
Another feature of this commit is a minor change which adds fetching
message stanza IDs from the DB. It allows correcting messages
fetched from history.
Fixes https://github.com/profanity-im/profanity/issues/1934
Use a singleton `Jid` inside the connection instead of always re-creating
a `Jid` from the same string.
Signed-off-by: Steffen Jaeckel <jaeckel-floss@eyet-services.de>
Enhance data consistency by updating the database to treat empty
strings as NULL values. This change simplifies queries and improves
overall database integrity.
**Please, backup your DB before performing any testing.**
Introduce new DB structure and DB migration mechanism.
Index `timestamp`, `to_jid`, `from_jid` columns to improve performance.
Add trigger for `replaced_by_db_id` calculation by DB on message insert.
Now LMC messages are interconnected with original messages,
this way we have fast access to last (hence correct) applicable edits,
as well as reference to the original message from any edit (in case of chained edits).
Change the way LMC messages are being displayed. Now we check if we
can replace a message from current buffer. If we don't have a message in
the buffer, it might've been lost, but we can still display it as a
new message.
Further information available here:
https://github.com/profanity-im/profanity/pull/1893https://github.com/profanity-im/profanity/issues/1899https://github.com/profanity-im/profanity/pull/1902
When we received a message correction via `XEP-0308: Last Message Correction`
we accepted the change without checking the sender
making it possible for anybody to replace the message if the ID was known.
This change has been proposed by @jubalh
https://github.com/profanity-im/profanity/pull/1893#issuecomment-1752434740
Add logging for cases when the message is not being inserted in the DB due to its ID.
Make diagnostic of unreliable message history much easier,
lay the groundwork for a fully reliable message history.
Further changes might include insertion of messages with non-unique stanza-id,
but as it's a big change with plentiful issues, decision was made
to further investigate potential causes of history unreliability.
For details see #1893 discussion
Add ability to completely turn logs off,
Needed since `/logging` and `/history` didn't stop history logging
in the DB, only in files.
Command might break something on usage,
hence it was primarily introduced for privacy geeks.
Privacy command discussion #1836
I think this PR already solves and improves the MAM situation a lot.
What's @MarcoPolo-PasTonMolo still wanted to do in this branch is:
* MAM for mucs
* Check if url and quotes autocompletion works fine
* Check if the api still works fine
* Resolve conflicts
Conflicts are solved with this commit.
MAM for mucs can be another feature PR.
The rest we can check while being on master. And more people can help
testing.
Did this by waiting for a batch of MAM messages to arrive before
prepending them to the buffer. Also limited the number of messages
to fetch to 10 so that the user gets more frequent updates.
As all parts of the code invoking the `files_get_account_data_path()`
function did the same afterwards, a function has been added with the same
behavior.
1. create path
2. `mkdir` of that path
3. return final path
Signed-off-by: Steffen Jaeckel <jaeckel-floss@eyet-services.de>
The original intention of the code was that in case archive_id is not set, NULL should be inserted. What is
inserted however is an empty string. This causes the condition to not insert messages with non-unique
archive_id insert only one message in total and ignore all further ones (if NULL was there, the condition
would work properly). And this in turn causes chat history not work properly.
This commit makes the SQL condition work properly and therefore fixes chat history.
Fixes#1589.
