Files
cproof/CONTRIBUTING.md
Jabber Developer 3f36c303c2
All checks were successful
CI Code / Check spelling (push) Successful in 21s
CI Code / Check coding style (push) Successful in 31s
CI Code / Code Coverage (push) Successful in 2m21s
CI Code / Linux (ubuntu) (push) Successful in 4m30s
CI Code / Linux (debian) (push) Successful in 6m43s
CI Code / Linux (arch) (push) Successful in 10m8s
feat(history): flat-file backend with bidirectional SQLite migration
A flat-file alternative to the SQLite chatlog backend with runtime
switching, full migration tooling, integrity verification, and a
synthetic load harness. SQLite remains the default; both backends share
one dispatch layer (db_backend_t vtable) so callers don't change.

Storage layout
- Per-contact append-only `flatlog/<account>/<contact>/history.log`
  under XDG_DATA_HOME, one line per message
- Single-line file header with embedded format-version marker
  (FLATFILE_FORMAT_VERSION); reader warns on missing or mismatched
  marker, writer and checker stay in sync via preprocessor
  stringification
- Deterministic key=value metadata (`id`, `aid`, `corrects`, `to`,
  `to_res`, `read`) plus escaped body \u2014 `\|`, `\]`, `\\`, `\n`, `\r`
  literals prevent log injection
- Sparse byte-offset index (FF_INDEX_STEP=500) per contact for
  O(log n) time-range lookups; rebuilt on inode / size / mtime
  change, extended in-place when the file just grew
- Per-contact GHashTable caches for archive_id presence and
  stanza_id \u2192 from_jid mapping (O(1) MAM dedup, O(1) LMC sender
  validation)

Hardening
- Path-traversal protection: JID directory name normalisation
  (`@` \u2192 `_at_`, slashes and `..` rejected at construction); every
  per-contact path is anchored under the account's flatlog/
  directory and validated before open
- Symlink-attack protection: every fopen / open uses O_NOFOLLOW; on
  ELOOP the operation aborts with an error rather than following
- Filesystem permissions: log files created with mode 0600,
  directories with mode 0700; both enforced at creation, verified
  on each open and reported on drift by `/history verify`
- Atomic crash-safe export: write to a temp file via mkstemp (mode
  0600, random suffix, no name collisions between concurrent
  exports), fsync, then rename \u2014 partial state never replaces the
  live file
- Concurrency: advisory flock(LOCK_EX) held for the duration of
  every write, including append from live messages and full rewrite
  from export, so two profanity processes can't interleave bytes
  on the same log
- DoS / abuse guards:
    * FF_MAX_LINE_LEN = 10 MB \u2014 lines longer than this are rejected
      at read with a warning; the parser will not allocate
      unbounded memory for a single record
    * FF_MAX_LMC_DEPTH = 100 \u2014 `corrects:` chain walk stops at this
      depth and emits a warning, preventing a malicious correction
      cycle from spinning the apply pass
    * FF_VERSION_SCAN_MAX = 16 \u2014 header version probe never reads
      past 16 leading comment lines, even on garbage input
    * Empty / inverted byte-range early-return in page-up read path
      so a malformed time filter cannot cause an unbounded scan
    * Zero-entry index guard so a file whose every line failed to
      parse cannot cause a NULL deref on later page-up
- LMC sender validation: an incoming correction whose sender does
  not match the original message's sender is rejected at write
  time and surfaced via cons_show_error; a cycle in the apply pass
  is broken via a visited-set
- jid_create_from_bare_and_resource treats NULL, empty string, and
  the literal "(null)" as no resource and returns a bare jid;
  similar normalisation for barejid eliminates the legacy
  "user@host/(null)" artefact that leaked into stored fulljids
  whenever g_strdup_printf("%s", NULL) ran inside create_fulljid

Commands
- `/history switch sqlite|flatfile` \u2014 runtime backend swap, closes
  the old backend and opens the new one without reconnecting
- `/history export [<jid>]` \u2014 SQLite -> flat-file, merging with any
  existing flatlog (dedup keyed on a SHA-256 hash mixing stanza_id,
  timestamp, from_jid, body \u2014 robust against id reuse by older
  clients)
- `/history import [<jid>]` \u2014 flat-file -> SQLite, same merge
  semantics, runs inside a single SQLite transaction with rollback
  on per-contact failure
- `/history verify [<jid>]` \u2014 integrity check; emits a structured
  list of issues (ERROR / WARNING / INFO) per file:
    * file-level: missing log, wrong permissions (\u2260 0600), UTF-8
      BOM present, CRLF line endings, empty file
    * line-level: invalid UTF-8 (with byte offset), embedded
      control characters, unparsable lines, timestamps out of
      order, duplicate `id:` and `aid:` (tracked separately so a
      stanza/archive id collision isn't double-reported)
    * cross-line: broken `corrects:` references whose target id is
      not present in the file
- `/history backend` \u2014 show currently active backend
- Active backend indicator `[sqlite]` / `[flatfile]` in the status
  bar next to the JID
- Roster-JID autocomplete for verify / export / import
- export and import open a SQLite handle on demand when the
  flatfile backend is currently active, so migration works
  regardless of which backend is live

Tests
- Unit: database_export (parser round-trip, escape/unescape, dedup
  key stability, JID normalisation), database_stress (14 cases
  exercising rapid writes, large messages, deep LMC chains, MAM
  dedup, concurrent contacts)
- Functional: history persistence across reconnects, export /
  import round-trip with content equality, MUC migration,
  timestamp normalisation across timezones
- Bench harness P1\u2013P5 (synthetic load: bulk insert, time-range
  read, page-up scroll, MAM ingest, mixed workload) and failure
  modes F1\u2013F17 (page-up cursor and forward-iteration symmetry,
  oversized lines, MAM dedup, LMC depth and cycles, BOM/CRLF,
  missing log, empty file, mtime+inode flip, broken corrects, etc.)
- All bench tests integrate with the existing make targets and
  emit CSV rows for baseline comparison

Author: jabber.developer2 <jabber.developer2@jabber.space>
Reviewed-by: jabber.developer <jabber.developer@jabber.space>
2026-05-05 19:26:07 +00:00

234 lines
9.4 KiB
Markdown

# Contributing to Profanity
## Build
Please follow the [build section](https://profanity-im.github.io/guide/latest/build.html) in our user guide.
You might also take a look at the `Dockerfile.*` in the root directory.
## Submitting patches
We recommend for people to always work on a dedicated git branch for each fix or feature.
Don't work on master.
So that they can easily pull master and rebase their work if needed.
For fixing (reported) bugs we usually use `git switch -c fix/issuenumber-somedescription`.
When working on a new feature we usually use `git switch -c feature/optionalissuenumber-somedescription`.
However this is not a rule just a recommendation to keep an overview of things.
If your change isn't a bugfix or new feature you can also just use any branch name.
### Commit messages
Write commit messages that make sense. Explain what and *why* you change.
Write in present tense.
Please give [this guideline](https://gist.github.com/robertpainsi/b632364184e70900af4ab688decf6f53) a read.
### GitHub
We would like to encourage people to use GitHub to create pull requests.
It makes it easy for us to review the patches, track WIP branches, organize branches with labels and milestones,
and help others to see what's being worked on.
Also see the blogpost [Contributing a Patch via GitHub](https://profanity-im.github.io/blog/post/contributing-a-patch-via-github/).
### E-Mail
In case GitHub is down or you can't use it for any other reason, you can send a patch to our [mailing list](https://lists.posteo.de/listinfo/profanity).
We recommend that you follow the workflow mentioned above.
And create your patch using the [`git-format-patch`](https://git-scm.com/docs/git-format-patch) tool: `git format-patch master --stdout > feature.patch`
### Another git service
We prefer if you create a pull request on GitHub.
Then our team can easily request reviews. And we have the history of the review saved in one place.
If using GitHub is out of the question but you are okay using another service (i.e.: GitLab, codeberg) then please message us in the MUC or send us an email.
We will then pull from your repository and merge manually.
### Rules
* When fixing a bug, describe it and how your patch fixes it.
* When fixing a reported issue add an `Fixes https://github.com/profanity-im/profanity/issues/23` in the commit body.
* When adding a new feature add a description of the feature and how it should be used (workflow).
* If your patch adds a new configuration option add this to the `profrc.example` file.
* If your patch adds a new theming option add this to the `theme_template` file.
* Each patch or pull request should only contain related modifications.
* Run the tests and code formatters before submitting (c.f. Chapter 'Check everything' of this README).
* When changing the UI it would be appreciated if you could add a before and after screenshot for comparison.
* Squash fixup commits into one
* If applicable, document how to test the functionality
## Hints and Pitfalls
* When adding a new hotkey/shortcut make sure it's not registered in Profanity already. And also that it's not a default shortcut of readline.
* We ship a `.git-blame-ignore-revs` file containing banal commits which you will most likely want to ignore when using `git blame`. In case you are using vim and [fugitive](https://github.com/tpope/vim-fugitive) `command Gblame Git blame --ignore-revs-file=.git-blame-ignore-revs` might be helpful in your vimrc. You can also set the `blame.ignoreRevsFile` option in your git config to have `git blame` generally ignore the listed commits.
## Coding style
Follow the style already present ;-)
To make this easier for you we created a `.clang-format` file.
You'll need to have `clang-format` installed.
Then just run `make format` before you do any commit.
It might be a good idea to add a git pre-commit hook.
So git automatically runs clang-format before doing a commit.
You can add the following snippet to `.git/hooks/pre-commit`:
```shell
for f in $(git diff --cached --name-only)
do
if [[ "$f" =~ \.(c|h)$ ]]; then
clang-format -i $f
fi
done
```
If you feel embarrassed every time the CI fails you can add the following
snippet to `.git/hooks/pre-push`:
```shell
#!/bin/sh
set -e
./ci-build.sh
```
This will run the same tests that the CI runs and refuse the push if it fails.
The CI script runs 4 parallel builds with different configurations:
- **Full** — all features enabled (+ coverage in `--coverage-only` mode)
- **Minimal** — all optional features disabled
- **NoEncrypt** — no encryption (OTR, PGP, OMEMO disabled)
- **Default** — default ./configure options
Each build runs Valgrind and functional tests on Linux.
Use `./ci-build.sh --coverage-only` to run only the Full build with coverage collection.
Output shows test results per build:
```
✓ Full PASSED
Unit tests: 437 passed, 0 failed
Functional tests: 69 passed, 0 failed
Coverage: Lines: 27.5% | Functions: 36.2% | Branches: 18.1%
Duration: 5m39s
```
Note that it will run on the actual content of the repository directory and not
what may have been staged/committed.
If you're in a hurry you can add the `--no-verify` flag when issuing `git push`
and the `pre-push` hook will be skipped.
*Note:* We provide a config file that describes our coding style for clang. But due to a mistake on their side it might happen that you can get a different result that what we expect. See [here](https://github.com/profanity-im/profanity/pull/1774) and [here](https://github.com/profanity-im/profanity/pull/1828) for details. We will try to always run latest clang-format.
## Finding mistakes
Test your changes with the following tools to find mistakes.
### unit tests
Run `make check` to run the unit tests with your current configuration or `./ci-build.sh` to check with different switches passed to configure.
### flat-file backend tests
To run functional tests with the flat-file database backend (instead of SQLite):
```bash
make check-functional-flatfile
```
Or manually for a single group:
```bash
PROF_FLATFILE=1 PROF_TEST_GROUP=1 ./tests/functionaltests/functionaltests 1
```
### valgrind
We provide a suppressions file `prof.supp`. It is a combination of the suppressions for shipped with glib2, python and custom rules.
`G_DEBUG=gc-friendly G_SLICE=always-malloc valgrind --tool=memcheck --track-origins=yes --leak-check=full --leak-resolution=high --num-callers=30 --show-leak-kinds=definite --log-file=profval --suppressions=prof.supp ./profanity`
There's also the option to create a "personalized" suppression file with the up-to-date glib2 and python suppressions.
`make my-prof.supp`
After executing this, you can replace the `--suppressions=prof.supp` argument in the above call, by `--suppressions=my-prof.supp`.
### clang
Running the clang static code analyzer helps improving the quality too.
```
make clean
scan-build make
scan-view ...
```
### Security checks
We have a static analyzer `check-cwe134.sh` that detects CWE-134 format string vulnerabilities. It runs automatically in CI but you can also run it locally:
```bash
./check-cwe134.sh
```
This checks for unsafe patterns where data could be passed directly as a format string to functions like `printf`, `cons_show`, etc. Never pass a raw string for formatting; use `"%s"` format specifier instead.
### Finding typos
We include a `.codespellrc` configuration file for `codespell` in the root directory.
Before committing it might make sense to run `codespell` to see if you made any typos.
You can run the `make spell` command for this.
### Check everything
`make doublecheck` will run the code formatter, spell checker and unit tests.
### Functional tests
The functional test suite uses [stabber](https://git.jabber.space/devs/stabber) as a mock XMPP server. Tests are located in `tests/functionaltests/`.
#### Running functional tests
Functional tests require stabber to be installed. Once installed, tests run as part of `make check`:
```bash
make check # Run all tests (unit + functional)
make check-functional-parallel # Run functional tests in parallel (~3x faster)
./tests/functionaltests/functionaltests # Run all functional tests sequentially
./tests/functionaltests/functionaltests 1 # Run specific group (1-4)
```
#### Test groups
Tests are organized into 4 groups for parallel execution:
| Group | Description |
|-------|-------------|
| 1 | Connect, Ping, Rooms, Software |
| 2 | Message, Receipts, Roster, Chat Session |
| 3 | Presence, Disconnect |
| 4 | MUC, Carbons |
To add a new group:
1. Define the test array in `functionaltests.c`
2. Add entry to `groups[]` array
3. Update `FUNC_TEST_GROUPS` in `Makefile.am`
#### Writing functional tests
Use content-based stubbing with stabber:
```c
// Use stbbr_for_query for IQ queries (roster, disco, etc.)
stbbr_for_query("jabber:iq:roster", "<iq type='result'>...</iq>");
// Use stbbr_send for presence, message, and push-style stanzas
stbbr_send("<presence from='buddy@localhost'>...</presence>");
```
Guidelines:
1. Use `stbbr_for_query(namespace, xml)` for IQ queries where the namespace is stable.
2. Use `stbbr_send(xml)` for presence, message, and other push-style stanzas.
3. Keep assertions tolerant of ordering when possible; use `prof_output_regex()` for flexible matching.
4. If timing issues appear, use `prof_timeout()` around critical expectations and reset afterwards.
5. When adding new tests, place them in the appropriate group based on functionality.