- bound _parse_responses scans to the output_text part's object and to
the content array, so a later sibling item's "text" (e.g. a reasoning
summary) can never be returned as the assistant reply
- do not retry the other flavour on a curl timeout: the request likely
reached the server and may still be generating, so a re-POST of the
conversation could trigger a second billed generation
- remember an unparseable 2xx from the first AUTO attempt and surface
stashed first-attempt errors in both error paths, instead of showing
only the final attempt's transport or HTTP error
- recognize Ollama's model-not-found wording in _names_model so a model
typo is not misread as a missing endpoint
- drop the dead, racy provider-lookup fallback in the generic request
thread: a missing provider ref is a caller bug and now fails loudly
- fix ai_providers_lists_defaults to expect the header the command
actually prints ("Configured providers:"); the test was broken since
its introduction but CI never ran it
- add functional test group 5 (AI command surface) to FUNC_TEST_GROUPS
so the CI parallel target runs it; proftest.c port ranges already
account for five groups
Parser:
- Never return reasoning text as the reply: the no-output_text path
anchors strictly on the message "content" array; a reasoning-only body
(truncated by max_output_tokens, carries only "summary") now fails the
parse instead of leaking the chain-of-thought. Regression test added.
- _find_json_field skips field-name occurrences that are string values
(no ':' after), so {"type":"text","text":"hi"} parses instead of
failing on the value of "type".
- _same_json_object tracks brace depth, so a nested object between the
"text" key and the "output_text" tag (e.g. "annotations") no longer
rejects the part's own text.
- Chat-completions extraction is bounded to the choices array (new
_json_array_end helper), so "message"/"content" in a sibling object
(e.g. a top-level "warning") is not returned as the reply.
AUTO fallback:
- Model errors are classified only from a structured JSON error
envelope; a route-missing 404 page merely mentioning models (e.g.
listing /v1/models) no longer suppresses the fallback. Trade-off: a
model error in a non-JSON body costs one extra fallback attempt.
- Fall back on a curl-level failure: an endpoint that accepts the
connection but never responds no longer burns the 60s timeout without
trying the flavour that works.
- Fall back on an unparseable 2xx: a gateway answering unknown paths
with a 200 error page previously reset the hint and re-probed the same
dead flavour on every request. The body is parsed inside the attempt
loop and no longer re-parsed after it.
- The second attempt's 401/403/429/5xx now wins over the stashed 400/422
payload rejection, so an invalid key or rate limit is not hidden
behind a payload-shape error; a second 404 still defers to the first.
- When the first flavour's endpoint is missing but the other answers
(even unsuccessfully, e.g. 401), the resolved hint moves to the
surviving flavour, so the dead endpoint is not probed first forever.
Settings:
- Reserved payload keys are rejected only when setting a value; removal
is allowed again, so a setting persisted before its key became
reserved (e.g. "input") is no longer stuck in the config.
The setting-name branch for /ai set custom never fired: it passed the
prefix "/ai set custom " with a trailing space to
autocomplete_param_with_ac(), which appends a space itself, so the
matched prefix contained a double space no real input ever has. Even
without that, the primitive completes the first token after the command
while the setting name is the second, and the num_args guard only held
before the name was started. The suggestion list (tools/search/memory/
plugins) predated the /ai set custom backend and matched no real
payload parameter.
Replace the branch with token-position completion (arg 5 via
autocomplete_param_no_with_func) fed by a list rebuilt on demand from
common payload parameters plus the keys already set on the provider,
exposed through the new ai_get_provider_setting_keys() (snapshot taken
under settings_lock). Drop the now-unused parse_args() call whose NULL
result was fed to g_strv_length() unguarded, spamming a glib CRITICAL
on TAB at /ai with zero or five-plus arguments.
Drive both OpenAI-compatible flavours from one code path. Requests
default to /v1/responses and fall back once to /v1/chat/completions when
the provider reports the endpoint missing (404/405/501) or rejects the
payload shape (400/422); the working flavour is cached for the rest of
the run as a first-attempt hint, with the other flavour kept as a
fallback so a backend change self-corrects in-request. A per-provider
override is available via
/ai set api-type <provider> responses|chat-completions|auto
and persisted as api_type= in the provider section.
One payload builder serves both flavours (messages/stream vs
input/stream/store); history and custom settings are serialized once per
request and the per-flavour envelope is assembled per attempt under a
single lock acquisition, so the fallback retry cannot double-count the
prompt. ai_parse_response_typed() dispatches on the request flavour and
splits extraction per envelope: chat completions anchored on
choices[].message.content, responses on the output_text part with a
string-aware backward scan bounded to that part's own object so a
reasoning summary or a truncated body is never returned as the reply.
Endpoint detection classifies wrong-model errors from the structured
error code/type/message when present, so route-missing 404s from
gateways no longer suppress the fallback; it preserves and surfaces the
first payload-rejection error when the fallback also fails, and re-probes
on an unparseable 2xx. resolved_api_type is written only through a locked
helper guarded by a URL/api-type epoch, and api_url is snapshotted under
settings_lock in the request and models-fetch threads, closing
use-after-free and stale-cache races against /ai set provider. The
models-fetch provider ref is taken on the main thread and released on
every worker exit path. Reserved custom-setting keys are extended with
input (store stays writable as a legitimate chat-completions parameter).
The chat-completions refactor removed the legacy Perplexity "text"
extraction but left two tests asserting the old behavior, breaking
make check on every CI flavor:
- test_ai_parse_response_perplexity_text expected the legacy /v1/agent
nested format to parse; it now yields NULL — renamed to
test_ai_parse_response_legacy_text_format_unsupported.
- test_ai_parse_response_text_preferred_over_content expected "text"
to win; "content" is authoritative now — renamed to
test_ai_parse_response_content_preferred_over_text.
Add test_ai_set_provider_setting_reserved_key covering the new
reserved-key rejection.
Add an AI client module that integrates with OpenAI-compatible API
providers (OpenAI, Perplexity, and custom endpoints) to provide
AI-assisted chat within CProof. Users can start sessions with /ai start,
send prompts, receive responses in a dedicated AI window, switch between
providers and models, and manage API keys — all with tab-completion.
Providers are configured via /ai set commands with per-provider API keys,
endpoints, default models, and custom settings. Two default providers
(openai, perplexity) are seeded on first use. Provider state persists in
[ai/<name>] sections of the preferences keyfile with automatic migration
from the previous flat-key format.
The /ai command integrates into the existing command system with 8
subcommands covering provider management, session lifecycle, model
fetching, and conversation clearing. Autocomplete uses the standard
flat prefix-matching chain for reliable tab-completion at every nesting
level. A new ProfAiWin window type is added to the window system.
Architecture:
Async design: HTTP requests run on a background thread (pthread) to avoid blocking the ncurses UI loop; results are displayed on the main thread via direct function calls
Thread safety: AIProvider and AISession use atomic ref-counting and mutex-protected session state; the request thread snapshots all session data before making the HTTP call
Window validation: wins_ai_exists() prevents use-after-free when the user closes the AI window during an in-flight HTTP request (~60s)
Privacy: store:false is sent with every request to prevent providers from persisting conversations or using them for training
Response size limit: 10MB cap with immediate curl abort via CURL_WRITEFUNC_ERROR to prevent OOM
JSON parsing uses unified helpers for both chat responses and error
envelopes with consistent escape decoding. The response parser tries
Perplexity /v1/responses "text" field first, then falls back to OpenAI
"content". Error parsing extracts provider error.message from the
standard envelope format. Model parsing handles multiple API response
formats (OpenAI list, Perplexity, array) including edge cases.
Tests include 470+ lines of unit tests covering provider management,
session lifecycle, JSON parsing (multiple formats), autocomplete cycling,
and error handling, plus functional tests for /ai command dispatch.
A stub_ai.c module isolates unit tests from UI dependencies.
A flat-file alternative to the SQLite chatlog backend with runtime
switching, full migration tooling, integrity verification, and a
synthetic load harness. SQLite remains the default; both backends share
one dispatch layer (db_backend_t vtable) so callers don't change.
Storage layout
- Per-contact append-only `flatlog/<account>/<contact>/history.log`
under XDG_DATA_HOME, one line per message
- Single-line file header with embedded format-version marker
(FLATFILE_FORMAT_VERSION); reader warns on missing or mismatched
marker, writer and checker stay in sync via preprocessor
stringification
- Deterministic key=value metadata (`id`, `aid`, `corrects`, `to`,
`to_res`, `read`) plus escaped body \u2014 `\|`, `\]`, `\\`, `\n`, `\r`
literals prevent log injection
- Sparse byte-offset index (FF_INDEX_STEP=500) per contact for
O(log n) time-range lookups; rebuilt on inode / size / mtime
change, extended in-place when the file just grew
- Per-contact GHashTable caches for archive_id presence and
stanza_id \u2192 from_jid mapping (O(1) MAM dedup, O(1) LMC sender
validation)
Hardening
- Path-traversal protection: JID directory name normalisation
(`@` \u2192 `_at_`, slashes and `..` rejected at construction); every
per-contact path is anchored under the account's flatlog/
directory and validated before open
- Symlink-attack protection: every fopen / open uses O_NOFOLLOW; on
ELOOP the operation aborts with an error rather than following
- Filesystem permissions: log files created with mode 0600,
directories with mode 0700; both enforced at creation, verified
on each open and reported on drift by `/history verify`
- Atomic crash-safe export: write to a temp file via mkstemp (mode
0600, random suffix, no name collisions between concurrent
exports), fsync, then rename \u2014 partial state never replaces the
live file
- Concurrency: advisory flock(LOCK_EX) held for the duration of
every write, including append from live messages and full rewrite
from export, so two profanity processes can't interleave bytes
on the same log
- DoS / abuse guards:
* FF_MAX_LINE_LEN = 10 MB \u2014 lines longer than this are rejected
at read with a warning; the parser will not allocate
unbounded memory for a single record
* FF_MAX_LMC_DEPTH = 100 \u2014 `corrects:` chain walk stops at this
depth and emits a warning, preventing a malicious correction
cycle from spinning the apply pass
* FF_VERSION_SCAN_MAX = 16 \u2014 header version probe never reads
past 16 leading comment lines, even on garbage input
* Empty / inverted byte-range early-return in page-up read path
so a malformed time filter cannot cause an unbounded scan
* Zero-entry index guard so a file whose every line failed to
parse cannot cause a NULL deref on later page-up
- LMC sender validation: an incoming correction whose sender does
not match the original message's sender is rejected at write
time and surfaced via cons_show_error; a cycle in the apply pass
is broken via a visited-set
- jid_create_from_bare_and_resource treats NULL, empty string, and
the literal "(null)" as no resource and returns a bare jid;
similar normalisation for barejid eliminates the legacy
"user@host/(null)" artefact that leaked into stored fulljids
whenever g_strdup_printf("%s", NULL) ran inside create_fulljid
Commands
- `/history switch sqlite|flatfile` \u2014 runtime backend swap, closes
the old backend and opens the new one without reconnecting
- `/history export [<jid>]` \u2014 SQLite -> flat-file, merging with any
existing flatlog (dedup keyed on a SHA-256 hash mixing stanza_id,
timestamp, from_jid, body \u2014 robust against id reuse by older
clients)
- `/history import [<jid>]` \u2014 flat-file -> SQLite, same merge
semantics, runs inside a single SQLite transaction with rollback
on per-contact failure
- `/history verify [<jid>]` \u2014 integrity check; emits a structured
list of issues (ERROR / WARNING / INFO) per file:
* file-level: missing log, wrong permissions (\u2260 0600), UTF-8
BOM present, CRLF line endings, empty file
* line-level: invalid UTF-8 (with byte offset), embedded
control characters, unparsable lines, timestamps out of
order, duplicate `id:` and `aid:` (tracked separately so a
stanza/archive id collision isn't double-reported)
* cross-line: broken `corrects:` references whose target id is
not present in the file
- `/history backend` \u2014 show currently active backend
- Active backend indicator `[sqlite]` / `[flatfile]` in the status
bar next to the JID
- Roster-JID autocomplete for verify / export / import
- export and import open a SQLite handle on demand when the
flatfile backend is currently active, so migration works
regardless of which backend is live
Tests
- Unit: database_export (parser round-trip, escape/unescape, dedup
key stability, JID normalisation), database_stress (14 cases
exercising rapid writes, large messages, deep LMC chains, MAM
dedup, concurrent contacts)
- Functional: history persistence across reconnects, export /
import round-trip with content equality, MUC migration,
timestamp normalisation across timezones
- Bench harness P1\u2013P5 (synthetic load: bulk insert, time-range
read, page-up scroll, MAM ingest, mixed workload) and failure
modes F1\u2013F17 (page-up cursor and forward-iteration symmetry,
oversized lines, MAM dedup, LMC depth and cycles, BOM/CRLF,
missing log, empty file, mtime+inode flip, broken corrects, etc.)
- All bench tests integrate with the existing make targets and
emit CSV rows for baseline comparison
Author: jabber.developer2 <jabber.developer2@jabber.space>
Reviewed-by: jabber.developer <jabber.developer@jabber.space>
As 9f2abc75 accidentally got the ordering of some of the includes wrong,
I decided to propose my initial solution again.
Additional to that, I've opened a MR against CMocka to solve this on
their side, since I believe that the current way this is done is not
sustainable [0].
[0] https://gitlab.com/cmocka/cmocka/-/merge_requests/91
Fixes: 9f2abc75 ("Fix tests with gcc15 (uintptr_t)")
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
CProof note: our new tests need to also be updated.
Added null check for jidp in cmd_sub to handle jid_create returning NULL.
Crash occurred when processing malformed JID inputs like @example.com.
Ensures robust handling of invalid JIDs.
Fixes#22
* Also pass `$*` to `configure` when invoking `ci-build.sh`, so one can
e.g. run `./ci-build.sh --without-xscreensaver`
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
While trying to get the unit tests working again I stumbled over all those
things that I thought could be better^TM.
Now we also know "TODO: why does this make the test fail?" - because
the unit tests are brittle AF ... and we have to init the subsystems
we use in the test, otherwise the cleanup will fail...
BTW. you can now also only run a single test ... or a pattern or so ...
you'd have to read how `cmocka_set_test_filter()` works exactly.
Signed-off-by: Steffen Jaeckel <s@jaeckel.eu>
## Summary
Fixes https://github.com/profanity-im/profanity/issues/1907
Update functional and unit test code to comply with the current cmocka test runner.
## Changes
- `UnitTest` struct to `CMUnitTest` struct
- `unit_test()` macro to `cmocka_unit_test(f)` macro
- `unit_test_setup_teardown()` macro to `cmocka_unit_test_setup_teardown` macro
- `run_tests()` macro to `cmocka_run_group_tests()` function
- Setup and teardown functions return `int` instead of `void`
## Testing
### Unit Tests
`make check`
### Functional Tests
I did not compile or run functional tests because they are *shelved* for now.
### Valgrind
I'm not entirely sure how to fun Valgrind in this case. I did not do fancy memory management, so it should be fine.
Don't lower account name before calling accounts_get_account().
Only lower if there is no account with the given name and the name is
interpreted as a jid.
Updated unittests to test this behaviour.
Fixes#725
.
The flag hides errors. But we want unit tests to be correct in order
to reveal errors in the main code.
The patch removes tests_unittests_unittests_CFLAGS which makes autotools
use AM_CFLAGS instead. Therefore, unit tests are built with flags
derived from configure.ac.
Move `p_sha1_hash()` from `common.c` to `xmpp/stanza.c` as it is only
used in this file and now depends on libstrophe so xmpp is a better
namespace folder.
Renaming it as `_stanza_create_sha1_hash()`. And making static since
only used here.
The function cannot be tested in the unit tests anymore.
Once functional tests are working again we should write a test for the
sha1 functionality.
