test(ai): expand unit and functional coverage for /ai feature
Unit tests (tests/unittests/test_ai_client.c): +50 cases on top of the
existing 50 — total 100. Covers:
- chat response parser (ai_parse_response): OpenAI content + Perplexity
text formats, escape decoding, empty/null/missing inputs, format-string
safety, multiline content
- error envelope parser (ai_parse_error_message): standard envelope,
nested escapes, missing fields, null/empty
- extended JSON escape: \b, \f, \r, all-specials, UTF-8 pass-through
- provider autocomplete cycling with >=2 matches and wrap-around
- session edge cases: NULL args, 100-message order preservation,
set_model(NULL), ref/unref(NULL)
- provider edge cases: get/remove with NULL, double-remove, survival
via session ref after ai_remove_provider
- settings: multi-key independence, missing key, cross-provider
isolation
- model parsing edges: data not array, empty data, "id" outside data,
multiple models
- prefs round-trip: set token -> shutdown -> init -> token reloaded
from disk (uses load_preferences fixture)
Functional tests:
- Tier A (test_ai.c): 15 cases for the /ai command surface that don't
need HTTP. Covers /ai help, providers list, set provider/token,
start with/without key/unknown provider, clear, remove, default
provider/model, switch without window, bad subcommand.
- Tier B (test_ai_http.c + ai_http_stub.{c,h,py}): 5 cases that
exercise the libcurl path against a local Python HTTP stub. The
stub serves canned bodies in five modes (ok, openai, 401, 500,
models) so each chat/models/error path is hit end-to-end without
network.
Infrastructure:
- TEST_GROUPS bumped to 5 in proftest.c; AI tests live in their own
Group 5 because mixing them with stabber-driven tests in Group 4
poisoned stbbr_stop() teardown.
- PROF_FUNC_TEST_AI macro in functionaltests.c registers AI tests
with ai_init_test() which wraps init_prof_test with a prof_connect()
so stabber sees a graceful disconnect at teardown.
- dist_check_DATA ships ai_http_stub.py with the source tarball.
Source-level changes to enable testing:
- src/ai/ai_client.{c,h}: dropped 'static' from _parse_ai_response
(renamed ai_parse_response) and _parse_error_response (renamed
ai_parse_error_message); declared under a "Parsing helpers (exposed
for testing)" section. Same approach as ai_parse_models_from_json.
Results in Docker (cproof-debian image):
- 595/595 unit tests pass
- 20/20 AI functional tests pass (Group 5)
This commit is contained in:
117
tests/functionaltests/ai_http_stub.py
Normal file
117
tests/functionaltests/ai_http_stub.py
Normal file
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Minimal HTTP stub for AI functional tests.
|
||||
|
||||
Listens on 127.0.0.1:<port> and serves canned JSON responses for any POST.
|
||||
Behaviour is selected via the URL path and the AI_STUB_MODE environment
|
||||
variable, set by the test runner before exec().
|
||||
|
||||
Modes:
|
||||
- "ok" : 200 with a valid OpenAI Responses-API body
|
||||
({"output":[{"content":[{"text":"...","type":"output_text"}]}]})
|
||||
- "openai" : 200 with a legacy OpenAI Chat Completions body
|
||||
({"choices":[{"message":{"content":"..."}}]})
|
||||
- "401" : 401 with an OpenAI-style error envelope
|
||||
- "500" : 500 with a body that triggers the parse-error fallback
|
||||
- "models" : 200 with a /v1/models list body
|
||||
|
||||
The reply text is taken from AI_STUB_REPLY (default: "MOCK-REPLY").
|
||||
|
||||
The stub writes its actual listening port to stdout as a single line
|
||||
"PORT=<n>" so the parent can read it back, then runs until SIGTERM.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
|
||||
|
||||
REPLY = os.environ.get("AI_STUB_REPLY", "MOCK-REPLY")
|
||||
MODE = os.environ.get("AI_STUB_MODE", "ok")
|
||||
|
||||
|
||||
def _dumps(body: object) -> bytes:
|
||||
"""Produce compact JSON without whitespace — profanity's hand-rolled
|
||||
parser does strstr for `"key":"`, which fails when json.dumps inserts
|
||||
a space after the colon by default."""
|
||||
return json.dumps(body, separators=(",", ":")).encode("utf-8")
|
||||
|
||||
|
||||
def _body_for(mode: str) -> tuple[int, bytes]:
|
||||
if mode == "ok":
|
||||
body = {
|
||||
"output": [
|
||||
{
|
||||
"content": [
|
||||
{"text": REPLY, "type": "output_text"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
return 200, _dumps(body)
|
||||
if mode == "openai":
|
||||
body = {
|
||||
"choices": [
|
||||
{"message": {"role": "assistant", "content": REPLY}}
|
||||
]
|
||||
}
|
||||
return 200, _dumps(body)
|
||||
if mode == "401":
|
||||
body = {
|
||||
"error": {
|
||||
"message": REPLY,
|
||||
"type": "invalid_request_error",
|
||||
"code": "invalid_api_key",
|
||||
}
|
||||
}
|
||||
return 401, _dumps(body)
|
||||
if mode == "500":
|
||||
return 500, b"<html><body>Internal Server Error</body></html>"
|
||||
if mode == "models":
|
||||
body = {
|
||||
"object": "list",
|
||||
"data": [
|
||||
{"id": "model-a", "object": "model"},
|
||||
{"id": "model-b", "object": "model"},
|
||||
{"id": "model-c", "object": "model"},
|
||||
],
|
||||
}
|
||||
return 200, _dumps(body)
|
||||
return 200, b"{}"
|
||||
|
||||
|
||||
class StubHandler(BaseHTTPRequestHandler):
|
||||
def _send(self):
|
||||
status, body = _body_for(MODE)
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_POST(self):
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
if length:
|
||||
self.rfile.read(length)
|
||||
self._send()
|
||||
|
||||
def do_GET(self):
|
||||
self._send()
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
return
|
||||
|
||||
|
||||
def main() -> None:
|
||||
port = int(sys.argv[1]) if len(sys.argv) > 1 else 0
|
||||
srv = HTTPServer(("127.0.0.1", port), StubHandler)
|
||||
actual = srv.server_address[1]
|
||||
print(f"PORT={actual}", flush=True)
|
||||
try:
|
||||
srv.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user