WIP: feat(tests): enable parallel execution of functional tests #68

Closed

jabber.developer2 wants to merge 15 commits from feat/parallel-functional-tests into master

pull from: feat/parallel-functional-tests

merge into: devs:master

devs:master

devs:feat/ai-api-type

devs:fix/editor-terminal-size

devs:ci/docker-hub-publishing

devs:feat/disco-ac

devs:feat/privacy-enhancements

devs:fix/clientid-regression

devs:fix/ai-chat-completions-followup

devs:feat/ai-custom

devs:fix/unencrypted-send

devs:rollback/pre-upstream-merge

devs:fix/autoping-warning-null-domain

devs:fix/pad-dead-space-reclaim

devs:feat/autoping-warning

devs:chore/untrack-gitversion

devs:fix/multiline-pad-clip

devs:fix/issue-112-followups

devs:fix/issue-128-migrate-v3-dedup

devs:fix/delay-timestamp-validation

devs:ref/light-cleanup

devs:fix/history-scroll-pad-redraw-storm

devs:fix/plugin-post-display-incoming-only

devs:merge/upstream-full

devs:merge-improve

devs:chore/remove-chatlog-stage-1

devs:fix/ai-json-encoding

devs:feat/no-db-backlog-114

devs:fix/scroll-non-chat-windows

devs:fix/paged-non-chat-windows

devs:fix/ai-leaks

devs:feat/ai

devs:fix/ai-followups

devs:test/ai-coverage-unit-only

devs:test/ai-coverage

devs:refactor/scroll-mechanism

devs:fix/verify-per-contact-context

devs:feat/no-db-mode

devs:feat/ai-json

devs:feat/pikaur-parity-arch

devs:fix-pikaur-build

devs:feat/upstream-sync

devs:feat/functest-speedup

devs:fix/cwe-134-format-string-audit

devs:ci/separate-build-step

devs:test/autoping-functional-tests

devs:test/db-functional-tests

devs:fix/arch-build

devs:fix/xep-0030-disco-items-error-handling

devs:fix/xep-0030-empty-disco-items

devs:playground/fix/src_refactoring

devs:tests/disco

devs:fix/test-CI-stability

devs:feat/parallel-tests-clean

devs:fix/functional_tests_v2

devs:fix/functional_tests

devs:fix/connect_max_args

devs:feat/extended_debug_info

devs:playground/fix/scroll-stuck

devs:build/multicore

devs:build/reenable-fedora

devs:build/autoupdate

Conversation 0 Commits 15 Files Changed 13 +392 -117

jabber.developer2 commented 2026-01-13 14:56:51 +00:00

Collaborator

Copy Link

Summary

Implement parallel functional test execution to reduce CI time (~3.2x speedup).

Changes

Test Infrastructure

• Dynamic XDG paths: Each test instance uses unique directories based on stabber port (./tests/functionaltests/files/{port}/xdg_*), enabling safe parallel execution
• Test groups: Split 69 tests into 4 balanced groups that can run independently
• CLI argument: ./functionaltests [1-4] to run specific group

Build System

• Added make check-functional-parallel target
• Updated ci-build.sh to use parallel execution by default

Test Groups

Group	Tests	Content
1	17	Connect, Ping, Rooms, Software
2	17	Message, Receipts, Roster, Chat Session
3	16	Presence, Disconnect
4	19	MUC, Carbons

Usage

# Parallel execution (default in CI)
make check-functional-parallel

# Or manually
./functionaltests 1 & ./functionaltests 2 & ./functionaltests 3 & ./functionaltests 4 & wait

# Sequential (legacy)
./functionaltests


Resolves #26

## Summary Implement parallel functional test execution to reduce CI time (~3.2x speedup). ## Changes ### Test Infrastructure - **Dynamic XDG paths**: Each test instance uses unique directories based on stabber port (`./tests/functionaltests/files/{port}/xdg_*`), enabling safe parallel execution - **Test groups**: Split 69 tests into 4 balanced groups that can run independently - **CLI argument**: `./functionaltests [1-4]` to run specific group ### Build System - Added `make check-functional-parallel` target - Updated `ci-build.sh` to use parallel execution by default ## Test Groups | Group | Tests | Content | |-------|-------|---------| | 1 | 17 | Connect, Ping, Rooms, Software | | 2 | 17 | Message, Receipts, Roster, Chat Session | | 3 | 16 | Presence, Disconnect | | 4 | 19 | MUC, Carbons | ## Usage ```bash # Parallel execution (default in CI) make check-functional-parallel # Or manually ./functionaltests 1 & ./functionaltests 2 & ./functionaltests 3 & ./functionaltests 4 & wait # Sequential (legacy) ./functionaltests Resolves #26

jabber.developer2 added 1 commit 2026-01-13 14:56:52 +00:00

feat(tests): enable parallel execution of functional tests fa17173e5c

jabber.developer2 added 1 commit 2026-01-15 14:22:37 +00:00

feat(tests): update Dockerfile and CI script for parallel builds and improved caching bed046e7ed

jabber.developer2 added this to the Plans (2025-2026) project 2026-01-16 09:35:12 +00:00

jabber.developer2 added 1 commit 2026-01-16 13:48:21 +00:00

feat: update compatibility header for cmocka 2.0 deprecation handling 5153e04f96

jabber.developer2 added 1 commit 2026-01-16 14:12:11 +00:00

fix(build): use $(srcdir)/tests for VPATH builds ... 7765f65be1

Fix include paths for out-of-tree builds (e.g., build-1/, build-2/)
to correctly find tests/prof_cmocka.h header file.

jabber.developer2 added 1 commit 2026-01-16 14:38:35 +00:00

test: remove cmocka 2.0 fix to verify CI failure ... e1a9f55a32

Temporarily removing CMOCKA_DISABLE_DEPRECATION_WARNINGS to demonstrate
that tests fail on debian:testing (cmocka 2.0.1) without this fix.

jabber.developer2 added 1 commit 2026-01-16 15:16:01 +00:00

ci: disable Docker cache to test cmocka 2.0 breakage ... 16d4efbccc

Add --no-cache to docker build to force fresh image build with
latest cmocka 2.0.1 from debian:testing.

jabber.developer2 referenced this pull request 2026-01-16 15:29:27 +00:00

Add fault injection code in tests #70

jabber.developer2 added 1 commit 2026-01-16 16:34:10 +00:00

perf: optimize build with configure cache and parallel make ... c125746f6e

- Add -C flag to ./configure for caching results
- Use --depth 1 for git clone (faster)
- Use make -j$(nproc) for stabber/libstrophe builds

Borrowed optimizations from build/multicore branch.

jabber.developer2 added 1 commit 2026-01-17 14:46:15 +00:00

feat: add code coverage support ... f41888b7a5

- Add --enable-coverage option to configure.ac
- Add coverage and coverage-html targets to Makefile.am
- Add coverage CI job with Codecov upload
- Add lcov to all Dockerfiles (arch, debian, fedora, tumbleweed, ubuntu)

Usage:
  ./configure --enable-coverage
  make check
  make coverage-html

jabber.developer2 added 1 commit 2026-01-17 15:03:28 +00:00

ci: improve test failure detection in parallel tests ... e40b89ad92

1. ci-build.sh: Add verify_test_failure_detection() function
   - Tests that single test failures are properly detected
   - Tests that parallel test failures are properly detected
   - Runs at CI start to catch framework issues early

2. Makefile.am: Fix check-functional-parallel exit code handling
   - OLD: used 'wait' without checking individual exit codes
   - NEW: capture each PID and check exit code with 'wait $pid'
   - On failure: print the log file and set failed flag
   - Exit with error if any group failed

This ensures CI correctly fails when:
- Any unit test fails
- Any of the 4 functional test groups fails
- Any parallel build configuration fails

jabber.developer2 added 1 commit 2026-01-17 15:13:57 +00:00

ci: run coverage on PRs too, not just master push 1d113c4d64

jabber.developer2 added 1 commit 2026-01-17 15:37:15 +00:00

ci: fix coverage lcov exclude pattern ... ca5835c58e

The code is built in /usr/src/profanity/ inside Docker, so
excluding '/usr/*' removes all source files!

Fix: exclude only system headers/libs (/usr/include/*, /usr/lib/*),
not the entire /usr/ tree.

Also add --ignore-errors empty to handle edge cases.

jabber.developer2 added 1 commit 2026-01-17 16:10:35 +00:00

ci: add functional tests to coverage + fix lcov errors ... d80092821b

1. Add 'make check-functional-parallel' to coverage job
   - Unit tests alone cover ~27% (mostly utilities)
   - Functional tests exercise UI, XMPP, and full application flow
   - Should significantly increase coverage

2. Fix lcov errors:
   - Remove /usr/lib/* pattern (unused, causes error)
   - Add 'unused' to --ignore-errors

jabber.developer2 added 1 commit 2026-01-19 12:54:20 +00:00

ci: add branch coverage to lcov ... f6b621ad40

Enable branch coverage with --rc lcov_branch_coverage=1.
This provides more detailed coverage metrics:
- Line coverage: % of code lines executed
- Function coverage: % of functions called
- Branch coverage: % of if/else/switch branches taken

Branch coverage is a step towards MC/DC (Modified Condition/Decision Coverage)
used in safety-critical systems.

Also add coverage/ to .gitignore.

jabber.developer2 added 1 commit 2026-01-19 13:29:51 +00:00

ci: fix deprecated lcov option ... 4789ada834

Change lcov_branch_coverage to branch_coverage (new syntax)

jabber.developer2 added 1 commit 2026-01-19 13:33:31 +00:00

revert: remove cmocka 2.0 compatibility workaround ... 01c3205f5d

Restore original prof_cmocka.h - the CMOCKA_DISABLE_DEPRECATION_WARNINGS
workaround is not needed for this project.

jabber.developer2 closed this pull request 2026-01-19 13:43:26 +00:00

jabber.developer removed this from the Plans (2025-2026) project 2026-01-21 15:55:18 +00:00

Some checks failed

Hide all checks

CI Code / Code Coverage (pull_request) Has been cancelled

Required

Details

CI Code / Linux (arch) (pull_request) Has been cancelled

Required

Details

CI Code / Linux (ubuntu) (pull_request) Has been cancelled

Required

Details

CI Code / Check coding style (pull_request) Has been cancelled

Required

Details

CI Code / Check spelling (pull_request) Has been cancelled

Required

Details

CI Code / Linux (debian) (pull_request) Has been cancelled

Required

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

admin jabber.developer jabber.developer2

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: devs/cproof#68

No description provided.