devs/cproof
1
2
Fork 1
Code Issues 43 Pull Requests 6 Actions Packages Projects 1 Releases Wiki Activity

Test execution on CI optimization #72

Manually merged
jabber.developer merged 4 commits from feat/parallel-tests-clean into master 2026-01-21 15:52:36 +00:00
Conversation 1 Commits 4 Files Changed 14 +418 -118
jabber.developer2 commented 2026-01-19 13:43:06 +00:00
Collaborator
Copy Link

Summary

Implement parallel functional test execution, build optimizations, and code coverage support. Combined ~3x CI speedup.

Changes

1. Parallel Functional Tests

  • Dynamic XDG paths: Each test instance uses unique directories based on stabber port (./tests/functionaltests/files/{port}/xdg_*), enabling safe parallel execution
  • Test groups: Split 69 tests into 4 balanced groups that can run independently
  • CLI argument: ./functionaltests [1-4] to run specific group
  • Proper exit code handling: Each parallel group's exit code is checked individually
  • Test failure detection: CI validates that test failures are properly detected before running actual tests

2. Build Performance

  • Configure caching: Added -C flag to configure for faster reconfiguration
  • Parallel compilation: make -j$(nproc) in all Dockerfiles
  • Shallow clone: --depth 1 for git clone in Dockerfiles
  • VPATH fix: Use $(srcdir)/tests for out-of-tree builds

3. Code Coverage

  • New configure option: --enable-coverage adds gcov instrumentation
  • Makefile targets: make coverage and make coverage-html
  • CI job: Runs unit + functional tests, uploads to Codecov
  • Branch coverage: Enabled via --rc branch_coverage=1 (step towards MC/DC)
  • lcov added to all Dockerfiles

Test Groups

Group Tests Content
1 17 Connect, Ping, Rooms, Software
2 17 Message, Receipts, Roster, Chat Session
3 16 Presence, Disconnect
4 19 MUC, Carbons

Usage

# Parallel execution (default in CI)
make check-functional-parallel

# Or manually
./functionaltests 1 & ./functionaltests 2 & ./functionaltests 3 & ./functionaltests 4 & wait

# Sequential (legacy)
./functionaltests

# Coverage
./configure --enable-coverage
make && make check
make coverage        # generates coverage.info
make coverage-html   # generates HTML report

Files Changed

  • functionaltests.c - test groups and CLI
  • proftest.c - dynamic XDG paths
  • Makefile.am - parallel target, coverage targets
  • configure.ac - --enable-coverage option
  • ci-build.sh - parallel execution, test failure detection
  • ci-code.yml - coverage CI job
  • Dockerfile.* - lcov, parallel make, shallow clone

Resolves #26
Resolves #73
Partially addresses #72

## Summary Implement parallel functional test execution, build optimizations, and code coverage support. Combined ~3x CI speedup. ## Changes ### 1. Parallel Functional Tests - **Dynamic XDG paths**: Each test instance uses unique directories based on stabber port (`./tests/functionaltests/files/{port}/xdg_*`), enabling safe parallel execution - **Test groups**: Split 69 tests into 4 balanced groups that can run independently - **CLI argument**: `./functionaltests [1-4]` to run specific group - **Proper exit code handling**: Each parallel group's exit code is checked individually - **Test failure detection**: CI validates that test failures are properly detected before running actual tests ### 2. Build Performance - **Configure caching**: Added `-C` flag to configure for faster reconfiguration - **Parallel compilation**: `make -j$(nproc)` in all Dockerfiles - **Shallow clone**: `--depth 1` for git clone in Dockerfiles - **VPATH fix**: Use `$(srcdir)/tests` for out-of-tree builds ### 3. Code Coverage - **New configure option**: `--enable-coverage` adds gcov instrumentation - **Makefile targets**: `make coverage` and `make coverage-html` - **CI job**: Runs unit + functional tests, uploads to Codecov - **Branch coverage**: Enabled via `--rc branch_coverage=1` (step towards MC/DC) - **lcov** added to all Dockerfiles ## Test Groups | Group | Tests | Content | |-------|-------|---------| | 1 | 17 | Connect, Ping, Rooms, Software | | 2 | 17 | Message, Receipts, Roster, Chat Session | | 3 | 16 | Presence, Disconnect | | 4 | 19 | MUC, Carbons | ## Usage ```bash # Parallel execution (default in CI) make check-functional-parallel # Or manually ./functionaltests 1 & ./functionaltests 2 & ./functionaltests 3 & ./functionaltests 4 & wait # Sequential (legacy) ./functionaltests # Coverage ./configure --enable-coverage make && make check make coverage # generates coverage.info make coverage-html # generates HTML report ``` ## Files Changed - functionaltests.c - test groups and CLI - proftest.c - dynamic XDG paths - Makefile.am - parallel target, coverage targets - configure.ac - `--enable-coverage` option - ci-build.sh - parallel execution, test failure detection - ci-code.yml - coverage CI job - `Dockerfile.*` - lcov, parallel make, shallow clone Resolves #26 Resolves #73 Partially addresses #72 ---
jabber.developer2 added 3 commits 2026-01-19 13:43:06 +00:00
feat: parallel functional tests (~3x faster CI) cb2cd1bbf2 
- Split 69 functional tests into 4 groups for parallel execution
- Add check-functional-parallel target to Makefile.am
- Fix VPATH builds with $(srcdir)/tests path
- Add test failure detection verification in CI
- Update Dockerfiles with parallel make -j$(nproc)
- Add --depth 1 for faster git clone in Dockerfiles
perf: optimize CI build performance 5a34a4a1d3 
- Add -C flag to configure for caching
- Use ccache for faster recompilation
feat: add code coverage with branch coverage
All checks were successful
CI Code / Check spelling (pull_request) Successful in 19s
Details
CI Code / Check coding style (pull_request) Successful in 39s
Details
CI Code / Code Coverage (pull_request) Successful in 15m36s
Details
CI Code / Linux (arch) (pull_request) Successful in 16m18s
Details
CI Code / Linux (debian) (pull_request) Successful in 16m44s
Details
CI Code / Linux (ubuntu) (pull_request) Successful in 17m42s
Details
c1a60dc603 
- Add --enable-coverage option to configure.ac
- Add coverage/coverage-html targets to Makefile.am
- Add coverage CI job with lcov + Codecov upload
- Include both unit and functional tests in coverage
- Enable branch coverage (step towards MC/DC)
- Add lcov to all Dockerfiles
- Add coverage/ to .gitignore
jabber.developer2 force-pushed feat/parallel-tests-clean from c1a60dc603 to 49b552dcb1 2026-01-19 17:50:09 +00:00 Compare
jabber.developer reviewed 2026-01-19 19:52:14 +00:00
jabber.developer left a comment
Owner
Copy Link

Generally, LGTM. However, certain parts might be slightly improved.

Generally, LGTM. However, certain parts might be slightly improved.
Makefile.am Outdated
@@ -308,0 +312,4 @@
@echo "Running functional tests in parallel (4 groups)..."
@mkdir -p $(builddir)/test-logs
@failed=0; \
./tests/functionaltests/functionaltests 1 > $(builddir)/test-logs/group1.log 2>&1 & pid1=$$!; \
jabber.developer commented 2026-01-19 15:55:35 +00:00
Owner
Copy Link

Can we make it a loop for extensibility?

Can we make it a loop for extensibility?
jabber.developer2 commented 2026-01-21 12:27:47 +00:00
Author
Collaborator
Copy Link

Corrected

Corrected
jabber.developer marked this conversation as resolved
tests/functionaltests/functionaltests.c Outdated
@@ -20,2 +17,2 @@
* Group 3: MUC (Multi-User Chat) functionality
* Group 4: Carbons, Chat sessions, Software version, Disconnect
* Tests are organized into groups for better maintainability and parallel execution:
* Group 1: Connect, Ping, Rooms, Software (17 tests)
jabber.developer commented 2026-01-19 16:06:08 +00:00
Owner
Copy Link

Please remove counts (such as 17 tests) from the commentary. It's highly likely that we will miss to update those and we'll have outdated docs. Updating them each time we add/remove test is going to be painful as well.

Please remove counts (such as 17 tests) from the commentary. It's highly likely that we will miss to update those and we'll have outdated docs. Updating them each time we add/remove test is going to be painful as well.
jabber.developer2 commented 2026-01-21 12:27:57 +00:00
Author
Collaborator
Copy Link

Corrected

Corrected
jabber.developer marked this conversation as resolved
tests/functionaltests/functionaltests.c Outdated
@@ -55,3 +74,1 @@
* GROUP 1: Connect, Ping, Rooms, Presence
* Basic XMPP session establishment and presence management
* ============================================================ */
/* GROUP 1: Connect, Ping, Rooms, Software (17 tests)
jabber.developer commented 2026-01-19 16:09:07 +00:00
Owner
Copy Link

Previous style (here and later) feels better for readability.

Previous style (here and later) feels better for readability.
jabber.developer2 commented 2026-01-21 12:28:07 +00:00
Author
Collaborator
Copy Link

Corrected

Corrected
jabber.developer marked this conversation as resolved
tests/functionaltests/functionaltests.c Outdated
@@ -119,2 +132,3 @@
};
/* Room join with various options */
/* GROUP 3: Presence (16 tests)
jabber.developer commented 2026-01-19 16:11:20 +00:00
Owner
Copy Link

Same as earlier: style and counts

Same as earlier: style and counts
jabber.developer2 commented 2026-01-21 12:28:13 +00:00
Author
Collaborator
Copy Link

Corrected

Corrected
jabber.developer marked this conversation as resolved
tests/functionaltests/functionaltests.c Outdated
@@ -176,0 +191,4 @@
int result = 0;
switch (group) {
case 1:
jabber.developer commented 2026-01-19 16:15:17 +00:00
Owner
Copy Link

Do we plan to increase count of groups? If so, maybe it would make sense to make a different structure to store group names and their content and then loop over it?

Do we plan to increase count of groups? If so, maybe it would make sense to make a different structure to store group names and their content and then loop over it?
jabber.developer2 commented 2026-01-21 12:28:22 +00:00
Author
Collaborator
Copy Link

Corrected

Corrected
jabber.developer marked this conversation as resolved
tests/functionaltests/proftest.h Outdated
@@ -4,2 +4,2 @@
#define XDG_CONFIG_HOME "./tests/functionaltests/files/xdg_config_home"
#define XDG_DATA_HOME "./tests/functionaltests/files/xdg_data_home"
/*
* XDG paths are now dynamic, generated per-test based on stub_port.
jabber.developer commented 2026-01-19 16:41:39 +00:00
Owner
Copy Link

Adjust comment so it reflects current state of things, not the change that has been made (E.g. "XDG paths are used for...").

Adjust comment so it reflects current state of things, not the change that has been made (E.g. "XDG paths are used for...").
jabber.developer2 commented 2026-01-21 12:28:38 +00:00
Author
Collaborator
Copy Link

Corrected

Corrected
jabber.developer marked this conversation as resolved
jabber.developer2 added 1 commit 2026-01-21 11:56:38 +00:00
refactor: address code review feedback
All checks were successful
CI Code / Check spelling (pull_request) Successful in 21s
Details
CI Code / Check coding style (pull_request) Successful in 34s
Details
CI Code / Code Coverage (pull_request) Successful in 15m30s
Details
CI Code / Linux (debian) (pull_request) Successful in 16m11s
Details
CI Code / Linux (ubuntu) (pull_request) Successful in 18m18s
Details
CI Code / Linux (arch) (pull_request) Successful in 20m36s
Details
d6388674d5 
Makefile.am:
- Replace 4 hardcoded parallel commands with loop over FUNC_TEST_GROUPS
- Easy to extend by adding group numbers to the variable

functionaltests.c:
- Remove test counts from comments (prone to becoming outdated)
- Restore previous comment style with === separators for readability
- Replace switch with groups[] array structure for extensibility
- Add ARRAY_SIZE macro for cleaner code

proftest.h:
- Fix comment to describe current state, not the change made

CONTRIBUTING.md:
- Document test groups and parallel execution commands
- Add instructions for adding new test groups
jabber.developer2 force-pushed feat/parallel-tests-clean from d6388674d5 to f779c157a7 2026-01-21 12:32:10 +00:00 Compare
jabber.developer2 force-pushed feat/parallel-tests-clean from f779c157a7 to 56ede1b45f 2026-01-21 12:33:58 +00:00 Compare
jabber.developer2 was assigned by jabber.developer 2026-01-21 14:34:07 +00:00
jabber.developer added the
Kind/Enhancement
 label 2026-01-21 14:53:49 +00:00
jabber.developer manually merged commit 85c817ee8c into master 2026-01-21 15:52:36 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label
Kind/Enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
admin jabber.developer jabber.developer2
No Assignees jabber.developer2
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: devs/cproof#72
No description provided.
Delete Branch "feat/parallel-tests-clean"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?