devs/cproof
1
2
Fork 1
Code Issues 43 Pull Requests 6 Actions Packages Projects 1 Releases Wiki Activity

Add fault injection code in tests #70

New Issue
Open
opened 2026-01-16 15:29:27 +00:00 by jabber.developer2 · 0 comments
jabber.developer2 commented 2026-01-16 15:29:27 +00:00
Collaborator
Copy Link

Summary

Add infrastructure for fault injection testing to verify that tests correctly detect failures and CI pipeline properly reports them.

Motivation

Currently there's no automated way to verify that:

  1. Unit tests actually fail when expected behavior is broken
  2. CI pipeline correctly reports test failures
  3. Docker images are properly rebuilt when dependencies change (e.g., cmocka 2.0 breaking changes)

We discovered this gap when testing cmocka 2.0 compatibility - cached Docker images masked the actual test failures.

Proposed Solution

  1. Test failure injection mode

    • Add --inject-fault flag to test runner
    • Introduce deliberate assertion failures to verify test harness
    • Example: make check INJECT_FAULT=1 should always fail

  2. CI verification job

    • Periodic job that runs fault injection tests
    • Verifies CI correctly detects and reports failures
    • Alerts if "expected failures" pass (indicating broken test detection)

  3. Docker cache invalidation strategy

    • Scheduled weekly builds with --no-cache
    • Or cache key based on dependency versions
    • Ensures fresh base images with latest packages

Implementation Ideas

# In Makefile.am
if INJECT_FAULT
AM_CPPFLAGS += -DINJECT_TEST_FAULT
endif

// In test file
#ifdef INJECT_TEST_FAULT
assert_true(false && "Deliberate fault injection - this test should fail");
#endif

Acceptance Criteria

  • make check INJECT_FAULT=1 fails with clear error message
  • CI job for fault injection runs weekly
  • Documentation for fault injection testing
  • Docker images rebuilt periodically without cache

Related

  • cmocka 2.0 compatibility issue (tests passed due to cached images)
  • PR #68 - parallel functional tests optimization
  • partially addressed by #72

Labels

testing, ci/cd, enhancement

### Summary Add infrastructure for fault injection testing to verify that tests correctly detect failures and CI pipeline properly reports them. ### Motivation Currently there's no automated way to verify that: 1. Unit tests actually fail when expected behavior is broken 2. CI pipeline correctly reports test failures 3. Docker images are properly rebuilt when dependencies change (e.g., cmocka 2.0 breaking changes) We discovered this gap when testing cmocka 2.0 compatibility - cached Docker images masked the actual test failures. ### Proposed Solution 1. **Test failure injection mode** - Add `--inject-fault` flag to test runner - Introduce deliberate assertion failures to verify test harness - Example: `make check INJECT_FAULT=1` should always fail 2. **CI verification job** - Periodic job that runs fault injection tests - Verifies CI correctly detects and reports failures - Alerts if "expected failures" pass (indicating broken test detection) 3. **Docker cache invalidation strategy** - Scheduled weekly builds with `--no-cache` - Or cache key based on dependency versions - Ensures fresh base images with latest packages ### Implementation Ideas ```makefile # In Makefile.am if INJECT_FAULT AM_CPPFLAGS += -DINJECT_TEST_FAULT endif ``` ```c // In test file #ifdef INJECT_TEST_FAULT assert_true(false && "Deliberate fault injection - this test should fail"); #endif ``` ### Acceptance Criteria - [ ] `make check INJECT_FAULT=1` fails with clear error message - [ ] CI job for fault injection runs weekly - [ ] Documentation for fault injection testing - [ ] Docker images rebuilt periodically without cache ### Related - cmocka 2.0 compatibility issue (tests passed due to cached images) - PR #68 - parallel functional tests optimization - partially addressed by #72 ### Labels `testing`, `ci/cd`, `enhancement`
jabber.developer2 added the
Kind/Testing
 label 2026-01-16 15:29:27 +00:00
jabber.developer2 added this to the Plans (2025-2026) project 2026-01-16 15:29:27 +00:00
jabber.developer moved this to In Progress in Plans (2025-2026) on 2026-01-16 15:29:45 +00:00
jabber.developer2 changed title from Add fault injection testing infrastructure to Add fault injection code in tests 2026-01-21 15:51:43 +00:00
jabber.developer moved this to To Do in Plans (2025-2026) on 2026-01-21 15:56:12 +00:00
jabber.developer added the
Priority
Low
4
Reviewed
Confirmed
1
 labels 2026-01-21 17:13:27 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label
Kind/Testing
Priority
Low
4
Reviewed
Confirmed
1
Milestone
No items
No Milestone
Projects
Clear projects
No project Plans (2025-2026)
Assignees
Clear assignees
admin jabber.developer jabber.developer2
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: devs/cproof#70
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?