Improve Functional Test Stability and CI Reliability #79
Reference in New Issue
Block a user
No description provided.
Delete Branch "fix/test-CI-stability"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
This PR significantly improves CI reliability, test stability, and code quality by reorganizing the build pipeline and adding comprehensive verification mechanisms.
Changes Made
CI Pipeline Improvements:
--coverage-onlymode for dedicated coverage jobCoverage Collection:
Test Infrastructure:
PROF_BUILD_INDEXport allocationTesting
Related Issues
Closes #78 - CI improvements for pull request testing.
Issue was introduced in #75
c573cdb26dto2fb9b3fed92fb9b3fed9to01c100a60b01c100a60bto52a805d1ce52a805d1cetoce4cb13470ce4cb13470to9a2a7614a9f0c9417471to6c5c523a1aef673c2527tod3526007d0d3526007d0to35f1902cbc35f1902cbctodc9aed00a6dc9aed00a6toe5330ab0a8e5330ab0a8to6c5c523a1a2c308e824eto77958773067795877306to42c04d28fc42c04d28fctoadc5e26689WIP: Improve Functional Test Stability and CI Reliabilityto Improve Functional Test Stability and CI ReliabilityThank you for the PR. It's a great functional improvement. On the other note, I'd like to see slightly better practices in terms of readability and general coding best practices (avoiding magic numbers, commentary that add context rather than repeat code, and, importantly, unified error handling stategy). Nonetheless, solid PR overall.
@@ -91,2 +91,4 @@This will run the same tests that the CI runs and refuse the push if it fails.The CI script runs 4 parallel builds with different configurations, each with Valgrind and functional tests.Use `./ci-build.sh --coverage-only` to run only build 1 with coverage collection.Could you clarify that "build 1" is a standard build without flags?
Corrected
@@ -26,0 +40,4 @@# Parse STATS line from build log and set global variables# Usage: parse_build_stats <log_file># Sets: STAT_UNIT_P, STAT_UNIT_F, STAT_FUNC_P, STAT_FUNC_F,Is it required? It needs to be updated each time the set variables of the method are updated, yet it does not provide additional data. It would make more sense to explain what each variable means next to their initiation place.
Refactored, moved to separate function
@@ -216,3 +261,3 @@# shellcheck disable=SC2086../configure -C $features $extra_argsif ! ../configure -C $features $extra_args; thenWouldn't it get caught in the error handler declared earlier?
It is done foe log readability.
@@ -219,3 +267,2 @@$MAKE CC="${CC}"$MAKE check-functional-parallelif ! $MAKE CC="${CC}"; then(same as above) Wouldn't it get caught in the error handler declared earlier?
It is done foe log readability.
@@ -224,0 +356,4 @@# Save coverage.info before cleanup (for CI artifact)if [ "$run_coverage" = "yes" ] && [ -f coverage.info ]; thencp coverage.info ../coverage.infoWouldn't it create trash in non-CI (local) run?
Added [ -n "$CI" ] check. Artifacts only created in CI environment now.
@@ -243,0 +400,4 @@run_valgrind="yes"extra_flags="--enable-valgrind"elseextra_flags=""Can we repeat order of the variables for readability? I.e.
run_valgrindand thenextra_flagsto match order aboveUnified
@@ -138,1 +138,4 @@{const char *group_env = getenv("PROF_TEST_GROUP");int group = group_env ? atoi(group_env) : 0;int dir_id = (group >= 1 && group <= 4) ? group : stub_port;IMPORTANT:
Here and everywhere else: can we avoid magic constant of "4"? Can we make "TEST_GROUPS" constant or something similar? This way it would be much easier to adjust in the future
Added #define TEST_GROUPS 4 constant.
@@ -229,0 +245,4 @@/* Calculate port base: each build uses a different range of 4 ports* Build 0/1: 5230-5233, Build 2: 5234-5237, Build 3: 5238-5241, Build 4: 5242-5245 */int port_base = 5230 + ((build_idx > 0 ? build_idx - 1 : 0) * 4);Why 2 builds (0 and 1) share the same ports range?
Added explanation: builds 0/1 share ports because they're mutually exclusive (sequential mode or parallel).
@@ -234,0 +255,4 @@if (group >= 1 && group <= 4) {/* Static allocation: each group gets a dedicated port */stub_port = port_base + group - 1;fprintf(stderr, "[PROF_TEST] Build %d, Group %d: trying port %d\n", build_idx, group, stub_port);It shouldn't be at the
stderr. It would make more sense to put it in the stdout, since it's informational/debug data.Fixed. Info messages now use printf() (stdout), stderr reserved for errors only.
@@ -234,2 +260,3 @@if (stbbr_start(STBBR_LOGDEBUG, stub_port, 0) == 0) {started = TRUE;break;fprintf(stderr, "[PROF_TEST] Started stabber on port %d\n", stub_port);It shouldn't be at the
stderr. It would make more sense to put it in the stdout, since it's informational/debug data.Fixed. Info messages now use printf() (stdout), stderr reserved for errors only.
@@ -238,2 +268,3 @@/* Fallback to dynamic allocation if static failed or group=0 */if (!started) {assert_true(FALSE); // could not start stabber on any port in rangefprintf(stderr, "[PROF_TEST] Using dynamic port allocation\n");It shouldn't be at the
stderr. It would make more sense to put it in the stdout, since it's informational/debug data.Fixed. Info messages now use printf() (stdout), stderr reserved for errors only.
@@ -240,0 +272,4 @@if (stbbr_start(STBBR_LOGDEBUG, p, 0) == 0) {stub_port = p;started = TRUE;fprintf(stderr, "[PROF_TEST] Started stabber on port %d\n", stub_port);It shouldn't be at the
stderr. It would make more sense to put it in the stdout, since it's informational/debug data.Fixed. Info messages now use printf() (stdout), stderr reserved for errors only.
@@ -240,0 +280,4 @@if (!started) {fprintf(stderr, "[PROF_TEST] ERROR: could not start stabber on any port\n");assert_true(FALSE);Why do we need to do this?
Removed.
@@ -247,1 +293,3 @@"./tests/functionaltests/files/%d/xdg_data_home", stub_port);"./test-files/%d/xdg_data_home", dir_id);fprintf(stderr, "[PROF_TEST] Group %d using directories: config=%s, data=%s\n",same as above
Removed
@@ -399,3 +448,3 @@}usleep(50000); /* 50ms */usleep(50000);50 ms clarifying comment was on point. But it should be formatted in the same manner as other comments (for inline comment, use either
/*or//, not both)corrected
suggested squash commit message
ci: improve CI stability with parallel builds and Valgrind
Major changes:
Code quality:
Generally LGTM, but errors should go to
stderr. Ideally, we should also change1 2 3 4to use a constant (something like1 ... TEST_GROUPS). Please fix these minor issues and I can merge it ASAP.@@ -229,0 +249,4 @@/* Calculate port base: each build uses a different range of TEST_GROUPS ports.* Build 0 (local/default): 5230-5233, Build 1: 5230-5233, Build 2: 5234-5237, etc.* Build 0 and 1 share the same range because build 0 is for local runs (no parallel builds),* while builds 1-4 are used in CI where they run in parallel. */Can we do something with the magic number 4 in the comments as well? Here and elsewhere
Config names added additional to predefined values.
@@ -235,1 +267,3 @@break;printf("[PROF_TEST] Started stabber on port %d\n", stub_port);} else {printf("[PROF_TEST] Failed to start stabber on port %d\n", stub_port);Should be in stderr. Please check analogous cases
Corrected.
LGTM