Skip to content

ci: parallelize the repeated test job; add concurrency cancellation#4309

Closed
ReneWerner87 wants to merge 2 commits into
mainfrom
ci/speed-up-test-and-benchmark
Closed

ci: parallelize the repeated test job; add concurrency cancellation#4309
ReneWerner87 wants to merge 2 commits into
mainfrom
ci/speed-up-test-and-benchmark

Conversation

@ReneWerner87
Copy link
Copy Markdown
Member

@ReneWerner87 ReneWerner87 commented May 22, 2026

Motivation

Reduce CI wall-clock per commit without changing results, output or coverage.

Key finding: the Test workflow's long pole is the repeated job, and it is dominated by fixed test latency (133 time.Sleep calls: 14x 3s, 17x 1s, ...), not CPU. Removing -race alone saved only ~33s. The effective lever is parallelism.

Changes

test.yml

  • Concurrency group that cancels superseded in-progress PR runs (never cancels main).
  • Coverage computed only on the ubuntu/1.25.x job (the only one that uploads it); the other 5 matrix combinations skip -coverprofile -covermode=atomic.
  • repeated job sharded into 3 parallel matrix shards of -count=5 each = still 15 shuffled runs total (3 distinct shuffle seeds), -race kept.

benchmark.yml

  • Same PR-only concurrency cancellation. Otherwise unchanged (single job, default benchtime).

Measured impact (real run, not estimate)

Job Before After
Test repeated ~9m41s ~3m49s (slowest of 3 shards)
Test workflow overall ~10 min ~4 min

Functionality preserved

  • Coverage: identical, still uploaded only from ubuntu/1.25.x.
  • Race detection: unit matrix unchanged; repeated keeps -race.
  • Flaky/order detection: 15 shuffled runs total, unchanged.

Why the benchmark job is NOT sharded

An attempt to shard benchmarks across 4 runners cut wall-clock to ~4min but failed the comparison: a no-code-change commit reported a uniform ~1.5-2.5x ns/op "regression" with identical B/op/allocs/op, i.e. pure CPU variance. The benchmark comparison relies on a cpu-model cache key so it only compares same-CPU results; spreading work across GitHub's heterogeneous runners breaks that guarantee. Benchmark therefore stays a single job (correct over fast).

🤖 Generated with Claude Code

@ReneWerner87 ReneWerner87 requested a review from a team as a code owner May 22, 2026 10:01
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (2)
  • .github/workflows/benchmark.yml is excluded by !**/*.yml
  • .github/workflows/test.yml is excluded by !**/*.yml

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: c928e4ce-e58d-462b-91d1-ef5751bbeb4f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/speed-up-test-and-benchmark

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes CI wall-clock time for Fiber’s test and benchmark workflows while keeping coverage upload behavior and existing validation intent intact.

Changes:

  • Add PR-only workflow concurrency cancellation to avoid wasted CI on superseded commits.
  • Pin gotestsum to v1.13.0 and reduce unnecessary coverage instrumentation to the single job that uploads coverage.
  • Speed up benchmark runs by reducing go test benchtime.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
.github/workflows/test.yml Cancels superseded PR runs, pins gotestsum, computes coverage only on the upload job, and removes -race from the repeated/flakiness job.
.github/workflows/benchmark.yml Cancels superseded PR runs and reduces benchmark runtime via -benchtime=100ms.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 22, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.40%. Comparing base (57cf8d2) to head (ee6e2dd).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4309   +/-   ##
=======================================
  Coverage   91.40%   91.40%           
=======================================
  Files         132      132           
  Lines       13120    13120           
=======================================
  Hits        11992    11992           
  Misses        711      711           
  Partials      417      417           
Flag Coverage Δ
unittests 91.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite Current: 5fad032 Previous: 86397b0 Ratio
Benchmark_NewError (github.com/gofiber/fiber/v3) 85.77 ns/op 24 B/op 1 allocs/op 33.9 ns/op 24 B/op 1 allocs/op 2.53
Benchmark_NewError (github.com/gofiber/fiber/v3) - ns/op 85.77 ns/op 33.9 ns/op 2.53
Benchmark_Ctx_AutoFormat_JSON (github.com/gofiber/fiber/v3) 428.9 ns/op 16 B/op 1 allocs/op 285.3 ns/op 16 B/op 1 allocs/op 1.50
Benchmark_Ctx_AutoFormat_JSON (github.com/gofiber/fiber/v3) - ns/op 428.9 ns/op 285.3 ns/op 1.50
Benchmark_Ctx_Write (github.com/gofiber/fiber/v3) 19.14 ns/op 70 B/op 0 allocs/op 12.74 ns/op 76 B/op 0 allocs/op 1.50
Benchmark_Ctx_Write (github.com/gofiber/fiber/v3) - ns/op 19.14 ns/op 12.74 ns/op 1.50
Benchmark_App_RebuildTree (github.com/gofiber/fiber/v3) 25290 ns/op 15960 B/op 48 allocs/op 16340 ns/op 15960 B/op 48 allocs/op 1.55
Benchmark_App_RebuildTree (github.com/gofiber/fiber/v3) - ns/op 25290 ns/op 16340 ns/op 1.55
Benchmark_Router_WithCompression (github.com/gofiber/fiber/v3) 320.6 ns/op 8 B/op 1 allocs/op 209.5 ns/op 8 B/op 1 allocs/op 1.53
Benchmark_Router_WithCompression (github.com/gofiber/fiber/v3) - ns/op 320.6 ns/op 209.5 ns/op 1.53
Benchmark_PaginateMiddlewareWithCustomConfig (github.com/gofiber/fiber/v3/middleware/paginate) 21657 ns/op 11492 B/op 43 allocs/op 14378 ns/op 11474 B/op 43 allocs/op 1.51
Benchmark_PaginateMiddlewareWithCustomConfig (github.com/gofiber/fiber/v3/middleware/paginate) - ns/op 21657 ns/op 14378 ns/op 1.51
Benchmark_Rewrite/Next_always_true (github.com/gofiber/fiber/v3/middleware/rewrite) 556.1 ns/op 24 B/op 2 allocs/op 360.7 ns/op 24 B/op 2 allocs/op 1.54
Benchmark_Rewrite/Next_always_true (github.com/gofiber/fiber/v3/middleware/rewrite) - ns/op 556.1 ns/op 360.7 ns/op 1.54
Benchmark_Bind_URI (github.com/gofiber/fiber/v3) 3593 ns/op 768 B/op 32 allocs/op 2349 ns/op 768 B/op 32 allocs/op 1.53
Benchmark_Bind_URI (github.com/gofiber/fiber/v3) - ns/op 3593 ns/op 2349 ns/op 1.53
Benchmark_Ctx_Cookie (github.com/gofiber/fiber/v3) 205.9 ns/op 0 B/op 0 allocs/op 135.7 ns/op 0 B/op 0 allocs/op 1.52
Benchmark_Ctx_Cookie (github.com/gofiber/fiber/v3) - ns/op 205.9 ns/op 135.7 ns/op 1.52
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckSimple (github.com/gofiber/fiber/v3) 68.96 ns/op 8 B/op 1 allocs/op 44.71 ns/op 8 B/op 1 allocs/op 1.54
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckSimple (github.com/gofiber/fiber/v3) - ns/op 68.96 ns/op 44.71 ns/op 1.54
Benchmark_Ctx_IsProxyTrusted/WithProxyCheck (github.com/gofiber/fiber/v3) 76.16 ns/op 8 B/op 1 allocs/op 50.32 ns/op 8 B/op 1 allocs/op 1.51
Benchmark_Ctx_IsProxyTrusted/WithProxyCheck (github.com/gofiber/fiber/v3) - ns/op 76.16 ns/op 50.32 ns/op 1.51
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowPrivate (github.com/gofiber/fiber/v3) 81.49 ns/op 8 B/op 1 allocs/op 52.33 ns/op 8 B/op 1 allocs/op 1.56
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowPrivate (github.com/gofiber/fiber/v3) - ns/op 81.49 ns/op 52.33 ns/op 1.56
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowAll (github.com/gofiber/fiber/v3) 103.8 ns/op 8 B/op 1 allocs/op 66.56 ns/op 8 B/op 1 allocs/op 1.56
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowAll (github.com/gofiber/fiber/v3) - ns/op 103.8 ns/op 66.56 ns/op 1.56
Benchmark_Router_Handler_Unescape (github.com/gofiber/fiber/v3) 181.6 ns/op 0 B/op 0 allocs/op 116 ns/op 0 B/op 0 allocs/op 1.57
Benchmark_Router_Handler_Unescape (github.com/gofiber/fiber/v3) - ns/op 181.6 ns/op 116 ns/op 1.57
Benchmark_Compress_Levels/Brotli_LevelBestCompression (github.com/gofiber/fiber/v3/middleware/compress) - B/op 17 B/op 11 B/op 1.55
Benchmark_Communication_Flow (github.com/gofiber/fiber/v3) 81.98 ns/op 0 B/op 0 allocs/op 49.1 ns/op 0 B/op 0 allocs/op 1.67
Benchmark_Communication_Flow (github.com/gofiber/fiber/v3) - ns/op 81.98 ns/op 49.1 ns/op 1.67

This comment was automatically generated by workflow using github-action-benchmark.

@ReneWerner87 ReneWerner87 force-pushed the ci/speed-up-test-and-benchmark branch from 93c1fcb to 706ae86 Compare May 22, 2026 10:08
@ReneWerner87 ReneWerner87 changed the title ci: speed up test and benchmark workflows without changing coverage ci: speed up test workflow without changing coverage May 22, 2026
@ReneWerner87 ReneWerner87 force-pushed the ci/speed-up-test-and-benchmark branch from 706ae86 to 5fad032 Compare May 22, 2026 10:33
@ReneWerner87 ReneWerner87 changed the title ci: speed up test workflow without changing coverage ci: parallelize test and benchmark workflows via sharding May 22, 2026
Reduce CI wall-clock per commit without changing results, output or coverage.

test.yml:
- add a concurrency group that cancels superseded in-progress PR runs while
  never cancelling pushes to main
- compute coverage only on the ubuntu/1.25.x job (the only one that uploads it);
  the other 5 matrix combinations skip the coverage instrumentation
- shard the repeated job into 3 parallel matrix shards of -count=5 each (15
  shuffled runs total, with -race). Measured: ~9m41s -> ~3m49s. The job is
  dominated by fixed test latency (sleeps/timeouts), not CPU, so trimming -race
  alone did not help; splitting the repetitions does.

benchmark.yml:
- add the same PR-only concurrency cancellation

The benchmark job stays a single job on purpose. It cannot be sharded on
GitHub-hosted runners: the regression comparison relies on a cpu-model cache key
so it only compares same-CPU results. Spreading benchmarks across multiple
runners with heterogeneous CPUs breaks that guarantee and produces false alerts
(a no-code-change PR showed a uniform ~1.5-2.5x ns/op "regression" with
identical B/op and allocs/op, i.e. pure CPU variance).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ReneWerner87 ReneWerner87 force-pushed the ci/speed-up-test-and-benchmark branch from 5fad032 to a55fb1f Compare May 22, 2026 10:40
@ReneWerner87 ReneWerner87 changed the title ci: parallelize test and benchmark workflows via sharding ci: parallelize the repeated test job; add concurrency cancellation May 22, 2026
@github-project-automation github-project-automation Bot moved this to Done in v3 Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants