Run checks in CI

CI repeats an experiment you can already run locally. Decide which result should fail the job, and keep the reports.

Measure existing test items

julia --startup-file=no --project=test -e 'using PerfChecker; exit(perfchecker_main(ARGS))' -- testitems --root=. --list
julia --startup-file=no --project=test -e 'using PerfChecker; exit(perfchecker_main(ARGS))' -- testitems --root=. --tags=small --samples=1 --threads=1 --reports=results/items

• The second command exits nonzero if an item does not validate.

• performance="not_compared" means no regression budget was applied.

• Do not gate on a single first-call item duration.

For functional jobs that must exclude :perf_only, load PerfChecker and TestItemRunner, then:

@run_package_tests filter=testitem_filter(:test)

GitHub Actions

name: Performance items
on: [push, pull_request]
permissions:
  contents: read
jobs:
  items:
    runs-on: ubuntu-latest
    env:
      JULIA_NUM_THREADS: '1'
      JULIA_NUM_GC_THREADS: '1'
      OPENBLAS_NUM_THREADS: '1'
    steps:
      - uses: actions/checkout@v4
      - uses: julia-actions/setup-julia@v2
        with:
          version: '1'
      - name: Prepare test environment
        run: julia --startup-file=no --project=test -e 'using Pkg; Pkg.develop(path=pwd()); Pkg.instantiate()'
      - name: Measure selected items
        run: julia --startup-file=no --project=test -e 'using PerfChecker; exit(perfchecker_main(ARGS))' -- testitems --root=. --tags=small --samples=1 --threads=1 --reports=results/items
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: performance-items
          path: results/items

Regression gates

Run a suite, then compare two saved bundles:

julia --startup-file=no --project=. -e 'using PerfChecker; exit(perfchecker_main(ARGS))' -- run --suite=perf/suite.jl --profile=ci --reports=results/ci --progress=jsonl

julia --startup-file=no --project=. -e 'using PerfChecker; exit(perfchecker_main(ARGS))' -- check --baseline=results/baseline --candidate=results/candidate --limit=julia.wall.time=0.05 --limit=julia.alloc.bytes=0.02 --min-samples=10 --reports=results/comparison

• Limits are relative fractions: 0.05 allows a 5% increase for a lower-is-better metric.

• They are illustrative, not universal thresholds. Inspect distributions before choosing one.

• check fails when a limit fails; compare does not.

Keep the right artifacts

• Native items: testitems.json.

• Suite runs: the whole report directory, including bundles/.

• Keep runners, thread settings and fixtures comparable between baseline and candidate.