Benchmarks

Solver and index performance is tracked with pytest-benchmark and published to Bencher.

Run locally

Run the existing solver and index benchmarks with the same Python environment as CI:

pixi run --locked -e test-py313 benchmark

Export the measurements as pytest-benchmark JSON:

pixi run --locked -e test-py313 benchmark --benchmark-json benchmark_results.json

The JSON file is ignored by Git. Local runs do not upload results to Bencher. To select a smaller group, append a pytest filter such as -k query_search. Compare local timings on the same machine with the same environment and cache state.

Solver overhead workloads

The solver overhead benchmarks use synthetic local channels. Run them with:

pixi run --locked -e test-py313 benchmark -k 'pinned_request or empty_package_cache'

Use -k pinned_request or -k empty_package_cache to select one workload:

• The pinned-request benchmark has 10 package names with 200 candidate builds each. Pins exclude the newest version. It times solve_final_state(), including metadata loading and pin validation. Channel and prefix creation, plus solver construction, happen before timing starts.

• The empty-cache benchmark requests 100 or 563 packages. It times index construction and a Rattler solve, including package-cache source callbacks. Channel creation and request-spec construction happen before timing starts.

Both workloads clear conda’s in-memory metadata and package-cache objects before each round, outside the timed operation. Filesystem caches are left intact.

Add benchmarks

Add tests marked with @pytest.mark.benchmark and time the operation with the benchmark fixture. See the index benchmarks and solver benchmarks for examples.

Keep fixture setup outside the timed callable unless it is part of the operation you intend to measure. Use fixed test data and stable test names and parameter IDs so Bencher can track the same workload across commits. For cache-sensitive benchmarks, define whether each case measures a cold or warm cache.

The existing tests cover index searches, installed-package solver handling, and end-to-end solve comparisons between rattler and libmamba.

Results and regression alerts

The Benchmarks workflow runs on pushes to main and pull requests targeting main, and can also be run manually. It uses Ubuntu 22.04, Python 3.13, and the Pixi lockfile. Its benchmark-results artifact contains the JSON measurements and is retained for seven days.

The Track Benchmarks workflow uploads successful runs to the Bencher project. Select main for the baseline or pr-<number> for a pull request. Manual runs are uploaded only when run from main.

PR reports compare against the base branch’s results. The latency threshold uses a t-test with an upper threshold of 0.99 and at most 64 historical samples. A first baseline run records measurements before comparisons can begin. Detected regressions fail the reporting workflow and generate PR feedback. Inspect the affected benchmark and its history before changing a threshold.

Input-state reuse

tests/test_state_benchmarks.py measures spec collection for 100, 563, and 1,000 installed packages. The first cases include input-state construction, output-state construction, and spec collection. The retry cases populate the input-derived values before timing, then rebuild the output state and collect specs again. Record creation stays outside both timed operations.

The same module includes a complete 563-package update fallback. A pinned app requires an older dependency build, forcing three Rattler attempts across the normal and fallback loops. This case checks the resulting transaction as well as the attempt count. Local channel and prefix creation are outside timing.

Run this group with -k state_ or pass tests/test_state_benchmarks.py to the benchmark task. Compare first collection, repeated collection, and complete solves separately. A faster repeated collection does not imply the same percentage improvement for the full solve. Use separate processes for Python allocation measurements, since tracing affects timing, and distinguish retained input-state data from allocation peaks and native Rattler memory.