| commit | d87571ae5597ab5d3aad148918399a7a2275c9cd | [log] [tgz] |
|---|---|---|
| author | Leszek Swirski <leszeks@google.com> | Mon Mar 04 12:27:22 2024 +0000 |
| committer | V8 LUCI CQ <v8-scoped@luci-project-accounts.iam.gserviceaccount.com> | Mon Mar 04 12:27:22 2024 +0000 |
| tree | 2641238463fa33a6827efa7ab14d3d32155c4a67 | |
| parent | bd75f61ede5a0ec9fe601985af75ae6e8b293ae0 [diff] |
Allow passing lists as comma-delimited strings
Make configs optionally a bit less verbose (esp. if passed on the
command line) by allowing lists to be passed as comma delimited strings.
In particular, this means that something like
--probe='tracing:{categories:["foo","bar"]}'
can be shortened to
--probe='tracing:{categories:"foo,bar"}'
and is a bit more amenable to copy and paste from lists of categories.
Change-Id: Id3d567c1ed21ade3442409c36e92bf2f6aacb45c
Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/5331662
Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Leszek Swirski <leszeks@chromium.org>
Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.
Mailing list: crossbench@chromium.org
Issues/Bugs: Tests > CrossBench
Supported Browsers: Chrome/Chromium, Firefox, Safari and Edge.
Supported OS: macOS, Android, linux and windows.
Use the ./cb.py script directly to run benchmarks (requires chrome's vpython3)
pip install crossbench,Run the latest speedometer benchmark 20 times with the system default browser (chrome-stable):
# Run chrome-stable by default: ./cb.py speedometer --repeat=20 # Compare chrome browser versions and a local chrome build on jetstream: ./cb.py jetstream --browser=chrome-stable --browser=chrome-m90 --browser=$PATH
Profile individual line items (with pprof on linux):
./cb.py speedometer --probe='profiling' --separate
Use a custom chrome build and only run a subset of the stories:
./cb.py speedometer --browser=$PATH --probe='profiling' --story='jQuery.*'
Profile a website for 17 seconds on Chrome M100 (auto-downloading on macOS and linux):
./cb.py loading --browser=chrome-m100 --probe='profiling' --url=www.cnn.com,17s
Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.
You can specify a browser with --browser=<name>. You can repeat the --browser argument to run multiple browser. If you need custom flags for multiple browsers use --browser-config (or pass simple flags after -- to the browser).
./cb.py speedometer --browser=$BROWSER -- --enable-field-trial-config
For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.
./cb.py speedometer --browser-config=config.hjson
The example file lists and explains all configuration details.
Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.
Multiple probes can be added with repeated --probe='XXX' options. You can use the describe probes subcommand to list all probes:
# List all probes: ./cb.py describe probes # List help for an individual probe: ./cb.py describe probe v8.log
Some probes can be configured, either with inline json when using --probe or in a separate --probe-config hjson file. Use the describe command to list all options.
# Get probe config details: ./cb.py describe probe v8.log # Use inline hjson to configure a probe: ./cb.py speedometer --probe='v8.log:{prof:true}'
For complex probe setups you can use --probe-config=<file>. The example file lists and explains all configuration details. For the specific probe configuration properties consult the describe command.
Use the describe command to list all benchmark details:
# List all benchmark info: ./cb.py describe benchmarks # List an individual benchmark info: ./cb.py describe benchmark speedometer_3.0 # List a benchmark's command line options: ./cb.py speedometer_3.0 --help
Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.
Use --help or describe to list all stories for a benchmark:
./cb.py speedometer --help
Use --stories to list individual story names, or use regular expression as filter.
./cb.py speedometer --browser=$BROWSER --stories='.*Angular.*'
This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.
# a) On debian: sudo apt-get install python3.10 python3-poetry # b) With python 3.8 to 3.10 installed already: pip3 install poetry
Check that you have poetry on your path and make sure you have the right $PATH settings.
poetry --help || echo "Please update your \$PATH to include poetry bin location"; # Depending on your setup, add one of the following to your $PATH: echo "`python3 -m site --user-base`/bin"; python3 -c "import sysconfig; print(sysconfig.get_path('scripts'))";
Install the necessary dependencies from the lock file using poetry:
# Select the python version you want to use (3.8 to 3.10): poetry env use 3.10 poetry install # For python 3.11 you have to skip pytype support: poetry env use 3.11 poetry install --without=dev-pytype
For local development / non-chromium installation you should use poetry run cb ... instead of ./cb.py ....
Side-note, beware that poetry eats up an empty --:
# With cb.py: ./cb.py speedometer ... -- --custom-chrome-flag ... # With poetry: poetry run cb speedometer ... -- -- --custom-chrome-flag ...
poetry run pytest
Run detailed test coverage:
poetry run pytest --cov=crossbench --cov-report=html
Run pytype type checker:
poetry run pytype -j auto .