Testing Dawn

(TODO)

Dawn Perf Tests

For benchmarking with dawn_perf_tests, it's best to build inside a Chromium checkout using the following GN args:

is_official_build = true  # Enables highest optimization level, using LTO on some platforms
use_dawn = true           # Required to build Dawn
use_cfi_icall=false       # Required because Dawn dynamically loads function pointers, and we don't sanitize them yet.

A Chromium checkout is required for the highest optimization flags. It is possible to build and run dawn_perf_tests from a standalone Dawn checkout as well, only using GN arg is_debug=false. For more information on building, please see building.md.

Terminology

  • Iteration: The unit of work being measured. It could be a frame, a draw call, a data upload, a computation, etc. dawn_perf_tests metrics are reported as time per iteration.
  • Step: A group of Iterations run together. The number of iterationsPerStep is provided to the constructor of DawnPerfTestBase.
  • Trial: A group of Steps run consecutively. kNumTrials are run for each test. A Step in a Trial is run repetitively for approximately kCalibrationRunTimeSeconds. Metrics are accumlated per-trial and reported as the total time divided by numSteps * iterationsPerStep. maxStepsInFlight is passed to the DawnPerfTestsBase constructor to limit the number of Steps pipelined.

(See //src/dawn/tests/perf_tests/DawnPerfTest.h for the values of the constants).

Metrics

dawn_perf_tests measures the following metrics:

  • wall_time: The time per iteration, including time waiting for the GPU between Steps in a Trial.
  • cpu_time: The time per iteration, not including time waiting for the GPU between Steps in a Trial.
  • validation_time: The time for CommandBuffer / RenderBundle validation.
  • recording_time: The time to convert Dawn commands to native commands.

Metrics are reported according to the format specified at [chromium]//build/recipes/performance_log_processor.py

Dumping Trace Files

The test harness supports a --trace-file=path/to/trace.json argument where Dawn trace events can be dumped. The traces can be viewed in Chrome's about://tracing viewer.

Test Runner

//scripts/perf_test_runner.py may be run to continuously run a test and report mean times and variances.

Currently the script looks in the out/Release build directory and measures the wall_time metric (hardcoded into the script). These should eventually become arguments.

Example usage:

scripts/perf_test_runner.py DrawCallPerf.Run/Vulkan__e_skip_validation

Tests

BufferUploadPerf

Tests repetitively uploading data to the GPU using either WriteBuffer or CreateBuffer with mappedAtCreation = true.

DrawCallPerf

DrawCallPerf tests drawing a simple triangle with many ways of encoding commands, binding, and uploading data to the GPU. The rationale for this is the following:

  • Static/Multiple/Dynamic vertex buffers: Tests switching buffer bindings. This has a state tracking cost as well as a GPU driver cost.
  • Static/Multiple/Dynamic bind groups: Same rationale as vertex buffers
  • Static/Dynamic pipelines: In addition to a change to GPU state, changing the pipeline layout incurs additional state tracking costs in Dawn.
  • With/Without render bundles: All of the above can have lower validation costs if precomputed in a render bundle.
  • Static/Dynamic data: Updating data for each draw is a common use case. It also tests the efficiency of resource transitions.

Testing with the CTS

There are various ways to test dawn against the WebGPU CTS.

With dawn.node

See src/dawn/node/README.md.

With a Chromium build manually

Go to https://gpuweb.github.io/cts/standalone/ or clone the CTS repo and serve it locally

git clone https://github.com/gpuweb/cts.git
cd cts
npm ci
npm start

Then open http://localhost:8080/standalone.

On the CQ

You can run a specific version of the CTS with top of tree Dawn in Chromium. First, upload the version you want to test to your own github fork of the CTS. Note the commit-hash. Then run:

./tools/run cts roll --max-attempts 0 --repo <repo-url> --revision <commit-hash>

Example:

./tools/run cts roll --max-attempts 0 --repo https://github.com/greggman/cts.git --revision 1b3173b36a1084917e0ff29e5f40c6290b6f8adc

The roll script take a couple of minutes as it downloads that version of the CTS and creates a roll patch. Once the roll script says it's waiting on the bots to finish you can kill (Ctrl-C) the script. The CQ will continue to run the tests.

Filtering - running less than the full CTS

You can also specify a CTS query with --test-query <query>, for example, test all the WGSL builtins ... --test-query "webgpu:shader,execution,expression,call,builtin,*". You can also filter tests with a glob --test-filter <glob>, for example, only run tests with BindGroup in the test name ... --test-filter "*BindGroup*".

See tools/src/cmd/cts/roll/roll.go for other options.

Testing with a different version of Dawn

The roll is just a normal CL on gerrit so, after creating a roll CL, follow the instructions on gerrit to download that CL. Apply any dawn changes. Then re-upload with git cl upload --force --bypass-hooks and start a dry run on gerrit.