Tint Fuzzing Experiments

The Tint fuzzing experiment/benchmarking framework is designed to automate the process of comparing changes to the Tint fuzzers. It measures hardware performance, fuzzer coverage percentage rates, and test case execution rates to come up with abstract performance scores to compare versions of the fuzzers.

This framework is part of the general fuzz utility for Tint found in ./tools/src/cmd/fuzz.

This document explains the high-level architecture of the experiments framework, the environment prerequisites, and a step-by-step walkthrough of how to define, execute, and process an experiment. Along with a reference section at the end for the experiment configuration JSON format,

Prerequisites

Running fuzzing experiments requires a local Dawn repository and a GN build directory configured for fuzzing and coverage.

To support correct instrumentation, your GN build (args.gn) must include at least the following flags:

use_libfuzzer = true
tint_build_wgsl_reader = true
use_clang_coverage = true
optimize_for_fuzzing = false  # Cannot be used with coverage enabled

Additional flags must be enabled depending on the fuzzers selected:

IR Fuzzers (tint_ir_fuzzer, tint_ir_mesa_fuzzer ): Require tint_build_ir_binary = true and tint_has_protobuf = true.
Mesa Fuzzers (tint_wgsl_mesa_fuzzer, tint_ir_mesa_fuzzer): Require "checkout_mesa": True in .gclient, as well as tint_build_mesa = true and tint_build_fuzzer_vulkan_support = true in args.gn (only supported on Linux x64 for Vulkan; see Building Mesa fuzzers for full setup instructions and limitations).

The tool will produce an error if the flags are not correctly configured. The tool does not support using a pre-built version of the Tint fuzzers, nor the CMake builds.

Note: For the most applicable results it is recommended that all backends are turned on (tint_build_hlsl_writer = true, tint_build_msl_writer = true, tint_build_spv_writer = true)

Note: Though the fuzzers can be built and run with sanitizers turned on (is_asan = true, etc), this is not recommended, because the sanitizers tend to dominate the runtime of the execution, so produce substantially lower quality data

Defining an Experiment

An experiment is represented by a dedicated directory containing:

An experiment.json configuration file.
A corpora/ directory holding the seed corpora for the various fuzzing modes.

Directory Structure

my_experiment/
├── experiment.json
└── corpora/
    ├── wgsl_seed/
    │   ├── shader1.wgsl
    │   └── shader2.wgsl
    └── empty/

Writing `experiment.json`

The experiment.json defines variables like the commit hash to check out, specific fuzzers to test, and duration of the experiments.

Here is an example setup for comparing fuzzer performance using both a standard corpus and an empty directory as starting points:

{
    "name": "example",
    "hash": "4c2395a860ab76b44f7256c25281dfd3a3680192",
    "fuzzers": ["tint_wgsl_fuzzer", "tint_ir_fuzzer"],
    "burnin_enabled": false,
    "normalization_duration": 10,
    "normalization_iterations": 2,
    "wgsl_normalization_corpus": "wgsl_seed",
    "ir_normalization_corpus": "empty",
    "wgsl_corpora": [
        { "name": "wgsl_full", "path": "wgsl_seed" },
        { "name": "wgsl_empty", "path": "empty" }
    ],
    "ir_corpora": [
        { "name": "ir_empty", "path": "empty" }
    ],
    "default_iterations": 5,
    "durations": [
        {"runs": 10000},
        {"runs": 50000},
        {"runs": 100000}
    ]
}

name is just a user visible title for this experiment which will appear in reports/logging, but has no semantic meaning
hash is the specific version of the Dawn repo that will be checked out to build the fuzzers. (The tool will be built and run from what ever hash the repo is at when you call it, not this version)
fuzzers are the specific fuzzer binaries to test
burnin_enabled, normalization_duration, and normalization_iterations are turned down or off in this example so that the setup phases run fast, but these should be removed or increased when generating statistically valid data
*_normalization_corpus are the corpora to use when establishing baselines for normalizing performance numbers between machines/environments
*_corpora are the various starting corpora to experiment using
default_iterations specifies how many times to run each experiment if not explicitly overridden
durations specifies the sets experiments to run, broadly you will want to specify a spread of values here to get a reasonable graph performance over time, since the framework doesn't sample these values during the experiment run

Full details on the various options are detailed below.

Note: The number of actual runs performed will be # of fuzzers X # of corpora X # of durations X # of iterations

Testing Mesa Fuzzers

To include "tint_wgsl_mesa_fuzzer" or "tint_ir_mesa_fuzzer" in an experiment, follow the Mesa fuzzer build instructions first. Keep the following in mind when designing and running Mesa experiments:

Platform & .gclient setup: Mesa fuzzers are only supported on Linux x64 and only test Mesa's software Vulkan driver (Lavapipe). Ensure "checkout_mesa": True is already configured in .gclient before starting tools/run fuzz experiment, because the tool checks out hash and runs gclient sync automatically during binary preparation.
Pass filtering & corpora: tint_wgsl_mesa_fuzzer uses wgsl_normalization_corpus and wgsl_corpora, while tint_ir_mesa_fuzzer uses ir_normalization_corpus and ir_corpora. Unlike the general Tint fuzzers, both Mesa targets force --filter=tint::spirv::writer::IRFuzzer and validate generated SPIR-V modules through Mesa's Vulkan ICD (libvulkan_lvp.so via lvp_icd.json), so tint_build_spv_writer = true must be enabled.
Execution rate & durations: Compiling SPIR-V through Mesa's Vulkan/LLVM pipeline makes per-input execution significantly slower than Tint-only fuzzing. runs-based durations (such as {"runs": 100000}) will take substantially longer in wall-clock time than on tint_wgsl_fuzzer or tint_ir_fuzzer; consider smaller runs targets or seconds-based durations.
Mixing Mesa and non-Mesa fuzzers: All fuzzers in an experiment are built from the same args.gn and staged together in <experiment>/bin/. When tint_build_fuzzer_vulkan_support = true is enabled and a Mesa fuzzer copies lvp_icd.json into bin/, the experiment runner automatically passes --vk_icd to every fuzzer in that experiment. As a result, tint_wgsl_fuzzer and tint_ir_fuzzer in the same experiment.json will also invoke Mesa Vulkan validation during their SPIR-V writer pass. To benchmark tint_wgsl_fuzzer or tint_ir_fuzzer without Mesa overhead, run them in a separate experiment configured with tint_build_mesa = false and tint_build_fuzzer_vulkan_support = false.

Running an Experiment

To start the experiment on your system, execute the fuzz helper with the experiment subcommand:

tools/run fuzz experiment -build out/fuzzers my_experiment/

where out/fuzzers is a GN configured build with the correct values set in args.gn

Note: experiment does support the -j flag for running concurrent operations, but defaults it to 1. This is because saturating the RAM or CPU in the execution environment (e.g. -j max num of cores) can significantly impact the results. It is recommended to experiment with a mini-version of an experiment.json to determine an appropriate value to use, (try num of cores / 2 or / 4 as a starting point)

Execution Pipeline

The framework guides execution through the following phases:

Validation & Sync: The tool checks the active GN arguments in your build directory to ensure they match the requirements. It then checks out the requested commit hash (hash field) and runs gclient sync automatically. (The state of repo should be returned to the original state)
Binary Preparation: It builds the selected fuzzer binaries and copies them, alongside their dynamic library dependencies and required LLVM tools (llvm-profdata, llvm-cov), into my_experiment/bin/.
Burn-in (Optional): If enabled, it executes multiple parallel workloads for 5 minutes (burnin_duration & burnin_enabled) to bring the physical host's CPU to a thermal steady state. This prevents throttling and normalization profiling skews during the experiment.
Normalization: It executes multiple normalization profiling runs for each fuzzer against its designated normalization corpus (controlled by normalization_duration and normalization_iterations). Each iteration first runs the fuzzer with -runs=0 to measure the corpus initialization cost (runs and elapsed seconds), then runs the timed benchmark for the discovered initialization duration plus normalization_duration, subtracting the initialization baseline to isolate the steady-state execution rate (runs/sec). The resulting average execution rate and standard error (SEM) are saved to normalization_scores.json and are used to compute “Normalized CPU Seconds” for each fuzzer. All individual profiling runs are archived in normalization_iterations.csv.
Task Execution: It calculates the Cartesian product of experiments (Fuzzer × Corpus × Duration × Iteration). These are run concurrently across available CPU cores (configurable via -j). Results are saved directly under my_experiment/results/. Each task directory stores a state.json file, captured logs, mutated corpus files, and .profraw coverage profiles.

Processing and Analyzing Results

Once execution has completed, you can aggregate and process results using the analyze subcommand:

tools/run fuzz analyze my_experiment/

Analysis Pipeline

Coverage Generation: The analyzer locates the .profraw file for each completed iteration, merges them using llvm-profdata, and executes coverage.py to produce standard .lcov coverage logs.
Component Mapping: It parses the .lcov profiles and aggregates line coverage metrics separately for Tint (src/tint/ and src/utils/), Mesa (third_party/mesa/), and DXC (third_party/directx-headers/).
Statistics Computation: It computes arithmetic means and standard errors for:
- Coverage percentage
- Normalized CPU seconds
- Coverage rates (% of lines covered per CPU second, using quadrature error propagation)
Report Export:
- raw_iteration_data.csv: A flat table of raw execution counts, actual runtimes, and coverage hits for every single iteration.
- calculated_statistics.csv: A flat table of calculated statistics for every experiment category (Fuzzer × Corpus).
- experiment_report.md: A human-readable Markdown summary report.

Appendix: `experiment.json` Reference

The following sections define the full configuration schema for experiment.json.

Root Attributes

Field	Type	Description
`name`	`string`	A human-readable identifier for the experiment, used in the logging/reports.
`hash`	`string`	The Git commit hash checkout from which the binaries should be compiled.
`fuzzers`	`array of strings`	Fuzzer target names to test(Supported values: `"tint_wgsl_fuzzer"`, `"tint_ir_fuzzer"`, `"tint_wgsl_mesa_fuzzer"`, `"tint_ir_mesa_fuzzer"`).
`timeout`	`integer` optional	Timeout limit in seconds for a single fuzzer execution on a test case in libFuzzer (defaults to libFuzzer's default of 1200).
`burnin_enabled`	`boolean` optional	If true, launches parallel workloads to warm up the machine before benching (defaults to true).
`burnin_duration`	`integer` optional	Target duration in seconds for the initial thermal burn-in (defaults to 300).
`normalization_duration`	`integer` optional	Execution duration in seconds for each iteration of the normalization microbenchmarking phase (defaults to 60).
`normalization_iterations`	`integer` optional	Number of iterations to perform for normalization microbenchmarking (defaults to 5).
`wgsl_normalization_corpus`	`string`	Directory path relative to the root `corpora/` to use when performing normalization on WGSL fuzzers.
`ir_normalization_corpus`	`string`	Directory path relative to the root `corpora/` to use when performing normalization on IR fuzzers.
`wgsl_corpora`	`array of CorpusDef`	Corpora definitions available to run with WGSL fuzzers.
`ir_corpora`	`array of CorpusDef`	Corpora definitions available to run with IR fuzzers.
`default_iterations`	`integer`	The default number of times to repeat every Fuzzer/Corpus/Duration combination if not otherwise specified.
`durations`	`array of DurationDef`	The target run lengths defining the experiment stopping criteria.

`CorpusDef` Format

Field	Type	Description
`name`	`string`	A human-readable identifier for the corpus, used in the logging/reports.
`path`	`string`	Directory path relative to the root `corpora/` directory.

`DurationDef` Format

Note: Must specify exactly one and only one of seconds or runs.

Note: Because of how libFuzzer operate seconds and runs are approximate values. Fuzzing should run for at least this limit, but will normally be a little over.

Field	Type	Description
`seconds`	`integer` (optional)	Number of seconds to run for (maps to `-max_total_time`).
`runs`	`integer` (optional)	Number of inputs/test cases to run for(maps to `-runs`).
`iterations`	`integer` (optional)	Override the experiment `default_iterations` specifically for this duration target. Useful for manually load balancing very long and very short durations