Tint Fuzzing Experiments

The Tint fuzzing experiment/benchmarking framework is designed to automate the process of comparing changes to the Tint fuzzers. It measures hardware performance, fuzzer coverage percentage rates, and test case execution rates to come up with abstract performance scores to compare versions of the fuzzers.

This framework is part of the general fuzz utility for Tint found in ./tools/src/cmd/fuzz.

This document explains the high-level architecture of the experiments framework, the environment prerequisites, and a step-by-step walkthrough of how to define, execute, and process an experiment. Along with a reference section at the end for the experiment configuration JSON format,

Prerequisites

Running fuzzing experiments requires a local Dawn repository and a GN build directory configured for fuzzing and coverage.

To support correct instrumentation, your GN build (args.gn) must include at least the following flags:

use_libfuzzer = true
tint_build_wgsl_reader = true
use_clang_coverage = true
optimize_for_fuzzing = false  # Cannot be used with coverage enabled

Additional flags must be enabled depending on the fuzzers selected:

IR Fuzzers (tint_ir_fuzzer, tint_ir_mesa_fuzzer ): Require tint_build_ir_binary = true and tint_has_protobuf = true.
Mesa Fuzzers (tint_wgsl_mesa_fuzzer, tint_ir_mesa_fuzzer): Require tint_build_mesa = true and tint_build_fuzzer_vulkan_support = true.

The tool will produce an error if the flags are not correctly configured. The tool does not support using a pre-built version of the Tint fuzzers, nor the CMake builds.

Note: For the most applicable results it is recommended that all backends are turned on (tint_build_hlsl_writer = true, tint_build_msl_writer = true, tint_build_spv_writer = true)

Note: Though the fuzzers can be built and run with sanitizers turned on (is_asan = true, etc), this is not recommended, because the sanitizers tend to dominate the runtime of the execution, so produce substantially lower quality data

Defining an Experiment

An experiment is represented by a dedicated directory containing:

An experiment.json configuration file.
A corpora/ directory holding the seed corpora for the various fuzzing modes.

Directory Structure

my_experiment/
├── experiment.json
└── corpora/
    ├── wgsl_seed/
    │   ├── shader1.wgsl
    │   └── shader2.wgsl
    └── empty/

Writing `experiment.json`

The experiment.json defines variables like the commit hash to check out, specific fuzzers to test, and duration of the experiments.

Here is an example setup for comparing fuzzer performance using both a standard corpus and an empty directory as starting points:

{
    "name": "example",
    "hash": "4c2395a860ab76b44f7256c25281dfd3a3680192",
    "fuzzers": ["tint_wgsl_fuzzer", "tint_ir_fuzzer"],
    "burnin_enabled": false,
    "normalization_duration": 10,
    "normalization_iterations": 2,
    "wgsl_normalization_corpus": "wgsl_seed",
    "ir_normalization_corpus": "empty",
    "wgsl_corpora": [
        { "name": "wgsl_full", "path": "wgsl_seed" },
        { "name": "wgsl_empty", "path": "empty" }
    ],
    "ir_corpora": [
        { "name": "ir_empty", "path": "empty" }
    ],
    "default_iterations": 5,
    "durations": [
        {"runs": 10000},
        {"runs": 50000},
        {"runs": 100000}
    ]
}

name is just a user visible title for this experiment which will appear in reports/logging, but has no semantic meaning
hash is the specific version of the Dawn repo that will be checked out to build the fuzzers. (The tool will be built and run from what ever hash the repo is at when you call it, not this version)
fuzzers are the specific fuzzer binaries to test
burnin_enabled, normalization_duration, and normalization_iterations are turned down or off in this example so that the setup phases run fast, but these should be removed or increased when generating statistically valid data
*_normalization_corpus are the corpora to use when establishing baselines for normalizing performance numbers between machines/environments
*_corpora are the various starting corpora to experiment using
default_iterations specifies how many times to run each experiment if not explicitly overridden
durations specifies the sets experiments to run, broadly you will want to specify a spread of values here to get a reasonable graph performance over time, since the framework doesn't sample these values during the experiment run

Full details on the various options are detailed below.

Note: The number of actual runs performed will be # of fuzzers X # of corpora X # of durations X # of iterations

Running an Experiment

To start the experiment on your system, execute the fuzz helper with the -experiment flag:

/tools/run fuzz -experiment -build out/fuzzers my_experiment/

where out/fuzzers is a GN configured build with the correct values set in args.gn

Note: -experiment does support the -j flag for running concurrent operations, but defaults it to 1. This is because saturating the RAM or CPU in the execution environment (e.g. -j max num of cores) can significantly impact the results. It is recommended to experiment with a mini-version of an experiment.json to determine an appropriate value to use, (try num of cores / 2 or / 4 as a starting point)

Execution Pipeline

The framework guides execution through the following phases:

Validation & Sync: The tool checks the active GN arguments in your build directory to ensure they match the requirements. It then checks out the requested commit hash (hash field) and runs gclient sync automatically. (The state of repo should be returned to the original state)
Binary Preparation: It builds the selected fuzzer binaries and copies them, alongside their dynamic library dependencies and required LLVM tools (llvm-profdata, llvm-cov), into my_experiment/bin/.
Burn-in (Optional): If enabled, it executes multiple parallel workloads for 5 minutes (burnin_duration & burnin_enabled) to bring the physical host's CPU to a thermal steady state. This prevents throttling and normalization profiling skews during the experiment.
Normalization: It executes multiple normalization profiling runs for each fuzzer against its designated normalization corpus (controlled by normalization_duration and normalization_iterations). The resulting average execution rate (runs/sec) and standard error (SEM) are saved to normalization_scores.json and are used to compute “Normalized CPU Seconds” for each fuzzer. All individual profiling runs are archived in normalization_iterations.csv.
Task Execution: It calculates the Cartesian product of experiments (Fuzzer × Corpus × Duration × Iteration). These are run concurrently across available CPU cores (configurable via -j). Results are saved directly under my_experiment/results/. Each task directory stores a state.json file, captured logs, mutated corpus files, and .profraw coverage profiles.

Processing and Analyzing Results

Once execution has completed, you can aggregate and process results using the -analyze flag:

tools/run fuzz -analyze my_experiment/

Analysis Pipeline

Coverage Generation: The analyzer locates the .profraw file for each completed iteration, merges them using llvm-profdata, and executes coverage.py to produce standard .lcov coverage logs.
Component Mapping: It parses the .lcov profiles and aggregates line coverage metrics separately for Tint (src/tint/ and src/utils/), Mesa (third_party/mesa/), and DXC (third_party/directx-headers/).
Statistics Computation: It computes arithmetic means and standard errors for:
- Coverage percentage
- Normalized CPU seconds
- Coverage rates (% of lines covered per CPU second, using quadrature error propagation)
Report Export:
- raw_iteration_data.csv: A flat table of raw execution counts, actual runtimes, and coverage hits for every single iteration.
- calculated_statistics.csv: A flat table of calculated statistics for every experiment category (Fuzzer × Corpus).
- experiment_report.md: A human-readable Markdown summary report.

Appendix: `experiment.json` Reference

The following sections define the full configuration schema for experiment.json.

Root Attributes

Field	Type	Description
`name`	`string`	A human-readable identifier for the experiment, used in the logging/reports.
`hash`	`string`	The Git commit hash checkout from which the binaries should be compiled.
`fuzzers`	`array of strings`	Fuzzer target names to test(Supported values: `"tint_wgsl_fuzzer"`, `"tint_ir_fuzzer"`, `"tint_wgsl_mesa_fuzzer"`, `"tint_ir_mesa_fuzzer"`).
`timeout`	`integer` optional	Timeout limit in seconds for a single fuzzer execution on a test case in libFuzzer (defaults to 5).
`burnin_enabled`	`boolean` optional	If true, launches parallel workloads to warm up the machine before benching (defaults to true).
`burnin_duration`	`integer` optional	Target duration in seconds for the initial thermal burn-in (defaults to 300).
`normalization_duration`	`integer` optional	Execution duration in seconds for each iteration of the normalization microbenchmarking phase (defaults to 60).
`normalization_iterations`	`integer` optional	Number of iterations to perform for normalization microbenchmarking (defaults to 5).
`wgsl_normalization_corpus`	`string`	Directory path relative to the root `corpora/` to use when performing normalization on WGSL fuzzers.
`ir_normalization_corpus`	`string`	Directory path relative to the root `corpora/` to use when performing normalization on IR fuzzers.
`wgsl_corpora`	`array of CorpusDef`	Corpora definitions available to run with WGSL fuzzers.
`ir_corpora`	`array of CorpusDef`	Corpora definitions available to run with IR fuzzers.
`default_iterations`	`integer`	The default number of times to repeat every Fuzzer/Corpus/Duration combination if not otherwise specified.
`durations`	`array of DurationDef`	The target run lengths defining the experiment stopping criteria.

`CorpusDef` Format

Field	Type	Description
`name`	`string`	A human-readable identifier for the corpus, used in the logging/reports.
`path`	`string`	Directory path relative to the root `corpora/` directory.

`DurationDef` Format

Note: Must specify exactly one and only one of seconds or runs.

Note: Because of how libFuzzer operate seconds and runs are approximate values. Fuzzing should run for at least this limit, but will normally be a little over.

Field	Type	Description
`seconds`	`integer` (optional)	Number of seconds to run for (maps to `-max_total_time`).
`runs`	`integer` (optional)	Number of inputs/test cases to run for(maps to `-runs`).
`iterations`	`integer` (optional)	Override the experiment `default_iterations` specifically for this duration target. Useful for manually load balancing very long and very short durations