[fuzz] Add documentation for fuzzer benchmarking Adds a Markdown file explaining the basic flow of `./tools/run fuzz` `-experiment` and `-analyze`, providing a simple example, and documentation for the JSON configuration format Also changes 'Tint Core' to 'Tint' and 'DirectX' to 'DXC' in the code/outputs. Bug: 524444910 Change-Id: I8c25a4c73d112adf0875c0da35a6dffae5daf08c Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/339115 Auto-Submit: Ryan Harrison <rharrison@chromium.org> Commit-Queue: Ryan Harrison <rharrison@chromium.org> Commit-Queue: dan sinclair <dsinclair@chromium.org> Reviewed-by: dan sinclair <dsinclair@chromium.org>
diff --git a/docs/tint/fuzzing-experiments.md b/docs/tint/fuzzing-experiments.md new file mode 100644 index 0000000..14c6fbf --- /dev/null +++ b/docs/tint/fuzzing-experiments.md
@@ -0,0 +1,264 @@ +# Tint Fuzzing Experiments + +The Tint fuzzing experiment/benchmarking framework is designed to +automate the process of comparing changes to the Tint fuzzers. It +measures hardware performance, fuzzer coverage percentage rates, and +test case execution rates to come up with abstract performance scores +to compare versions of the fuzzers. + +This framework is part of the general fuzz utility for Tint found in +`./tools/src/cmd/fuzz`. + +This document explains the high-level architecture of the experiments +framework, the environment prerequisites, and a step-by-step +walkthrough of how to define, execute, and process an +experiment. Along with a reference section at the end for the +experiment configuration JSON format, + +--- + +## Prerequisites + +Running fuzzing experiments requires a local Dawn repository and a GN +build directory configured for fuzzing and coverage. + +To support correct instrumentation, your GN build (`args.gn`) must +include at least the following flags: + +```gn +use_libfuzzer = true +tint_build_wgsl_reader = true +use_clang_coverage = true +optimize_for_fuzzing = false # Cannot be used with coverage enabled +``` + +Additional flags must be enabled depending on the fuzzers selected: +- **IR Fuzzers (`tint_ir_fuzzer`, `tint_ir_mesa_fuzzer` )**: Require + `tint_build_ir_binary = true` and `tint_has_protobuf = true`. +- **Mesa Fuzzers (`tint_wgsl_mesa_fuzzer`, `tint_ir_mesa_fuzzer`)**: + Require `tint_build_mesa = true` and + `tint_build_fuzzer_vulkan_support = true`. + +The tool will produce an error if the flags are not correctly +configured. The tool does not support using a pre-built version of the +Tint fuzzers, nor the CMake builds. + +**Note:** For the most applicable results it is recommended that all +backends are turned on (`tint_build_hlsl_writer = true`, +`tint_build_msl_writer = true`, `tint_build_spv_writer = true`) + +**Note:** Though the fuzzers can be built and run with sanitizers +turned on (`is_asan = true`, etc), this is not recommended, because +the sanitizers tend to dominate the runtime of the execution, so +produce substantially lower quality data + +--- + +## Defining an Experiment + +An experiment is represented by a dedicated directory containing: +1. An `experiment.json` configuration file. +2. A `corpora/` directory holding the seed corpora for the various + fuzzing modes. + +### Directory Structure + +```directory +my_experiment/ +├── experiment.json +└── corpora/ + ├── wgsl_seed/ + │ ├── shader1.wgsl + │ └── shader2.wgsl + └── empty/ +``` + +### Writing `experiment.json` + +The `experiment.json` defines variables like the commit hash to check +out, specific fuzzers to test, and duration of the experiments. + +Here is an example setup for comparing fuzzer performance using both a +standard corpus and an empty directory as starting points: + +```json +{ + "name": "example", + "hash": "4c2395a860ab76b44f7256c25281dfd3a3680192", + "fuzzers": ["tint_wgsl_fuzzer", "tint_ir_fuzzer"], + "burnin_enabled": false, + "wgsl_benchmark_corpus": "wgsl_seed", + "ir_benchmark_corpus": "empty", + "wgsl_corpora": [ + { "name": "wgsl_full", "path": "wgsl_seed" }, + { "name": "wgsl_empty", "path": "empty" } + ], + "ir_corpora": [ + { "name": "ir_empty", "path": "empty" } + ], + "default_iterations": 5, + "durations": [ + {"runs": 10000}, + {"runs": 50000}, + {"runs": 100000} + ] +} +``` + +- `name` is just a user visible title for this experiment which will + appear in reports/logging, but has no semantic meaning +- `hash` is the specific version of the Dawn repo that will be checked + out to build the fuzzers. (The tool will be built and run from what + ever hash the repo is at when you call it, not this version) +- `fuzzers` are the specific fuzzer binaries to test +- `burnin_enabled` is turning off burn-in so that the there isn't a 5 + minute lag when running this experiment, but this should be removed + when generating statistically valid data +- `*_benchmark_corpus` are the a corpora to use when establishing + baselines for normalizing performance numbers between + machines/environments +- `*_corpora` are the various starting corpora to experiment using +- `default_iterations` specifies how many times to run each experiment + if not explicitly overridden +- `durations` specifies the sets experiments to run, broadly you will + want to specify a spread of values here to get a reasonable + graph performance over time, since the framework doesn't + sample these values during the experiment run + +Full details on the various options are detailed below. + +**Note:** The number of actual runs performed will be `# of fuzzers` X +`# of corpora` X `# of durations` X `# of iterations` + +--- + +## Running an Experiment + +To start the experiment on your system, execute the `fuzz` helper with +the `-experiment` flag: + +```bash +/tools/run fuzz -experiment -build out/fuzzers my_experiment/ +``` + +where `out/fuzzers` is a GN configured build with the correct values +set in `args.gn` + +**Note:** `-experiment` does support the `-j` flag for running +concurrent operations, but defaults it to **1**. This is because +saturating the RAM or CPU in the execution environment (e.g. -j `max +num of cores`) can significantly impact the results. It is recommended +to experiment with a mini-version of an experiment.json to determine +an appropriate value to use, (try num of cores / 2 or / 4 as a +starting point) + +### Execution Pipeline + +The framework guides execution through the following phases: + +1. **Validation & Sync**: The tool checks the active GN arguments in + your build directory to ensure they match the requirements. It then + checks out the requested commit hash (`hash` field) and runs + `gclient sync` automatically. (The state of repo should be returned + to the original state) +2. **Binary Preparation**: It builds the selected fuzzer binaries and + copies them, alongside their dynamic library dependencies and + required LLVM tools (`llvm-profdata`, `llvm-cov`), into + `my_experiment/bin/`. +3. **Burn-in (Optional)**: If enabled, it executes multiple parallel + workloads for 5 minutes (`burnin_duration` & `burnin_enabled`) to + bring the physical host's CPU to a thermal steady state. This + prevents throttling and microbenchmark skews during the experiment. +4. **Microbenchmarking**: It executes a short benchmark (60 seconds) + for each fuzzer against its designated benchmark corpus. The + resulting execution rate (runs/sec) is saved to `perf_scores.json` + and used as a normalization factor to compute "Normalized CPU + Seconds" for each fuzzer. +5. **Task Execution**: It calculates the Cartesian product of + experiments (Fuzzer × Corpus × Duration × Iteration). These are run + concurrently across available CPU cores (configurable via + `-j`). Results are saved directly under + `my_experiment/results/`. Each task directory stores a `state.json` + file, captured logs, mutated corpus files, and `.profraw` coverage + profiles. + +--- + +## Processing and Analyzing Results + +Once execution has completed, you can aggregate and process results +using the `-analyze` flag: + +```bash +tools/run fuzz -analyze my_experiment/ +``` + +### Analysis Pipeline + +1. **Coverage Generation**: The analyzer locates the `.profraw` file + for each completed iteration, merges them using `llvm-profdata`, + and executes `coverage.py` to produce standard `.lcov` coverage + logs. +2. **Component Mapping**: It parses the `.lcov` profiles and + aggregates line coverage metrics separately for **Tint** + (`src/tint/` and `src/utils/`), **Mesa** (`third_party/mesa/`), and + **DXC** (`third_party/directx-headers/`). +3. **Statistics Computation**: It computes arithmetic means and + standard errors for: + - Coverage percentage + - Normalized CPU seconds + - Coverage rates (% of lines covered per CPU second, using + quadrature error propagation) +4. **Report Export**: + - `raw_iteration_data.csv`: A flat table of raw execution counts, + actual runtimes, and coverage hits for every single iteration. + - `calculated_statistics.csv`: A flat table of calculated + statistics for every experiment category (Fuzzer × Corpus). + - `experiment_report.md`: A human-readable Markdown summary report. + +--- + +## Appendix: `experiment.json` Reference + +The following sections define the full configuration schema for +`experiment.json`. + +### Root Attributes + +| Field | Type | Description | +|:------------------------|:-----------------------|:---------------------------------------------------------------------------------------------------------------------------------------------| +| `name` | `string` | A human-readable identifier for the experiment, used in the logging/reports. | +| `hash` | `string` | The Git commit hash checkout from which the binaries should be compiled. | +| `fuzzers` | `array of strings` | Fuzzer target names to test(Supported values: `"tint_wgsl_fuzzer"`, `"tint_ir_fuzzer"`, `"tint_wgsl_mesa_fuzzer"`, `"tint_ir_mesa_fuzzer"`). | +| `timeout` | `integer` *optional* | Timeout limit in seconds for a single fuzzer execution on a test case in libFuzzer (defaults to 5). | +| `benchmark_duration` | `integer` *optional* | Execution duration in seconds for the microbenchmarking phase (defaults to 60). | +| `burnin_duration` | `integer` *optional* | Target duration in seconds for the initial thermal burn-in (defaults to 300). | +| `burnin_enabled` | `boolean` *optional* | If true, launches parallel workloads to warm up the machine before benching (defaults to true). | +| `wgsl_benchmark_corpus` | `string` | Directory path relative to the root `corpora/` to use when microbenchmarking WGSL fuzzers. | +| `ir_benchmark_corpus` | `string` | Directory path relative to the root `corpora/` to use when microbenchmarking IR fuzzers. | +| `wgsl_corpora` | `array of CorpusDef` | Corpora definitions available to run with WGSL fuzzers. | +| `ir_corpora` | `array of CorpusDef` | Corpora definitions available to run with IR fuzzers. | +| `default_iterations` | `integer` | The default number of times to repeat every Fuzzer/Corpus/Duration combination if not otherwise specified. | +| `durations` | `array of DurationDef` | The target run lengths defining the experiment stopping criteria. | + +### `CorpusDef` Format + +| Field | Type | Description | +|:-------|:---------|:-------------------------------------------------------------------------| +| `name` | `string` | A human-readable identifier for the corpus, used in the logging/reports. | +| `path` | `string` | Directory path relative to the root `corpora/` directory. | + +### `DurationDef` Format + +**Note**: Must specify exactly one and only one of `seconds` or +`runs`. + +**Note**: Because of how libFuzzer operate `seconds` and `runs` are +approximate values. Fuzzing should run for at least this limit, but +will normally be a little over. + +| Field | Type | Description | +|:-------------|:---------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------| +| `seconds` | `integer` (optional) | Number of seconds to run for (maps to `-max_total_time`). | +| `runs` | `integer` (optional) | Number of inputs/test cases to run for(maps to `-runs`). | +| `iterations` | `integer` (optional) | Override the experiment `default_iterations` specifically for this duration target. Useful for manually load balancing very long and very short durations |
diff --git a/tools/src/cmd/fuzz/analyze.go b/tools/src/cmd/fuzz/analyze.go index 5689c78..bcfcfc3 100644 --- a/tools/src/cmd/fuzz/analyze.go +++ b/tools/src/cmd/fuzz/analyze.go
@@ -59,13 +59,31 @@ Percentage float64 `json:"percentage"` } -// IterationCoverage aggregates coverage statistics across different project components. -type IterationCoverage struct { - TintCore CoverageStats `json:"tint_core"` - Mesa CoverageStats `json:"mesa"` - DirectX CoverageStats `json:"directx"` +type CoverageComponent uint8 + +const ( + CoverageComponentTint = iota + CoverageComponentMesa + CoverageComponentDXC +) + +func (c CoverageComponent) String() string { + switch c { + case CoverageComponentTint: + return "Tint" + case CoverageComponentMesa: + return "Mesa" + case CoverageComponentDXC: + return "DXC" + default: + return "<unknown>" + } + } +// IterationCoverage aggregates coverage statistics across different project components. +type IterationCoverage map[CoverageComponent]*CoverageStats + // IterationData holds the raw performance and coverage metrics for a single fuzzer iteration. type IterationData struct { Machine string @@ -387,9 +405,9 @@ func (ac *analyzeConfig) printRawCSV() error { var csvBuilder strings.Builder csvBuilder.WriteString("Machine,Fuzzer,Corpus,LimitType,LimitValue,Iteration,PerfScore,ActualSeconds,ActualRuns,NormalizedCPUSeconds," + - "TintCore_LinesFound,TintCore_LinesHit,TintCore_CoveragePercent," + + "Tint_LinesFound,Tint_LinesHit,Tint_CoveragePercent," + "Mesa_LinesFound,Mesa_LinesHit,Mesa_CoveragePercent," + - "DirectX_LinesFound,DirectX_LinesHit,DirectX_CoveragePercent\n") + "DXC_LinesFound,DXC_LinesHit,DXC_CoveragePercent\n") for _, d := range ac.data { csvBuilder.WriteString(fmt.Sprintf("%s,%s,%s,%s,%d,%d,%.4f,%.2f,%d,%.4f,%d,%d,%.2f,%d,%d,%.2f,%d,%d,%.2f\n", @@ -403,15 +421,15 @@ d.ActualSeconds, d.ActualRuns, d.NormalizedSecs, - d.Coverage.TintCore.LinesFound, - d.Coverage.TintCore.LinesHit, - d.Coverage.TintCore.Percentage, - d.Coverage.Mesa.LinesFound, - d.Coverage.Mesa.LinesHit, - d.Coverage.Mesa.Percentage, - d.Coverage.DirectX.LinesFound, - d.Coverage.DirectX.LinesHit, - d.Coverage.DirectX.Percentage, + d.Coverage[CoverageComponentTint].LinesFound, + d.Coverage[CoverageComponentTint].LinesHit, + d.Coverage[CoverageComponentTint].Percentage, + d.Coverage[CoverageComponentMesa].LinesFound, + d.Coverage[CoverageComponentMesa].LinesHit, + d.Coverage[CoverageComponentMesa].Percentage, + d.Coverage[CoverageComponentDXC].LinesFound, + d.Coverage[CoverageComponentDXC].LinesHit, + d.Coverage[CoverageComponentDXC].Percentage, )) } @@ -443,17 +461,8 @@ accumulations := make(map[accumulatorKey]*accumulatorVal) for _, d := range data { - components := []struct { - Name string - Stats CoverageStats - }{ - {"Tint Core", d.Coverage.TintCore}, - {"Mesa", d.Coverage.Mesa}, - {"DirectX", d.Coverage.DirectX}, - } - - for _, c := range components { - if c.Stats.LinesFound == 0 { + for comp, stats := range d.Coverage { + if stats.LinesFound == 0 { continue // skip reporting empty components (e.g. Mesa if not a Mesa fuzzer) } @@ -462,7 +471,7 @@ Corpus: d.Corpus, LimitType: d.LimitType, LimitValue: d.LimitValue, - Component: c.Name, + Component: comp.String(), } val, ok := accumulations[key] if !ok { @@ -471,7 +480,7 @@ } val.NormalizedSecs = append(val.NormalizedSecs, d.NormalizedSecs) - val.CoveragePercent = append(val.CoveragePercent, c.Stats.Percentage) + val.CoveragePercent = append(val.CoveragePercent, stats.Percentage) } } @@ -803,48 +812,43 @@ } // parseLcov parses the contents of an LCOV file and categorizes coverage metrics -// into Tint Core, Mesa, and DirectX components based on file paths. +// into Tint, Mesa, and DXC components based on file paths. func parseLcov(content string) IterationCoverage { lines := strings.Split(content, "\n") var currentFile string - var inTintCore, inMesa, inDirectX bool - - metrics := map[string]*CoverageStats{ - "tint_core": {}, - "mesa": {}, - "directx": {}, + currentlyIn := map[CoverageComponent]bool{ + CoverageComponentTint: false, + CoverageComponentMesa: false, + CoverageComponentDXC: false, } + metrics := IterationCoverage{ + CoverageComponentTint: {}, + CoverageComponentMesa: {}, + CoverageComponentDXC: {}, + } for _, line := range lines { line = strings.TrimSpace(line) if after, ok := strings.CutPrefix(line, "SF:"); ok { currentFile = after currentFile = filepath.ToSlash(currentFile) - inTintCore = strings.Contains(currentFile, "src/tint/") || strings.Contains(currentFile, "src/utils/") - inMesa = strings.Contains(currentFile, "third_party/mesa/") - inDirectX = strings.Contains(currentFile, "third_party/directx") + currentlyIn[CoverageComponentTint] = strings.Contains(currentFile, "src/tint/") || strings.Contains(currentFile, "src/utils/") + currentlyIn[CoverageComponentMesa] = strings.Contains(currentFile, "third_party/mesa/") + currentlyIn[CoverageComponentDXC] = strings.Contains(currentFile, "third_party/directx") } else if after, ok := strings.CutPrefix(line, "LF:"); ok { val, _ := strconv.Atoi(after) - if inTintCore { - metrics["tint_core"].LinesFound += val - } - if inMesa { - metrics["mesa"].LinesFound += val - } - if inDirectX { - metrics["directx"].LinesFound += val + for comp, in := range currentlyIn { + if in { + metrics[comp].LinesFound += val + } } } else if after, ok := strings.CutPrefix(line, "LH:"); ok { val, _ := strconv.Atoi(after) - if inTintCore { - metrics["tint_core"].LinesHit += val - } - if inMesa { - metrics["mesa"].LinesHit += val - } - if inDirectX { - metrics["directx"].LinesHit += val + for comp, in := range currentlyIn { + if in { + metrics[comp].LinesHit += val + } } } } @@ -855,11 +859,7 @@ } } - return IterationCoverage{ - TintCore: *metrics["tint_core"], - Mesa: *metrics["mesa"], - DirectX: *metrics["directx"], - } + return metrics } // computeAvgAndStdDev calculates the arithmetic mean and sample standard deviation for a slice of float64.
diff --git a/tools/src/cmd/fuzz/analyze_test.go b/tools/src/cmd/fuzz/analyze_test.go index 93987f6..2b375e5 100644 --- a/tools/src/cmd/fuzz/analyze_test.go +++ b/tools/src/cmd/fuzz/analyze_test.go
@@ -56,20 +56,20 @@ ` cov := parseLcov(lcovContent) - // Tint Core: src/tint (10 LF, 5 LH) + src/utils (20 LF, 15 LH) = 30 LF, 20 LH - require.Equal(t, 30, cov.TintCore.LinesFound) - require.Equal(t, 20, cov.TintCore.LinesHit) - require.InDelta(t, 66.67, cov.TintCore.Percentage, 0.01) + // Tint: src/tint (10 LF, 5 LH) + src/utils (20 LF, 15 LH) = 30 LF, 20 LH + require.Equal(t, 30, cov[CoverageComponentTint].LinesFound) + require.Equal(t, 20, cov[CoverageComponentTint].LinesHit) + require.InDelta(t, 66.67, cov[CoverageComponentTint].Percentage, 0.01) // Mesa: third_party/mesa (100 LF, 80 LH) = 100 LF, 80 LH - require.Equal(t, 100, cov.Mesa.LinesFound) - require.Equal(t, 80, cov.Mesa.LinesHit) - require.InDelta(t, 80.0, cov.Mesa.Percentage, 0.01) + require.Equal(t, 100, cov[CoverageComponentMesa].LinesFound) + require.Equal(t, 80, cov[CoverageComponentMesa].LinesHit) + require.InDelta(t, 80.0, cov[CoverageComponentMesa].Percentage, 0.01) - // DirectX: third_party/directx-headers (50 LF, 10 LH) = 50 LF, 10 LH - require.Equal(t, 50, cov.DirectX.LinesFound) - require.Equal(t, 10, cov.DirectX.LinesHit) - require.InDelta(t, 20.0, cov.DirectX.Percentage, 0.01) + // DXC: third_party/directx-headers (50 LF, 10 LH) = 50 LF, 10 LH + require.Equal(t, 50, cov[CoverageComponentDXC].LinesFound) + require.Equal(t, 10, cov[CoverageComponentDXC].LinesHit) + require.InDelta(t, 20.0, cov[CoverageComponentDXC].Percentage, 0.01) } func TestStats(t *testing.T) { @@ -101,7 +101,7 @@ Iteration: 1, NormalizedSecs: 9.0, Coverage: IterationCoverage{ - TintCore: CoverageStats{ + CoverageComponentTint: { LinesFound: 100, LinesHit: 50, Percentage: 50.0, @@ -116,7 +116,7 @@ Iteration: 2, NormalizedSecs: 11.0, Coverage: IterationCoverage{ - TintCore: CoverageStats{ + CoverageComponentTint: { LinesFound: 100, LinesHit: 60, Percentage: 60.0, @@ -131,7 +131,7 @@ pt := summaries[0] require.Equal(t, "fuzzerA", pt.Fuzzer) require.Equal(t, "corpusA", pt.Corpus) - require.Equal(t, "Tint Core", pt.Component) + require.Equal(t, "Tint", pt.Component) require.Equal(t, "seconds", pt.LimitType) require.Equal(t, 10, pt.LimitValue) require.Equal(t, 2, pt.N) @@ -230,7 +230,7 @@ { Fuzzer: "fuzzerA", Corpus: "corpusA", - Component: "Tint Core", + Component: "Tint", LimitType: "seconds", LimitValue: 10, NormSecsAvg: 9.5, @@ -244,7 +244,7 @@ { Fuzzer: "fuzzerA", Corpus: "corpusA", - Component: "Tint Core", + Component: "Tint", LimitType: "seconds", LimitValue: 20, NormSecsAvg: 19.5, @@ -265,8 +265,8 @@ require.NoError(t, err) expectedCSV := "Fuzzer,Corpus,Component,Samples,LimitType,LimitValue,NormalizedCPUSecondsAvg,NormalizedCPUSecondsSEM,CoveragePercentAvg,CoveragePercentSEM,CoverageRateAvg,CoverageRateSEM\n" + - "fuzzerA,corpusA,Tint Core,5,seconds,10,9.5000,0.5000,55.00,1.20,5.780000,0.100000\n" + - "fuzzerA,corpusA,Tint Core,5,seconds,20,19.5000,0.8000,65.00,1.50,3.330000,0.120000\n" + "fuzzerA,corpusA,Tint,5,seconds,10,9.5000,0.5000,55.00,1.20,5.780000,0.100000\n" + + "fuzzerA,corpusA,Tint,5,seconds,20,19.5000,0.8000,65.00,1.50,3.330000,0.120000\n" require.Equal(t, expectedCSV, string(csvContent)) @@ -280,7 +280,7 @@ reportStr := string(reportContent) require.Contains(t, reportStr, "# Experiment Performance and Coverage Report: test_experiment") require.Contains(t, reportStr, "- **Git Hash**: `abcdef123`") - require.Contains(t, reportStr, "### fuzzerA - corpusA (Tint Core)") + require.Contains(t, reportStr, "### fuzzerA - corpusA (Tint)") require.Contains(t, reportStr, "| Samples (N) | Target Limit | Normalized CPU Seconds (Avg ± SEM) | Coverage % (Avg ± SEM) | Coverage Rate (%/sec) (Avg ± SEM) |") require.Contains(t, reportStr, "| 5 | 10 seconds | 9.50 ± 0.50 | 55.00% ± 1.20% | 5.780000 ± 0.100000 |") require.Contains(t, reportStr, "| 5 | 20 seconds | 19.50 ± 0.80 | 65.00% ± 1.50% | 3.330000 ± 0.120000 |")