Ben Clayton | 11ab43d | 2021-11-05 22:47:16 +0000 | [diff] [blame] | 1 | # Tint end-to-end tests |
| 2 | |
| 3 | This repo contains a large number of end-to-end tests at `<tint>/test`. |
| 4 | |
| 5 | ## Test files |
| 6 | |
| 7 | Test input files have either the `.wgsl`, `.spv` or `.spvasm` file extension. |
| 8 | |
| 9 | Each test input file is tested against each of the Tint backends. There are `<number-of-input-files>` × `<number-of-tint-backends>` tests that are performed on an unfiltered end-to-end test run. |
| 10 | |
| 11 | Each backend test can have an **expectation file**. This expectation file sits next to the input file, with a `<input-file>.expected.<format>` extension. For example the test `test/foo.wgsl` would have the HLSL expectation file `test/foo.wgsl.expected.hlsl`. |
| 12 | |
| 13 | An expectation file contains the expected output of Tint, when passed the input file for the given backend. |
| 14 | |
| 15 | If the first line of the expectation file starts `SKIP`, then the test will be skipped instead of failing the end-to-end test run. It is good practice to include after the `SKIP` a reason for why the test is being skipped, along with any additional details, such as compiler error messages. |
| 16 | |
| 17 | ## Running |
| 18 | |
| 19 | To run the end-to-end tests use the `<tint>/test/test-all.sh` script, passing the path to the tint executable as the first command line argument. |
| 20 | |
| 21 | You can pass `--help` to see the full list of command line flags.\ |
| 22 | The most commonly used flags are: |
| 23 | |
| 24 | | flag | description | |
| 25 | |----------------------|-------------| |
| 26 | |`--filter` | Filters the testing to subset of the tests. The filter argument is a glob pattern that can include `*` for any substring of a file or directory, and `**` for any number of directories.<br>Example: `--filter 'expressions/**/i32.wgsl'` will test all the `i32.wgsl` expression tests. |
| 27 | |`--format` | Filters the tests to the particular backend.<br>Example: `--format hlsl` will just test the HLSL backend. |
| 28 | |`--generate-expected` | Generate expectation files for the tests that previously had no expectation file, or were marked as `SKIP` but now pass. |
| 29 | |`--generate-skip` | Generate `SKIP` expectation files for tests that are not currently passing. |
| 30 | |
| 31 | ## Authoring guidelines |
| 32 | |
| 33 | Each test should be as small as possible, and focused on the particular feature being tested. |
| 34 | |
| 35 | Use sub-directories whenever possible to group similar tests, and try to keep the pattern of directories as consistent as possible between different tests. This helps filter tests using the `--filter` glob patterns. |