This pull request rewrites the entire Azure DevOps build system.
The guiding principles behind this rewrite are:
- No pipeline definitions should contain steps (or tasks) directly.
- All jobs should be in template files.
- Any set of steps that is reused across multiple jobs must be in
template files.
- All artifact names can be customized (via a property called
`artifactStem` on all templates that produce or consume artifacts).
- No compilation happens outside of the "Build" phase, to consolidate
the production and indexing of PDBs.
- **Building the project produces a `bin` directory.** That `bin`
directory is therefore the primary currency of the build. Jobs will
either produce or consume `bin` if they want to do anything with the
build outputs.
- All step and job templates are named with `step` or `job` _first_,
which disambiguates them in the templates directory.
- Most jobs can be run on different `pool`s, so that we can put
expensive jobs on expensive build agents and cheap jobs on cheap
build agents. Some jobs handle pool selection on their own, however.
Our original build pipelines used the `VSBuild` task _all over the
place._ This resulted in Terminal being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where `ci.yml` consumed jobs and steps templates... but when
`release.yml` was added, all of that went out the window.
The new pipelines are consistent and focus on a small, well-defined set
of jobs:
- `job-build-project`
- This is the big one!
- Takes a list of build configurations and platforms.
- Produces an artifact named `build-PLATFORM-CONFIG` for the entire
matrix of possibilities.
- Optionally signs the output and produces a bill of materials.
- Admittedly has a lot going on.
- `job-build-package-wpf`
- Takes a list of build configurations and platforms.
- Consumes the `build-` artifact for every config/platform
possibility, plus one for "Any CPU" (hardcoded; this is where the
.NET code builds)
- Produces one `wpf-nupkg-CONFIG` for each configuration, merging
all platforms.
- Optionally signs the output and produces a bill of materials.
- `job-merge-msix-into-bundle`
- Takes a list of build configurations and platforms.
- Consumes the `build-` artifact for every config/platform
- Produces one `appxbundle-CONFIG` for each configuration, merging
all platforms for that config into one `msixbundle`.
- Optionally signs the output and produces a bill of materials.
- `job-package-conpty`
- Takes a list of build configurations and platforms.
- Consumes the `build-` artifact for every config/platform
- Produces one `conpty-nupkg-CONFIG` for each configuration, merging
all platforms.
- Optionally signs the output and produces a bill of materials.
- `job-test-project`
- Takes **one** build config and **one** platform.
- Consumes `build-PLATFORM-CONFIG`
- Selects its own pools (hardcoded) because it knows about
architectures and must choose the right agent arch.
- Runs tests (directly on the build agent).
- `job-run-pgo-tests`
- Just like the above, but runs tests where `IsPgo` is `true`
- Collects all of the PGO counts and publishes a `pgc-intermediates`
artifact for that platform and configuration.
- `job-pgo-merge-pgd`
- Takes **one** build config and multiple platforms.
- Consumes `build-$platform-CONFIG` for each platform.
- Consumes `pgc-intermediates-$platform-CONFIG` for each platform.
- Merges the `pgc` files into `pgd` files
- Produces a new `pgd-` artifact.
- `job-pgo-build-nuget-and-publish`
- Consumes the `pgd-` artifact from above.
- Packs it into a `nupkg` and publishes it.
- `job-submit-windows-vpack`
- Only expected to run against `Release`.
- Consumes the `appxbundle-CONFIG` artifact.
- Publishes it to a vpack for Windows to consume.
- `job-check-code-format`
- Does not use artifacts. Runs `clang-format`.
- `job-index-github-codenav`
- Does not use artifacts.
Fuzz submission is broken due to changes in the `onefuzz` client.
I have removed the compliance and security build because it is no longer
supported.
Finally, this pull request has some additional benefits:
- I've expanded the PGO build phase to cover ARM64!
- We can remove everything Helix-related except the WTT parser
- We no longer depend on Helix submission or Helix pools
- The WPF control's inner DLLs are now codesigned (#15404)
- Symbols for the WPF control, both .NET and C++, are published
alongside all other symbols.
- The files we submit to ESRP for signing are batched up into a single
step[^1]
Closes#11874Closes#11974Closes#15404
[^1]: This will have to change if we want to sign the individual
per-architecture `.appx` files before bundling so that they can be
directly installed.
Upgrades check-spelling to v0.0.21
The command to apply changes should now work on Windows (it requires
Perl, but I believe that's more or less present most of the time, and it
should walk you through the rest of the required tools).
There are a bunch of new features, the most important here are probably
being able to update the metadata from Windows. (If it doesn't work,
please @ me).
Also, candidate.patterns will automatically suggest patterns. You can
see them in patterns.txt, e.g.:
```
# Automatically suggested patterns
# hit-count: 3831 file-count: 582
# IServiceProvider
\bI(?=(?:[A-Z][a-z]{2,})+\b)
```
The metadata bits (the hit count/file count) don't have to be retained
(I hope they'll be useful in deciding whether/or not to add a pattern,
i.e. "how applicable is it?"), the comment hinting at what the pattern
does is probably worth retaining.
We've been using more or less this version for a while internally
(including talk-to-bot, and, I do have a pattern that could be used to
let people use that in forks, but, I'm going to skip that for now).
This weekend, I did some cleanup for `act` (to run check-spelling
locally), and some minor polish.
You can see the runs I made in
https://github.com/check-spelling/terminal/actions
While working on another PR related to this I noticed that my VS
generates `.vcxproj` files that are a bit different to the ones we have.
This commit is a quick search & replace of all our project files to make
(primarily) their `ToolsVersion` more in line with what VS does itself:
No `ToolsVersion` for `.vcxproj`, `ToolsVersion="15.0"`
for `.csproj` and `ToolsVersion="4.0"` for `.filters` files.
The PGO helpers NuGet had the Y2K22 bug. This receives and integrates the updated package in our project to restore NuGet functionality.
## PR Checklist
* [x] Closes#12261
* [x] I work here
* [x] If it builds it sits.
## Validation Steps Performed
* [x] Build new PGO instrument data with this pipeline update: https://dev.azure.com/microsoft/Dart/_build/results?buildId=44304850&view=results
To unify with WinUI, we're going to share an engineering component of this particular NuGet package full of scripts and utilities to make PGOing things easier.
This basically removes all of the scripts that I ~blatantly stole~ copied from https://github.com/microsoft/microsoft-ui-xaml and moves to the NuGet package that the team generates instead. A bunch of build things had to be massaged to make it work in our pipeline.
