Migrate spelling-0.0.21 changes from main

We now require Visual Studio 2022 to build our solution, and VS 2022
only supports 64-bit processors. Therefore, the "native" tools are the
64-bit tools.

This property causes an issue in building for ARM64, wherein Terminal
is built with the x64 tools targeting ARM64 (cross-compilation under
emulation) rather than the native ARM64 tools.

.git-blame-ignore-revs

@@ -1,15 +0,0 @@
-# Commits mentioned in this file will automatically
-# be skipped by GitHub's blame view.
-# To use it with "git", run
-# > git blame --ignore-revs-file ./.git-blame-ignore-revs
-
-# Reformatted the entire codebase
-9b92986b49bed8cc41fde4d6ef080921c41e6d9e
-
-# Line Endings changes
-cb7a76d96c92aa9fc7b03f69148fb0c75dff191d
-5bbf61af8c8f12e6c05d07a696bf7d411b330a67
-d07546a6fef73fa4e1fb1c2f01535843d1fcc212
-
-# UTF-8 BOM changes
-ddae2a1d49d604487d3c963e5eacbeb73861d986

.github/ISSUE_TEMPLATE/Bug_Report.yml

@@ -14,7 +14,7 @@ body:
     label: Windows Terminal version
     placeholder: "1.7.3651.0"
     description: |
-      You can copy the version number from the About dialog. Open the About dialog by opening the menu with the "V" button (to the right of the "+" button that opens a new tab) and choosing About from the end of the list.
+      You can find the version in the about dialog, or by running `wt -v` at the commandline (for the Stable/Preview version you’re reporting a bug about).
   validations:
     required: false
     

doc/cascadia/profiles.schema.json

@@ -2396,10 +2396,7 @@
         },
         "startingDirectory": {
           "description": "The directory the shell starts in when it is loaded.",
-          "type": [
-            "string",
-            "null"
-          ]
+          "type": "string"
         },
         "suppressApplicationTitle": {
           "description": "When set to true, tabTitle overrides the default title of the tab and any title change messages from the application will be suppressed. When set to false, tabTitle behaves as normal.",

doc/specs/#1571 - New Tab Menu Customization/#1571 - New Tab Menu Customization.md

@@ -1,7 +1,7 @@
 ---
 author: Mike Griese @zadjii-msft
 created on: 2020-5-13
-last updated: 2022-11-18
+last updated: 2020-08-04
 issue id: 1571
 ---
 
@@ -76,34 +76,6 @@ There are five `type`s of objects in this menu:
   - The `"entries"` property specifies a list of menu entries that will appear
     nested under this entry. This can contain other `"type":"folder"` groups as
     well!
-  - The `"inline"` property accepts two values
-    - `auto`: When the folder only has one entry in it, don't actually create a
-      nested layer to then menu. Just place the single entry in the layer that
-      folder would occupy. (Useful for dynamic profile sources with only a
-      single entry).
-    - `never`: (**default**) Always create a nested entry, even for a single
-      sub-item.
-  - The `allowEmpty` property will force this entry to show up in the menu, even
-    if it doesn't have any profiles in it. This defaults to `false`, meaning
-    that folders without any entries in them will just be ignored when
-    generating the menu. This will be more useful with the `matchProfile` entry,
-    below.
-
-    When this is true, and the folder is empty, we should add a
-    placeholder `<empty>` entry to the menu, to indicate that no profiles were
-    in that folder.
-    - _This setting is probably pretty niche, and not a requirement_. More of a
-      theoretical suggestion than anything.
-    - In the case of no entries for this folder, we should make sure to also
-      reflect the `inline` property:
-      - `allowEmpty:true`, `inline:auto`: just ignore the entry at all. Don't
-        add a placeholder to the parent list.
-      - `allowEmpty:true`, `inline:never`: Add a nested entry, with an
-        `<empty>` placeholder.
-      - `allowEmpty:false`, `inline:auto`: just ignore the entry at all. Don't
-        add a placeholder to the parent list.
-      - `allowEmpty:false`, `inline:never`: just ignore the entry at all. Don't
-        add a placeholder to the parent list.
 * `"type":"action"`: This represents a menu entry that should execute a specific
   `ShortcutAction`.
   - the `id` property will specify the global action ID (see [#6899], [#7175])
@@ -125,16 +97,6 @@ There are five `type`s of objects in this menu:
     enabling all other profiles to also be accessible.
   - The "name" of these entries will simply be the name of the profile
   - The "icon" of these entries will simply be the profile's icon
-  - This won't include any profiles that have been included via `matchProfile`
-    entries (below)
-* `"type": "matchProfile"`: Expands to all the profiles that match a given
-  string. This lets the user easily specify a whole collection of profiles for a
-  folder, without needing to add them all manually.
-  - `"name"`, `"commandline"` or `"source"`: These three properties are used to
-    filter the list of profiles, based on the matching property in the profile
-    itself. The value is a string to compare with the corresponding property in
-    the profile. A full string comparison is done - not a regex or partial
-    string match.
 
 The "default" new tab menu could be imagined as the following blob of json:
 
@@ -146,42 +108,6 @@ The "default" new tab menu could be imagined as the following blob of json:
 }
 ```
 
-Alternatively, we could consider something like the following. This would place
-CMD, PowerShell, and all PowerShell cores in the root at the top, followed by
-nested entries for each subsequent dynamic profile generator.
-
-```jsonc
-{
-    "newTabMenu": [
-        { "type":"profile", "profile": "cmd" },
-        { "type":"profile", "profile": "Windows PowerShell" },
-        { "type": "matchProfile", "source": "Microsoft.Terminal.PowerShellCore" }
-        {
-            "type": "folder",
-            "name": "WSL",
-            "entries": [ { "type": "matchProfile", "source": "Microsoft.Terminal.Wsl" } ]
-        },
-        {
-            "type": "folder",
-            "name": "Visual Studio",
-            "entries": [ { "type": "matchProfile", "source": "Microsoft.Terminal.VisualStudio" } ]
-        },
-        // ... etc for other profile generators
-        { "type": "remainingProfiles" }
-    ]
-}
-```
-
-I might only recommend that for `userDefaults.json`, which is the json files
-used as a template for a user's new settings file. This would prevent us from
-moving the user's cheese too much, if they're already using the Terminal and
-happy with their list as is. Especially consider someone who's default profile
-is a WSL distro, which would now need two clicks to get to.
-
-> _note_: We will also want to support the same `{ "key": "SomeResourceString"}`
-> syntax used by the Command Palette commands
-> for specifying localizable names, if we chose to pursue this route.
-
 ### Other considerations
 
 Also considered during the investigation for this feature was re-using the list
@@ -228,42 +154,6 @@ The design chosen in this spec more cleanly separates the responsibilities of
 the list of profiles and the contents of the new tab menu. This way, each object
 can be defined independent of the structure of the other.
 
-Regarding implementation of `matchProfile` entries: In order to build the menu,
-we'll evaluate the entries in the following order:
-
-* all explicit `profile` entries
-* then all `matchProfile` entries, using profiles not already specified
-* then expand out `remainingProfiles` with anything not found above.
-
-As an example:
-
-```jsonc
-{
-    "newTabMenu": [
-        { "type": "matchProfile", "source": "Microsoft.Terminal.Wsl" }
-        {
-            "type": "folder",
-            "name": "WSLs",
-            "entries": [ { "type": "matchProfile", "source": "Microsoft.Terminal.Wsl" } ]
-        },
-        { "type": "remainingProfiles" }
-    ]
-}
-```
-
-For profiles { "Profile A", "Profile B (WSL)", "Profile C (WSL)" }, This would
-expand to:
-
-```
-New Tab Button ▽
-├─ Profile A
-├─ Profile B (WSL)
-├─ Profile C (WSL)
-└─ WSLs
-   └─ Profile B (WSL)
-   └─ Profile C (WSL)
-```
-
 ## UI/UX Design
 
 See the above _figure 1_.
@@ -399,39 +289,7 @@ And assuming the user has bound:
     - Close Tab: `{ "action": "closeTab", "index": "${selectedTab.index}" }`
     - Close Other Tabs: `{ "action": "closeTabs", "otherThan": "${selectedTab.index}" }`
     - Close Tabs to the Right: `{ "action": "closeTabs", "after": "${selectedTab.index}" }`
-* We may want to consider regex, tag-based, or some other type of matching for
-  `matchProfile` entries in the future. We originally considered using regex for
-  `matchProfile` by default, but decided instead on full string matches to leave
-  room for regex matching in the future. Should we chose to pursue something
-  like that, we should use a settings structure like:
 
-  ```json
-  "type": "profileMatch",
-  "source": { "type": "regex", "value": ".*wsl.*" }
-  ```
-* We may want to expand `matchProfile` to match on other properties too. (`title`?)
-* We may want to consider adding support for capture groups, e.g.
-  ```json
-  {
-    "type": "profileMatch",
-    "name": { "type": "regex", "value": "^ssh: (.*)" }
-  }
-  ```
-  for matching to all your `ssh: ` profiles, but populate the name in the entry
-  with that first capture group. So, ["ssh: foo", "ssh: bar"] would just expand
-  to a "foo" and "bar" entry.
-
-## Updates
-
-_February 2022_: Doc updated in response to some discussion in [#11326] and
-[#7774]. In those PRs, it became clear that there needs to be a simple way of
-collecting up a whole group of profiles automatically for sorting in these
-menus. Although discussion centered on how hard it would be for extensions to
-provide that customization themselves, the `match` statement was added as a way
-to allow the user to easily filter those profiles themselves.
-
-This was something we had originally considered as a "future consideration", but
-ultimately deemed it to be out of scope for the initial spec review.
 
 <!-- Footnotes -->
 [#2046]: https://github.com/microsoft/terminal/issues/2046
@@ -440,5 +298,3 @@ ultimately deemed it to be out of scope for the initial spec review.
 [#3337]: https://github.com/microsoft/terminal/issues/3337
 [#6899]: https://github.com/microsoft/terminal/issues/6899
 [#7175]: https://github.com/microsoft/terminal/issues/7175
-[#11326]: https://github.com/microsoft/terminal/issues/11326
-[#7774]: https://github.com/microsoft/terminal/issues/7774

src/buffer/out/OutputCellIterator.cpp

@@ -5,9 +5,8 @@
 
 #include "OutputCellIterator.hpp"
 
-#include <til/unicode.h>
-
 #include "../../types/inc/convert.hpp"
+#include "../../types/inc/Utf16Parser.hpp"
 #include "../../types/inc/GlyphWidth.hpp"
 #include "../../inc/conattrs.hpp"
 
@@ -393,7 +392,7 @@ OutputCellView OutputCellIterator::s_GenerateView(const std::wstring_view view,
                                                   const TextAttribute attr,
                                                   const TextAttributeBehavior behavior)
 {
-    const auto glyph = til::utf16_next(view);
+    const auto glyph = Utf16Parser::ParseNext(view);
     const auto dbcsAttr = IsGlyphFullWidth(glyph) ? DbcsAttribute::Leading : DbcsAttribute::Single;
     return OutputCellView(glyph, dbcsAttr, attr, behavior);
 }

src/buffer/out/search.cpp

@@ -5,9 +5,8 @@
 
 #include "search.h"
 
-#include <til/unicode.h>
-
 #include "textBuffer.hpp"
+#include "../types/inc/Utf16Parser.hpp"
 #include "../types/inc/GlyphWidth.hpp"
 
 using namespace Microsoft::Console::Types;
@@ -193,11 +192,12 @@ bool Search::_FindNeedleInHaystackAt(const til::point pos, til::point& start, ti
 
     auto bufferPos = pos;
 
-    for (const auto& needleChars : _needle)
+    for (const auto& needleCell : _needle)
     {
         // Haystack is the buffer. Needle is the string we were given.
         const auto hayIter = _uiaData.GetTextBuffer().GetTextDataAt(bufferPos);
         const auto hayChars = *hayIter;
+        const auto needleChars = std::wstring_view(needleCell.data(), needleCell.size());
 
         // If we didn't match at any point of the needle, return false.
         if (!_CompareChars(hayChars, needleChars))
@@ -328,12 +328,13 @@ void Search::_UpdateNextPosition()
 // - wstr - String that will be our search term
 // Return Value:
 // - Structured text data for comparison to screen buffer text data.
-std::vector<std::wstring> Search::s_CreateNeedleFromString(const std::wstring_view wstr)
+std::vector<std::vector<wchar_t>> Search::s_CreateNeedleFromString(const std::wstring_view wstr)
 {
-    std::vector<std::wstring> cells;
-    for (const auto& chars : til::utf16_iterator{ wstr })
+    const auto charData = Utf16Parser::Parse(wstr);
+    std::vector<std::vector<wchar_t>> cells;
+    for (const auto chars : charData)
     {
-        if (IsGlyphFullWidth(chars))
+        if (IsGlyphFullWidth(std::wstring_view{ chars.data(), chars.size() }))
         {
             cells.emplace_back(chars);
         }

src/buffer/out/search.h

@@ -68,7 +68,7 @@ private:
 
     static til::point s_GetInitialAnchor(const Microsoft::Console::Types::IUiaData& uiaData, const Direction dir);
 
-    static std::vector<std::wstring> s_CreateNeedleFromString(const std::wstring_view wstr);
+    static std::vector<std::vector<wchar_t>> s_CreateNeedleFromString(const std::wstring_view wstr);
 
     bool _reachedEnd = false;
     til::point _coordNext;
@@ -76,7 +76,7 @@ private:
     til::point _coordSelEnd;
 
     const til::point _coordAnchor;
-    const std::vector<std::wstring> _needle;
+    const std::vector<std::vector<wchar_t>> _needle;
     const Direction _direction;
     const Sensitivity _sensitivity;
     Microsoft::Console::Types::IUiaData& _uiaData;

src/buffer/out/textBuffer.cpp

@@ -6,11 +6,11 @@
 #include "textBuffer.hpp"
 
 #include <til/hash.h>
-#include <til/unicode.h>
 
 #include "../renderer/base/renderer.hpp"
 #include "../types/inc/utils.hpp"
 #include "../types/inc/convert.hpp"
+#include "../../types/inc/Utf16Parser.hpp"
 #include "../../types/inc/GlyphWidth.hpp"
 
 namespace
@@ -2810,14 +2810,16 @@ PointTree TextBuffer::GetPatterns(const til::CoordType firstRow, const til::Coor
             // match and the previous match, so we use the size of the prefix
             // along with the size of the match to determine the locations
             til::CoordType prefixSize = 0;
-            for (const auto str = i->prefix().str(); const auto& glyph : til::utf16_iterator{ str })
+            for (const auto parsedGlyph : Utf16Parser::Parse(i->prefix().str()))
             {
+                const std::wstring_view glyph{ parsedGlyph.data(), parsedGlyph.size() };
                 prefixSize += IsGlyphFullWidth(glyph) ? 2 : 1;
             }
             const auto start = lenUpToThis + prefixSize;
             til::CoordType matchSize = 0;
-            for (const auto str = i->str(); const auto& glyph : til::utf16_iterator{ str })
+            for (const auto parsedGlyph : Utf16Parser::Parse(i->str()))
             {
+                const std::wstring_view glyph{ parsedGlyph.data(), parsedGlyph.size() };
                 matchSize += IsGlyphFullWidth(glyph) ? 2 : 1;
             }
             const auto end = start + matchSize;

src/buffer/out/ut_textbuffer/ReflowTests.cpp

@@ -7,6 +7,7 @@
 
 #include "../textBuffer.hpp"
 #include "../../renderer/inc/DummyRenderer.hpp"
+#include "../../types/inc/Utf16Parser.hpp"
 #include "../../types/inc/GlyphWidth.hpp"
 
 #include <IDataSource.h>

src/cascadia/TerminalControl/ControlCore.cpp

@@ -10,6 +10,7 @@
 
 #include <DefaultSettings.h>
 #include <unicode.hpp>
+#include <Utf16Parser.hpp>
 #include <WinUser.h>
 #include <LibraryResources.h>
 
@@ -1743,7 +1744,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
             const auto strEnd = rowText.find_last_not_of(UNICODE_SPACE);
             if (strEnd != decltype(rowText)::npos)
             {
-                str.append(rowText.substr(0, strEnd + 1));
+                str.append(rowText.substr(strEnd + 1));
             }
 
             if (!row.WasWrapForced())

src/cascadia/TerminalControl/ControlInteractivity.cpp

@@ -5,6 +5,7 @@
 #include "ControlInteractivity.h"
 #include <DefaultSettings.h>
 #include <unicode.hpp>
+#include <Utf16Parser.hpp>
 #include <Utils.h>
 #include <LibraryResources.h>
 #include "../../types/inc/GlyphWidth.hpp"

src/cascadia/TerminalControl/TermControl.cpp

@@ -5,6 +5,7 @@
 #include "TermControl.h"
 
 #include <unicode.hpp>
+#include <Utf16Parser.hpp>
 #include <LibraryResources.h>
 
 #include "TermControlAutomationPeer.h"

src/cascadia/TerminalSettingsEditor/Interaction.xaml

@@ -88,11 +88,6 @@
                 <ToggleSwitch IsOn="{x:Bind ViewModel.DetectURLs, Mode=TwoWay}"
                               Style="{StaticResource ToggleSwitchInExpanderStyle}" />
             </local:SettingContainer>
-
-            <local:SettingContainer x:Uid="Globals_ConfirmCloseAllTabs">
-                <ToggleSwitch IsOn="{x:Bind ViewModel.ConfirmCloseAllTabs, Mode=TwoWay}"
-                              Style="{StaticResource ToggleSwitchInExpanderStyle}" />
-            </local:SettingContainer>
         </StackPanel>
     </ScrollViewer>
 </Page>

src/cascadia/TerminalSettingsEditor/InteractionViewModel.h

@@ -27,7 +27,6 @@ namespace winrt::Microsoft::Terminal::Settings::Editor::implementation
         PERMANENT_OBSERVABLE_PROJECTED_SETTING(_GlobalSettings, FocusFollowMouse);
         PERMANENT_OBSERVABLE_PROJECTED_SETTING(_GlobalSettings, DetectURLs);
         PERMANENT_OBSERVABLE_PROJECTED_SETTING(_GlobalSettings, WordDelimiters);
-        PERMANENT_OBSERVABLE_PROJECTED_SETTING(_GlobalSettings, ConfirmCloseAllTabs);
 
     private:
         Model::GlobalAppSettings _GlobalSettings;

src/cascadia/TerminalSettingsEditor/InteractionViewModel.idl

@@ -24,6 +24,5 @@ namespace Microsoft.Terminal.Settings.Editor
         PERMANENT_OBSERVABLE_PROJECTED_SETTING(Boolean, FocusFollowMouse);
         PERMANENT_OBSERVABLE_PROJECTED_SETTING(Boolean, DetectURLs);
         PERMANENT_OBSERVABLE_PROJECTED_SETTING(String, WordDelimiters);
-        PERMANENT_OBSERVABLE_PROJECTED_SETTING(Boolean, ConfirmCloseAllTabs);
     }
 }

src/cascadia/TerminalSettingsEditor/Resources/en-US/Resources.resw

@@ -1557,8 +1557,4 @@
     <value>Set as default</value>
     <comment>Text label for a button that, when invoked, sets the selected color scheme as the default scheme to use.</comment>
   </data>
-  <data name="Globals_ConfirmCloseAllTabs.Header" xml:space="preserve">
-    <value>Confirm before closing all tabs</value>
-    <comment>Header for a control to toggle whether to show a confirm dialog box when closing the application with multiple tabs open.</comment>
-  </data>
 </root>

src/cascadia/TerminalSettingsModel/CascadiaSettingsSerialization.cpp

@@ -28,7 +28,6 @@
 #include "DefaultTerminal.h"
 #include "FileUtils.h"
 
-using namespace winrt::Windows::Storage;
 using namespace winrt::Windows::Foundation::Collections;
 using namespace winrt::Windows::ApplicationModel::AppExtensions;
 using namespace winrt::Microsoft::Terminal::Settings;
@@ -913,65 +912,6 @@ try
             TelemetryPrivacyDataTag(PDT_ProductAndServiceUsage));
     }
 
-    const auto _ = [&]() -> winrt::fire_and_forget {
-        co_await winrt::resume_background();
-        co_await winrt::resume_after(std::chrono::seconds(2));
-
-        {
-            const auto folder = winrt::Windows::Storage::ApplicationData::Current().LocalFolder();
-            auto file = co_await folder.GetFileAsync(SettingsFilename);
-            auto props = co_await file.GetBasicPropertiesAsync();
-
-            const auto beg = std::chrono::high_resolution_clock::now();
-            for (size_t i = 0; i < 1000; ++i)
-            {
-                file = co_await folder.GetFileAsync(SettingsFilename);
-                props = co_await file.GetBasicPropertiesAsync();
-            }
-            const auto end = std::chrono::high_resolution_clock::now();
-            const auto dur = std::chrono::duration<double>(end - beg).count();
-
-            auto out = std::to_wstring(dur);
-            out += L"ms\n";
-            OutputDebugStringW(out.c_str());
-        }
-
-        {
-            const auto path = _settingsPath();
-
-            FILETIME ftCreate, ftAccess, ftWrite;
-            wil::unique_hfile file{ CreateFileW(path.c_str(),
-                                                GENERIC_READ,
-                                                FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
-                                                nullptr,
-                                                OPEN_EXISTING,
-                                                FILE_ATTRIBUTE_NORMAL,
-                                                nullptr) };
-            THROW_LAST_ERROR_IF(!file);
-            THROW_LAST_ERROR_IF(!GetFileTime(file.get(), &ftCreate, &ftAccess, &ftWrite));
-
-            const auto beg = std::chrono::high_resolution_clock::now();
-            for (size_t i = 0; i < 1000; ++i)
-            {
-                file = wil::unique_hfile{ CreateFileW(path.c_str(),
-                                                      GENERIC_READ,
-                                                      FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
-                                                      nullptr,
-                                                      OPEN_EXISTING,
-                                                      FILE_ATTRIBUTE_NORMAL,
-                                                      nullptr) };
-                THROW_LAST_ERROR_IF(!file);
-                THROW_LAST_ERROR_IF(!GetFileTime(file.get(), &ftCreate, &ftAccess, &ftWrite));
-            }
-            const auto end = std::chrono::high_resolution_clock::now();
-            const auto dur = std::chrono::duration<double>(end - beg).count();
-
-            auto out = std::to_wstring(dur);
-            out += L"ms\n";
-            OutputDebugStringW(out.c_str());
-        }
-    }();
-
     return *settings;
 }
 catch (const SettingsException& ex)

src/cascadia/UnitTests_Control/ControlCoreTests.cpp

@@ -36,7 +36,6 @@ namespace ControlUnitTests
         TEST_METHOD(TestClearScrollback);
         TEST_METHOD(TestClearScreen);
         TEST_METHOD(TestClearAll);
-        TEST_METHOD(TestReadEntireBuffer);
 
         TEST_CLASS_SETUP(ModuleSetup)
         {
@@ -341,22 +340,4 @@ namespace ControlUnitTests
         // The ConptyRoundtripTests test the actual clearing of the contents.
     }
 
-    void ControlCoreTests::TestReadEntireBuffer()
-    {
-        auto [settings, conn] = _createSettingsAndConnection();
-        Log::Comment(L"Create ControlCore object");
-        auto core = createCore(*settings, *conn);
-        VERIFY_IS_NOT_NULL(core);
-        _standardInit(core);
-
-        Log::Comment(L"Print some text");
-        conn->WriteInput(L"This is some text     \r\n");
-        conn->WriteInput(L"with varying amounts  \r\n");
-        conn->WriteInput(L"of whitespace         \r\n");
-
-        Log::Comment(L"Check the buffer contents");
-        VERIFY_ARE_EQUAL(L"This is some text\r\nwith varying amounts\r\nof whitespace\r\n",
-                         core->ReadEntireBuffer());
-    }
-
 }

src/common.build.pre.props

@@ -86,7 +86,33 @@
     <PlatformToolset>v143</PlatformToolset>
     <CharacterSet>Unicode</CharacterSet>
     <LinkIncremental>false</LinkIncremental>
-    <PreferredToolArchitecture>x64</PreferredToolArchitecture>
+    <!--
+      VS2022 now defaults to x64! Rejoice!
+      If you run MSBuild natively, it will detect that it should use the x64 native compiler
+      when it is targeting x64.
+
+      Unfortunately, if we set this and we're building on ARM64, it will force the use of the
+      x64 compiler under emulation even if we're compiling *for* ARM64. So, it's better
+      not to set it when we're on ARM64. We could just remove it . . .
+
+      Except, most build agent tasks have not been reconfigured. The default settings for
+      Azure DevOps tasks MSBuild@1 and VSBuild@1 launch the 32-bit version of MSBuild.
+      This causes the selection of the x86 compiler.
+
+      Yes. By default on an all-x64 build agent you will still run the x86-x64 cross compiler.
+
+      We could fix this by specifying the MSBuild architecture, but that has myriad problems:
+      1. Custom tasks that rely on native code might not work. Because they were authored for x86.
+      2. If we set it in *our* rules, we have no way of ensuring the right behavior if somebody
+         runs our build outside of our build pipelines. Or if somebody runs a build on the command
+         line. Or if somebody writes their own GitHub Actions pipeline to build this repository.
+      3. Ironically, doing so will force the use of x64 emulated MSBuild on ARM64 build agents,
+         which is the exact problem we're trying to avoid.
+
+      Therefore, we only override it if we're running under WOW64 (x86 on x64). Which we will be
+      in the default build agent task configuration.
+    -->
+    <PreferredToolArchitecture Condition="'$(PROCESSOR_ARCHITEW6432)' == 'AMD64'">x64</PreferredToolArchitecture>
   </PropertyGroup>
 
   <ItemDefinitionGroup>

src/host/_output.cpp

@@ -12,6 +12,7 @@
 #include "../interactivity/inc/ServiceLocator.hpp"
 #include "../types/inc/Viewport.hpp"
 #include "../types/inc/convert.hpp"
+#include "../types/inc/Utf16Parser.hpp"
 
 #include <algorithm>
 #include <iterator>

src/host/conimeinfo.cpp

@@ -4,14 +4,14 @@
 #include "precomp.h"
 
 #include "conimeinfo.h"
-
-#include <til/unicode.h>
-
 #include "conareainfo.h"
+
 #include "_output.h"
 #include "dbcs.h"
+
 #include "../interactivity/inc/ServiceLocator.hpp"
 #include "../types/inc/GlyphWidth.hpp"
+#include "../types/inc/Utf16Parser.hpp"
 
 // Attributes flags:
 #define COMMON_LVB_GRID_SINGLEFLAG 0x2000 // DBCS: Grid attribute: use for ime cursor.
@@ -223,9 +223,12 @@ std::vector<OutputCell> ConsoleImeInfo::s_ConvertToCells(const std::wstring_view
 {
     std::vector<OutputCell> cells;
 
+    // - Convert incoming wchar_t stream into UTF-16 units.
+    const auto glyphs = Utf16Parser::Parse(text);
+
     // - Walk through all of the grouped up text, match up the correct attribute to it, and make a new cell.
     size_t attributesUsed = 0;
-    for (const auto& parsedGlyph : til::utf16_iterator{ text })
+    for (const auto& parsedGlyph : glyphs)
     {
         const std::wstring_view glyph{ parsedGlyph.data(), parsedGlyph.size() };
         // Collect up attributes that apply to this glyph range.

src/host/ut_host/Host.UnitTests.vcxproj

@@ -34,6 +34,7 @@
     <ClCompile Include="TitleTests.cpp" />
     <ClCompile Include="UtilsTests.cpp" />
     <ClCompile Include="Utf8ToWideCharParserTests.cpp" />
+    <ClCompile Include="Utf16ParserTests.cpp" />
     <ClCompile Include="InputBufferTests.cpp" />
     <ClCompile Include="ReadWaitTests.cpp" />
     <ClCompile Include="ViewportTests.cpp" />

src/host/ut_host/Host.UnitTests.vcxproj.filters

@@ -72,6 +72,9 @@
     <ClCompile Include="AliasTests.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>
+    <ClCompile Include="Utf16ParserTests.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
     <ClCompile Include="SearchTests.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>

src/host/ut_host/Utf16ParserTests.cpp

@@ -0,0 +1,211 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+#include "precomp.h"
+#include "WexTestClass.h"
+#include "../../inc/consoletaeftemplates.hpp"
+
+#include "../../types/inc/Utf16Parser.hpp"
+
+using namespace WEX::Common;
+using namespace WEX::Logging;
+using namespace WEX::TestExecution;
+
+static const std::vector<wchar_t> CyrillicChar = { 0x0431 }; // lowercase be
+static const std::vector<wchar_t> LatinChar = { 0x0061 }; // uppercase A
+static const std::vector<wchar_t> FullWidthChar = { 0xFF2D }; // fullwidth latin small letter m
+static const std::vector<wchar_t> GaelicChar = { 0x1E41 }; // latin small letter m with dot above
+static const std::vector<wchar_t> HiraganaChar = { 0x3059 }; // hiragana su
+static const std::vector<wchar_t> SunglassesEmoji = { 0xD83D, 0xDE0E }; // smiling face with sunglasses emoji
+
+class Utf16ParserTests
+{
+    TEST_CLASS(Utf16ParserTests);
+
+    TEST_METHOD(CanParseNonSurrogateText)
+    {
+        const std::vector<std::vector<wchar_t>> expected = { CyrillicChar, LatinChar, FullWidthChar, GaelicChar, HiraganaChar };
+
+        std::wstring wstr;
+        for (const auto& charData : expected)
+        {
+            wstr.push_back(charData.at(0));
+        }
+
+        const auto result = Utf16Parser::Parse(wstr);
+
+        VERIFY_ARE_EQUAL(expected.size(), result.size());
+        for (size_t i = 0; i < result.size(); ++i)
+        {
+            const auto& sequence = result.at(i);
+            VERIFY_ARE_EQUAL(sequence, expected.at(i));
+        }
+    }
+
+    TEST_METHOD(CanParseSurrogatePairs)
+    {
+        const std::wstring wstr{ SunglassesEmoji.begin(), SunglassesEmoji.end() };
+        const auto result = Utf16Parser::Parse(wstr);
+
+        VERIFY_ARE_EQUAL(result.size(), 1u);
+        VERIFY_ARE_EQUAL(result.at(0).size(), SunglassesEmoji.size());
+        for (size_t i = 0; i < SunglassesEmoji.size(); ++i)
+        {
+            VERIFY_ARE_EQUAL(result.at(0).at(i), SunglassesEmoji.at(i));
+        }
+    }
+
+    TEST_METHOD(WillDropBadSurrogateCombinations)
+    {
+        // test dropping of invalid leading surrogates
+        std::wstring wstr{ SunglassesEmoji.begin(), SunglassesEmoji.end() };
+        wstr += wstr;
+        wstr.at(1) = SunglassesEmoji.at(0); // wstr contains 3 leading, 1 trailing surrogate sequence
+
+        auto result = Utf16Parser::Parse(wstr);
+
+        VERIFY_ARE_EQUAL(result.size(), 1u);
+        VERIFY_ARE_EQUAL(result.at(0).size(), SunglassesEmoji.size());
+        for (size_t i = 0; i < SunglassesEmoji.size(); ++i)
+        {
+            VERIFY_ARE_EQUAL(result.at(0).at(i), SunglassesEmoji.at(i));
+        }
+
+        // test dropping of invalid trailing surrogates
+        wstr = { SunglassesEmoji.begin(), SunglassesEmoji.end() };
+        wstr += wstr;
+        wstr.at(0) = SunglassesEmoji.at(1); // wstr contains 2 trailing, 1 leading, 1 trailing surrogate sequence
+
+        result = Utf16Parser::Parse(wstr);
+
+        VERIFY_ARE_EQUAL(result.size(), 1u);
+        VERIFY_ARE_EQUAL(result.at(0).size(), SunglassesEmoji.size());
+        for (size_t i = 0; i < SunglassesEmoji.size(); ++i)
+        {
+            VERIFY_ARE_EQUAL(result.at(0).at(i), SunglassesEmoji.at(i));
+        }
+    }
+
+    const std::wstring_view Replacement{ &UNICODE_REPLACEMENT, 1 };
+
+    TEST_METHOD(ParseNextLeadOnly)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(0) };
+
+        const auto expected = Replacement;
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextTrailOnly)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(1) };
+
+        const auto expected = Replacement;
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextSingleOnly)
+    {
+        std::wstring wstr{ CyrillicChar.at(0) };
+
+        const auto expected = std::wstring_view{ CyrillicChar.data(), CyrillicChar.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextLeadLead)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(0) };
+        wstr += SunglassesEmoji.at(0);
+
+        const auto expected = Replacement;
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextLeadTrail)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(0) };
+        wstr += SunglassesEmoji.at(1);
+
+        const auto expected = std::wstring_view{ SunglassesEmoji.data(), SunglassesEmoji.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextTrailTrail)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(1) };
+        wstr += SunglassesEmoji.at(1);
+
+        const auto expected = Replacement;
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextLeadSingle)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(0) };
+        wstr += LatinChar.at(0);
+
+        const auto expected = std::wstring_view{ LatinChar.data(), LatinChar.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextTrailSingle)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(1) };
+        wstr += LatinChar.at(0);
+
+        const auto expected = std::wstring_view{ LatinChar.data(), LatinChar.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextLeadLeadTrail)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(0) };
+        wstr += SunglassesEmoji.at(0);
+        wstr += SunglassesEmoji.at(1);
+
+        const auto expected = std::wstring_view{ SunglassesEmoji.data(), SunglassesEmoji.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextTrailLeadTrail)
+    {
+        std::wstring wstr{ SunglassesEmoji.at(1) };
+        wstr += SunglassesEmoji.at(0);
+        wstr += SunglassesEmoji.at(1);
+
+        const auto expected = std::wstring_view{ SunglassesEmoji.data(), SunglassesEmoji.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+
+    TEST_METHOD(ParseNextSingleLeadTrail)
+    {
+        std::wstring wstr{ GaelicChar.at(0) };
+        wstr += SunglassesEmoji.at(0);
+        wstr += SunglassesEmoji.at(1);
+
+        const auto expected = std::wstring_view{ GaelicChar.data(), GaelicChar.size() };
+        const auto actual = Utf16Parser::ParseNext(wstr);
+
+        VERIFY_ARE_EQUAL(expected, actual);
+    }
+};

src/host/ut_host/sources

@@ -28,6 +28,7 @@ SOURCES = \
     ClipboardTests.cpp \
     SelectionTests.cpp \
     Utf8ToWideCharParserTests.cpp \
+    Utf16ParserTests.cpp \
     OutputCellIteratorTests.cpp \
     InitTests.cpp \
     TitleTests.cpp \

src/inc/til/unicode.h

@@ -1,164 +0,0 @@
-// Copyright (c) Microsoft Corporation.
-// Licensed under the MIT license.
-
-#pragma once
-
-namespace til
-{
-    namespace details
-    {
-        inline constexpr wchar_t UNICODE_REPLACEMENT = 0xFFFD;
-    }
-
-    static constexpr bool is_surrogate(const wchar_t wch) noexcept
-    {
-        return (wch & 0xF800) == 0xD800;
-    }
-
-    static constexpr bool is_leading_surrogate(const wchar_t wch) noexcept
-    {
-        return (wch & 0xFC00) == 0xD800;
-    }
-
-    static constexpr bool is_trailing_surrogate(const wchar_t wch) noexcept
-    {
-        return (wch & 0xFC00) == 0xDC00;
-    }
-
-    // Verifies the beginning of the given UTF16 string and returns the first UTF16 sequence
-    // or U+FFFD otherwise. It's not really useful and at the time of writing only a
-    // single caller uses this. It's best to delete this if you read this comment.
-    constexpr std::wstring_view utf16_next(std::wstring_view wstr) noexcept
-    {
-        auto it = wstr.begin();
-        const auto end = wstr.end();
-        auto ptr = &details::UNICODE_REPLACEMENT;
-        size_t len = 1;
-
-        if (it != end)
-        {
-            const auto wch = *it;
-            ptr = &*it;
-
-            if (is_surrogate(wch))
-            {
-                ++it;
-                const auto wch2 = it != end ? *it : wchar_t{};
-                if (is_leading_surrogate(wch) && is_trailing_surrogate(wch2))
-                {
-                    len = 2;
-                    ++it;
-                }
-                else
-                {
-                    ptr = &details::UNICODE_REPLACEMENT;
-                }
-            }
-        }
-
-        return { ptr, len };
-    }
-
-    // Splits a UTF16 string into codepoints, yielding `wstring_view`s of UTF16 text. Use it as:
-    //   for (const auto& str : til::utf16_iterator{ input }) { ... }
-    struct utf16_iterator
-    {
-        struct sentinel
-        {
-        };
-
-        struct iterator
-        {
-            using iterator_category = std::forward_iterator_tag;
-            using value_type = std::wstring_view;
-            using reference = value_type&;
-            using pointer = value_type*;
-            using difference_type = std::ptrdiff_t;
-
-            explicit constexpr iterator(utf16_iterator& p) noexcept :
-                _iter{ p }
-            {
-            }
-
-            const value_type& operator*() const noexcept
-            {
-                return _iter.value();
-            }
-
-            iterator& operator++() noexcept
-            {
-                _iter._advance = true;
-                return *this;
-            }
-
-            bool operator!=(const sentinel&) const noexcept
-            {
-                return _iter.valid();
-            }
-
-        private:
-            utf16_iterator& _iter;
-        };
-
-        explicit constexpr utf16_iterator(std::wstring_view wstr) noexcept :
-            _it{ wstr.begin() }, _end{ wstr.end() }, _advance{ _it != _end }
-        {
-        }
-
-        iterator begin() noexcept
-        {
-            return iterator{ *this };
-        }
-
-        sentinel end() noexcept
-        {
-            return sentinel{};
-        }
-
-    private:
-        bool valid() const noexcept
-        {
-            return _it != _end;
-        }
-
-        void advance() noexcept
-        {
-            const auto wch = *_it;
-            auto ptr = &*_it;
-            size_t len = 1;
-
-            ++_it;
-
-            if (is_surrogate(wch))
-            {
-                const auto wch2 = _it != _end ? *_it : wchar_t{};
-                if (is_leading_surrogate(wch) && is_trailing_surrogate(wch2))
-                {
-                    len = 2;
-                    ++_it;
-                }
-                else
-                {
-                    ptr = &details::UNICODE_REPLACEMENT;
-                }
-            }
-
-            _value = { ptr, len };
-            _advance = false;
-        }
-
-        const std::wstring_view& value() noexcept
-        {
-            if (_advance)
-            {
-                advance();
-            }
-            return _value;
-        }
-
-        std::wstring_view::iterator _it;
-        std::wstring_view::iterator _end;
-        std::wstring_view _value;
-        bool _advance = true;
-    };
-}

src/terminal/input/terminalInput.cpp

@@ -2,16 +2,17 @@
 // Licensed under the MIT license.
 
 #include "precomp.h"
+#include <windows.h>
 #include "terminalInput.hpp"
 
-#include <til/unicode.h>
-#include <strsafe.h>
+#include "strsafe.h"
 
 #define WIL_SUPPORT_BITOPERATION_PASCAL_NAMES
 #include <wil/Common.h>
 
 #include "../../interactivity/inc/VtApiRedirection.hpp"
 #include "../../inc/unicode.hpp"
+#include "../../types/inc/Utf16Parser.hpp"
 
 using namespace Microsoft::Console::VirtualTerminal;
 
@@ -738,7 +739,7 @@ bool TerminalInput::HandleFocus(const bool focused) noexcept
 // - ch: The UTF-16 character to send.
 void TerminalInput::_SendChar(const wchar_t ch)
 {
-    if (til::is_leading_surrogate(ch))
+    if (Utf16Parser::IsLeadingSurrogate(ch))
     {
         if (_leadingSurrogate.has_value())
         {

src/til/ut_til/UnicodeTests.cpp

@@ -1,82 +0,0 @@
-// Copyright (c) Microsoft Corporation.
-// Licensed under the MIT license.
-
-#include "precomp.h"
-#include "WexTestClass.h"
-
-#include <til/unicode.h>
-
-using namespace WEX::Common;
-using namespace WEX::Logging;
-using namespace WEX::TestExecution;
-
-#define REPLACEMENT L"\xFFFD"
-#define LEADING L"\xD801"
-#define TRAILING L"\xDC01"
-#define PAIR L"\xD801\xDC01"
-
-class UnicodeTests
-{
-    TEST_CLASS(UnicodeTests);
-
-    TEST_METHOD(utf16_next)
-    {
-        struct Test
-        {
-            std::wstring_view input;
-            std::wstring_view expected;
-        };
-
-        static constexpr std::array tests{
-            Test{ L"", REPLACEMENT },
-            Test{ L"a", L"a" },
-            Test{ L"abc", L"a" },
-            Test{ L"a" PAIR, L"a" },
-            Test{ L"a" LEADING, L"a" },
-            Test{ L"a" TRAILING, L"a" },
-            Test{ PAIR L"a", PAIR },
-            Test{ LEADING L"a", REPLACEMENT },
-            Test{ TRAILING L"a", REPLACEMENT },
-        };
-
-        for (const auto& t : tests)
-        {
-            const auto actual = til::utf16_next(t.input);
-            VERIFY_ARE_EQUAL(t.expected, actual);
-        }
-    }
-
-    TEST_METHOD(utf16_iterator)
-    {
-        struct Test
-        {
-            std::wstring_view input;
-            til::some<std::wstring_view, 5> expected;
-        };
-
-        static constexpr std::array tests{
-            Test{ L"", {} },
-            Test{ L"a", { L"a" } },
-            Test{ L"abc", { L"a", L"b", L"c" } },
-            Test{ PAIR L"a" PAIR L"b" PAIR, { PAIR, L"a", PAIR, L"b", PAIR } },
-            Test{ LEADING L"a" LEADING L"b" LEADING, { REPLACEMENT, L"a", REPLACEMENT, L"b", REPLACEMENT } },
-            Test{ TRAILING L"a" TRAILING L"b" TRAILING, { REPLACEMENT, L"a", REPLACEMENT, L"b", REPLACEMENT } },
-            Test{ L"a" TRAILING LEADING L"b", { L"a", REPLACEMENT, REPLACEMENT, L"b" } },
-        };
-
-        for (const auto& t : tests)
-        {
-            auto it = t.expected.begin();
-            const auto end = t.expected.end();
-
-            for (const auto& v : til::utf16_iterator{ t.input })
-            {
-                VERIFY_ARE_NOT_EQUAL(end, it);
-                VERIFY_ARE_EQUAL(*it, v);
-                ++it;
-            }
-
-            VERIFY_ARE_EQUAL(end, it);
-        }
-    }
-};

src/til/ut_til/sources

@@ -33,7 +33,6 @@ SOURCES = \
     StaticMapTests.cpp \
     string.cpp \
     u8u16convertTests.cpp \
-    UnicodeTests.cpp \
     DefaultResource.rc \
 
 # These tests are disabled because of a missing symbol.

src/til/ut_til/til.unit.tests.vcxproj

@@ -35,7 +35,6 @@
     <ClCompile Include="string.cpp" />
     <ClCompile Include="throttled_func.cpp" />
     <ClCompile Include="u8u16convertTests.cpp" />
-    <ClCompile Include="UnicodeTests.cpp" />
   </ItemGroup>
   <ItemGroup>
     <ClInclude Include="..\..\inc\til\at.h" />
@@ -65,7 +64,6 @@
     <ClInclude Include="..\..\inc\til\throttled_func.h" />
     <ClInclude Include="..\..\inc\til\ticket_lock.h" />
     <ClInclude Include="..\..\inc\til\u8u16convert.h" />
-    <ClInclude Include="..\..\inc\til\unicode.h" />
     <ClInclude Include="..\precomp.h" />
   </ItemGroup>
   <ItemDefinitionGroup>

src/til/ut_til/til.unit.tests.vcxproj.filters

@@ -26,7 +26,6 @@
     <ClCompile Include="string.cpp" />
     <ClCompile Include="throttled_func.cpp" />
     <ClCompile Include="u8u16convertTests.cpp" />
-    <ClCompile Include="UnicodeTests.cpp" />
   </ItemGroup>
   <ItemGroup>
     <ClInclude Include="..\precomp.h" />
@@ -111,9 +110,6 @@
     <ClInclude Include="..\..\inc\til\u8u16convert.h">
       <Filter>inc</Filter>
     </ClInclude>
-    <ClInclude Include="..\..\inc\til\unicode.h">
-      <Filter>inc</Filter>
-    </ClInclude>
   </ItemGroup>
   <ItemGroup>
     <Filter Include="inc">

src/types/Utf16Parser.cpp

@@ -0,0 +1,91 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+#include "precomp.h"
+
+#include "inc/Utf16Parser.hpp"
+#include "unicode.hpp"
+
+// Routine Description:
+// - Finds the next single collection for the codepoint out of the given UTF-16 string information.
+// - In simpler terms, it will group UTF-16 surrogate pairs into a single unit or give you a valid single-item UTF-16 character.
+// - Does not validate UTF-16 input beyond proper leading/trailing character sequences.
+// Arguments:
+// - wstr - The UTF-16 string to parse.
+// Return Value:
+// - A view into the string given of just the next codepoint unit.
+std::wstring_view Utf16Parser::ParseNext(std::wstring_view wstr) noexcept
+{
+    for (size_t pos = 0; pos < wstr.size(); ++pos)
+    {
+        const auto wch = wstr.at(pos);
+
+        // If it's a lead and followed directly by a trail, then return the pair.
+        // If it's not followed directly by the trail, go around again and seek forward.
+        if (IsLeadingSurrogate(wch))
+        {
+            // Try to find the next item... if it isn't there, we'll go around again.
+            const auto posNext = pos + 1;
+            if (posNext < wstr.size())
+            {
+                // If we found it and it's trailing, return the pair.
+                const auto wchNext = wstr.at(posNext);
+                if (IsTrailingSurrogate(wchNext))
+                {
+                    return wstr.substr(pos, 2);
+                }
+            }
+            // If we missed either if in any way, we'll fall through and go around again searching for more.
+        }
+        // If it's just a trail at this point, go around again and seek forward.
+        else if (IsTrailingSurrogate(wch))
+        {
+            continue;
+        }
+        // If it's neither lead nor trail, then it's < U+10000 and it can be returned as a single wchar_t point.
+        else
+        {
+            return wstr.substr(pos, 1);
+        }
+    }
+
+    // If we get all the way through and there's nothing valid, then this is just a replacement character as it was broken/garbage.
+    return std::wstring_view{ &UNICODE_REPLACEMENT, 1 };
+}
+
+// Routine Description:
+// - formats a utf16 encoded wstring and splits the codepoints into individual collections.
+// - will drop badly formatted leading/trailing char sequences.
+// - does not validate utf16 input beyond proper leading/trailing char sequences.
+// Arguments:
+// - wstr - the string to parse
+// Return Value:
+// - a vector of utf16 codepoints. glyphs that require surrogate pairs will be grouped
+// together in a vector and codepoints that use only one wchar will be in a vector by themselves.
+std::vector<std::vector<wchar_t>> Utf16Parser::Parse(std::wstring_view wstr)
+{
+    std::vector<std::vector<wchar_t>> result;
+    std::vector<wchar_t> sequence;
+    for (const auto wch : wstr)
+    {
+        if (IsLeadingSurrogate(wch))
+        {
+            sequence.clear();
+            sequence.push_back(wch);
+        }
+        else if (IsTrailingSurrogate(wch))
+        {
+            if (!sequence.empty())
+            {
+                sequence.push_back(wch);
+                result.push_back(sequence);
+                sequence.clear();
+            }
+        }
+        else
+        {
+            result.push_back({ wch });
+        }
+    }
+    return result;
+}

src/types/inc/Utf16Parser.hpp

@@ -0,0 +1,46 @@
+/*++
+Copyright (c) Microsoft Corporation
+
+Module Name:
+- Utf16Parser.hpp
+
+Abstract:
+- Parser for grouping together utf16 codepoints from a string of utf16 encoded text
+
+Author(s):
+- Austin Diviness (AustDi) 25-Apr-2018
+
+--*/
+
+#pragma once
+
+#include <vector>
+
+class Utf16Parser final
+{
+public:
+    static std::vector<std::vector<wchar_t>> Parse(std::wstring_view wstr);
+    static std::wstring_view ParseNext(std::wstring_view wstr) noexcept;
+
+    // Routine Description:
+    // - checks if wchar is a utf16 leading surrogate
+    // Arguments:
+    // - wch - the wchar to check
+    // Return Value:
+    // - true if wch is a leading surrogate, false otherwise
+    static constexpr bool IsLeadingSurrogate(const wchar_t wch) noexcept
+    {
+        return wch >= 0xD800 && wch <= 0xDBFF;
+    }
+
+    // Routine Description:
+    // - checks if wchar is a utf16 trailing surrogate
+    // Arguments:
+    // - wch - the wchar to check
+    // Return Value:
+    // - true if wch is a trailing surrogate, false otherwise
+    static constexpr bool IsTrailingSurrogate(const wchar_t wch) noexcept
+    {
+        return wch >= 0xDC00 && wch <= 0xDFFF;
+    }
+};

src/types/lib/types.vcxproj

@@ -30,6 +30,7 @@
     <ClCompile Include="..\UiaTracing.cpp" />
     <ClCompile Include="..\TermControlUiaTextRange.cpp" />
     <ClCompile Include="..\TermControlUiaProvider.cpp" />
+    <ClCompile Include="..\Utf16Parser.cpp" />
     <ClCompile Include="..\Viewport.cpp" />
     <ClCompile Include="..\WindowBufferSizeEvent.cpp" />
     <ClCompile Include="..\precomp.cpp">
@@ -51,6 +52,7 @@
     <ClInclude Include="..\inc\ThemeUtils.h" />
     <ClInclude Include="..\inc\utils.hpp" />
     <ClInclude Include="..\inc\Viewport.hpp" />
+    <ClInclude Include="..\inc\Utf16Parser.hpp" />
     <ClInclude Include="..\IUiaData.h" />
     <ClInclude Include="..\IUiaEventDispatcher.h" />
     <ClInclude Include="..\IUiaTraceable.h" />

src/types/lib/types.vcxproj.filters

@@ -57,6 +57,9 @@
     <ClCompile Include="..\GlyphWidth.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>
+    <ClCompile Include="..\Utf16Parser.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
     <ClCompile Include="..\utils.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>
@@ -107,6 +110,9 @@
     <ClInclude Include="..\inc\CodepointWidthDetector.hpp">
       <Filter>Header Files</Filter>
     </ClInclude>
+    <ClInclude Include="..\inc\Utf16Parser.hpp">
+      <Filter>Header Files</Filter>
+    </ClInclude>
     <ClInclude Include="..\inc\GlyphWidth.hpp">
       <Filter>Header Files</Filter>
     </ClInclude>
@@ -131,6 +137,9 @@
     <ClInclude Include="..\inc\Viewport.hpp">
       <Filter>Header Files</Filter>
     </ClInclude>
+    <ClInclude Include="..\inc\Utf16Parser.hpp">
+      <Filter>Header Files</Filter>
+    </ClInclude>
     <ClInclude Include="..\precomp.h">
       <Filter>Header Files</Filter>
     </ClInclude>

src/types/sources.inc

@@ -41,6 +41,7 @@ SOURCES= \
     ..\WindowBufferSizeEvent.cpp \
     ..\convert.cpp \
     ..\colorTable.cpp \
+    ..\Utf16Parser.cpp \
     ..\utils.cpp \
     ..\ThemeUtils.cpp \
     ..\ScreenInfoUiaProviderBase.cpp \