- Various spelling fixes
- Refresh metadata (including dictionaries)
- Upgrade to v0.0.25
## Validation Steps Performed
- check-spelling has been automatically testing this repository for a
while now on a daily basis to ensure that it works fairly reliably:
https://github.com/check-spelling-sandbox/autotest-check-spelling/actions/workflows/microsoft-terminal-spelling2.yml
Specific in-code fixes:
- winget
- whereas
- tl;dr
- set up
- otherwise,
- more,
- macbook
- its
- invalid
- in order to
- if
- if the
- for this tab,...
- fall back
- course,
- cch
- aspect
- archaeologists
- an
- all at once
- a
- `...`
- ; otherwise,
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
## Summary of the Pull Request
Updates the New Tab Menu's Match Profiles entry to support regex instead
of doing a direct match. Also adds validation to ensure the regex is
valid. Updated the UI to help make it more clear that this supports
regexes and even added a link to some helpful docs.
## Validation Steps Performed
✅ Invalid regex displays a warning
✅ Valid regex works nicely
✅ profile matcher with source=`Windows.Terminal.VisualStudio` still
works as expected
## PR Checklist
Closes#18553
The `SGR 1` VT attribute can either be interpreted as a brighter color,
or as a bolder font, depending on the _Intense text style_ setting.
However, the concept of brightness only applies to the eight standard
ANSI colors, so when `SGR 1` is configured as _bright_, it has no effect
on the ITU T.416 colors (RGB and the 256 index colors).
To address that, we now interpret `SGR 1` as a bolder font when applied
to ITU colors, regardless of whether the _Intense text style_ option is
set to bold or not.
Note that this only applies to the Atlas render engine, since the GDI
engine doesn't support bold fonts.
## Validation Steps Performed
I've manually tested `SGR 1` applied to different color formats with the
_Intense text style_ option set to _None_, and confirmed that the text
is now rendered with a bold font for ITU colors, but not for ANSI/AIX
colors.
Closes#18284
This PR fixes two cases where image content wasn't correctly erased when
overwritten.
1. When legacy console APIs fill an area of the buffer using a starting
coordinate and a length, the affected area could potentially wrap over
multiple rows, but we were only erasing the overwritten image content on
the first affected row.
2. When copying an area of the buffer with text content over another
area that contained image content, the image in the target area would
sometimes not be erased, because we ignored the `_eraseCells` return
value which indicated that the image slice needed to be removed.
## References and Relevant Issues
The original code was from the Sixel implementation in PR #17421.
## Validation Steps Performed
I've manually verified that these two cases are now working as expected.
## PR Checklist
- [x] Closes#18568
After taking in 1.22, our CodeQL process caught a few locations where we
weren't following the right guidance:
- Performing integer comparisons of different sizes which could lead to
an infinite loop if the larger integer goes out of range of the smaller
integer
- Not checking HResult of a called method
Co-authored-by: aphistra <102989060+aphistra@users.noreply.github.com>
Selection is generally stored as an inclusive start and end. This PR
makes the end exclusive which now allows degenerate selections, namely
in mark mode. This also modifies mouse selection to round to the nearest
cell boundary (see #5099) and improves word boundaries to be a bit more
modern and make sense for degenerate selections (similar to #15787).
Closes#5099Closes#13447Closes#17892
## Detailed Description of the Pull Request / Additional comments
- Buffer, Viewport, and Point
- Introduced a few new functions here to find word boundaries, delimiter
class runs, and glyph boundaries.
- 📝These new functions should be able to replace a few other functions
(i.e. `GetWordStart` --> `GetWordStart2`). That migration is going to be
a part of #4423 to reduce the risk of breaking UIA.
- Viewport: added a few functions to handle navigating the _exclusive_
bounds (namely allowing RightExclusive as a position for buffer
coordinates). This is important for selection to be able to highlight
the entire line.
- 📝`BottomInclusiveRightExclusive()` will replace `EndExclusive` in the
UIA code
- Point: `iterate_rows_exclusive` is similar to `iterate_rows`, except
it has handling for RightExclusive
- Renderer
- Use `iterate_rows_exclusive` for proper handling (this actually fixed
a lot of our issues)
- Remove some workarounds in `_drawHighlighted` (this is a boundary
where we got inclusive coords and made them exclusive, but now we don't
need that!)
- Terminal
- fix selection marker rendering
- `_ConvertToBufferCell()`: add a param to allow for RightExclusive or
clamp it to RightInclusive (original behavior). Both are useful!
- Use new `GetWordStart2` and `GetWordEnd2` to improve word boundaries
and make them feel right now that the selection an exclusive range.
- Convert a few `IsInBounds` --> `IsInExclusiveBounds` for safety and
correctness
- Add `TriggerSelection` to `SelectNewRegion`
- 📝 We normally called `TriggerSelection` in a different layer, but it
turns out, UIA's `Select` function wouldn't actually update the
renderer. Whoops! This fixes that.
- TermControl
- `_getTerminalPosition` now has a new param to round to the nearest
cell (see #5099)
- UIA
- `TermControlUIAProvider::GetSelectionRange` no need to convert from
inclusive range to exclusive range anymore!
- `TextBuffer::GetPlainText` now works on an exclusive range, so no need
to convert the range anymore!
## Validation Steps Performed
This fundamental change impacts a lot of scenarios:
- ✅Rendering selections
- ✅Selection markers
- ✅Copy text
- ✅Session restore
- ✅Mark mode navigation (i.e. character, word, line, buffer)
- ✅Mouse selection (i.e. click+drag, shift+click, multi-click,
alt+click)
- ✅Hyperlinks (interaction and rendering)
- ✅Accessibility (i.e. get selection, movement, text extraction,
selecting text)
- [ ] Prev/Next Command/Output (untested)
- ✅Unit tests
## Follow-ups
- Refs #4423
- Now that selection and UIA are both exclusive ranges, it should be a
lot easier to deduplicate code between selection and UIA. We should be
able to remove `EndExclusive` as well when we do that. This'll also be
an opportunity to modernize that code and use more `til` classes.
## Summary of the Pull Request
This extends the copy command to be able to include control sequences,
for use in tools that subsequently know how to parse and display that.
## References and Relevant Issues
https://github.com/microsoft/terminal/issues/15703
## Detailed Description of the Pull Request / Additional comments
At a high level, this:
- Expands the `CopyTextArgs` to have a `withControlSequences` bool.
- Plumbs that bool down through many layers to where we actuall get
data out of the text buffer.
- Modifies the existing `TextBuffer::Serialize` to be more generic
and renames it to `TextBuffer::ChunkedSerialize`.
- Uses the new `ChunkedSerialize` to generate the data for the copy
request.
## Validation Steps Performed
To test this I've manually:
- Generated some styled terminal contents, copied it with the control
sequences, pasted it into a file, `cat`ed the file and seen that it
looks the same.
- Set `"firstWindowPreference": "persistedWindowLayout"` and
validated that the contents of windows are saved and
restored with styling intact.
I also checked that `Invoke-OpenConsoleTests` passed.
## PR Checklist
- [x] Closes#15703
- [ ] Tests added/passed
- [x] Documentation updated
- If checked, please file a pull request on [our docs
repo](https://github.com/MicrosoftDocs/terminal) and link it here:
https://github.com/MicrosoftDocs/terminal/pull/756
- [x] Schema updated (if necessary)
* Don't reset the position entirely when changing the needle
* Don't change the scroll position when output arrives
* Don't interfere with the search when output arrives constantly
Closes#17301
## Validation Steps Performed
* In pwsh, run `10000..20000 | % { sleep 0.25; $_ }`
* You can search for e.g. `1004` and it'll find 10 results. ✅
* You can scroll up and down past it and it won't snap back
when new output arrives. ✅
* `while ($true) { Write-Host -NoNewline "`e[Ha"; sleep 0.0001; }`
* You can cycle between the hits effortlessly. ✅ (This tests that
the constantly reset `OutputIdle` event won't interfere.)
* On input change, the focused result is near the previous one. ✅
Because `_layoutLine` would never return `column == columnLimit` for
control character visualizers, we'd get a deadlock in `_redisplay`,
as it tries to fill the line until it's full, but never achieve it.
Closes#17893
## Validation Steps Performed
* Press Ctrl-A to insert "^A"
* Press Home to get to the start of the prompt
* Press and hold "A" until the line wraps
* The line wraps and there's no deadlock ✅
This is particularly relevant to pwsh with the "ghost text" enabled. In
that scenario, pwsh writes out the predicted command to the right of the
cursor. With `showSuggestions(useCommandline=true)`, we'd auto-include
that text in the filter, and that was effectively useless.
This instead defaults us to not use anything to the right of the cursor
(inclusive) for what we consider "the current commandline"
closes#17772
This pull request adds support for setting and querying the selection
color with `OSC 17`.
To make this possible, I had to move selection color down into the color
table where it always belonged. This lets us get rid of the special
`SetSelectionColor` method from the surface of AtlasEngine, and reunites
selection colors with the rest of the special colors.
When we have a series of image slices of differing widths, which also
don't align with the cell boundaries, we can get rounding errors in the
scaling which makes the different slices appear misaligned.
This PR fixes the issue by removing the 4 pixel width alignment that was
enforced in the `ImageSlice` class, since that's not actually necessary
when the pixels themselves are already 4 bytes in size. And without
that, the widths should be correctly aligned with the cell boundaries.
## References and Relevant Issues
The initial Sixel implementation was added in PR #17421.
## Validation Steps Performed
I've confirmed that this fixes the rendering glitches reported in
#17711, and all my existing Sixel tests still work as expected.
Closes#17711
`RealUnicodeToFalseUnicode` was described as:
> This routine converts a unicode string into the correct characters
> for an OEM (cp 437) font. This code is needed because the gdi glyph
> mapper converts unicode to ansi using codepage 1252 to index font.
> This is how the data is stored internally.
In other words, it takes a UCS2 string, translates it to the current
codepage and translates it back to UCS2 in the US version of Windows.
In the "eastern" DBCS version it "reinterprets" the DBCS string as
`CP_USA` (a particularly weird quirk).
The original implementation used to do this translation at every
opportunity where text went into or out of conhost.
The translation was weird, but it was consistent.
In Windows 10 RS1 conhost got a new UCS2-aware text buffer and
this translation was removed from most places, as the text buffer
was converted to store proper UCS2. This broke the entire concept
of the translation though. Whatever data you previously wrote with
something like `WriteConsoleOutputCharacter` now came back with
something entirely else via `ReadConsoleOutput`.
In other words, I believe past RS1 there was technically never any
point in "munging" `CHAR_INFO`s, as this only covered 2 API functions.
Still, this does mean that this PR represents an API breaking change.
It's a minor one though, because it only affects 2 API functions.
And more importantly, it's a necessary breaking change as we move
further and further away from correlating codepoint and column counts.
## Validation Steps Performed
* Remaining tests pass ✅
Roughly 4 years ago we gave Windows Terminal the ability to
differentiate between black/white and the default colors.
One of the victims was PowerShell and most importantly PSReadLine,
which emit SRG 37 & 40 when what they really want is 38 & 48.
We fixed this on our side by adding a shim.
Since the addition of VT passthrough in #17510 we now intentionally
lost the ability to translate VT sequences from one thing to another.
This meant we also lost the ability to do this shim and as such
this PR removes it. Luckily Windows 11 now ships PSReadLine 2.0.0,
which contains a proper fix for this.
Unfortunately, this is not the case for Windows 10, which ships
PSReadLine 2.0.0-beta2. Users affected by this will have to install
a newer version of PSReadLine or use the default black/white theme.
See 1bf4c082b4Closes#13037
The idea is that we can translate Console API calls directly to VT at
least as well as the current VtEngine setup can. For instance, a call
to `SetConsoleCursorPosition` clearly translates directly to a `CUP`
escape sequence. Effectively, instead of translating output
asynchronously in the renderer thread, we'll do it synchronously
right during the Console API call.
Most importantly, the this means that any VT output that an
application generates will now be given to the terminal unmodified.
Aside from reducing our project's complexity quite a bit and opening
the path towards various interesting work like sixels, Device Control
Strings, buffer snapshotting, synchronized updates, and more, it also
improves performance for mixed text output like enwik8.txt in conhost
to 1.3-2x and in Windows Terminal via ConPTY to roughly 20x.
This adds support for overlapped IO, because now that output cannot
be "skipped" anymore (VtEngine worked like a renderer after all)
it's become crucial to block conhost as little as possible.
⚠️ Intentionally unresolved changes/quirks:
* To force a delayed EOL wrap to wrap, `WriteCharsLegacy` emits a
`\r\n` if necessary. This breaks text reflow on window resize.
We cannot emit ` \r` the way readline does it, because this would
overwrite the first column in the next row with a whitespace.
The alternative is to read back the affected cell from the buffer
and emit that character and its attributes followed by a `\r`.
I chose to not do that, because buffer read-back is lossy (= UCS2).
Unless the window is resized, the difference is unnoticeable
and historically, conhost had no support for buffer reflow anyway.
* If `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is set while
`DISABLE_NEWLINE_AUTO_RETURN` is reset, we'll blindly replace all
LF with CRLF. This may hypothetically break DCS sequences, but it's
the only way to do this without parsing the given VT string and
thus the only way we can achieve passthrough mode in the future.
* `ENABLE_WRAP_AT_EOL_OUTPUT` is translated to `DECAWM`.
Between Windows XP and Windows 11 21H2, `ENABLE_WRAP_AT_EOL_OUTPUT`
being reset would cause the cursor position to reset to wherever
a write started, _if_ the write, including expanded control chars,
was less than 100 characters long. If it was longer than that,
the cursor position would end up in an effectively random position.
After lengthy research I believe that this is a bug introduced in
Windows XP and that the original intention was for this mode to be
equivalent to `DECAWM`. This is compounded by MSDN's description
(emphasis mine):
> If this mode is disabled, the **last character** in the row is
> overwritten with any subsequent characters.
⚠️ Unresolved issues/quirks:
* Focus/Unfocus events are injected into the output stream without
checking whether the VT output is currently in a ground state.
This may break whatever VT sequence is currently ongoing.
This is an existing issue.
* `VtIo::Writer::WriteInfos` should properly verify the width of
each individual character.
* Using `SetConsoleActiveScreenBuffer` destroys surrogate pairs
and extended (VT) attributes. It could be translated to VT pages
in the long term.
* Similarly, `ScrollConsoleScreenBuffer` results in the same and
could be translated to `DECCRA` and `DECFRA` in the near term.
This is important because otherwise `vim` output may loose
its extended attributes during scrolling.
* Reflowing a long line until it wraps results in the cooked read
prompt to be misaligned vertically.
* `SCREEN_INFORMATION::s_RemoveScreenBuffer` should trigger a
buffer switch similar to `SetConsoleActiveScreenBuffer`.
* Translation of `COMMON_LVB_GRID_HORIZONTAL` to `SGR 53` was dropped
and may be reintroduced alongside `UNDERSCORE` = `SGR 4`.
* Move the `OSC 0 ; P t BEL` sequence to `WriteWindowTitle`
and swap the `BEL` with the `ST` (`ESC \`).
* PowerShell on Windows 10 ships with PSReadLine 2.0.0-beta2
which emits SGR 37/40 instead of 39/49. This results in black
spaces when typing and there's no good way to fix that.
* A test is missing that ensures that `FillConsoleOutputCharacterW`
results in a `CSI n J` during the PowerShell shim.
* A test is missing that ensures that `PtySignal::ClearBuffer`
does not result in any VT being generated.
Closes#262Closes#1173Closes#3016Closes#4129Closes#5228Closes#8698Closes#12336Closes#15014Closes#15888Closes#16461Closes#16911Closes#17151Closes#17313
The "copy the remaining attributes" loop assumes that it has full
ownership over the rows that it copies. For that to be true,
we have to of course make sure that the current write-cursor
is at a fresh, new row in the first place.
## Validation Steps Performed
* In a new pwsh tab with 120 colums:
``Write-Host -NoNewline "`e[36m$('a'*120)`e[m"; sleep 10``
* Resize the window wider
* Color doesn't get lost
* Add a revision to `ImageSlice` so that the renderers
can use it to cache them as bitmaps across frames.
* Hooked up the revision tracking to AtlasEngine to cache the
slices into `Buffer`s so we can own them into the `Present`.
* Hooked up those snapshots to BackendD3D with a straightforward
hashmap -> atlas-rect logic. Just like rendering text.
* Hooked up BackendD2D with a bad, but simple & direct drawing logic.
* Bonus: Modify `ImageSlice` to be returned as a raw pointers
as this helps performance slightly. (Trivial type == good.)
* Bonus: Fixed the `_debugShowDirty` code (disabled by default).
## Validation Steps Performed
* `mpv --really-quiet --vo=sixel foo.mp4` looks good ✅
* Scroll up down & observe dirty rects ✅
This abstraction will help #17510 inject its ConPTY-specific behavior
into all 6 relevant console API functions simultaneously. This avoids
having to repeat the same prologue and epilogue 4 times.
Ideally, we'd use composition here, but I found it to be a bad fit.
* Wide glyphs that don't fit into the last column got treated
as narrow glyphs which broke line layout.
* Wide glyphs that don't fit into the last column got manually
padded with whitespace which broke Ctrl+A + Ctrl+C in conhost.
* Sudden increases/decreases in the pager height would leave
parts of the viewport with leftover text and not clear it away.
* Deleting an entire word at the start of a line would only delete
its first two characters.
Closes#17554
STL iterators have a significant overhead. This improves performance
of `GetLastNonSpaceColumn` by >100x (it's too large to measure),
and reflow by ~15x in debug builds. This makes text reflow in debug
builds today ~10x faster than it used to be in release builds before
the large rewrites in #15701 and #13626.
By rewriting `COOKED_READ_DATA` to use VT for its output we make it
possible to pass this VT output 1:1 straight to the hosting terminal
if we're running under ConPTY. This is also possible with the current
console APIs it uses, but it's somewhat janky. In particular the
usage of `ReadConsoleOutput` to backup/restore the popup contents
could be considered bad faith "rules for thee, not for me",
given that we're telling people to move away from those APIs.
The new implementation contains a bare bones "pager" to fit even
very long prompt contents into the VT viewport.
I fully expect this initial PR to not be entirely bug free, because
writing a proper pager with line wrapping is a little bit complex.
This PR takes some significant shortcuts by leveraging the fact
that the prompt line is always left-to-right and always a series
of fully filled lines followed by one potentially semi-full line.
This allows us to skip using a front/back-buffer for diffing the
contents between two redisplay calls.
Part of #14000
## Validation Steps Performed
* ASCII input
* Chinese input (中文維基百科) ✅
* Surrogate pair input (🙂) ✅
* In cmd.exe
* Create 2 files: "a😊b.txt" and "a😟b.txt"
* Press tab: Autocomplete to "a😊b.txt" ✅
* Navigate the cursor right past the "a"
* Press tab twice: Autocomplete to "a😟b.txt" ✅
* Execute `printf(" "); gets(buffer);` in C (or equivalent)
* Press Tab, A, Ctrl+V, Tab, A ✅
* The prompt is " A^V A" ✅
* Cursor navigation works ✅
* Backspacing/Deleting random parts of it works ✅
* It never deletes the initial 4 spaces ✅
* Backspace deletes preceding glyphs ✅
* Ctrl+Backspace deletes preceding words ✅
* Escape clears input ✅
* Home navigates to start ✅
* Ctrl+Home deletes text between cursor and start ✅
* End navigates to end ✅
* Ctrl+End deletes text between cursor and end ✅
* Left navigates over previous code points ✅
* Ctrl+Left navigates to previous word-starts ✅
* Right and F1 navigate over next code points ✅
* Pressing right at the end of input copies characters
from the previous command ✅
* Ctrl+Right navigates to next word-ends ✅
* Insert toggles overwrite mode ✅
* Delete deletes next code point ✅
* Up and F5 cycle through history ✅
* Doesn't crash with no history ✅
* Stops at first entry ✅
* Down cycles through history ✅
* Doesn't crash with no history ✅
* Stops at last entry ✅
* PageUp retrieves the oldest command ✅
* PageDown retrieves the newest command ✅
* F2 starts "copy to char" prompt ✅
* Escape dismisses prompt ✅
* Typing a character copies text from the previous command up
until that character into the current buffer (acts identical
to F3, but with automatic character search) ✅
* F3 copies the previous command into the current buffer,
starting at the current cursor position,
for as many characters as possible ✅
* Doesn't erase trailing text if the current buffer
is longer than the previous command ✅
* Puts the cursor at the end of the copied text ✅
* F4 starts "copy from char" prompt ✅
* Escape dismisses prompt ✅
* Erases text between the current cursor position and the
first instance of a given char (but not including it) ✅
* F6 inserts Ctrl+Z ✅
* F7 without modifiers starts "command list" prompt ✅
* Escape dismisses prompt ✅
* Entries wider than the window width are truncated ✅
* Height expands up to 20 rows with longer histories ✅
* F9 starts "command number" prompt ✅
* Left/Right replace the buffer with the given command ✅
* And put cursor at the end of the buffer ✅
* Up/Down navigate selection through history ✅
* Stops at start/end with <10 entries ✅
* Stops at start/end with >20 entries ✅
* Scrolls through the entries if there are too many ✅
* Shift+Up/Down moves history items around ✅
* Home navigates to first entry ✅
* End navigates to last entry ✅
* PageUp navigates by $height items at a time or to first ✅
* PageDown navigates by $height items at a time or to last ✅
* Alt+F7 clears command history ✅
* F8 cycles through commands that start with the same text as
the current buffer up until the current cursor position ✅
* Doesn't crash with no history ✅
* F9 starts "command number" prompt ✅
* Escape dismisses prompt ✅
* Ignores non-ASCII-decimal characters ✅
* Allows entering between 1 and 5 digits ✅
* Pressing Enter fetches the given command from the history ✅
* Alt+F10 clears doskey aliases ✅
* In cmd.exe, with an empty prompt in an empty directory:
Pressing tab produces an audible bing and prints no text ✅
* When Narrator is enabled, in cmd.exe:
* Typing individual characters announces only
exactly each character that is being typed ✅
* Backspacing at the end of a prompt announces
only exactly each deleted character ✅
This removes all of the 2D iteration machinery. Imagine the text buffer
as a `Cell[w][h]` grid. Clearly, this is identical to a `Cell[w*h]`
array, which shows that copying between overlapping ranges only needs
either forward or backward copying, and not left/right/top/down.
With `WalkDir` removed, `WalkInBounds` can be rewritten with basic
arithmetic which allows `pos` to be an exclusive end coordinate.
## Summary of the Pull Request
This PR introduces basic support for the Sixel graphics protocol in
conhost, limited to the GDI renderer.
## References and Relevant Issues
This is a first step towards supporting Sixel graphics in Windows
Terminal (#448), but that will first require us to have some form of
ConPTY passthrough (#1173).
## Detailed Description of the Pull Request / Additional comments
There are three main parts to the architecture:
* The `SixelParser` class takes care of parsing the incoming Sixel `DCS`
sequence.
* The resulting image content is stored in the text buffer in a series
of `ImageSlice` objects, which represent per-row image content.
* The renderer then takes care of painting those image slices for each
affected row.
The parser is designed to support multiple conformance levels so we can
one day provide strict compatibility with the original DEC hardware. But
for now the default behavior is intended to work with more modern Sixel
applications. This is essentially the equivalent of a VT340 with 256
colors, so it should still work reasonably well as a VT340 emulator too.
## Validation Steps Performed
Thanks to the work of @hackerb9, who has done extensive testing on a
real VT340, we now have a fairly good understanding of how the original
Sixel hardware terminals worked, and I've tried to make sure that our
implementation matches that behavior as closely as possible.
I've also done some testing with modern Sixel libraries like notcurses
and jexer, but those typically rely on the terminal implementing certain
proprietary Xterm query sequences which I haven't included in this PR.
---------
Co-authored-by: Dustin L. Howett <dustin@howett.net>
First, this adds `GraphemeTableGen` which
* parses `ucd.nounihan.grouped.xml`
* computes the cluster break property for each codepoint
* computes the East Asian Width property for each codepoint
* compresses everything into a 4-stage trie
* computes a LUT of cluster break rules between 2 codepoints
* and serializes everything to C++ tables and helper functions
Next, this adds `GraphemeTestTableGen` which
* parses `GraphemeBreakTest.txt`
* splits each test into graphemes and break opportunities
* and serializes everything to a C++ table for use as unit tests
`CodepointWidthDetector.cpp` was rewritten from scratch to
* use an iterator struct (`GraphemeState`) to maintain state
* accumulate codepoints until a break opportunity arises
* accumulate the total width of a grapheme
* support 3 different measurement modes: Grapheme clusters,
`wcswidth`-style, and a mode identical to the old conhost
With this in place the following changes were made:
* `ROW::WriteHelper::_replaceTextUnicode` now uses the new
grapheme cluster text iterators
* The same function was modified to join new text with existing
contents of the current cell if they join to form a cluster
* Otherwise, a ton of places were modified to funnel the selection
of the measurement mode over from WT's settings to ConPTY
This is part of #1472
## Validation Steps Performed
* So many tests ✅
* https://github.com/apparebit/demicode works fantastic ✅
* UTF8-torture-test.txt works fantastic ✅
If `VtEngine` gets removed from conhost, we need to be able to run
without any renderer present whatsoever. To make this possible,
I've turned all `Renderer&` into `Renderer*`.
Part of #14000
I think I forgot to complete that section of the code...
The parentheses were missing and `beg` was repeated twice. The last
line in the comment above this explains what I intended it to be.
Closes#17365
## Validation Steps Performed
* In a new PowerShell tab
* Run ``"`e[999C`e[2D`e[42mfoo`e[m"``
* Newline until it scrolls
* Run it again
* Close and reopen
* The green "foo" is still green ✅
`utextAccess` apparently doesn't actually need to clamp the
`chunkOffset` to be in range of the current chunk. Also, I missed to
implement the part of the spec that says to leave the iterator on the
first/last chunk of the `UText` in case of an out-of-bounds index.
This PR fixes the issue by simply not returning early, doing a more
liberal clamp of the offset, and then checking whether it was in range.
As an aside, this also fixes a one-off bug when hovering URLs that
end on the very last cell of the viewport (or are cut off).
Closes#17343
## Validation Steps Performed
* Write an URL that wraps across the last 2 lines in the buffer
* Scroll 1 line up
* No assert ✅
* Hovering the URL shows the full, still visible parts of the URL ✅
It may be more accurate to say: "Fix _known_ remaining buffer
serialization bugs", but I'll try to be positive about my code.
Initially, the buffer is initialized with the default attributes,
but once it begins to scroll, newly scrolled in rows are initialized
with the current attributes. This means we need to set the current
attributes to those of the upcoming row before the row comes up.
This is related to #17074.
## Validation Steps Performed
* Persist and restore a buffer 10 times
* All previous "Restore" status messages look correct ✅
* The escape sequences in the buffer file look correct ✅
This addresses a review comment left by tusharsnx in #17092 which I
forgot to fix before merging the PR. The fix itself is somewhat simple:
`Terminal::SetSearchHighlightFocused` triggers a scroll if the target
is outside of the current (scrolled) viewport and avoiding the call
unless necessary fixes it. To do it properly though, I've split up
`Search::ResetIfStale` into `IsStale` and `Reset`. Now we can properly
detect staleness in advance and branch out the search reset cleanly.
Additionally, I've taken the liberty to replace the `IVector` in
`SearchResultRows` with a direct `const std::vector&` into `Searcher`.
This removes a bunch of code and makes it faster to boot.
## Validation Steps Performed
* Print lots of text
* Search a common letter
* Scroll up
* Doesn't scroll back down ✅
* Hold enter to search more occurrences scrolls up as needed ✅
* `showMarksOnScrollbar` still works ✅
This PR achieves two things:
* When encountering rows with newlines (`WasForceWrapped` = `false`)
we'll now copy the contents out of the row and append a `\n`.
To make `utext_clone` cheap, it adds a reference counted buffer.
* Text extraction in `Terminal::GetHyperlinkAtBufferPosition`
was fixed by using a higher level `TextBuffer::GetPlainText`
instead of iterating through each cell.
Closes#16676Closes#17065
## Validation Steps Performed
* In pwsh execute the following:
``"`e[999C`e[22Dhttps://example.com/foo`nbar"``
* Hovering over the URL only underlines `.../foo` and not `bar` ✅
* The tooltip ends in `.../foo` and not `.../fo` ✅
This PR extends `til::throttled_func` to also support debouncing:
* throttling: "At most 1 call every N seconds"
* debouncing: "Exactly 1 call after N seconds of inactivity"
Based on the latter the following series of changes were made:
* An `OutputIdle` event was added to `ControlCore` which is
raised once there hasn't been any incoming data in 100ms.
This also triggers an update of our regex patterns (URL detection).
* The event is then caught by `TermControl` which calls `Search()`.
* `Search()` in turn was modified to return its results by-value
as a struct, which avoids the need for a search-update event
and simplifies how we update the UI.
This architectural change, most importantly the removal of the
`TextLayoutUpdated` event, fixes a DoS bug in Windows Terminal:
As the event leads to UI thread activity, printing lots of text
continuously results in the UI thread becoming unresponsive.
On top of these, a number of improvements were made:
* `IRenderEngine::InvalidateHighlight` was changed to take the
`TextBuffer` by-reference which avoids the need to accumulate the
line renditions in a `std::vector` first. This improves Debug build
performance during reflow by what I guess must be roughly
a magnitude faster. This difference is very noticeable.
* When closing the search box, `ClearSearch()` is called to remove
the highlights. The search text is restored when it's reopened,
however the current search position isn't.
Closes#17073Closes#17089
## Validation Steps Performed
* UIA announcements:
* Pressing Ctrl+Shift+F the first time does not lead to one ✅
* Typing the first letter does ✅
* Closing doesn't ✅
* Reopening does (as it restores the letter) ✅
* Closing the search box dismisses the highlights ✅
* Resizing the window recalculates the highlights ✅
* Changing the terminal output while the box is open
recalculates the highlights ✅
Next in the popular series of minor refactorings:
Out with the old, in with the new!
This PR removes all of the existing TSF code, both for conhost and
Windows Terminal. conhost's TSF implementation was awful:
It allocated an entire text buffer _per line_ of input.
Additionally, its implementation spanned a whopping 40 files and
almost 5000 lines of code. Windows Terminal's implementation was
absolutely fine in comparison, but it was user unfriendly due to
two reasons: Its usage of the `CoreTextServices` WinRT API indirectly
meant that it used a non-transitory TSF document, which is not the
right choice for a terminal. A `TF_SS_TRANSITORY` document (-context)
indicates to TSF that it cannot undo a previously completed composition
which is exactly what we need: Once composition has completed we send
the result to the shell and we cannot undo this later on.
The WinRT API does not allow us to use `TF_SS_TRANSITORY` and so it's
unsuitable for our application. Additionally, the implementation used
XAML to render the composition instead of being part of our text
renderer, which resulted in the text looking weird and hard to read.
The new implementation spans just 8 files and is ~1000 lines which
should make it significantly easier to maintain. The architecture is
not particularly great, but it's certainly better than what we had.
The implementation is almost entirely identical between both conhost
and Windows Terminal and thus they both also behave identical.
It fixes an uncountable number of subtle bugs in the conhost TSF
implementation, as it failed to check for status codes after calls.
It also adds several new features, like support for wavy underlines
(as used by the Japanese IME), dashed underlines (the default for
various languages now, like Vietnamese), colored underlines,
colored foreground/background controlled by the IME, and more!
I have tried to replicate the following issues and have a high
confidence that they're resolved now:
Closes#1304Closes#3730Closes#4052Closes#5007 (as it is not applicable anymore)
Closes#5110Closes#6186Closes#6192Closes#13805Closes#14349Closes#14407Closes#16180
For the following issues I'm not entirely sure if it'll fix it,
but I suspect it's somewhat likely:
#13681#16305#16817
Lastly, there's one remaining bug that I don't know how to resolve.
However, that issue also plagues conhost and Windows Terminal
right now, so it's at least not a regression:
* Press Win+. (emoji picker) and close it
* Move the window around
* Press Win+.
This will open the emoji picker at the old window location.
It also occurs when the cursor moves within the window.
While this is super annoying, I could not find a way to fix it.
## Validation Steps Performed
* See the above closed issues
* Use Vietnamese Telex and type "xin choaf"
Results in "xin chào" ✅
* Use the MS Japanese IME and press Alt+`
Toggles between the last 2 modes ✅
* Use the MS Japanese IME, type "kyouhaishaheiku", and press Space
* The text is converted, underlined and the first part is
doubly underlined ✅
* Left/Right moves between the 3 segments ✅
* Home/End moves between start/end ✅
* Esc puts a wavy line under the current segment ✅
* Use the Korean IME, type "gksgks"
This results in "한한" ✅
* Use the Korean IME, type "gks", and press Right Ctrl
Opens a popup which allows you to navigate with Arrow/Tab keys ✅
### The changeset involves:
- Decoupling Selection and Search Highlighting code paths.
- We no longer invalidate search highlights when:
- Left-clicking on terminal
- A new selection is made
- Left-clicking on Search-box
- Dispatching Find Next/Prev Match Action. (The search highlight was
removed after pressing the first key of the Action's key combination)
- And, anything that doesn't change buffer content, shouldn't invalidate
the highlighted region (E.g. Cursor movement)
- Highlighting foreground color is *actually* applied to the highlighted
text.
- Double-clicking on SearchBox no longer starts a text selection in the
terminal.
- Selected text is properly populated in the Search Box (#16355)
Closes: #16355
## Some Implementation Details
### Detecting text layout changes in the Control layer
As Search Highlight regions need to be removed when new text is added,
or the existing text is re-arranged due to window resize or similar
events, a new event `TextLayoutUpdated` is added that notifies
`CoreControl` of any text layout changes. The event is used to
invalidate and remove all search highlight regions from the buffer
(because the regions might not be _fresh_ anymore.
The new event is raised when:
1. `AdaptDispatch` writes new text into the buffer.
2. MainBuffer is switched to AltBuffer or vice-versa.
3. The user resized the window.
4. Font size changed.
5. Zoom level changed.
(Intensionally,) It's not raised when:
1. Buffer is scrolled.
2. The text cursor is moved.
When `ControlCore` receives a `TextLayoutUpdated` event, it clears the
Search Highlights in the *render data*, and raises an
`UpdateSearchResults` event to notify `TermControl` to update the Search
UI (`SearchBoxControl`).
In the future, we can use `TextLayoutUpdated` event to start a new
search which would refresh the results automatically after a slight
delay (throttled). *VSCode already does this today*.
### How does AtlasEngine draw the highlighted regions?
We follow a similar idea as for drawing the Selection region. When new
regions are available, the old+new regions are marked invalidated.
Later, a call to `_drawHighlighted()` is made at the end of
`PaintBufferLine()` to override the highlighted regions' colors with
highlight colors. The highlighting colors replace the buffer colors
while search highlights are active.
Note that to paint search highlights, we currently invalidate the row
completely. This forces text shaping for the rows in the viewport that
have at least one highlighted region. This is done to keep the (already
lengthy) PR... simple. We could take advantage of the fact that only
colors have changed and not the characters (or glyphs). I'm expecting
that this could be improved like:
1. When search regions are added, we add the highlighting colors to the
color bitmaps without causing text shaping.
2. When search regions are removed, we re-fill the color bitmaps with
the original colors from the Buffer.
## Validation Steps:
- New text, window resize, font size changes, zooming, and pasting
content into the terminal removes search highlights.
- highlighting colors override the foreground and background color of
the text (in the rendered output).
- Blinking, faded, reverse video, Intense text is highlighted as
expected.
Performance of printing enwik8.txt at the following block sizes:
4KiB (printf): 53MB/s -> 58MB/s
128KiB (cat): 170MB/s -> 235MB/s
This commit is imperfect. Support for more than one rendering
engine was "hacked" into `Renderer` and is not quite correct.
As such, this commit cannot fix cursor invalidation correctly either,
and while some bugs are fixed (engines may see highly inconsistent
TextBuffer and Cursor states), it introduces others (an error in the
first engine may result in the second engine not executing).
Neither of those are good and the underlying issue remains to be fixed.
## Validation Steps Performed
* Seems ok? ✅
This gets rid off the implicit dependency on `IsGlyphFullWidth`
for the IRM and DECSWL/DECDWL/DECDHL implementations.
## Validation Steps Performed
In pwsh:
* ``"`e[31mab`e[m`b`e[4h`e[32m$('*'*10)`e[m`e[4l"``
prints a red "a", 10 green "*" and a red "b" ✅
* ``"`e[31mab`e[m`b`e[4h`e[32m$('*'*1000)`e[m`e[4l"``
prints a red "a" and a couple lines of green "*" ✅
* ``"`e[31mf$('o'*70)`e[m`e#6`e#5"``
the right half of the row is erased ✅
This is pretty much a huge refactoring of how marks are stored in the
buffer.
Gone is the list of `ScrollMark`s in the buffer that store regions of
text as points marking the ends. Those would be nigh impossible to
reflow nicely.
Instead, we're going to use `TextAttribute`s to store the kind of output
we've got - `Prompt`, `Command`, `Output`, or, the default, `None`.
Those already reflow nicely!
But we also need to store things like, the exit code for the command.
That's why we've now added `ScrollbarData` to `ROW`s. There's really
only going to be one prompt->output on a single row. So, we only need to
store one ScrollbarData per-row. When a command ends, we can just go
update the mark on the row that started that command.
But iterating over the whole buffer to find the next/previous
prompt/command/output region sounds complicated. So, to avoid everyone
needing to do some variant of that, we've added `MarkExtents` (which is
literally just the same mark structure as before). TextBuffer can figure
out where all the mark regions are, and hand that back to callers. This
allows ControlCore to be basically unchanged.
_But collecting up all the regions for all the marks sounds expensive!
We need to update the scrollbar frequently, we can't just collect those
up every time!_ No we can't! But we also don't need to. The scrollbar
doesn't need to know where all the marks start and end and if they have
commands and this and that - no. We only need to know the rows that have
marks on them. So, we've now also got `ScrollMark` to represent just a
mark on a scrollbar at a specific row on the buffer. We can get those
quickly.
* [x] I added a bunch of tests for this.
* [x] I played with it and it feels good, even after a reflow (finally)
* See:
* #11000
* #15057 (I'm not marking this as closed. The stacked PR will close
this, when I move marks to Stable)
I forgot to `buffer.clear()` after a write. Whoops.
This includes 2 additional, smaller improvements that I just happened
to notice: The `GenRTF` code calls `to_string` despite using `fmt`.
Closes#16782
### Validation Steps Performed
- Double-clicking on a Double-Width row selects the word (identified by
delimiters) under the cursor.
- Tripple-clicking on a Double-Width row selects the whole line under
the cursor.
- The same works for Double-Height rows also.
- The same works for Single-Width rows also.
This takes care of an edge case in regards to SGR 22: It turns off
both intense and faint attributes which means that we may need to
turn on one of the two if only one of them turned off.
Additionally, this removes the mapping for `BottomGridline` which
has no real VT equivalent anyway.
## Validation Steps Performed
* Turn session restore on
* In pwsh write:
```pwsh
"`e[1;2mboth`e[0;1mintense`e[m`n`e[1;2mboth`e[0;2mfaint`e[m"
```
* Close the app and open the `buffer_*.txt` file next to settings.json
* It contains... ✅
```
␛[1m␛[2mboth␛[22;1mintense␛[22m
␛[1m␛[2mboth␛[22;2mfaint␛[22m
```
This changeset allows Windows Terminal to dump its buffer contents as
UTF-16LE VT text onto disk and restore it later. This functionality is
enabled whenever `persistedWindowLayout` is being used.
Closes#961Closes#16741
## Validation Steps Performed
* Open multiple windows with multiple tabs and restart the app
Everything's restored ✅
* Reopen a tab with output from `RenderingTests.exe`
Everything's restored ✅
* Closing tabs and windows with Ctrl+W deletes their buffer dumps ✅
* Closing tabs doesn't create buffer dumps ✅
* `[[nodiscard]]` and `[[maybe_unused]]` must come before `virtual` and
`static` qualifiers
* MSVC and Clang disagree on how `gsl::suppress` should look;
fortunately, GSL provides a macro to paper over the difference
* Clang throws "pessimizing move" warnings when you `std::move` a
temporary, as it makes copy elision impossible
* The fuzzing logic was using an unspecified template expansion
`CFuzzLogic<>` before the type had been declared
* LibraryResources was emitting most of the `.util` section with
read-write permissions and some of it with read-only
Refs #15952
Aside from overall simplifying `CharToColumnMapper` this fixes 2 bugs:
* The backward search loop may have iterated 1 column too far,
because it didn't stop at `*current <= *target`, but rather at
`*(current - 1) <= *target`. This issue was only apparent when
surrogate pairs were being used in a row.
* When the target offset is that of a trailing surrogate pair
the forward search loop may have iterated 1 column too far.
It's somewhat unlikely for this to happen since this code is
only used through ICU, but you never know.
This is a continuation of PR #16775.
