The idea is that we can translate Console API calls directly to VT at
least as well as the current VtEngine setup can. For instance, a call
to `SetConsoleCursorPosition` clearly translates directly to a `CUP`
escape sequence. Effectively, instead of translating output
asynchronously in the renderer thread, we'll do it synchronously
right during the Console API call.
Most importantly, the this means that any VT output that an
application generates will now be given to the terminal unmodified.
Aside from reducing our project's complexity quite a bit and opening
the path towards various interesting work like sixels, Device Control
Strings, buffer snapshotting, synchronized updates, and more, it also
improves performance for mixed text output like enwik8.txt in conhost
to 1.3-2x and in Windows Terminal via ConPTY to roughly 20x.
This adds support for overlapped IO, because now that output cannot
be "skipped" anymore (VtEngine worked like a renderer after all)
it's become crucial to block conhost as little as possible.
⚠️ Intentionally unresolved changes/quirks:
* To force a delayed EOL wrap to wrap, `WriteCharsLegacy` emits a
`\r\n` if necessary. This breaks text reflow on window resize.
We cannot emit ` \r` the way readline does it, because this would
overwrite the first column in the next row with a whitespace.
The alternative is to read back the affected cell from the buffer
and emit that character and its attributes followed by a `\r`.
I chose to not do that, because buffer read-back is lossy (= UCS2).
Unless the window is resized, the difference is unnoticeable
and historically, conhost had no support for buffer reflow anyway.
* If `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is set while
`DISABLE_NEWLINE_AUTO_RETURN` is reset, we'll blindly replace all
LF with CRLF. This may hypothetically break DCS sequences, but it's
the only way to do this without parsing the given VT string and
thus the only way we can achieve passthrough mode in the future.
* `ENABLE_WRAP_AT_EOL_OUTPUT` is translated to `DECAWM`.
Between Windows XP and Windows 11 21H2, `ENABLE_WRAP_AT_EOL_OUTPUT`
being reset would cause the cursor position to reset to wherever
a write started, _if_ the write, including expanded control chars,
was less than 100 characters long. If it was longer than that,
the cursor position would end up in an effectively random position.
After lengthy research I believe that this is a bug introduced in
Windows XP and that the original intention was for this mode to be
equivalent to `DECAWM`. This is compounded by MSDN's description
(emphasis mine):
> If this mode is disabled, the **last character** in the row is
> overwritten with any subsequent characters.
⚠️ Unresolved issues/quirks:
* Focus/Unfocus events are injected into the output stream without
checking whether the VT output is currently in a ground state.
This may break whatever VT sequence is currently ongoing.
This is an existing issue.
* `VtIo::Writer::WriteInfos` should properly verify the width of
each individual character.
* Using `SetConsoleActiveScreenBuffer` destroys surrogate pairs
and extended (VT) attributes. It could be translated to VT pages
in the long term.
* Similarly, `ScrollConsoleScreenBuffer` results in the same and
could be translated to `DECCRA` and `DECFRA` in the near term.
This is important because otherwise `vim` output may loose
its extended attributes during scrolling.
* Reflowing a long line until it wraps results in the cooked read
prompt to be misaligned vertically.
* `SCREEN_INFORMATION::s_RemoveScreenBuffer` should trigger a
buffer switch similar to `SetConsoleActiveScreenBuffer`.
* Translation of `COMMON_LVB_GRID_HORIZONTAL` to `SGR 53` was dropped
and may be reintroduced alongside `UNDERSCORE` = `SGR 4`.
* Move the `OSC 0 ; P t BEL` sequence to `WriteWindowTitle`
and swap the `BEL` with the `ST` (`ESC \`).
* PowerShell on Windows 10 ships with PSReadLine 2.0.0-beta2
which emits SGR 37/40 instead of 39/49. This results in black
spaces when typing and there's no good way to fix that.
* A test is missing that ensures that `FillConsoleOutputCharacterW`
results in a `CSI n J` during the PowerShell shim.
* A test is missing that ensures that `PtySignal::ClearBuffer`
does not result in any VT being generated.
Closes#262Closes#1173Closes#3016Closes#4129Closes#5228Closes#8698Closes#12336Closes#15014Closes#15888Closes#16461Closes#16911Closes#17151Closes#17313
This is pretty much a huge refactoring of how marks are stored in the
buffer.
Gone is the list of `ScrollMark`s in the buffer that store regions of
text as points marking the ends. Those would be nigh impossible to
reflow nicely.
Instead, we're going to use `TextAttribute`s to store the kind of output
we've got - `Prompt`, `Command`, `Output`, or, the default, `None`.
Those already reflow nicely!
But we also need to store things like, the exit code for the command.
That's why we've now added `ScrollbarData` to `ROW`s. There's really
only going to be one prompt->output on a single row. So, we only need to
store one ScrollbarData per-row. When a command ends, we can just go
update the mark on the row that started that command.
But iterating over the whole buffer to find the next/previous
prompt/command/output region sounds complicated. So, to avoid everyone
needing to do some variant of that, we've added `MarkExtents` (which is
literally just the same mark structure as before). TextBuffer can figure
out where all the mark regions are, and hand that back to callers. This
allows ControlCore to be basically unchanged.
_But collecting up all the regions for all the marks sounds expensive!
We need to update the scrollbar frequently, we can't just collect those
up every time!_ No we can't! But we also don't need to. The scrollbar
doesn't need to know where all the marks start and end and if they have
commands and this and that - no. We only need to know the rows that have
marks on them. So, we've now also got `ScrollMark` to represent just a
mark on a scrollbar at a specific row on the buffer. We can get those
quickly.
* [x] I added a bunch of tests for this.
* [x] I played with it and it feels good, even after a reflow (finally)
* See:
* #11000
* #15057 (I'm not marking this as closed. The stacked PR will close
this, when I move marks to Stable)
Silent MIDI notes can be used to seemingly deny a user's input for long
durations (multiple minutes). This commit improves the situation by ignoring
all DECPS sequences for a second when Ctrl+C/Ctrl+Break is pressed.
Additionally it fixes a regression introduced in 666c446:
When we close a tab we need to unblock/shutdown `MidiAudio` early,
so that `ConptyConnection::Close()` can run down as fast as possible.
## Validation Steps Performed
* In pwsh in Windows Terminal 1.16 run ``while ($True) { echo "`e[3;8;3,~" }``
* Ctrl+C doesn't do anything ❎
* Closing the tab doesn't do anything ❎
* With these modifications in Windows Terminal:
* Ctrl+C stops the output ✅
* Closing the tab completes instantly ✅
* With these modifications in OpenConsole:
* Ctrl+C stops the output ✅
* Closing the window completes instantly ✅
Make a VTApiRoutines servicer that does minimal translations instead of
environmental simulation for some output methods. Remaining methods are
backed on the existing console host infrastructure (primarily input
related methods).
## PR Checklist
* [x] I work here
* [x] It's Fix-Hack-Learn quality so it's behind a feature gate so we
can keep refining it. But it's a start!
To turn this on, you will have to be in the Dev or Preview rings
(feature staged). Then add `experimental.connection.passthroughMode:
true` to a profile and on the next launch, the flags will propagate down
through the `ConptyConnection` into the underlying `Openconsole.exe`
startup and tell it to use the passthrough mode instead of the full
simulation mode.
## Validation Steps Performed
- Played with it manually in CMD.exe, it seems to work mostly.
- Played with it manually in Ubuntu WSL, it seems to work.
- Played with it manually in Powershell and it's mostly sad. It'll get
there.
Starts #1173
## Summary of the Pull Request
This PR implements a pair of shims for `cmd` and `powershell`, so that their `cls` and `Clear-Host` functions will clear the entire terminal buffer (like they do in conhost), instead of just the viewport. With the conpty viewport and buffer being the same size, there's effectively no way to know if an application is calling these API's in this way with the intention of clearing the buffer or the viewport. We absolutely have to guess.
Each of these shims checks to see if the way that the API is being called exactly matches the way `cmd` or `powershell` would call these APIs. If it does, we manually write a `^[[3J` to the connected terminal, to get he Terminal to clear it's own scrollback.
~~_⚠️ If another application were trying to clear the **viewport** with an exactly similar API call, this would also cause the terminal scrollback to get cleared ⚠️_~~
* [x] Should these shims be restricted to when the process that's calling them is actually `cmd.exe` or `powershell.exe`? Can I even do this? I think we've done such a good job of isolating the client process information from the rest of the host code that I can't figure out how to do this.
- YES, this can be done, and I did it.
* [ ] **TODO**: _While I'm here_, should I have `DoSrvPrivateEraseAll` (the implementation for `^[[2J`, in `getset.cpp`) also manually trigger a EraseAll in the terminal in conpty mode?
## PR Checklist
* [x] Closes#3126
* [x] Actually closes#1305 too, which is really the same thing, but probably deserves a callout
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Validation Steps Performed
* ran tests
* checked `cls` in the Terminal
* checked `Clear-Host` in the Terminal
* Checked running `powershell clear-host` from `cmd.exe`
## Summary of the Pull Request
#4354 is a pretty complicated PR. It's got a bunch of conpty changes, but what it also has was some critical improvements to the roundtrip test suite. I'm working on some other bugfixes in the same area currently, and need these tests enhancements in those branches _now_. The rest of #4354 is complex enough that I don't trust it will get merged soon (if ever). However, these fixes _should_ be in regardless.
## PR Checklist
* [x] Taken directly from #4354
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
This is four main changes:
* Enable conpty to be fully enabled in unittests. Just setting up a VT renderer isn't enough to trick the host into being in conpty mode - it also needs to have some other flags set.
* Some minor changes to `CommonState` to better configure the common test state for conpty
* Move some of the verify helpers from `ConptyRoundtripTests` into their own helper class, to be shared in multiple tests
* Add a `TerminalBufferTests` class, for testing the Terminal buffer directly (without conpty).
This change is really easier than
would suggest, I promise.
