This massive refactoring has two goals:
* Enable us to go beyond UCS-2 support for input editing
* Bring clarity into `COOKED_READ_DATA`'s inner workings
Unfortunately, over time, knowledge about its exact operation was lost.
While the new code is still complex it reduces the amount of code by 4x
which will make preserving knowledge hopefully significantly easier.
The new implementation is simpler and slower than the old one in a way,
because every time the input line is modified it's rewritten to the text
buffer from scratch. This however massively simplifies the underlying
algorithm and the amount of state that needs to be tracked and results
in a significant reduction in code size. It also makes it more robust,
because there's less code now that can be incorrect.
This "optimization laziness" can be afforded due the recent >10x
improvements to `TextBuffer`'s text ingestion performance.
For short inputs (<1000 characters) I still expect this implementation
to outperform the conhost from the past.
It has received one optimization already however: While reading text
from the `InputBuffer` we'll now defer writing into the `TextBuffer`
until we've stopped reading. This improves the overhead of pasting text
from O(n^2) to O(n), which is immediately noticeable for inputs >100kB.
Resizing the text buffer still ends up corrupting the input line
however, which unfortunately cannot be fixed in `COOKED_READ_DATA`.
The issue occurs due to bugs in `TextBuffer::Reflow` itself, as it
misplaces the cursor if the prompt is on the last line of the buffer.
Closes#1377Closes#1503Closes#4628Closes#4975Closes#5033Closes#8008
This commit is required to fix#797
## Validation Steps Performed
* ASCII input ✅
* Chinese input (中文維基百科) ❔
* Resizing the window properly wraps/unwraps wide glyphs ❌
Broken due to `TextBuffer::Reflow` bugs
* Surrogate pair input (🙂) ❔
* Resizing the window properly wraps/unwraps surrogate pairs ❌
Broken due to `TextBuffer::Reflow` bugs
* In cmd.exe
* Create 2 file: "a😊b.txt" and "a😟b.txt"
* Press tab: Autocompletes "a😊b.txt" ✅
* Navigate the cursor right past the "a"
* Press tab twice: Autocompletes "a😟b.txt" ✅
* Backspace deletes preceding glyphs ✅
* Ctrl+Backspace deletes preceding words ✅
* Escape clears input ✅
* Home navigates to start ✅
* Ctrl+Home deletes text between cursor and start ✅
* End navigates to end ✅
* Ctrl+End deletes text between cursor and end ✅
* Left navigates over previous code points ✅
* Ctrl+Left navigates to previous word-starts ✅
* Right and F1 navigate over next code points ✅
* Pressing right at the end of input copies characters
from the previous command ✅
* Ctrl+Right navigates to next word-ends ✅
* Insert toggles overwrite mode ✅
* Delete deletes next code point ✅
* Up and F5 cycle through history ✅
* Doesn't crash with no history ✅
* Stops at first entry ✅
* Down cycles through history ✅
* Doesn't crash with no history ✅
* Stops at last entry ✅
* PageUp retrieves the oldest command ✅
* PageDown retrieves the newest command ✅
* F2 starts "copy to char" prompt ✅
* Escape dismisses prompt ✅
* Typing a character copies text from the previous command up
until that character into the current buffer (acts identical
to F3, but with automatic character search) ✅
* F3 copies the previous command into the current buffer,
starting at the current cursor position,
for as many characters as possible ✅
* Doesn't erase trailing text if the current buffer
is longer than the previous command ✅
* Puts the cursor at the end of the copied text ✅
* F4 starts "copy from char" prompt ✅
* Escape dismisses prompt ✅
* Erases text between the current cursor position and the
first instance of a given char (but not including it) ✅
* F6 inserts Ctrl+Z ✅
* F7 without modifiers starts "command list" prompt ✅
* Escape dismisses prompt ✅
* Minimum size of 40x10 characters ✅
* Width expands to fit the widest history command ✅
* Height expands up to 20 rows with longer histories ✅
* F9 starts "command number" prompt ✅
* Left/Right paste replace the buffer with the given command ✅
* And put cursor at the end of the buffer ✅
* Up/Down navigate selection through history ✅
* Stops at start/end with <10 entries ✅
* Stops at start/end with >20 entries ✅
* Wide text rendering during pagination with >20 entries ✅
* Shift+Up/Down moves history items around ✅
* Home navigates to first entry ✅
* End navigates to last entry ✅
* PageUp navigates by 20 items at a time or to first ✅
* PageDown navigates by 20 items at a time or to last ✅
* Alt+F7 clears command history ✅
* F8 cycles through commands that start with the same text as
the current buffer up until the current cursor position ✅
* Doesn't crash with no history ✅
* F9 starts "command number" prompt ✅
* Escape dismisses prompt ✅
* Ignores non-ASCII-decimal characters ✅
* Allows entering between 1 and 5 digits ✅
* Pressing Enter fetches the given command from the history ✅
* Alt+F10 clears doskey aliases ✅
This commit slightly modernizes `CommandHistory` by leaning more heavily
on the STL container functionalities. For one, it uses for-range
iterations to loop through `_commands` instead of using `GetNth`
on every iteration. Another major improvement however is that
the code previously copied entire `CommandHistory` instances out of
the linked list `s_historyLists`, then removed the slot and copied
(not moved!) that instance into the front again. Now it uses the
`splice` function from `std::list` to do it in `O(1)` and virtually
cost-free.
Another major improvement (and the one I'm personally interested in)
is the switch from `SHORT` to `int32_t`. This will greatly simplify
the implementation of the future `COOKED_READ_DATA` class, as the
larger integer type will remove worries about over/underflow.
For instance, we can then just blindly increment/decrement the history
position and then only later clamp it to the expected range.
## Validation Steps Performed
* Existing history tests ✅
* History cycling with F8 ✅
* Navigating history with F7 ✅
This change is a fairly subjective one. It was done because
`IsValidStringBuffer` will very soon be the only function left
in `cmdline.cpp`. Removing it allows removing `cmdline.cpp`.
While the code that replaces it is somewhat tricky, it's also much
more straightforward, as the `IsValidStringBuffer` function didn't
just check if the string buffer is valid - it also retrieved the
pointers to each of the strings contained in the buffer.
## Validation Steps Performed
Exhaustively covered by conhost feature tests ✅
This is a complete rewrite of the old `WriteCharsLegacy` function
which is used when VT mode is disabled as well as for all interactive
console input handling on Windows. The previous code was almost
horrifying in some aspects as it first wrote the incoming text into a
local buffer, stripping/replacing any control characters. That's not
particular fast and never was. It's unknown why it was like that.
It also measured the width of each glyph to correctly determine the
cursor position and line wrapping. Presumably this used to work quite
well in the original console code, because it would then just copy
that local buffer into the destination text buffer, but with the
introduction of the broken and extremely slow `OutputCellIterator`
abstraction this would end up measuring all text twice and cause
disagreements between `WriteCharsLegacy`'s idea of the cursor position
and `OutputCellIterator`'s cursor position. Emoji input was basically
entirely broken. This PR fixes it by passing any incoming text
straight to the `TextBuffer` as well as by using its cursor positioning
facilities to correctly implement wrapping and backspace handling.
Backspacing over Emojis and an array of other aspects still don't work
correctly thanks to cmdline.cpp, but it works quite a lot better now.
Related to #8000Closes#8839Closes#10808
## Validation Steps Performed
* Printing various Unicode text ✅
* On an fgets() input line
* Typing text works ✅
* Inserting text works anywhere ✅
* Ctrl+X is translated to ^X ✅
* Null is translated to ^@ ✅
This was tested by hardcoding the `OutputMode` to 3 instead of 7.
* Backspace only advances to start of the input ✅
* Backspace deletes the entire preceding tab ✅
* Backspace doesn't delete whitespace preceding a tab ✅
* Backspacing a force-wrapped wide glyph unwraps the line break ✅
* Backspacing ^X deletes both glyphs ✅
* Backspacing a force-wrapped tab deletes trailing whitespace ✅
* When executing
```cpp
fputs("foo: ", stdout);
fgets(buffer, stdin);
```
pressing tab and then backspace does not delete the whitespace
that follows after the "foo:" string (= `sOriginalXPosition`).
I've removed these because it made some of my new code pretty
convoluted for now good reason as most of these functions aren't
exception safe to begin with. Basically, their boolean status
is often just a pretense because they can crash or throw anyways.
Furthermore, `WriteCharsLegacy` failed to check the status code
returned by `AdjustCursorPosition` in some of its parts too.
In the future we should instead probably strive to continue
make our legacy code more exception safe.
This is a follow-up of #13025 to make the members of `til::point/size/rect`
uniform and consistent without the use of `unions`. The only file that has
any changes is `src/host/getset.cpp` where an if condition was simplified.
## Validation Steps Performed
* Host unit tests ✅
* Host feature tests ✅
* ControlCore feature tests ✅
This pull request adds a couple `const` keywords, simplifies a bit of boolean logic,
adds the `static` keyword to `Commandline::IsEditLineEmpty`, and a couple more things.
Previously this project used a great variety of types to present text buffer
coordinates: `short`, `unsigned short`, `int`, `unsigned int`, `size_t`,
`ptrdiff_t`, `COORD`/`SMALL_RECT` (aka `short`), and more.
This massive commit migrates almost all use of those types over to the
centralized types `til::point`/`size`/`rect`/`inclusive_rect` and their
underlying type `til::CoordType` (aka `int32_t`).
Due to the size of the changeset and statistics I expect it to contain bugs.
The biggest risk I see is that some code potentially, maybe implicitly, expected
arithmetic to be mod 2^16 and that this code now allows it to be mod 2^32.
Any narrowing into `short` later on would then throw exceptions.
## PR Checklist
* [x] Closes#4015
* [x] I work here
* [x] Tests added/passed
## Validation Steps Performed
Casual usage of OpenConsole and Windows Terminal. ✅
#4015 requires sweeping changes in order to allow a migration of our buffer
coordinates from `int16_t` to `int32_t`. This commit reduces the size of
future commits by using type inference wherever possible, dropping the
need to manually adjust types throughout the project later.
As an added bonus this commit standardizes the alignment of cv qualifiers
to be always left of the type (e.g. `const T&` instead of `T const&`).
The migration to type inference with `auto` was mostly done
using JetBrains Resharper with some manual intervention and the
standardization of cv qualifier alignment using clang-format 14.
## References
This is preparation work for #4015.
## Validation Steps Performed
* Tests pass ✅
This commit clarifies some things inside WriteCharsLegacy by adding
comments and renaming parameter and enum names. It does not change any
logic.
`WC_ECHO` was used extensively to mean only one thing: whether to print
a control character like `\x18` (Ctrl+X>) as `^X`. It's been renamed
to make that abundantly clear.
Many include statements use forward slashes, while others use backwards
slashes. This is inconsistent formatting. For this reason, I changed the
backward slashes to forward slashes since that is the standard.
This PR provides a faster algorithm for converting 8-bit and 24-bit
colors into the 4-bit legacy values that are required by the Win32
console APIs. It also fixes areas of the code that were incorrectly
using a simple 16-color conversion that didn't handle 8-bit and 24-bit
values.
The faster conversion algorithm should be an improvement for issues #783
and #3950.
One of the main points of this PR was to fix the
`ReadConsoleOutputAttribute` API, which was using a simplified legacy
color conversion (the original `TextAttribute:GetLegacyAttributes`
method), which could only handle values from the 16-color table. RGB
values, and colors from the 256-color table, would be mapped to
completely nonsensical values. This API has now been updated to use the
more correct `Settings::GenerateLegacyAttributes` method.
But there were also a couple of other places in the code that were using
`GetLegacyAttributes` when they really had no reason to be working with
legacy attributes at all. This could result in colors being downgraded
to 4-bit values (often badly, as explained above), when the code was
already perfectly capable of displaying the full 24-bits.
This included the fill colors in the IME composer (in `ConsoleImeInfo`),
and the construction of the highlighting colors in the color
search/selection handler (`Selection::_HandleColorSelection`). I also
got rid of some legacy attribute code in the `Popup` class, which was
originally intended to update colors below the popup when the settings
changed, but actually caused more problems than it solved.
The other major goal of this PR was to improve the performance of the
`GenerateLegacyAttributes` method, since the existing implementation
could be quite slow when dealing with RGB values.
The simple cases are handled much the same as they were before. For an
`IsDefault` color, we get the default index from the
`Settings::_wFillAttribute` field. For an `IsIndex16` color, the index
can just be returned as is.
For an `IsRgb` color, the RGB components are compressed down to 8 bits
(3 red, 3 green, 2 blue), simply by dropping the least significant bits.
This 8-bit value is then used to lookup a representative 16-color value
from a hard-coded table. An `IsIndex256` color is also converted with a
lookup table, just using the existing 8-bit index.
The RGB mapping table was calculated by taking each compressed 8-bit
color, and picking a entry from the _Campbell_ palette that best
approximated that color. This was done by looking at a range of 24-bit
colors that mapped to the 8-bit value, finding the best _Campbell_ match
for each of them (using a [CIEDE2000] color difference calculation), and
then the most common match became the index that the 8-bit value would
map to.
The 256-color table was just a simpler version of this process. For each
entry in the table, we take the default RGB palette value, and find it's
closest match in the _Campbell_ palette.
Because these tables are hard-coded, the results won't adjust to changes
in the palette. However, they should still produce reasonable results
for palettes that follow the standard ANSI color range. And since
they're only a very loose approximation of the colors anyway, the exact
value really isn't that important.
That said, I have tried to make sure that if you take an RGB value for a
particular index in a reasonable color scheme, then the legacy color
mapped from that value should ideally match the same index. This will
never be possible for all color schemes, but I have tweaked a few of the
table entries to improve the results for some of the common schemes.
One other point worth making regarding the hard-coded tables: even if we
wanted to take the active palette into account, that wouldn't actually
be possible over a conpty connection, because we can't easily know what
color scheme the client application is using. At least this way the
results in conhost are guaranteed to be the same as in the Windows
Terminal.
[CIEDE2000]: https://en.wikipedia.org/wiki/Color_difference#CIEDE2000
## Validation Steps Performed
This code still passes the `TextAttributeTests` that check the basic
`GetLegacyAttribute` behaviour and verify the all legacy attributes
roundtrip correctly. However, some of the values in the `RgbColorTests`
had to be updated, since we're now intentionally returning different
values as a result of the changes to the RGB conversion algorithm.
I haven't added additional unit tests, but I have done a lot of manual
testing to see how well the new algorithm works with a range of colors
and a variety of different color schemes. It's not perfect in every
situation, but I think it works well enough for the purpose it serves.
I've also confirmed that the issues reported in #5940 and #6247 are now
fixed by these changes.
Closes#5940Closes#6247
* Removed using namespace directive from header files and put these in cpp files where they are used
* Fixed tabbing issues by replacing them with spaces.
Also regrouped the using directives.
* Update src/host/exemain.cpp
Co-Authored-By: Mike Griese <migrie@microsoft.com>
* Update src/interactivity/win32/find.cpp
Co-Authored-By: Mike Griese <migrie@microsoft.com>
