Strange behavior with ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT #6907

@alexrp commented on GitHub (Mar 17, 2020):

OK, so to be clear, is the implication here that ENABLE_VIRTUAL_TERMINAL_INPUT in and of itself effectively means "raw mode"? In other words, that I'm pretty much expected/required to unset ENABLE_ECHO_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT at the same time as I set the former (because otherwise things just don't work properly)?

@alexrp commented on GitHub (Mar 17, 2020): OK, so to be clear, is the implication here that `ENABLE_VIRTUAL_TERMINAL_INPUT` in and of itself effectively means "raw mode"? In other words, that I'm pretty much expected/required to unset `ENABLE_ECHO_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT` at the same time as I set the former (because otherwise things just don't work properly)?

claunia commented

@DHowett-MSFT commented on GitHub (Mar 17, 2020):

It would be ideal if you do so, yeah -- if all you want is actual VT input, unmanipulated by the console host, you're going to want to disable line and processed input.

Most VT input consumers are doing their own line editing and want to get hold of the arrow keys and other keys germane to line editing, or they want to get mouse input in a cross-platform way, and having the processed or line input modes enabled will make it difficult to do that.

@DHowett-MSFT commented on GitHub (Mar 17, 2020): It would be ideal if you do so, yeah -- if all you want is actual VT input, unmanipulated by the console host, you're going to want to disable line and processed input. Most VT input consumers are doing their own line editing and want to get hold of the arrow keys and other keys germane to line editing, or they want to get mouse input in a cross-platform way, and having the processed or line input modes enabled will make it difficult to do that.

claunia commented

@alexrp commented on GitHub (Mar 17, 2020):

It would be ideal if you do so, yeah -- if all you want is actual VT input, unmanipulated by the console host, you're going to want to disable line and processed input.

But the thing is, that's not what I want. 🙂 For this project I'm working on, I'm trying to expose a consistent terminal interaction API that works the same way on Windows and Unix systems, and this issue touches on a pretty fundamental aspect of that.

I'm attempting to get the same behavior on Windows that you get in the Unix world: VT keyboard input at all times, with cooked mode by default, and raw mode optionally available via tcsetattr/termios. That's why I'm enabling ENABLE_VIRTUAL_TERMINAL_INPUT but leaving the other flags enabled, only disabling them when the user of the library explicitly requests raw mode.

Is there nothing that can be done to make the Windows console host handle this situation properly?

@alexrp commented on GitHub (Mar 17, 2020): > It would be ideal if you do so, yeah -- if all you want is actual VT input, unmanipulated by the console host, you're going to want to disable line and processed input. But the thing is, that's not what I want. 🙂 For this project I'm working on, I'm trying to expose a consistent terminal interaction API that works the same way on Windows and Unix systems, and this issue touches on a pretty fundamental aspect of that. I'm attempting to get the same behavior on Windows that you get in the Unix world: VT keyboard input at all times, with cooked mode by default, and raw mode optionally available via `tcsetattr`/`termios`. That's why I'm enabling `ENABLE_VIRTUAL_TERMINAL_INPUT` but leaving the other flags enabled, only disabling them when the user of the library explicitly requests raw mode. Is there nothing that can be done to make the Windows console host handle this situation properly?

claunia commented

@DHowett-MSFT commented on GitHub (Mar 17, 2020):

Sorry, I misunderstood your use case. I'll try to break down a couple of the scenarios and the flags that'll light them up. Would you let me know if I've captured the ones you're looking for?

If you want completely raw input, with VT sequences, as they are generated by a terminal:
- use ENABLE_VIRTUAL_TERMINAL_INPUT, disable ENABLE_LINE_INPUT, ENABLE_ECHO_INPUT, ENABLE_PROCESSED_INPUT
- Try to always use ReadConsoleInput, as that's the only way to be notified of window size changes (we have a workitem tracking a real SIGWINCH-like signal)
If you want input that is broken up by line...
- ... containing VT and control characters (^C, ^H, ^M):
  - use ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT, disable ENABLE_PROCESSED_INPUT, your choice for echo
  - lines will be delimited by ^M
- ... containing VT, but no control characters (I think this is the Unix TTY vt-cooked mode?):
  - use ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT, your choice for echo
  - lines will be delimited by ^M ^J (the Windows way)
  - you will never receive ^H in your buffer
- ... with Console-mediated line editing, but no VT:
  - use ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT
  - lines will be delimited by ^M ^J (the Windows way)
  - you will never receive ^H or ^[[A in your buffer

@DHowett-MSFT commented on GitHub (Mar 17, 2020): Sorry, I misunderstood your use case. I'll try to break down a couple of the scenarios and the flags that'll light them up. Would you let me know if I've captured the ones you're looking for? * If you want completely raw input, with VT sequences, as they are generated by a terminal: * use `ENABLE_VIRTUAL_TERMINAL_INPUT`, disable `ENABLE_LINE_INPUT`, `ENABLE_ECHO_INPUT`, `ENABLE_PROCESSED_INPUT` * Try to always use `ReadConsoleInput`, as that's the only way to be notified of window size changes (we have a workitem tracking a real SIGWINCH-like signal) * If you want input that is broken up by line... * ... containing VT and control characters (`^C`, `^H`, `^M`): * use `ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT`, disable `ENABLE_PROCESSED_INPUT`, your choice for echo * lines will be delimited by `^M` * ... containing VT, but no control characters (_I think this is the Unix TTY vt-cooked mode?_): * use `ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT`, your choice for echo * lines will be delimited by `^M` `^J` (the Windows way) * you will never receive `^H` in your buffer * ... with Console-mediated line editing, but no VT: * use `ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT` * lines will be delimited by `^M` `^J` (the Windows way) * you will never receive `^H` or `^[[A` in your buffer

claunia commented

@DHowett-MSFT commented on GitHub (Mar 17, 2020):

Okay, now I get it. Item 2 sub 3, the one I think is vt-cooked mode, has the wrong behavior for backspace; this seems to be true back a couple major releases of windows. That's not excellent.

I'll check this out versus the version that introduced ENABLE_VIRTUAL_TERMINAL_INPUT.

Sorry, I had to talk myself through your problem. 😄

@DHowett-MSFT commented on GitHub (Mar 17, 2020): Okay, now I get it. Item 2 sub 3, the one I think is vt-cooked mode, has the wrong behavior for backspace; this seems to be true back a couple major releases of windows. That's not excellent. I'll check this out versus the version that introduced `ENABLE_VIRTUAL_TERMINAL_INPUT`. Sorry, I had to talk myself through your problem. :smile:

claunia commented

@alexrp commented on GitHub (Mar 17, 2020):

Maybe it's easier to explain with this C program:

#include <unistd.h>

int main()
{
    char buf[4096];

    write(STDOUT_FILENO, "reading: ", 9);
    ssize_t len = read(STDIN_FILENO, buf, sizeof(buf));
    write(STDOUT_FILENO, buf, len);

    return 0;
}

Run on a Linux system (or WSL) with gcc test.c && ./a.out.

You will notice that you are able to do fairly basic line editing while the program reads from stdin, i.e. the TTY is in cooked mode. The exact level of line editing available will probably depend on your terminal and termios settings (the latter of which is usually the shell's responsibility). In my case, I'm able to use Backspace, but not arrow keys.

But, you will also notice that if you type some special keys, they are picked up as VT sequences by the program. For example, aaa followed by Ctrl+X and Enter prints aaa for me.

This is the behavior I'm trying to achieve.

To be clear, though, I don't particularly care what amount of editing features is available in cooked mode. It's okay that those are different between terminals and platforms. The important thing here is achieving the cooked + VT behavior, but not breaking Backspace / Ctrl+Backspace behavior on Windows in the process.

@alexrp commented on GitHub (Mar 17, 2020): Maybe it's easier to explain with this C program: ```c #include <unistd.h> int main() { char buf[4096]; write(STDOUT_FILENO, "reading: ", 9); ssize_t len = read(STDIN_FILENO, buf, sizeof(buf)); write(STDOUT_FILENO, buf, len); return 0; } ``` Run on a Linux system (or WSL) with `gcc test.c && ./a.out`. You will notice that you are able to do fairly basic line editing while the program reads from stdin, i.e. the TTY is in cooked mode. The exact level of line editing available will probably depend on your terminal and `termios` settings (the latter of which is usually the shell's responsibility). In my case, I'm able to use Backspace, but not arrow keys. But, you will also notice that if you type some special keys, they are picked up as VT sequences by the program. For example, `aaa` followed by Ctrl+X and Enter prints `aaa` for me. This is the behavior I'm trying to achieve. To be clear, though, I don't particularly care what amount of editing features is available in cooked mode. It's okay that those are different between terminals and platforms. The important thing here is achieving the cooked + VT behavior, but *not* breaking Backspace / Ctrl+Backspace behavior on Windows in the process.

claunia commented

1e22bfa8d6/src/terminal/input/terminalInput.cpp (L80)

@xoofx commented on GitHub (Dec 30, 2021):

@DHowett-MSFT This issue is quite annoying as it is making ENABLE_VIRTUAL_TERMINAL_INPUT quite useless when in cooked mode.

I have tried to investigate this and the end story is that the key is mapped by this table here for some reasons that I don't know:

I could make a PR and remove this line, but I'm not sure about the consequences. Thoughts?

@xoofx commented on GitHub (Dec 30, 2021): @DHowett-MSFT This issue is quite annoying as it is making `ENABLE_VIRTUAL_TERMINAL_INPUT` quite useless when in cooked mode. I have tried to investigate this and the end story is that the key is mapped by this table here for some reasons that I don't know: https://github.com/microsoft/terminal/blob/1e22bfa8d6f86d0546c3f162d171274cf453db65/src/terminal/input/terminalInput.cpp#L80 I could make a PR and remove this line, but I'm not sure about the consequences. Thoughts?

claunia commented

1e22bfa8d6/src/terminal/input/terminalInput.cpp (L78-L81)

@xoofx commented on GitHub (Dec 30, 2021):

I tested by modifying these two places in src\terminal\input\terminalInput.cpp:

static constexpr std::array<TermKeyMap, 19> s_keypadNumericMapping{
    TermKeyMap{ VK_TAB, L"\x09" },
-    TermKeyMap{ VK_BACK, L"\x7f" }, 
    TermKeyMap{ VK_PAUSE, L"\x1a" },

1e22bfa8d6/src/terminal/input/terminalInput.cpp (L218-L221)

static constexpr std::array<TermKeyMap, 14> s_simpleModifiedKeyMapping{
-    TermKeyMap{ VK_BACK, CTRL_PRESSED, L"\x8" },
-    TermKeyMap{ VK_BACK, ALT_PRESSED, L"\x1b\x7f" },
-    TermKeyMap{ VK_BACK, CTRL_PRESSED | ALT_PRESSED, L"\x1b\x8" },
+    TermKeyMap{ VK_BACK, CTRL_PRESSED, L"\x7f"},
+    TermKeyMap{ VK_BACK, ALT_PRESSED, L"\x1b\x8" },
+    TermKeyMap{ VK_BACK, CTRL_PRESSED | ALT_PRESSED, L"\x1b\x7f" },

And the terminal seems to behave correctly with or without ENABLE_VIRTUAL_TERMINAL_INPUT

There are a few other places in the file where VK_BACK is mapped to x7f, so not sure If I should change these as well.

@xoofx commented on GitHub (Dec 30, 2021): I tested by modifying these two places in `src\terminal\input\terminalInput.cpp`: https://github.com/microsoft/terminal/blob/1e22bfa8d6f86d0546c3f162d171274cf453db65/src/terminal/input/terminalInput.cpp#L78-L81 ```diff static constexpr std::array<TermKeyMap, 19> s_keypadNumericMapping{ TermKeyMap{ VK_TAB, L"\x09" }, - TermKeyMap{ VK_BACK, L"\x7f" }, TermKeyMap{ VK_PAUSE, L"\x1a" }, ``` https://github.com/microsoft/terminal/blob/1e22bfa8d6f86d0546c3f162d171274cf453db65/src/terminal/input/terminalInput.cpp#L218-L221 ```diff static constexpr std::array<TermKeyMap, 14> s_simpleModifiedKeyMapping{ - TermKeyMap{ VK_BACK, CTRL_PRESSED, L"\x8" }, - TermKeyMap{ VK_BACK, ALT_PRESSED, L"\x1b\x7f" }, - TermKeyMap{ VK_BACK, CTRL_PRESSED | ALT_PRESSED, L"\x1b\x8" }, + TermKeyMap{ VK_BACK, CTRL_PRESSED, L"\x7f"}, + TermKeyMap{ VK_BACK, ALT_PRESSED, L"\x1b\x8" }, + TermKeyMap{ VK_BACK, CTRL_PRESSED | ALT_PRESSED, L"\x1b\x7f" }, ``` And the terminal seems to behave correctly with or without `ENABLE_VIRTUAL_TERMINAL_INPUT` There are a few other places in the file where VK_BACK is mapped to `x7f`, so not sure If I should change these as well.

claunia commented

@alexrp commented on GitHub (Dec 31, 2021):

@xoofx I think this might be relevant context: https://github.com/microsoft/terminal/issues/5957#issuecomment-630493811

@alexrp commented on GitHub (Dec 31, 2021): @xoofx I *think* this might be relevant context: https://github.com/microsoft/terminal/issues/5957#issuecomment-630493811

claunia commented

@xoofx commented on GitHub (Dec 31, 2021):

wow, ok, thanks for the link, what a mess 😅

So basically, the official terminal mapping is:

CTRL-H is usually mapped to \x8, e.g deletes a char on the left in a prompt shell
Backspace is usually mapped to \x7f, e.g deletes a char on the left in a prompt shell
CTRL+Backspace maps to \x8
CTRL+W maps to \x17 so Backword

While in powershell/cmd on Windows to, it is interpreted as:

CTRL-H = \x8
Backspace = \x8
CTRL+Backspace = \x7f (so Backword)
CTRL+W = \x17 (but not Backword)

And the issue here is that, when ENABLE_VIRTUAL_TERMINAL_INPUT we use the official terminal mapping, but with the Windows Terminal interpretation:

CTRL-H = \x8
Backspace = \x7f as Backword
CTRL+Backspace = \x8

But it's incompatible with vt/unix terminals and CTRL+Backspace is anyway mapped to \x8... 🤦

Could Windows Terminal interpret \x17as the actual Backword instead? and so \x8 and \x7f would both be interpreted as backspaces, and the terminal would choose to map CTRL+Backspace to \x17 (instead of \x8)

[Edit]I have edited multiple times this message to get my head around the problem, and I will end it as I started, what a mess 😞 )

@xoofx commented on GitHub (Dec 31, 2021): wow, ok, thanks for the link, what a mess 😅 So basically, the official terminal mapping is: - `CTRL-H` is usually mapped to `\x8`, e.g deletes a char on the left in a prompt shell - `Backspace` is usually mapped to `\x7f`, e.g deletes a char on the left in a prompt shell - `CTRL+Backspace` maps to `\x8` - `CTRL+W` maps to `\x17` so `Backword` While in powershell/cmd on Windows to, it is interpreted as: - `CTRL-H` = `\x8` - `Backspace` = `\x8` - `CTRL+Backspace` = `\x7f` (so `Backword`) - `CTRL+W` = `\x17` (but not `Backword`) And the issue here is that, when `ENABLE_VIRTUAL_TERMINAL_INPUT` we use the official terminal mapping, but with the Windows Terminal interpretation: - `CTRL-H` = `\x8` - `Backspace` = `\x7f` as `Backword` - `CTRL+Backspace` = `\x8` But it's incompatible with vt/unix terminals and `CTRL+Backspace` is anyway mapped to `\x8`... 🤦 Could Windows Terminal interpret `\x17`as the actual `Backword` instead? and so `\x8` and `\x7f` would both be interpreted as `backspaces`, and the terminal would choose to map `CTRL+Backspace` to `\x17` (instead of `\x8`) [Edit]I have edited multiple times this message to get my head around the problem, and I will end it as I started, what a mess 😞 )

claunia commented

@xoofx commented on GitHub (Dec 31, 2021):

So thinking more about it, in the presence of ENABLE_VIRTUAL_TERMINAL_INPUT, I would propose that wt:

Keeps CTRL-H = \x8
Keeps BackSpace = \x7f but interpret it as real backspace instead of transforming it to a backword
Could translate CTRL+Backspace to \x17 (so CTRL+W, Backword). If not keep CTRL+Backspace = \x8. I don't know how much Unix folks cares about CTRL+Backspace mapping to \x8.

@xoofx commented on GitHub (Dec 31, 2021): So thinking more about it, in the presence of `ENABLE_VIRTUAL_TERMINAL_INPUT`, I would propose that wt: - Keeps `CTRL-H` = `\x8` - Keeps `BackSpace` = `\x7f` but interpret it as real backspace instead of transforming it to a backword - Could translate `CTRL+Backspace` to `\x17` (so `CTRL+W`, `Backword`). If not keep `CTRL+Backspace` = `\x8`. I don't know how much Unix folks cares about `CTRL+Backspace` mapping to `\x8`.

claunia commented

@xoofx commented on GitHub (Dec 31, 2021):

Tested this fix proposal for readDataCooked.cpp by having both \x8 and \x7f deleting previous space when ENABLE_VIRTUAL_TERMINAL_INPUT is setup:

diff --git a/src/host/readDataCooked.cpp b/src/host/readDataCooked.cpp
index 169b0973..c33e6f2f 100644
--- a/src/host/readDataCooked.cpp
+++ b/src/host/readDataCooked.cpp
@@ -503,6 +503,8 @@ bool COOKED_READ_DATA::ProcessInput(const wchar_t wchOrig,
         return true;
     }
 
+    bool isInVirtualTerminalInputMode = _pInputBuffer->IsInVirtualTerminalInputMode();
+
     if (wch == EXTKEY_ERASE_PREV_WORD)
     {
         wch = UNICODE_BACKSPACE;
@@ -558,7 +560,7 @@ bool COOKED_READ_DATA::ProcessInput(const wchar_t wchOrig,
                     _currentPosition -= 1;
 
                     // Repeat until it hits the word boundary
-                    if (wchOrig == EXTKEY_ERASE_PREV_WORD &&
+                    if (!isInVirtualTerminalInputMode && wchOrig == EXTKEY_ERASE_PREV_WORD &&
                         _bufPtr != _backupLimit &&
                         fStartFromDelim ^ !IsWordDelim(_bufPtr[-1]))
                     {
@@ -636,7 +638,7 @@ bool COOKED_READ_DATA::ProcessInput(const wchar_t wchOrig,
                     NumSpaces = 0;
 
                     // Repeat until it hits the word boundary
-                    if (wchOrig == EXTKEY_ERASE_PREV_WORD &&
+                    if (!isInVirtualTerminalInputMode && wchOrig == EXTKEY_ERASE_PREV_WORD &&
                         _bufPtr != _backupLimit &&
                         fStartFromDelim ^ !IsWordDelim(_bufPtr[-1]))
                     {

@xoofx commented on GitHub (Dec 31, 2021): Tested this fix proposal for `readDataCooked.cpp` by having both `\x8` and `\x7f` deleting previous space when `ENABLE_VIRTUAL_TERMINAL_INPUT` is setup: ```diff diff --git a/src/host/readDataCooked.cpp b/src/host/readDataCooked.cpp index 169b0973..c33e6f2f 100644 --- a/src/host/readDataCooked.cpp +++ b/src/host/readDataCooked.cpp @@ -503,6 +503,8 @@ bool COOKED_READ_DATA::ProcessInput(const wchar_t wchOrig, return true; } + bool isInVirtualTerminalInputMode = _pInputBuffer->IsInVirtualTerminalInputMode(); + if (wch == EXTKEY_ERASE_PREV_WORD) { wch = UNICODE_BACKSPACE; @@ -558,7 +560,7 @@ bool COOKED_READ_DATA::ProcessInput(const wchar_t wchOrig, _currentPosition -= 1; // Repeat until it hits the word boundary - if (wchOrig == EXTKEY_ERASE_PREV_WORD && + if (!isInVirtualTerminalInputMode && wchOrig == EXTKEY_ERASE_PREV_WORD && _bufPtr != _backupLimit && fStartFromDelim ^ !IsWordDelim(_bufPtr[-1])) { @@ -636,7 +638,7 @@ bool COOKED_READ_DATA::ProcessInput(const wchar_t wchOrig, NumSpaces = 0; // Repeat until it hits the word boundary - if (wchOrig == EXTKEY_ERASE_PREV_WORD && + if (!isInVirtualTerminalInputMode && wchOrig == EXTKEY_ERASE_PREV_WORD && _bufPtr != _backupLimit && fStartFromDelim ^ !IsWordDelim(_bufPtr[-1])) { ```

claunia commented

@alexrp commented on GitHub (Feb 15, 2023):

Hi folks, any thoughts on @xoofx's comments above?

@alexrp commented on GitHub (Feb 15, 2023): Hi folks, any thoughts on @xoofx's comments above?

claunia commented