Default all G-sets to ASCII unless ISO-2022 is requested (was: Terminal breaks on \x1b O) #14211

Closed
opened 2026-01-31 04:03:53 +00:00 by claunia · 24 comments
Owner

Originally created by @iamdbychkov on GitHub (Jun 11, 2021).

Windows Terminal version (or Windows build number)

1.8.1521.0

Other Software

OpenSSH_5.3p1

Steps to reproduce

  1. SSH to linux server (for me it's CentOS, if it matters)
  2. echo -e '\x1bo'

Expected Behavior

I expect that terminal will continue to work properly

Actual Behavior

I can no longer type "actual" text in terminal, every key press transforms in a mess seen on screenshot. CTRL+Something sequences are also no longer working.

изображение

Stumbled upon this issue when I was debugging gsm7 encodings in python program (\x1b is a common byte for this encoding). On yet another 'print()' terminal became a mess.

Issue is not reproduced if i'm using powershell.exe or cmd.exe without it being embeded in WT.

Originally created by @iamdbychkov on GitHub (Jun 11, 2021). ### Windows Terminal version (or Windows build number) 1.8.1521.0 ### Other Software OpenSSH_5.3p1 ### Steps to reproduce 1. SSH to linux server (for me it's CentOS, if it matters) 2. `echo -e '\x1bo'` ### Expected Behavior I expect that terminal will continue to work properly ### Actual Behavior I can no longer type "actual" text in terminal, every key press transforms in a mess seen on screenshot. CTRL+Something sequences are also no longer working. ![изображение](https://user-images.githubusercontent.com/53076075/121656464-2c363a80-caa8-11eb-9afa-4dbc3209debf.png) Stumbled upon this issue when I was debugging gsm7 encodings in python program (\x1b is a common byte for this encoding). On yet another 'print()' terminal became a mess. Issue is not reproduced if i'm using powershell.exe or cmd.exe without it being embeded in WT.
Author
Owner

@KalleOlaviNiemitalo commented on GitHub (Jun 11, 2021):

ESC 06/15 is LS3 - LOCKING-SHIFT THREE. That was implemented in https://github.com/microsoft/terminal/pull/4496.

Have you tried the same command in a different terminal, such as xterm or in the Linux console? I expect you'd get the same result with those.

@KalleOlaviNiemitalo commented on GitHub (Jun 11, 2021): ESC 06/15 is LS3 - LOCKING-SHIFT THREE. That was implemented in <https://github.com/microsoft/terminal/pull/4496>. Have you tried the same command in a different terminal, such as xterm or in the Linux console? I expect you'd get the same result with those.
Author
Owner

@iamdbychkov commented on GitHub (Jun 11, 2021):

ESC 06/15 is LS3 - LOCKING-SHIFT THREE. That was implemented in #4496.

Have you tried the same command in a different terminal, such as xterm or in the Linux console? I expect you'd get the same result with those.

I have no problem running mentioned command on the same server while using putty as my gui ssh client / winsshterm (which is putty)/penguinet and no problems with Debian's default console.

@iamdbychkov commented on GitHub (Jun 11, 2021): > > > ESC 06/15 is LS3 - LOCKING-SHIFT THREE. That was implemented in #4496. > > Have you tried the same command in a different terminal, such as xterm or in the Linux console? I expect you'd get the same result with those. I have no problem running mentioned command on the same server while using putty as my gui ssh client / winsshterm (which is putty)/penguinet and no problems with Debian's default console.
Author
Owner

@KalleOlaviNiemitalo commented on GitHub (Jun 11, 2021):

From https://www.man7.org/linux/man-pages/man4/console_codes.4.html, I get the impression that the Linux console does not implement LS3, while xterm does.

@KalleOlaviNiemitalo commented on GitHub (Jun 11, 2021): From <https://www.man7.org/linux/man-pages/man4/console_codes.4.html>, I get the impression that the Linux console does not implement LS3, while xterm does.
Author
Owner

@KalleOlaviNiemitalo commented on GitHub (Jun 11, 2021):

Is there already a feature request for an action that resets the character sets to the defaults? Similar to https://github.com/microsoft/terminal/issues/1882 for the scrollback.

@KalleOlaviNiemitalo commented on GitHub (Jun 11, 2021): Is there already a feature request for an action that resets the character sets to the defaults? Similar to <https://github.com/microsoft/terminal/issues/1882> for the scrollback.
Author
Owner

@j4james commented on GitHub (Jun 11, 2021):

FYI, typing reset in bash should fix this. I also sometimes include a soft reset (\x1b[!p) in my prompt, which fix most issues like this automatically. But an action would be good too - I think most terminal emulators have something like that. I'm sure we've discussed this before actually, but I can't find an existing issue for it now, so maybe I'm misremembering.

@j4james commented on GitHub (Jun 11, 2021): FYI, typing `reset` in bash should fix this. I also sometimes include a soft reset (`\x1b[!p`) in my prompt, which fix most issues like this automatically. But an action would be good too - I think most terminal emulators have something like that. I'm sure we've discussed this before actually, but I can't find an existing issue for it now, so maybe I'm misremembering.
Author
Owner

@iamdbychkov commented on GitHub (Jun 13, 2021):

Yes, thank you, typing reset in bash indeed worked, but unfortunately it won't work in the environments like python console, one in which I initially discovered the issue

@iamdbychkov commented on GitHub (Jun 13, 2021): Yes, thank you, typing reset in bash indeed worked, but unfortunately it won't work in the environments like python console, one in which I initially discovered the issue
Author
Owner

@j4james commented on GitHub (Jun 13, 2021):

If it happens again in a python console, you can use print '\x1bc' to reset. Not suggesting that's a good long term solution, but it's better than having to exit and restart.

@j4james commented on GitHub (Jun 13, 2021): If it happens again in a python console, you can use `print '\x1bc'` to reset. Not suggesting that's a good long term solution, but it's better than having to exit and restart.
Author
Owner

@WSLUser commented on GitHub (Jun 14, 2021):

Other Software
OpenSSH_5.3p1

Can you bump your Win32-OpenSSH to 8.6? https://github.com/PowerShell/Win32-OpenSSH/releases

Before 8.1, it was handling VT sequences itself instead of allowing conhost to do so instead.

@WSLUser commented on GitHub (Jun 14, 2021): > Other Software > OpenSSH_5.3p1 Can you bump your Win32-OpenSSH to 8.6? https://github.com/PowerShell/Win32-OpenSSH/releases Before 8.1, it was handling VT sequences itself instead of allowing conhost to do so instead.
Author
Owner

@iamdbychkov commented on GitHub (Jun 15, 2021):

Can you bump your Win32-OpenSSH to 8.6? https://github.com/PowerShell/Win32-OpenSSH/releases

Before 8.1, it was handling VT sequences itself instead of allowing conhost to do so instead.

Sadly, it didn't make a difference:

PS C:\Program Files\OpenSSH> ./ssh -V
OpenSSH_for_Windows_8.6p1, LibreSSL 3.3.3
PS C:\Program Files\OpenSSH> ./ssh dev35p4
[dbychkov@dev35p4 ~]$ echo -e '\x1bo'

ÛäâùãèëïöÀäåö³µð´ þݤ 

@iamdbychkov commented on GitHub (Jun 15, 2021): > Can you bump your Win32-OpenSSH to 8.6? https://github.com/PowerShell/Win32-OpenSSH/releases > > Before 8.1, it was handling VT sequences itself instead of allowing conhost to do so instead. Sadly, it didn't make a difference: >PS C:\Program Files\OpenSSH> ./ssh -V >OpenSSH_for_Windows_8.6p1, LibreSSL 3.3.3 >PS C:\Program Files\OpenSSH> ./ssh dev35p4 >[dbychkov@dev35p4 ~]$ echo -e '\x1bo' > >ÛäâùãèëïöÀäåö³µð´ þݤ 
Author
Owner

@WSLUser commented on GitHub (Jun 15, 2021):

Did you refresh env or start a new Terminal instance before trying again? Also make sure if you installed through Optional Features, that you remove the Optional Feature.

@WSLUser commented on GitHub (Jun 15, 2021): Did you refresh env or start a new Terminal instance before trying again? Also make sure if you installed through Optional Features, that you remove the Optional Feature.
Author
Owner

@j4james commented on GitHub (Jun 15, 2021):

I'm fairly certain OpenSSH has nothing to do with this. As KalleOlaviNiemitalo pointed out at the start, LS3 is a standard control sequence that is intended to change the character set - it's working as designed. You'll get the exact same behaviour in a local shell. What we're missing, is a way for users to easily reset the terminal when it gets into a state that they weren't expecting.

@j4james commented on GitHub (Jun 15, 2021): I'm fairly certain OpenSSH has nothing to do with this. As KalleOlaviNiemitalo pointed out at the start, LS3 is a standard control sequence that is intended to change the character set - it's working as designed. You'll get the exact same behaviour in a local shell. What we're missing, is a way for users to easily reset the terminal when it gets into a state that they weren't expecting.
Author
Owner

@zadjii-msft commented on GitHub (Jun 15, 2021):

It seems totally reasonable to me for us to have a "Reset Terminal" command ~~that's just a sendInput("\x1bc") command,~~bound by default. Good thinking.

It can't just be a sendInput command - it'll need to immediately trigger it like it was sent as client output, but yea it's a good idea.

@zadjii-msft commented on GitHub (Jun 15, 2021): It seems totally reasonable to me for us to have a "Reset Terminal" command ~~that's just a `sendInput("\x1bc")` command,~~bound by default. Good thinking. It can't just be a sendInput command - it'll need to immediately trigger it like it was sent as client _output_, but yea it's a good idea.
Author
Owner

@KalleOlaviNiemitalo commented on GitHub (Jun 15, 2021):

I'm not sure LS3 is really designed to work here, because it is defined in ECMA-35 (which claims to be identical to ISO/IEC 2022:1994), and UTF-8 is registered as a "Coding system different from that of ISO 2022". Perhaps a terminal that starts in UTF-8 mode could ignore LS3 and other ECMA-35 features until it receives DOCS (DESIGNATE OTHER CODING SYSTEM) for switching to ECMA-35 mode.

On the other hand, if Windows Terminal impersonates xterm, and xterm supports LS3, then I suppose Windows Terminal has to likewise support LS3.

@KalleOlaviNiemitalo commented on GitHub (Jun 15, 2021): I'm not sure LS3 is really designed to work here, because it is defined in ECMA-35 (which claims to be identical to ISO/IEC 2022:1994), and UTF-8 is [registered](<https://www.itscj-ipsj.jp/custom_contents/cms/linkfile/ISO-IR.pdf> "ISO International Register of Coded Character Sets To Be Used With Escape Sequences") as a "Coding system different from that of ISO 2022". Perhaps a terminal that starts in UTF-8 mode could ignore LS3 and other ECMA-35 features until it receives DOCS (DESIGNATE OTHER CODING SYSTEM) for switching to ECMA-35 mode. On the other hand, if Windows Terminal impersonates xterm, and xterm supports LS3, then I suppose Windows Terminal has to likewise support LS3.
Author
Owner

@j4james commented on GitHub (Jun 15, 2021):

Perhaps a terminal that starts in UTF-8 mode could ignore LS3 and other ECMA-35 features until it receives DOCS.

I had considered that, but the problem is that a lot of existing software expects a certain amount of ISO-2022 functionality even in UTF-8 mode - the most common case being to select the DEC line drawing character set.

@j4james commented on GitHub (Jun 15, 2021): > Perhaps a terminal that starts in UTF-8 mode could ignore LS3 and other ECMA-35 features until it receives DOCS. I had considered that, but the problem is that a lot of existing software expects a certain amount of ISO-2022 functionality even in UTF-8 mode - the most common case being to select the DEC line drawing character set.
Author
Owner

@j4james commented on GitHub (Jun 15, 2021):

It seems totally reasonable to me for us to have a "Reset Terminal" command ~~that's just a sendInput("\x1bc") command,~~bound by default.

Note that this has similar problems to #1882, in that we need the reset to occur on the conhost side of the conpty pipe. So we either need something like #1193, or in the long term #1173.

@j4james commented on GitHub (Jun 15, 2021): > It seems totally reasonable to me for us to have a "Reset Terminal" command ~~that's just a `sendInput("\x1bc")` command,~~bound by default. Note that this has similar problems to #1882, in that we need the reset to occur on the conhost side of the conpty pipe. So we either need something like #1193, or in the long term #1173.
Author
Owner

@orcmid commented on GitHub (Jun 15, 2021):

@KalleOlaviNiemitalo LS3 is defined in ECMA-35 (which claims to be identical to ISO/IEC 2022:1994), and UTF-8 is registered as a "Coding system different from that of ISO 2022". Perhaps a terminal that starts in UTF-8 mode could ignore LS3 and other ECMA-35 features until it receives [a change of scheme sequence]

On the other hand, if Windows Terminal impersonates xterm, and xterm supports LS3, then I suppose Windows Terminal has to likewise support LS3.

It is stated in the roadmap that WT VT (Virtual Terminal) relies on ECMA-48, which does appeal to ECMA-35. If UTF8 is usable with VT, then some ECMA-35 fidgets might not be directly possible, since UTF8 commandeers the higher order bit of octets and the client/shell might need to cope with that.

Is the line-drawing different than the pixel-graphics that have also been asked for? There are line-drawing code points in Unicode though.

[I have not dug deep into this, just pondering that the ECNA-48 and UTF8 and Virtual Terminal need to be examined for exactly how they apply for Window Terminal.]

@orcmid commented on GitHub (Jun 15, 2021): > @KalleOlaviNiemitalo LS3 is defined in ECMA-35 (which claims to be identical to ISO/IEC 2022:1994), and UTF-8 is [registered](https://www.itscj-ipsj.jp/custom_contents/cms/linkfile/ISO-IR.pdf) as a "Coding system different from that of ISO 2022". Perhaps a terminal that starts in UTF-8 mode could ignore LS3 and other ECMA-35 features until it receives [a change of scheme sequence] > > On the other hand, if Windows Terminal impersonates xterm, and xterm supports LS3, then I suppose Windows Terminal has to likewise support LS3. It is stated in the [roadmap](https://docs.microsoft.com/en-us/windows/console/ecosystem-roadmap) that WT VT (Virtual Terminal) relies on ECMA-48, which does appeal to ECMA-35. If UTF8 is usable with VT, then some ECMA-35 fidgets might not be directly possible, since UTF8 commandeers the higher order bit of octets and the client/shell might need to cope with that. Is the line-drawing different than the pixel-graphics that have also been asked for? There are line-drawing code points in Unicode though. [I have not dug deep into this, just pondering that the ECNA-48 and UTF8 and Virtual Terminal need to be examined for exactly how they apply for Window Terminal.]
Author
Owner

@DHowett commented on GitHub (Jun 17, 2021):

I'm of two minds on this. The one that's currently winning the debate for me right now is that these are control sequences. They're not supposed to be emitted unless an attempt is being made to control a thing.

Is the line-drawing different than the pixel-graphics that have also been asked for? There are line-drawing code points in Unicode though.

Yep- line drawing (emitting full-cell glyphs from the unicode range U+2500) is a tried-and-true method of drawing TUI-like interfaces. One of the character sets, the DEC Technical Set, allows you to access them in the sub-0x7F range.

I'm inclined to follow what other terminal emulators do here. What do gnome-terminal, Konsole and Alacritty do here?

@DHowett commented on GitHub (Jun 17, 2021): I'm of two minds on this. The one that's currently winning the debate for me right now is that _these are control sequences_. They're not supposed to be emitted unless an attempt is being made to _control_ a thing. > Is the line-drawing different than the pixel-graphics that have also been asked for? There are line-drawing code points in Unicode though. Yep- line drawing (emitting full-cell glyphs from the unicode range U+2500) is a tried-and-true method of drawing TUI-like interfaces. One of the character sets, the DEC Technical Set, allows you to access them in the sub-0x7F range. I'm inclined to follow what other terminal emulators do here. What do gnome-terminal, Konsole and Alacritty do here?
Author
Owner

@KalleOlaviNiemitalo commented on GitHub (Jun 20, 2021):

Gnome-terminal uses VTE, from which most ISO 2022 support was removed in 2014 after discussion at https://bugzilla.gnome.org/show_bug.cgi?id=732586

vte/src/parser.cc has a lot of code for parsing the escape sequences that designate or invoke character sets, but vte/src/vteseq.cc now supports only the default charset and line drawing. It uses SI=LS0 and SO=LS1 for this purpose and ignores all other shifts and all character set designations. Also, vte/src/vte.cc decodes UTF-8 and passes Unicode scalar values to the parser, so the line-drawing charset does not disable UTF-8 decoding. Instead, if the line-drawing charset is in use, then vte/src/vte.cc replaces some ASCII characters with line-drawing characters after they come from the parser.

@KalleOlaviNiemitalo commented on GitHub (Jun 20, 2021): Gnome-terminal uses VTE, from which most ISO 2022 support was removed in 2014 after discussion at <https://bugzilla.gnome.org/show_bug.cgi?id=732586> vte/src/parser.cc has a lot of code for parsing the escape sequences that designate or invoke character sets, but vte/src/vteseq.cc now supports only the default charset and line drawing. It uses SI=LS0 and SO=LS1 for this purpose and ignores all other shifts and all character set designations. Also, vte/src/vte.cc decodes UTF-8 and passes Unicode scalar values to the parser, so the line-drawing charset does not disable UTF-8 decoding. Instead, if the line-drawing charset is in use, then vte/src/vte.cc replaces some ASCII characters with line-drawing characters after they come from the parser.
Author
Owner

@j4james commented on GitHub (Jun 20, 2021):

I thought of a couple of possible solutions to this.

  1. We just default all the G-sets to ASCII, instead of having Latin1 in G2 and G3 (even XTerm seems to do this). For anyone that needs strict DEC character set compatibility we could always set them correctly when ISO-2022 mode is enabled.
  2. We add support for DECAUPSS (assign user preference supplemental set), and default that to ASCII. Since G2 and G3 are meant to be initialized with UPSS, they'd be ASCII by default, but users would still have a way to change that if necessary.

This gives us the best of both worlds. Users that don't care about character set designations will be much less likely to accidentally switch them, but that functionality is still there for anyone that actually needs it.

@j4james commented on GitHub (Jun 20, 2021): I thought of a couple of possible solutions to this. 1. We just default all the G-sets to ASCII, instead of having Latin1 in G2 and G3 (even XTerm seems to do this). For anyone that needs strict DEC character set compatibility we could always set them correctly when ISO-2022 mode is enabled. 2. We add support for `DECAUPSS` (assign user preference supplemental set), and default that to ASCII. Since G2 and G3 are meant to be initialized with UPSS, they'd be ASCII by default, but users would still have a way to change that if necessary. This gives us the best of both worlds. Users that don't care about character set designations will be much less likely to accidentally switch them, but that functionality is still there for anyone that actually needs it.
Author
Owner

@ghost commented on GitHub (Jun 24, 2021):

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

@ghost commented on GitHub (Jun 24, 2021): This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for **4 days**. It will be closed if no further activity occurs **within 3 days of this comment**.
Author
Owner

@DHowett commented on GitHub (Jul 6, 2021):

It seems like if we do 1, we can later do 2 with no perceived change to the user experience. Should we be storing the user preference supplemental set somewhere, when we do do 2?

@DHowett commented on GitHub (Jul 6, 2021): It seems like if we do 1, we can later do 2 with no perceived change to the user experience. Should we be storing the user preference supplemental set somewhere, when we _do_ do 2?
Author
Owner

@j4james commented on GitHub (Jul 7, 2021):

It seems like if we do 1, we can later do 2 with no perceived change to the user experience.

Yeah, that make sense to me. Although we may want to skip the bit about the setting the G-sets correctly when switching to ISO-2022 mode if we want to remain forward-compatible with DECAUPSS being added later. I'm not totally sure about that, but either way is OK with me. As long as there is some series of escape sequences a user can send to get things the way they want, then I think we're good.

Should we be storing the user preference supplemental set somewhere, when we do do 2?

If you mean in a setting, then ideally yes. It was certainly a user-controlled setting in the original hardware terminals, and I suspect most real terminal emulators would have a setting for it. If combined with a way to start up in ISO-2022 mode, it would enable users to easily run legacy 8-bit applications designed for a particular code page.

That said, I don't think there's a huge demand for that sort of thing anymore, so it's not essential if you don't want WT to go that route. Also, it would have to wait until we have a proper conpty pass-through implementation, since all the character set mapping is currently handled on the conhost side.

@j4james commented on GitHub (Jul 7, 2021): > It seems like if we do 1, we can later do 2 with no perceived change to the user experience. Yeah, that make sense to me. Although we may want to skip the bit about the setting the G-sets correctly when switching to ISO-2022 mode if we want to remain forward-compatible with `DECAUPSS` being added later. I'm not totally sure about that, but either way is OK with me. As long as there is some series of escape sequences a user can send to get things the way they want, then I think we're good. > Should we be storing the user preference supplemental set somewhere, when we _do_ do 2? If you mean in a setting, then ideally yes. It was certainly a user-controlled setting in the original hardware terminals, and I suspect most real terminal emulators would have a setting for it. If combined with a way to start up in ISO-2022 mode, it would enable users to easily run legacy 8-bit applications designed for a particular code page. That said, I don't think there's a huge demand for that sort of thing anymore, so it's not essential if you don't want WT to go that route. Also, it would have to wait until we have a proper conpty pass-through implementation, since all the character set mapping is currently handled on the conhost side.
Author
Owner

@ghost commented on GitHub (Dec 14, 2021):

:tada:This issue was addressed in #11658, which has now been successfully released as Windows Terminal Preview v1.12.3472.0.🎉

Handy links:

@ghost commented on GitHub (Dec 14, 2021): :tada:This issue was addressed in #11658, which has now been successfully released as `Windows Terminal Preview v1.12.3472.0`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.12.3472.0) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Author
Owner

@ghost commented on GitHub (Dec 14, 2021):

:tada:This issue was addressed in #11658, which has now been successfully released as Windows Terminal v1.11.3471.0.🎉

Handy links:

@ghost commented on GitHub (Dec 14, 2021): :tada:This issue was addressed in #11658, which has now been successfully released as `Windows Terminal v1.11.3471.0`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.11.3471.0) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#14211