Add support for or ignore legacy SI/SO encodings #4874

@zadjii-msft commented on GitHub (Nov 8, 2019):

@otmos Could you copy and paste that character here, or tell us what character it is?

Does that repro if you ssh to the same server using the legacy console window (by launching Debian directly)?

Does htop display that character in the Terminal without sshing?

Does the character appear in other commandline applications as well, or just htop?

@zadjii-msft commented on GitHub (Nov 8, 2019): @otmos Could you copy and paste that character here, or tell us what character it is? Does that repro if you ssh to the same server using the legacy console window (by launching Debian directly)? Does `htop` display that character in the Terminal _without_ `ssh`ing? Does the character appear in other commandline applications as well, or just `htop`?

claunia commented

@otmos commented on GitHub (Nov 12, 2019):

On local , the character doesn't show at all , everything is normal both on terminal and legacy console:

but in SSH , This character appears everywhere , in nano and even in the shell itself .

tried to copy it from nano , pasting it on notepad gave this

but searching it on google chrome gave a question mark

@otmos commented on GitHub (Nov 12, 2019): On local , the character doesn't show at all , everything is normal both on terminal and legacy console: ![Screenshot (13)](https://user-images.githubusercontent.com/43470049/68655912-288bac00-0531-11ea-874c-53281d05c0a6.png) but in SSH , This character appears everywhere , in nano and even in the shell itself . ![Screenshot (12)](https://user-images.githubusercontent.com/43470049/68655935-317c7d80-0531-11ea-8718-c1e557b51c2d.png) tried to copy it from nano , pasting it on notepad gave this ![Screenshot (14)](https://user-images.githubusercontent.com/43470049/68655958-3d683f80-0531-11ea-850f-5d6a5de6d834.png) but searching it on google chrome gave a question mark ![Screenshot (15)](https://user-images.githubusercontent.com/43470049/68655971-42c58a00-0531-11ea-80fe-4434be74f20c.png)

claunia commented

@zadjii-msft commented on GitHub (Nov 12, 2019):

@miniksa does this look like the replacement character thing you fixed?

Helpful to know that A: this isn't unique to the Windows Terminal and B: this only repros over ssh.

@zadjii-msft commented on GitHub (Nov 12, 2019): @miniksa does this look like the replacement character thing you fixed? Helpful to know that A: this isn't unique to the Windows Terminal and B: this only repros over ssh.

claunia commented

@miniksa commented on GitHub (Nov 12, 2019):

No, not quite, @zadjii-msft.

@otmos... fill me in on the entire chain here. It could be any of the pieces involved.

I need you to fill in the blanks on this set of steps for me.

I'm starting from a Windows 10 1903 machine
I open a WSL ____(v1/v2)_____ instance in (___default conhost, I think___) that is using ____(Debian from the Windows Store, I think)____
Then I run ____(ssh.exe username@remotehost, with no other args, I think)____
After I log in to the ____(What version and flavor of Linux machine)____, I immediately run ____(htop, I think)_____.
Then I see the boxes.
I close the program. Then I run it again, redirecting it to a file by calling program > foo.txt.
It either runs to completion or I press Ctrl+C to end it after a moment or two.
I attached foo.txt here: ____(uploaded foo.txt)____

@miniksa commented on GitHub (Nov 12, 2019): No, not quite, @zadjii-msft. @otmos... fill me in on the entire chain here. It could be any of the pieces involved. I need you to fill in the blanks on this set of steps for me. 1. I'm starting from a Windows 10 1903 machine 2. I open a WSL `____(v1/v2)_____` instance in (`___default conhost, I think___`) that is using `____(Debian from the Windows Store, I think)____` 3. Then I run `____(ssh.exe username@remotehost, with no other args, I think)____` 4. After I log in to the `____(What version and flavor of Linux machine)____`, I immediately run `____(htop, I think)_____`. 5. Then I see the boxes. 6. I close the program. Then I run it again, redirecting it to a file by calling `program > foo.txt`. 7. It either runs to completion or I press <kbd>Ctrl</kbd>+<kbd>C</kbd> to end it after a moment or two. 8. I attached `foo.txt` here: `____(uploaded foo.txt)____`

claunia commented

@otmos commented on GitHub (Nov 13, 2019):

I'm starting from a Windows 10 1903 machine
I open a WSL V1 instance in default conhost or Windows Terminal that is using Debian from the Windows Store
Then I run ssh.exe username@remotehost
After I log in to the DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS" , I immediately run Htop or nano then i begin to see the boxes.

I close the program.
Then I run it again, redirecting it to a file by calling program > foo.txt.
It either runs to completion or I press Ctrl+C to end it after a moment or two.
I attached foo.txt here:
foo.txt
Same machine on ssh from a Cloud9 IDE :

@otmos commented on GitHub (Nov 13, 2019): I'm starting from a Windows 10 1903 machine I open a WSL **V1** instance in **default conhost** or **Windows Terminal** that is using **Debian from the Windows Store** Then I run **ssh.exe username@remotehost** After I log in to the **DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"** , I immediately run Htop or nano then i begin to see the boxes. ![image](https://user-images.githubusercontent.com/43470049/68748639-0bbcaa80-05fd-11ea-932c-de07041901d7.png) I close the program. Then I run it again, redirecting it to a file by calling program > foo.txt. It either runs to completion or I press Ctrl+C to end it after a moment or two. I attached foo.txt here: [foo.txt](https://github.com/microsoft/terminal/files/3840276/foo.txt) Same machine on ssh from a Cloud9 IDE : ![image](https://user-images.githubusercontent.com/43470049/68749797-1f691080-05ff-11ea-8ff8-09526fac37e8.png)

claunia commented

@miniksa commented on GitHub (Nov 13, 2019):

The boxes are literally 0x0f in the stream, per what's in foo.txt. @zadjii-msft, any ideas? Could it be some setting in the termcap?

EDIT. There's also 0x5e 0x4f pairs for a ^O in the stream a bunch too.

@miniksa commented on GitHub (Nov 13, 2019): The boxes are literally `0x0f` in the stream, per what's in `foo.txt`. @zadjii-msft, any ideas? Could it be some setting in the termcap? EDIT. There's also `0x5e 0x4f` pairs for a ^O in the stream a bunch too.

claunia commented

@zadjii-msft commented on GitHub (Nov 13, 2019):

I haven't the darndest clue why we'd be getting "Shift In" (0xf, ^O) in the buffer ever. @j4james/ @egmontkob and ideas where those might come from?

@zadjii-msft commented on GitHub (Nov 13, 2019): I haven't the darndest clue why we'd be getting "Shift In" (0xf, ^O) in the buffer ever. @j4james/ @egmontkob and ideas where those might come from?

claunia commented

@egmontkob commented on GitHub (Nov 13, 2019):

I was staring at this the other day, and no, I have no idea. I don't recall ever seeing something like this.

foo.txt doesn't contain what you guys were asking for. It contains a 0x0f at the beginning and near its end, but the rest are literal caret ^ character followed by literal letter O or bracket [ or alike, no escape character in sight. It's sure not the raw output of htop. However, it might easily be the output of htop converted to these human-friendly ^-notations, e.g. by cat -v. And apparently whoever did this convertion also inserted actual ^O (0x0f aka. "shift in") characters??? Someone somehow broadcasting this character to all open file descriptors? Doesn't make too much sense.

Let's gzip it before uploading, that guarantees binary safe transfer (or, at least, it can be detected if the file gets corrupted).

Also, chances are that the app produces different output when redirected. Instead of redirection I recommend to launch script, then reproduce the bug, then quit script's shell (it tells you "Script done"), and then attach the (gzipped) typescript file.

The nano screenshot is IMO pretty misleading, as I have no idea what file is opened there, e.g. whether those rectangles are part of the file's contents or the actual bug we're hunting. Only the bottom rows prove that it's indeed buggy.

Could be a broken terminfo. Would be nice to see $TERM here vs. from that other terminal. Although the other terminal could just support ^O "shift in" and thus silently swallow these. I can't even exclude the possibility of a hardware issue, e.g. a transient memory failure resulting in broken terminfo residing in the memory. I'm not saying it's likely, but possible. Would be to see if the problem persists after the remote machine is rebooted, or is gone for good and we're left with an unresolved mystery.

@egmontkob commented on GitHub (Nov 13, 2019): I was staring at this the other day, and no, I have no idea. I don't recall ever seeing something like this. `foo.txt` doesn't contain what you guys were asking for. It contains a 0x0f at the beginning and near its end, but the rest are literal caret `^` character followed by literal letter `O` or bracket `[` or alike, no escape character in sight. It's sure not the raw output of htop. However, it might easily be the output of htop converted to these human-friendly `^`-notations, e.g. by `cat -v`. And apparently whoever did this convertion also inserted actual `^O` (0x0f aka. "shift in") characters??? Someone somehow broadcasting this character to all open file descriptors? Doesn't make too much sense. Let's `gzip` it before uploading, that guarantees binary safe transfer (or, at least, it can be detected if the file gets corrupted). Also, chances are that the app produces different output when redirected. Instead of redirection I recommend to launch `script`, then reproduce the bug, then quit script's shell (it tells you "Script done"), and then attach the (gzipped) `typescript` file. --- The nano screenshot is IMO pretty misleading, as I have no idea what file is opened there, e.g. whether those rectangles are part of the file's contents or the actual bug we're hunting. Only the bottom rows prove that it's indeed buggy. Could be a broken terminfo. Would be nice to see `$TERM` here vs. from that other terminal. Although the other terminal could just support `^O` "shift in" and thus silently swallow these. I can't even exclude the possibility of a hardware issue, e.g. a transient memory failure resulting in broken terminfo residing in the memory. I'm not saying it's likely, but possible. Would be to see if the problem persists after the remote machine is rebooted, or is gone for good and we're left with an unresolved mystery.

claunia commented

@egmontkob commented on GitHub (Nov 13, 2019):

In foo.txt, exactly the SGR sequences that reset all the attributes (and maybe enable bold), namely ^[[m, ^[[0m and ^[[0;1m are followed by ^O.

Sequences that tamper with the colors, such as ^[[30m, ^[[39;49m etc. are not followed by ^O.

(There's no SGR sequence in the file that would contain both 0/1 and 3x/4x.)

My best bet at this moment is an ncurses/terminfo bug (or hardware bug triggering faulty ncurses behavior), combined with that other emulator supporting and thus no-op'ing on SI.

What's the output of tput sgr0 | cat -v?

Oh, wait a sec!!!

On my Ubuntu 19.10, the output of this is ^[(B^[[m (designates US charset to G0), which escape sequence is closely related to SI.

Could it be that emitting that SI was a feature of some older ncurses/terminfo?

@egmontkob commented on GitHub (Nov 13, 2019): In foo.txt, exactly the SGR sequences that reset all the attributes (and maybe enable bold), namely `^[[m`, `^[[0m` and `^[[0;1m` are followed by `^O`. Sequences that tamper with the colors, such as `^[[30m`, `^[[39;49m` etc. are _not_ followed by `^O`. (There's no SGR sequence in the file that would contain both 0/1 and 3x/4x.) My best bet at this moment is an ncurses/terminfo bug (or hardware bug triggering faulty ncurses behavior), combined with that other emulator supporting and thus no-op'ing on SI. What's the output of `tput sgr0 | cat -v`? Oh, wait a sec!!! On my Ubuntu 19.10, the output of this is `^[(B^[[m` (designates US charset to G0), which escape sequence is closely related to SI. Could it be that emitting that SI was a _feature_ of some older ncurses/terminfo?

claunia commented

@egmontkob commented on GitHub (Nov 13, 2019):

Excerpt from ncurses-6.1/misc/terminfo.src:

iTerm2.app [...] sgr0=\E[m\017

and similar for e.g. linux-vt, putty, rxvt and quite a few others.

Apparently emitting SI is a feature of ncurses/terminfo.

At this point you can either blame the user for using an incorrect terminfo (which a perfectly valid blame as it might result in other problems too), or implement SI/SO :)

@egmontkob commented on GitHub (Nov 13, 2019): Excerpt from ncurses-6.1/misc/terminfo.src: iTerm2.app [...] sgr0=\E[m\017 and similar for e.g. linux-vt, putty, rxvt and quite a few others. Apparently emitting SI _is_ a feature of ncurses/terminfo. At this point you can either blame the user for using an incorrect terminfo (which a perfectly valid blame as it might result in other problems too), or implement SI/SO :)

claunia commented

@zadjii-msft commented on GitHub (Nov 13, 2019):

@egmontkob thanks for the great breakdown. I think there's probably two steps for us here:

Eat the SI/SO characters, to have users with this terminfo not totally broken.
Actually implement SI/SO.

I'd prefer break it down like this. Yea the terminfo is probably a bad terminfo and yea the user should change that, but also, we shouldn't just explode like this in that scenario. 1 is trivial enough to unblock this case, while (it sounds like) 2 is probably not that high of a priority feature that we can put off for a while.

@zadjii-msft commented on GitHub (Nov 13, 2019): @egmontkob thanks for the great breakdown. I think there's probably two steps for us here: 1. Eat the SI/SO characters, to have users with this terminfo not totally broken. 2. Actually implement SI/SO. I'd prefer break it down like this. Yea the terminfo is probably a bad terminfo and yea the user should change that, but also, we shouldn't just explode like this in that scenario. 1 is trivial enough to unblock this case, while (it sounds like) 2 is probably not that high of a priority feature that we can put off for a while.

claunia commented

@egmontkob commented on GitHub (Nov 13, 2019):

I'd prefer break it down like this.

This would've been my conclusion too if it wasn't past midnight here :D

@egmontkob commented on GitHub (Nov 13, 2019): > I'd prefer break it down like this. This would've been my conclusion too if it wasn't past midnight here :D

claunia commented

@otmos commented on GitHub (Nov 14, 2019):

Here is a screenshot of nano trying to write into a new file if that will help

I tried to unset the $term variable too but nothing changed

@otmos commented on GitHub (Nov 14, 2019): Here is a screenshot of nano trying to write into a new file if that will help ![image](https://user-images.githubusercontent.com/43470049/68846037-ad5efd00-06cc-11ea-9b0d-93fdc3fd60ed.png) I tried to unset the $term variable too but nothing changed

claunia commented

@egmontkob commented on GitHub (Nov 15, 2019):

And just to complete the picture:

Something that initially confused me was that the behavior seemed inconsistent. E.g. most of the time there's a rectangle when switching from grey to white, but no rectangle when switching from white to blue. However, in the very first screenshot, the row of PID 1334 is an exception to both of these "patterns".

The explanation to this is that probably we don't see the initial screen right after htop starts up, but after a couple of repaints, during which ncurses probably optimized by jumping over areas that it believed hadn't changed. And of course the reality was already off compared to ncurses's belief, so it could even matter if ncurses used relative or absolute cursor movement. Anyway, it's easy to imagine the aforementioned visual "pattern" breaking during such optimized partial repaints.

@egmontkob commented on GitHub (Nov 15, 2019): And just to complete the picture: Something that initially confused me was that the behavior seemed inconsistent. E.g. most of the time there's a rectangle when switching from grey to white, but no rectangle when switching from white to blue. However, in the very first screenshot, the row of PID 1334 is an exception to both of these "patterns". The explanation to this is that probably we don't see the initial screen right after htop starts up, but after a couple of repaints, during which ncurses probably optimized by jumping over areas that it believed hadn't changed. And of course the reality was already off compared to ncurses's belief, so it could even matter if ncurses used relative or absolute cursor movement. Anyway, it's easy to imagine the aforementioned visual "pattern" breaking during such optimized partial repaints.

claunia commented

@egmontkob commented on GitHub (Nov 15, 2019):

@otmos Thanks for the report and help, the mystery is solved now, no more screenshots are needed.

You might want to check your TERM (uppercase!) variable (echo $TERM) and share with us (I'm simply curious), and test if setting it to xterm-256color (export TERM=xterm-256color) fixes the issue for you (I think it should).

@egmontkob commented on GitHub (Nov 15, 2019): @otmos Thanks for the report and help, the mystery is solved now, no more screenshots are needed. You might want to check your TERM (uppercase!) variable (`echo $TERM`) and share with us (I'm simply curious), and test if setting it to xterm-256color (`export TERM=xterm-256color`) fixes the issue for you (I think it should).

claunia commented

@otmos commented on GitHub (Nov 15, 2019):

After setting TERM to xterm-256color , everything works as it should , and i understand how i came across this bug , it was a result of changing TERM to screen in order to fix a bug when scrolling in any app such as nano when in ssh mode from an AWS cloud9 instance

@otmos commented on GitHub (Nov 15, 2019): After setting TERM to xterm-256color , everything works as it should , and i understand how i came across this bug , it was a result of changing TERM to screen in order to fix a bug when scrolling in any app such as nano when in ssh mode from an AWS cloud9 instance

claunia commented

@egmontkob commented on GitHub (Nov 15, 2019):

I think we'd love to hear that bug as well :)

Generally, TERM=screen or friends should be used if and only if you're inside screen or tmux.

@egmontkob commented on GitHub (Nov 15, 2019): I think we'd love to hear _that_ bug as well :) Generally, `TERM=screen` or friends should be used if and only if you're inside screen or tmux.

claunia commented

@otmos commented on GitHub (Nov 15, 2019):

Here is a video showcasing the bug
data - Cloud9 - Google Chrome 2019-11-15 16-40-28.zip

@otmos commented on GitHub (Nov 15, 2019): Here is a video showcasing the bug [data - Cloud9 - Google Chrome 2019-11-15 16-40-28.zip](https://github.com/microsoft/terminal/files/3852184/data.-.Cloud9.-.Google.Chrome.2019-11-15.16-40-28.zip)

claunia commented

@j4james commented on GitHub (Dec 10, 2019):

Note that SI and SO are covered by issue #3377, and I have a work-in-progress PR for that which I'm hoping to get back to soon.

@j4james commented on GitHub (Dec 10, 2019): Note that SI and SO are covered by issue #3377, and I have a work-in-progress PR for that which I'm hoping to get back to soon.

claunia commented

@andres-pcg commented on GitHub (Apr 29, 2020):

Also if it helps, when $TERM = linux; and top is called, broken characters shows up too:

Thanks for your detailed and great analyst @egmontkob 👍 👌

@andres-pcg commented on GitHub (Apr 29, 2020): Also if it helps, when ```$TERM = linux```; and top is called, broken characters shows up too: ![image](https://user-images.githubusercontent.com/28060986/80605493-2c3cbd80-8a01-11ea-9b53-f8e6082e0240.png) Thanks for your detailed and great analyst @egmontkob 👍 👌

claunia commented

@nicm commented on GitHub (May 6, 2020):

The way it works is that enacs is \033(B\033)0 which configures G0 to be US-ASCII and G1 to be the ACS line drawing set. Then smacs and rmacs (and sgr0 as you have seen) use SI and SO to switch between G0 and G1 .

It is different in some entries (like xterm-256color) where enacs does nothing and smacs and rmacs just switch G0 between US-ASCII and ACS line drawing. This seems to be the new way of doing it, certainly for all the xterm entries that are still relevant.

@nicm commented on GitHub (May 6, 2020): The way it works is that `enacs` is `\033(B\033)0` which configures G0 to be US-ASCII and G1 to be the ACS line drawing set. Then `smacs` and `rmacs` (and `sgr0` as you have seen) use SI and SO to switch between G0 and G1 . It is different in some entries (like `xterm-256color`) where `enacs` does nothing and `smacs` and `rmacs` just switch G0 between US-ASCII and ACS line drawing. This seems to be the new way of doing it, certainly for all the `xterm` entries that are still relevant.

claunia commented

@ismail commented on GitHub (May 22, 2020):

The way it works is that enacs is \033(B\033)0 which configures G0 to be US-ASCII and G1 to be the ACS line drawing set. Then smacs and rmacs (and sgr0 as you have seen) use SI and SO to switch between G0 and G1 .

It is different in some entries (like xterm-256color) where enacs does nothing and smacs and rmacs just switch G0 between US-ASCII and ACS line drawing. This seems to be the new way of doing it, certainly for all the xterm entries that are still relevant.

Would it be possible to change tmux-256color to do something similar? I'm not privy to technical details, but looks like getting this bug fixed will probably take a long time.

@ismail commented on GitHub (May 22, 2020): > The way it works is that `enacs` is `\033(B\033)0` which configures G0 to be US-ASCII and G1 to be the ACS line drawing set. Then `smacs` and `rmacs` (and `sgr0` as you have seen) use SI and SO to switch between G0 and G1 . > > It is different in some entries (like `xterm-256color`) where `enacs` does nothing and `smacs` and `rmacs` just switch G0 between US-ASCII and ACS line drawing. This seems to be the new way of doing it, certainly for all the `xterm` entries that are still relevant. Would it be possible to change tmux-256color to do something similar? I'm not privy to technical details, but looks like getting this bug fixed will probably take a long time.

claunia commented

@DHowett commented on GitHub (May 22, 2020):

If I recall correctly, this is in pull request. We deferred looking at that PR (with the blessing of the author) until after we launched v1 of Terminal, but we’ll make sure it gets back on our list.

@DHowett commented on GitHub (May 22, 2020): If I recall correctly, this is in pull request. We deferred looking at that PR (with the blessing of the author) until after we launched v1 of Terminal, but we’ll make sure it gets back on our list.

claunia commented