Window line-wrapping can cause colored output to break copied text #20672

@lindhe commented on GitHub (Oct 13, 2023):

Alright, I've figured it out. And as always, it's a very very very specific edge case. It's when I use bat (aka. batcat) to output colored text, with certain flags set.

Install bat.

Create EXAMPLE.md with the following contents (including the markdown code fence, because we want syntax highlighting to color the output):

```shell
echo '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000'
```

Resize the terminal to be narrower than that long echo line.

Run the following:

/usr/bin/batcat --color=always --style=header --decorations=always EXAMPLE.md

Copy the output.
Paste the output in a wider window.

It's worth noting that it works fine (i.e. "as expected") if the window is wide enough to not wrap when the command is run. If I resize the window later, the problem is not there.

Something that really surprises me is that piping the output via cat fixes the issue while looking exactly the same visually:

@lindhe commented on GitHub (Oct 13, 2023): Alright, I've figured it out. And as always, it's a very very _very_ specific edge case. It's when I use [bat (aka. batcat)](https://github.com/sharkdp/bat) to output colored text, _with certain flags set_. 1. Install [bat](https://github.com/sharkdp/bat). 1. Create `EXAMPLE.md` with the following contents (_including_ the markdown code fence, because we want syntax highlighting to color the output): ~~~ ```shell echo '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000' ``` ~~~ 1. Resize the terminal to be narrower than that long `echo` line. 1. Run the following: ```shell /usr/bin/batcat --color=always --style=header --decorations=always EXAMPLE.md ``` 1. Copy the output. 1. Paste the output in a wider window. It's worth noting that it works fine (i.e. "as expected") if the window is wide enough to not wrap when the command is run. If I resize the window later, the problem is not there. Something that _really_ surprises me is that piping the output via `cat` fixes the issue while looking exactly the same visually: ![image](https://github.com/microsoft/terminal/assets/7773090/49dd15a7-294a-4942-bbed-8af03063185c)

claunia commented

@zadjii-msft commented on GitHub (Oct 13, 2023):

Can you try this again on Terminal Preview, v1.19/? (manual download link: https://github.com/microsoft/terminal/releases/tag/v1.19.2831.0) We just recently re-wrote how resize/reflow works, and I have a feeling that it may have just fixed this ☺️

@zadjii-msft commented on GitHub (Oct 13, 2023): Can you try this again on [Terminal Preview](https://aka.ms/terminal-preview), v1.19/? (manual download link: https://github.com/microsoft/terminal/releases/tag/v1.19.2831.0) We just recently re-wrote how resize/reflow works, and I have a feeling that it may have just fixed this ☺️

claunia commented

@j4james commented on GitHub (Oct 13, 2023):

I don't think v1.19 will make any difference. If you run batcat inside a script session and look at the log, you can see it's actually splitting the content into multiple lines with a line break between them, so it's expected that a resize won't treat that as a wrapped line.

I also noticed there's an issue in their bug tracker which sounds like it might be related (see https://github.com/sharkdp/bat/issues/1740).

@j4james commented on GitHub (Oct 13, 2023): I don't think v1.19 will make any difference. If you run `batcat` inside a [`script`](https://www.man7.org/linux/man-pages/man1/script.1.html) session and look at the log, you can see it's actually splitting the content into multiple lines with a line break between them, so it's expected that a resize won't treat that as a wrapped line. I also noticed there's an issue in their bug tracker which sounds like it might be related (see https://github.com/sharkdp/bat/issues/1740).

claunia commented

@lindhe commented on GitHub (Oct 15, 2023):

Can you try this again on Terminal Preview, v1.19/? (manual download link: https://github.com/microsoft/terminal/releases/tag/v1.19.2831.0) We just recently re-wrote how resize/reflow works, and I have a feeling that it may have just fixed this ☺️

Thanks for the suggestion. I just tried with Preview version 1.19.2682.0 and it had the exact same behavior, unfortunately.

I don't think v1.19 will make any difference. If you run batcat inside a script session and look at the log, you can see it's actually splitting the content into multiple lines with a line break between them, so it's expected that a resize won't treat that as a wrapped line.

I also noticed there's an issue in their bug tracker which sounds like it might be related (see sharkdp/bat#1740).

Interesting! That bug does indeed sound very similar to my problems. It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible?

@lindhe commented on GitHub (Oct 15, 2023): > Can you try this again on [Terminal Preview](https://aka.ms/terminal-preview), v1.19/? (manual download link: https://github.com/microsoft/terminal/releases/tag/v1.19.2831.0) We just recently re-wrote how resize/reflow works, and I have a feeling that it may have just fixed this ☺️ Thanks for the suggestion. I just tried with Preview version 1.19.2682.0 and it had the exact same behavior, unfortunately. > I don't think v1.19 will make any difference. If you run `batcat` inside a [`script`](https://www.man7.org/linux/man-pages/man1/script.1.html) session and look at the log, you can see it's actually splitting the content into multiple lines with a line break between them, so it's expected that a resize won't treat that as a wrapped line. > > I also noticed there's an issue in their bug tracker which sounds like it might be related (see [sharkdp/bat#1740](https://github.com/sharkdp/bat/issues/1740)). Interesting! That bug does indeed sound _very_ similar to my problems. It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible?

claunia commented

@j4james commented on GitHub (Oct 15, 2023):

It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible?

The fact that the resizing "works" for you is actually a bug. When you have a line that's the exact width of the screen, it is mistakenly considered to be wrapping when the screen is resized (see issue #3088). That was supposed to have been fixed recently, but it still doesn't seem to be working correctly in Windows Terminal.

@j4james commented on GitHub (Oct 15, 2023): > It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible? The fact that the resizing "works" for you is actually a bug. When you have a line that's the exact width of the screen, it is mistakenly considered to be wrapping when the screen is resized (see issue #3088). That was supposed to have been fixed recently, but it still doesn't seem to be working correctly in Windows Terminal.

claunia commented

@zadjii-msft commented on GitHub (Oct 25, 2023):

Can you try repro'ing this with the debug tap to get a trace of all the input and output? Once the bug starts occurring, send us a screenshot/copy-paste of the output and we might be able to figure out what the Terminal thinks it's getting here.

@zadjii-msft commented on GitHub (Oct 25, 2023): Can you try repro'ing this with the [debug tap](https://github.com/microsoft/terminal/wiki/Toubleshooting-Tips#enabling-the-debug-tap) to get a trace of all the input and output? Once the bug starts occurring, send us a screenshot/copy-paste of the output and we might be able to figure out what the Terminal thinks it's getting here.

claunia commented

@lindhe commented on GitHub (Oct 26, 2023):

I'd love to, but:

@lindhe commented on GitHub (Oct 26, 2023): I'd love to, but: !["You do not have permission to update this wiki."](https://github.com/microsoft/terminal/assets/7773090/52475624-2b7b-45f3-8110-c7081b23c2ad)

claunia commented

@j4james commented on GitHub (Oct 26, 2023):

It looks like there was a typo in Mike's link. It should be:
https://github.com/microsoft/terminal/wiki/Troubleshooting-Tips#enabling-the-debug-tap

@j4james commented on GitHub (Oct 26, 2023): It looks like there was a typo in Mike's link. It should be: https://github.com/microsoft/terminal/wiki/Troubleshooting-Tips#enabling-the-debug-tap

claunia commented

@zadjii-msft commented on GitHub (Oct 26, 2023):

(thanks - forgot that I had fixed that typo but not the links in my quick replies 🤦)

@zadjii-msft commented on GitHub (Oct 26, 2023): (thanks - forgot that I had fixed that typo but not the links in my quick replies 🤦)

claunia commented

@lindhe commented on GitHub (Oct 28, 2023):

Here you go! :)

␛[K␍␊
␛[K␛[5;3H␛[?25h␛[13;28;13;0;32;1_␛[200~/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[201~␛[7m/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[27m␛[K␛[13;28;13;1;32;1_␛[?25l␛[?1l␛>␛[?2004l␛[5;3H/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␍␊
␛[?25h␛[13;28;13;0;32;1_File:␣␛[1m/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[38;5;231m␛[22m␍␊
```␛[38;5;141mshell␛[38;5;231m␍␊
echo␣'1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣0000000000␣1␣22␣␍␊
333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣0000000000'␍␊
```␛[m␍␊
␛[K␍␊

@lindhe commented on GitHub (Oct 28, 2023): Here you go! :) ```text ␛[K␍␊ ␛[K␛[5;3H␛[?25h␛[13;28;13;0;32;1_␛[200~/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[201~␛[7m/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[27m␛[K␛[13;28;13;1;32;1_␛[?25l␛[?1l␛>␛[?2004l␛[5;3H/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␍␊ ␛[?25h␛[13;28;13;0;32;1_File:␣␛[1m/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[38;5;231m␛[22m␍␊ ```␛[38;5;141mshell␛[38;5;231m␍␊ echo␣'1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣0000000000␣1␣22␣␍␊ 333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣0000000000'␍␊ ```␛[m␍␊ ␛[K␍␊ ```

claunia commented

@zadjii-msft commented on GitHub (Nov 9, 2023):

Oh whoops I clearly never hit enter on my reply last week. The debug tap obviously showed a manual line break, that's what we're trying to figure out 🤦

Can you try with script instead/? https://github.com/microsoft/terminal/wiki/Troubleshooting-Tips#script

That'll actually capture what was emitted straight from the application, rather than what conpty re-emitted.

(thanks for your patience)

@zadjii-msft commented on GitHub (Nov 9, 2023): Oh whoops I clearly never hit enter on my reply last week. The debug tap obviously showed a manual line break, [that's what we're trying to figure out](https://i.kym-cdn.com/photos/images/newsfeed/002/161/331/28f.jpg) 🤦 Can you try with `script` instead/? https://github.com/microsoft/terminal/wiki/Troubleshooting-Tips#script That'll actually capture what was emitted straight from the application, rather than what conpty re-emitted. _(thanks for your patience)_

claunia commented

@lindhe commented on GitHub (Nov 9, 2023):

(thanks for your patience)

Thanks for the help!

typescript without debug tap

Script started on 2023-11-09 13:45:48+01:00 [TERM="xterm-256color" TTY="/dev/pts/3" COLUMNS="76" LINES="37"]
[1m[7m%[27m[1m[0m                                                                           
 
]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS\]7;file://conoa/tmp/tmp.pxjFyYQaGS\
[0m[27m[24m[JÔò¡ÔöÇ[1m[32mandreas@conoa[00m [1m[34m/tmp/tmp.pxjFyYQaGS [00m
Ôò░ÔöÇ[1m$[0m [K[71C[1m[0m[71D[?1h=[?2004h[7m/usr/bin/batcat --color=always --style=header --decorations=always EXAMP[7mL[7mE.md[27m[K[A[27m/[27mu[27ms[27mr[27m/[27mb[27mi[27mn[27m/[27mb[27ma[27mt[27mc[27ma[27mt[27m [27m-[27m-[27mc[27mo[27ml[27mo[27mr[27m=[27ma[27ml[27mw[27ma[27my[27ms[27m [27m-[27m-[27ms[27mt[27my[27ml[27me[27m=[27mh[27me[27ma[27md[27me[27mr[27m [27m-[27m-[27md[27me[27mc[27mo[27mr[27ma[27mt[27mi[27mo[27mn[27ms[27m=[27ma[27ml[27mw[27ma[27my[27ms[27m [27mE[27mX[27mA[27mM[27mPL[27mE[27m.[27mm[27md[?1l>[?2004l


File: [1mEXAMPLE.md[0m[m
[38;5;231m```[0m[38;5;141mshell[0m[m
[38;5;231mecho '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 [0m[m
[38;5;231m333 4444 55555 666666 7777777 88888888 999999999 0000000000'[0m[m
[38;5;231m```[0m[m

[K[1m[7m%[27m[1m[0m                                                                           
 
]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS\]7;file://conoa/tmp/tmp.pxjFyYQaGS\
[0m[27m[24m[JÔò¡ÔöÇ[1m[32mandreas@conoa[00m [1m[34m/tmp/tmp.pxjFyYQaGS [00m
Ôò░ÔöÇ[1m$[0m [K[71C[1m[0m[71D[?1h=[?2004heexit[?1l>[?2004l


Script done on 2023-11-09 13:45:59+01:00 [COMMAND_EXIT_CODE="0"]

debug tap output, without script

␛[K␍␊
␛[K␛[8;5H␛[?25h␛[O␛[200~/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␛[201~␛[I␛[7m/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␛[27m␛[K␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␛[?25l␛[?1l␛>␛[?2004l␛[8;5H/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␍␊
␛[?25h␛[?25l␛]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS␛\␛]7;file://conoa/tmp/tmp.pxjFyYQaGS␛\␛[?1h␛=␛[?2004hFile:␣␛[1mEXAMPLE.md␛[38;5;231m␛[22m␍␊
```␛[38;5;141mshell␛[38;5;231m␍␊
echo␣'1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣00000␍␊
00000␣1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣00000␍␊
00000'␍␊
```␛[m␍␊
╭─␛[32m␛[1mandreas@conoa␛[m␣␛[34m␛[1m/tmp/tmp.pxjFyYQaGS␣␛[m␛[K␍␊
╰─␛[1m$␛[22m␛[K␍␊
␛[K␍␊

@lindhe commented on GitHub (Nov 9, 2023): > (thanks for your patience) Thanks for the help! **typescript without debug tap** ```text Script started on 2023-11-09 13:45:48+01:00 [TERM="xterm-256color" TTY="/dev/pts/3" COLUMNS="76" LINES="37"] [1m[7m%[27m[1m[0m ]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS\]7;file://conoa/tmp/tmp.pxjFyYQaGS\ [0m[27m[24m[JÔò¡ÔöÇ[1m[32mandreas@conoa[00m [1m[34m/tmp/tmp.pxjFyYQaGS [00m Ôò░ÔöÇ[1m$[0m [K[71C[1m[0m[71D[?1h=[?2004h[7m/usr/bin/batcat --color=always --style=header --decorations=always EXAMP[7mL[7mE.md[27m[K[A[27m/[27mu[27ms[27mr[27m/[27mb[27mi[27mn[27m/[27mb[27ma[27mt[27mc[27ma[27mt[27m [27m-[27m-[27mc[27mo[27ml[27mo[27mr[27m=[27ma[27ml[27mw[27ma[27my[27ms[27m [27m-[27m-[27ms[27mt[27my[27ml[27me[27m=[27mh[27me[27ma[27md[27me[27mr[27m [27m-[27m-[27md[27me[27mc[27mo[27mr[27ma[27mt[27mi[27mo[27mn[27ms[27m=[27ma[27ml[27mw[27ma[27my[27ms[27m [27mE[27mX[27mA[27mM[27mPL[27mE[27m.[27mm[27md[?1l>[?2004l File: [1mEXAMPLE.md[0m[m [38;5;231m```[0m[38;5;141mshell[0m[m [38;5;231mecho '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 [0m[m [38;5;231m333 4444 55555 666666 7777777 88888888 999999999 0000000000'[0m[m [38;5;231m```[0m[m [K[1m[7m%[27m[1m[0m ]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS\]7;file://conoa/tmp/tmp.pxjFyYQaGS\ [0m[27m[24m[JÔò¡ÔöÇ[1m[32mandreas@conoa[00m [1m[34m/tmp/tmp.pxjFyYQaGS [00m Ôò░ÔöÇ[1m$[0m [K[71C[1m[0m[71D[?1h=[?2004heexit[?1l>[?2004l Script done on 2023-11-09 13:45:59+01:00 [COMMAND_EXIT_CODE="0"] ``` **debug tap output, without script** ```text ␛[K␍␊ ␛[K␛[8;5H␛[?25h␛[O␛[200~/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␛[201~␛[I␛[7m/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␛[27m␛[K␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␛[?25l␛[?1l␛>␛[?2004l␛[8;5H/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␍␊ ␛[?25h␛[?25l␛]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS␛\␛]7;file://conoa/tmp/tmp.pxjFyYQaGS␛\␛[?1h␛=␛[?2004hFile:␣␛[1mEXAMPLE.md␛[38;5;231m␛[22m␍␊ ```␛[38;5;141mshell␛[38;5;231m␍␊ echo␣'1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣00000␍␊ 00000␣1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣00000␍␊ 00000'␍␊ ```␛[m␍␊ ╭─␛[32m␛[1mandreas@conoa␛[m␣␛[34m␛[1m/tmp/tmp.pxjFyYQaGS␣␛[m␛[K␍␊ ╰─␛[1m$␛[22m␛[K␍␊ ␛[K␍␊ ```

claunia commented

@carlos-zamora commented on GitHub (Nov 15, 2023):

Looks like the bat utility inserts hard line breaks. This issue on the bat repository might be helpful: https://github.com/sharkdp/bat/issues/897

Thanks for filing!

@carlos-zamora commented on GitHub (Nov 15, 2023): Looks like the bat utility inserts hard line breaks. This issue on the bat repository might be helpful: https://github.com/sharkdp/bat/issues/897 Thanks for filing!

claunia commented