Bash command line editing gets wrong cursor position #19523

@dwmuller commented on GitHub (Apr 30, 2023):

I wonder if this is related to #3546

@dwmuller commented on GitHub (Apr 30, 2023): I wonder if this is related to #3546

claunia commented

@dwmuller commented on GitHub (Apr 30, 2023):

Oh good lord. I started digging deeper into what Oh My Posh sets up using the --init option, and there's way more going on here than is apparent from the PS1 value. It defines PS0, PS2, a shell function, and sets PROMPT_COMMAND to execute the shell function. The function sets up some context, puts the cursor at a calculated position, and defines PS1.

So there's more going on here than is apparent from my PS1. I won't bore you with the details just yet, but I'm analyzing it. Sorry about missing all this relevant information.

@dwmuller commented on GitHub (Apr 30, 2023): Oh good lord. I started digging deeper into what Oh My Posh sets up using the --init option, and there's way more going on here than is apparent from the PS1 value. It defines PS0, PS2, a shell function, and sets PROMPT_COMMAND to execute the shell function. The function sets up some context, puts the cursor at a calculated position, and defines PS1. So there's more going on here than is apparent from my PS1. I won't bore you with the details just yet, but I'm analyzing it. Sorry about missing all this relevant information.

claunia commented

@dwmuller commented on GitHub (Apr 30, 2023):

So here's what I found.

The PS0 value is irrelevant, it gets set to an empty string, but via side effects it records a start time for a command so that the elapsed time can be calculated later.

The PS2 value is irrelevant for the purposes of this bug.

The function executed by PROMPT_COMMAND has the following relevant effects:

Calculates elapsed time if relevant, and stashes it in a local variable.
Calculates the current cursor row and column and stashes those in exported variables POSH_CURSOR_LINE and POSH_CURSOR_COLUMN. More on this below.
Runs oh-my-posh to calculate a new PS1 value. Oh My Posh presumably incorporates the line and column values.

The method of calculating the row and column values:

local oldstty=$(stty -g)
stty raw -echo min 0

local COL
local ROW
IFS=';' read -sdR -p $'\E[6n' ROW COL

stty $oldstty

export POSH_CURSOR_LINE=${ROW#*[}
export POSH_CURSOR_COLUMN=${COL}

Clever.

This makes me wonder if the problem arises in the terminal's response to the \E[6n sequence. But I don't think that's the problem. OMP seems to be working OK, it's really readline that's manipulating the text after the prompt when you navigate history. It seems more likely that readline is interrogating the terminal for cursor position, and is getting an incorrect response.

BTW, I also removed the option in my OMP config that was sending the PWD to the terminal. This removed some cruft from the end of the PS1 value, but had no effect on the bug.

@dwmuller commented on GitHub (Apr 30, 2023): So here's what I found. The PS0 value is irrelevant, it gets set to an empty string, but via side effects it records a start time for a command so that the elapsed time can be calculated later. The PS2 value is irrelevant for the purposes of this bug. The function executed by PROMPT_COMMAND has the following relevant effects: 1. Calculates elapsed time if relevant, and stashes it in a local variable. 2. Calculates the current cursor row and column and stashes those in exported variables POSH_CURSOR_LINE and POSH_CURSOR_COLUMN. More on this below. 3. Runs oh-my-posh to calculate a new PS1 value. Oh My Posh presumably incorporates the line and column values. The method of calculating the row and column values: local oldstty=$(stty -g) stty raw -echo min 0 local COL local ROW IFS=';' read -sdR -p $'\E[6n' ROW COL stty $oldstty export POSH_CURSOR_LINE=${ROW#*[} export POSH_CURSOR_COLUMN=${COL} Clever. This makes me wonder if the problem arises in the terminal's response to the `\E[6n` sequence. But I don't think that's the problem. OMP seems to be working OK, it's really readline that's manipulating the text after the prompt when you navigate history. It seems more likely that readline is interrogating the terminal for cursor position, and is getting an incorrect response. BTW, I also removed the option in my OMP config that was sending the PWD to the terminal. This removed some cruft from the end of the PS1 value, but had no effect on the bug.

claunia commented

@zadjii-msft commented on GitHub (May 3, 2023):

Alright, yea we're basically at a loss. We don't think it should be a bad response to 6n, that wouldn't have to traverse ConPTY. Maybe someone has messed up the cursor position off the >, but even then, that would only be a single cell, not two.

I honestly don't know what else for us to do here except get this under a local repro and a live debugger.

Wait as a quick sanity check - if you set the default terminal emulator to Windows console host:

Does that repro this issue? That might be able to narrow it down to conpty or not.

@zadjii-msft commented on GitHub (May 3, 2023): Alright, yea we're basically at a loss. We don't think it should be a bad response to `6n`, that wouldn't have to traverse ConPTY. Maybe someone has messed up the cursor position off the `>`, but even then, that would only be a single cell, not two. I honestly don't know what else for us to do here except get this under a local repro and a live debugger. Wait as a quick sanity check - if you set the default terminal emulator to Windows console host: ![](https://user-images.githubusercontent.com/18356694/236057321-d2338e9e-3b8c-4eb3-b9b4-bea9d858ba65.png) Does _that_ repro this issue? That might be able to narrow it down to conpty or not.

claunia commented

@dwmuller commented on GitHub (May 7, 2023):

Wait as a quick sanity check - if you set the default terminal emulator to Windows console host:

Does that repro this issue? That might be able to narrow it down to conpty or not.

Tried it, it has no effect on the bug.

@dwmuller commented on GitHub (May 7, 2023): > > Wait as a quick sanity check - if you set the default terminal emulator to Windows console host: ![](https://user-images.githubusercontent.com/18356694/236057321-d2338e9e-3b8c-4eb3-b9b4-bea9d858ba65.png) > > Does _that_ repro this issue? That might be able to narrow it down to conpty or not. Tried it, it has no effect on the bug.

claunia commented

@rioka commented on GitHub (Jun 1, 2023):

I'm facing pretty much the same issue (using OMP, too): depending on the lenght of the prompt, the length of the command, and the width of the windows, the cursor get stuck

somewhere in the command line
in the previous line on the screen (i.e. the line above the one I'm trying to enter a command)
in the next line on the screen

Moreover, the cursor ends up in different positions depending on the key you enter

when I move one character at a time, the cursor stops somewhere in the middle of my command
when I move it using Home, the final position (column) is correct (but it may be on the current line, or in the previous one)
when I move it End, the final position is somewhere in the next line (which is empty of course)

As a side note, when used in Conemu or mintty , it works as expected (or at least I did not notice this behavior so far).

@rioka commented on GitHub (Jun 1, 2023): I'm facing pretty much the same issue (using OMP, too): depending on the lenght of the prompt, the length of the command, and the width of the windows, the cursor get stuck - somewhere in the command line - in the previous line on the screen (i.e. the line above the one I'm trying to enter a command) - in the next line on the screen Moreover, the cursor ends up in different positions depending on the key you enter - when I move one character at a time, the cursor stops somewhere in the middle of my command - when I move it using `Home`, the final position (column) is correct (but it may be on the current line, or in the previous one) - when I move it `End`, the final position is somewhere in the next line (which is empty of course) As a side note, when used in Conemu or mintty , it works as expected (or at least I did not notice this behavior so far).

claunia commented

@DavideCanton commented on GitHub (Jun 15, 2023):

Don't know if it is really related, but I have the same problem without using OMP.

Basically I have a PS1="λ ", and it is enough to reproduce the issue:

I believe @zadjii-msft's guess of unicode characters being not handled correctly is correct.

Reproduced with version 1.17.230526001, win 10, git bash, apparently it doesn't reproduce with powershell or cmd (but I don't use them too often, maybe I simply did not try enough).

@DavideCanton commented on GitHub (Jun 15, 2023): Don't know if it is really related, but I have the same problem without using OMP. Basically I have a `PS1="λ "`, and it is enough to reproduce the issue: ![image](https://github.com/microsoft/terminal/assets/3042187/1701f670-77fb-428d-b53d-073e8262b765) I believe @zadjii-msft's guess of unicode characters being not handled correctly is correct. Reproduced with version 1.17.230526001, win 10, git bash, apparently it doesn't reproduce with powershell or cmd (but I don't use them too often, maybe I simply did not try enough).

claunia commented

@j4james commented on GitHub (Jun 15, 2023):

@DavideCanton In your case it looks like git bash is miscalculating the length of the prompt. I added some logging to OpenConsole to see exactly what it is sending us, and when you press up arrow the second time we get this:

CR (moves the cursor to the start of the line)
CUF CUF CUF (moves the cursor forward 3 columns, up to the d in cd foo)
ICH 3 (inserts 3 spaces at that point, giving you c d foo
mkdir (overwrites the spaces and d, giving you cmkdirfoo
CUF CUF CUF CUF (moves the cursor forward 4 columns, one past the end of the line)

Those three CUF sequences at the start are the problem. It looks like it thought the lambda was two columns wide, making the prompt three columns wide (when it should have been two).

After some experimentation with other characters, I think it might be determining the width based on the UTF-8 byte count. For example, if you set the prompt to PS1="ᴀ ", it thinks it's four columns wide, because the ᴀ (U+1D00) is a three byte UTF-8 sequence.

It's possible this is a misconfiguration of the locale, and it thinks its using an 8-bit code page. I don't know enough about git bash to know how you might fix that.

@j4james commented on GitHub (Jun 15, 2023): @DavideCanton In your case it looks like git bash is miscalculating the length of the prompt. I added some logging to OpenConsole to see exactly what it is sending us, and when you press up arrow the second time we get this: `CR` (moves the cursor to the start of the line) `CUF` `CUF` `CUF` (moves the cursor forward 3 columns, up to the `d` in `cd foo`) `ICH 3` (inserts 3 spaces at that point, giving you `c d foo` `mkdir` (overwrites the spaces and `d`, giving you `cmkdirfoo` `CUF` `CUF` `CUF` `CUF` (moves the cursor forward 4 columns, one past the end of the line) Those three `CUF` sequences at the start are the problem. It looks like it thought the lambda was two columns wide, making the prompt three columns wide (when it should have been two). After some experimentation with other characters, I think it might be determining the width based on the UTF-8 byte count. For example, if you set the prompt to `PS1="ᴀ "`, it thinks it's four columns wide, because the `ᴀ` (U+1D00) is a three byte UTF-8 sequence. It's possible this is a misconfiguration of the locale, and it thinks its using an 8-bit code page. I don't know enough about git bash to know how you might fix that.

claunia commented

@DavideCanton commented on GitHub (Jun 15, 2023):

Hi, thanks for your response.

I suspected the problem was related to a wrong character length calculation, since adding many lambdas increases the count of wrong characters displayed when navigating through the history.

The strange thing is that opening the git bash standalone does not reproduce the issue, it happens only inside the windows terminal.

I fixed it by using tput sc and tput rc.

@DavideCanton commented on GitHub (Jun 15, 2023): Hi, thanks for your response. I suspected the problem was related to a wrong character length calculation, since adding many lambdas increases the count of wrong characters displayed when navigating through the history. The strange thing is that opening the git bash standalone does not reproduce the issue, it happens only inside the windows terminal. I fixed it by using `tput sc` and `tput rc`.

claunia commented

@j4james commented on GitHub (Jun 15, 2023):

The strange thing is that opening the git bash standalone does not reproduce the issue, it happens only inside the windows terminal.

I reproduced it in a recent build of OpenConsole, so it's definitely not limited to Windows Terminal. But it's possible git bash is using the legacy console APIs rather than VT sequences when it detects an older versions of Windows.

I fixed it by using tput sc and tput rc.

This didn't help on my system. I can still reproduce the issue after executing those commands.

@j4james commented on GitHub (Jun 15, 2023): > The strange thing is that opening the git bash standalone does not reproduce the issue, it happens only inside the windows terminal. I reproduced it in a recent build of OpenConsole, so it's definitely not limited to Windows Terminal. But it's possible git bash is using the legacy console APIs rather than VT sequences when it detects an older versions of Windows. > I fixed it by using `tput sc` and `tput rc`. This didn't help on my system. I can still reproduce the issue after executing those commands.

claunia commented

@DHowett commented on GitHub (Jun 15, 2023):

it's possible git bash is using the legacy console APIs rather than VT sequences when it detects an older versions of Windows.

It may also be using mintty in its default configuration, which could have a different (and less correct) understanding of the width of that character!

@DHowett commented on GitHub (Jun 15, 2023): > it's possible git bash is using the legacy console APIs rather than VT sequences when it detects an older versions of Windows. It may also be using mintty in its default configuration, which could have a different (and less correct) understanding of the width of that character!

claunia commented

@DavideCanton commented on GitHub (Jun 15, 2023):

I reproduced it in a recent build of OpenConsole, so it's definitely not limited to Windows Terminal. But it's possible git bash is using the legacy console APIs rather than VT sequences when it detects an older versions of Windows.

Probably.

This didn't help on my system. I can still reproduce the issue after executing those commands.

My bad, they are part of the fix but executing them is not enough.

~~I would link the stackoverflow response where I read it, but it seems SO is down :/~~
This is the link of the SO response.

I fixed by setting PS1='\n\['"`tput sc`"'\] \['"`tput rc`"'λ \]'

@DavideCanton commented on GitHub (Jun 15, 2023): > I reproduced it in a recent build of OpenConsole, so it's definitely not limited to Windows Terminal. But it's possible git bash is using the legacy console APIs rather than VT sequences when it detects an older versions of Windows. Probably. > This didn't help on my system. I can still reproduce the issue after executing those commands. My bad, they are part of the fix but executing them is not enough. ~~I would link the stackoverflow response where I read it, but it seems SO is down :/~~ [This](https://stackoverflow.com/a/7123564) is the link of the SO response. I fixed by setting ``PS1='\n\['"`tput sc`"'\] \['"`tput rc`"'λ \]'``

claunia commented

@j4james commented on GitHub (Jun 15, 2023):

I fixed by setting PS1='\n\['"`tput sc`"'\] \['"`tput rc`"'λ \]'

OK, that makes more sense. That works for me too. Interestingly they're not actually implementing the sc and rc operations with escape sequences, but with direct calls to the console SetConsoleCursorPosition API. I'm assuming this is all just legacy code that's evolved over time and only partially transitioned from the old console APIs to VT sequences.

It may also be using mintty in its default configuration

@DHowett I initially thought that too, but that's not the case. If I request a DA report I get back \x1b[?1;0c, which is what I would expect from old versions of conhost. Mintty would report something like \x1b[?64;...c.

@j4james commented on GitHub (Jun 15, 2023): > I fixed by setting `` PS1='\n\['"`tput sc`"'\] \['"`tput rc`"'λ \]' `` OK, that makes more sense. That works for me too. Interestingly they're not actually implementing the `sc` and `rc` operations with escape sequences, but with direct calls to the console `SetConsoleCursorPosition` API. I'm assuming this is all just legacy code that's evolved over time and only partially transitioned from the old console APIs to VT sequences. > It may also be using mintty in its default configuration @DHowett I initially thought that too, but that's not the case. If I request a DA report I get back `\x1b[?1;0c`, which is what I would expect from old versions of conhost. Mintty would report something like `\x1b[?64;...c`.

claunia commented

@j4james commented on GitHub (Jun 15, 2023):

I just realised there are actually two versions of git bash. The one that shows up in the start menu is C:\Program Files\Git\git-bash.exe, and the one in Windows Terminal is C:\Program Files\Git\bin\bash.exe (this is also what I was testing in OpenConsole, which explains why it had the same issue as Windows Terminal). The former version appears to use console APIs, while the latter version is mostly VT sequences.

@j4james commented on GitHub (Jun 15, 2023): I just realised there are actually two versions of git bash. The one that shows up in the start menu is `C:\Program Files\Git\git-bash.exe`, and the one in Windows Terminal is `C:\Program Files\Git\bin\bash.exe` (this is also what I was testing in OpenConsole, which explains why it had the same issue as Windows Terminal). The former version appears to use console APIs, while the latter version is mostly VT sequences.

claunia commented

@DavideCanton commented on GitHub (Jun 15, 2023):

I can confirm the problem arises with the bash.exe inside bin/.

@DavideCanton commented on GitHub (Jun 15, 2023): I can confirm the problem arises with the `bash.exe` inside `bin/`.

claunia commented

@dwmuller commented on GitHub (Jun 30, 2023):

I realized that Oh My Posh configuration files can be in YAML, which makes it a lot easier to comment out parts for experimenting. I subsequently stripped my config down to almost nothing, and am still seeing the problem.

I also noticed this FAQ entry for Oh My Posh: https://ohmyposh.dev/docs/faq#windows-terminal-unexpected-space-between-segmentstext

I figured that might be related, so I preceded each special Unicode character in the remaining configuration with \u200a. That didn't help either.

My prompt is down to this, piped through od so you can be sure you're seeing a representation of every character:

❯ echo -n "$PS1" | od -c
0000000  \n   \   [ 033   [   3   8   ;   2   ;   2   5   5   ;   2   5
0000020   5   ;   2   5   5   m   \   ] 342 235 257       \   [ 033   [
0000040   0   m   \   ]   \   [ 033   ]   9   ;   9   ;   C   :   \   U
0000060   s   e   r   s   \   d   a   n   m 033   \   \   \   ]
0000076

@dwmuller commented on GitHub (Jun 30, 2023): I realized that Oh My Posh configuration files can be in YAML, which makes it a lot easier to comment out parts for experimenting. I subsequently stripped my config down to almost nothing, and am still seeing the problem. I also noticed this FAQ entry for Oh My Posh: https://ohmyposh.dev/docs/faq#windows-terminal-unexpected-space-between-segmentstext I figured that might be related, so I preceded each special Unicode character in the remaining configuration with \u200a. That didn't help either. My prompt is down to this, piped through od so you can be sure you're seeing a representation of every character: ``` ❯ echo -n "$PS1" | od -c 0000000 \n \ [ 033 [ 3 8 ; 2 ; 2 5 5 ; 2 5 0000020 5 ; 2 5 5 m \ ] 342 235 257 \ [ 033 [ 0000040 0 m \ ] \ [ 033 ] 9 ; 9 ; C : \ U 0000060 s e r s \ d a n m 033 \ \ \ ] 0000076 ```

claunia commented

@dwmuller commented on GitHub (Jun 30, 2023):

I tried one more thing. The remaining visible character in my prompt was a Unicode character, which I didn't realize at first because it looks a lot like a greater-than sign. It shows up in the octal dump above as the sequence of bytes 342 235 257. Replacing that with an ASCII character makes the problem go away. Preceding the Unicode character with \u200a does not.

Further experimentation showed that putting any Unicode character in PS1 causes the problem. Try this:

export PS1="❤ "

@dwmuller commented on GitHub (Jun 30, 2023): I tried one more thing. The remaining visible character in my prompt was a Unicode character, which I didn't realize at first because it looks a lot like a greater-than sign. It shows up in the octal dump above as the sequence of bytes 342 235 257. Replacing that with an ASCII character makes the problem go away. Preceding the Unicode character with \u200a does not. Further experimentation showed that putting *any* Unicode character in PS1 causes the problem. Try this: ``` export PS1="❤ " ```

claunia commented

@j4james commented on GitHub (Jul 1, 2023):

Further experimentation showed that putting any Unicode character in PS1 causes the problem.

Yeah. This is what I was saying in https://github.com/microsoft/terminal/issues/14982#issuecomment-1592842298. My guess is that it's calculating the prompt length based on the number of bytes in the UTF-8 representation, rather than the actual glyph count.

@j4james commented on GitHub (Jul 1, 2023): > Further experimentation showed that putting _any_ Unicode character in PS1 causes the problem. Yeah. This is what I was saying in https://github.com/microsoft/terminal/issues/14982#issuecomment-1592842298. My guess is that it's calculating the prompt length based on the number of bytes in the UTF-8 representation, rather than the actual glyph count.

claunia commented

@dwmuller commented on GitHub (Jul 1, 2023):

Holy frijole, I finally found a simple fix for this.

I was experimenting some more and was befuddled as to why this was happening:

\u2764 should be equivalent to \342\235\244, so bash clearly wasn't interpreting the \u prefix correctly. A search for answers turned up a Stackoverflow article with an interesting top answer that pointed to locale settings.

I checked the output of locale and saw that LANG wasn't set:

❤ locale
LANG=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

This appears to be the default state in Git Bash on Windows - my dotfiles don't change anything related to locale, and I've seen this problem on a variety of systems, including a new laptop that I freshly set up just a few weeks ago.

Executing set LANG='C.UTF-8' fixed the \u problem and the positioning problem! So as @j4james guessed, this does seem to be a bash issue. Exporting LANG to the environment also seems to work and is visible to subordinate bash shells.

In a bash shell on WSL2, where I wasn't seeing the problem, LANG is set to C.UTF-8 already.

Also, I'll mention another Stackoverflow article that I ran into along the way, because it had a very clever workaround for a similar problem, using the sc and rc escape sequences. See the first answer here.

I could close this, but I'll let the devs do that after verifying my assumption that it's purely a bash problem. Maybe you can think of some way to defend users against this .

@dwmuller commented on GitHub (Jul 1, 2023): Holy frijole, I finally found a simple fix for this. I was experimenting some more and was befuddled as to why this was happening: ![image](https://github.com/microsoft/terminal/assets/17508429/c8dfcde3-e305-4e10-97a4-d2cb390a5a33) \u2764 should be equivalent to \342\235\244, so bash clearly wasn't interpreting the \u prefix correctly. A search for answers turned up a Stackoverflow article with [an interesting top answer](https://stackoverflow.com/questions/10651975/unicode-utf-8-with-git-bash) that pointed to locale settings. I checked the output of ``locale`` and saw that LANG wasn't set: ``` ❤ locale LANG= LC_CTYPE="C.UTF-8" LC_NUMERIC="C.UTF-8" LC_TIME="C.UTF-8" LC_COLLATE="C.UTF-8" LC_MONETARY="C.UTF-8" LC_MESSAGES="C.UTF-8" LC_ALL= ``` This appears to be the default state in Git Bash on Windows - my dotfiles don't change anything related to locale, and I've seen this problem on a variety of systems, including a new laptop that I freshly set up just a few weeks ago. Executing ``set LANG='C.UTF-8'`` fixed the \u problem *and* the positioning problem! So as @j4james guessed, this does seem to be a bash issue. Exporting LANG to the environment also seems to work and is visible to subordinate bash shells. In a bash shell on WSL2, where I wasn't seeing the problem, LANG is set to C.UTF-8 already. Also, I'll mention another Stackoverflow article that I ran into along the way, because it had a very clever workaround for a similar problem, using the sc and rc escape sequences. See the first answer [here](https://stackoverflow.com/questions/7112774/how-to-escape-unicode-characters-in-bash-prompt-correctly). I could close this, but I'll let the devs do that after verifying my assumption that it's purely a bash problem. Maybe you can think of some way to defend users against this .

claunia commented

@DHowett commented on GitHub (Jul 1, 2023):

AGH! Just today, I was going to ask you what LANG was set to. I decided that I must have already asked, and didn't choose to.

So instead... can you share the commandline for your Git Bash profile?

If it doesn't contain -l -i, it isn't starting with the correct environment. 😄

@DHowett commented on GitHub (Jul 1, 2023): AGH! Just today, I was going to ask you what `LANG` was set to. I decided that I must have already asked, and didn't choose to. So instead... can you share the commandline for your Git Bash profile? If it doesn't contain `-l -i`, it isn't starting with the correct environment. 😄

claunia commented

@dwmuller commented on GitHub (Jul 1, 2023):

Ugh. The -l option causes LANG to be set, -i does not. The shortcut installed by (the choco version of) Git for Windows looks like this:

git-bash.exe seems to be a wrapper around bash.exe, which does set up the locale correctly.

I actually usually start it from Microsoft Terminal, where I set it up to run bash.exe directly without any options at all. Doh.

@dwmuller commented on GitHub (Jul 1, 2023): Ugh. The -l option causes LANG to be set, -i does not. The shortcut installed by (the choco version of) Git for Windows looks like this: ![image](https://github.com/microsoft/terminal/assets/17508429/8fbae4c3-59e6-4d35-847a-d478d7996334) git-bash.exe seems to be a wrapper around bash.exe, which *does* set up the locale correctly. I actually usually start it from Microsoft Terminal, where I set it up to run bash.exe directly without any options at all. Doh.

claunia commented