starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

Narrow emoji >=U+10000 make WriteCharsLegacy sad ("wrong character insertion when scrolling to bash history") #9097

New Issue
Closed
opened 2026-01-31 01:45:57 +00:00 by claunia · 13 comments
claunia commented 2026-01-31 01:45:57 +00:00
Owner
Copy Link

Originally created by @francogp on GitHub (Jun 18, 2020).

Originally assigned to: @lhecker on GitHub.

Notes from https://github.com/microsoft/terminal/issues/6555#issuecomment-646206984

[0x4]   OpenConsoleFullPDB!IsGlyphFullWidth + 0x9   
[0x5]   OpenConsoleFullPDB!WriteCharsLegacy + 0x2a3                  <<<<<<<<<<<<<<<<<
[0x6]   OpenConsoleFullPDB!WriteBuffer::_DefaultStringCase + 0xaf   
[0x7]   OpenConsoleFullPDB!WriteBuffer::PrintString + 0x16   
[0x8]   OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString + 0x61   
[0x9]   OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString + 0x42   
[0xa]   OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString + 0x197

We're calling IsGlyphFullWidth once for each half of the surrogate pair. You can well imagine how nicely that works out.

Environment

Windows build number: Win32NT 10.0.19041.0 Microsoft Windows NT 10.0.19041.0
Windows Terminal version (if applicable): 1.0.1402.0

Using WSL Ubuntu LTS version 20.04

Steps to reproduce

  1. Install Microsoft Terminal from Windows Store. Keep the default settings.
  2. Enable WSL and install Ubuntu 20.04 from Windows Store.
  3. Start Ubuntu 20.04 via WSL2.
  4. When bash prompt appears, run exactly this (from the first character until the end):
echo 🔥🌡

the utf-16 equivalent (just to see the exact characters)

\u0065\u0063\u0068\u006f\u0020\ud83d\udd25\ud83c\udf21\u0020\u0020
  1. Press enter to see the result.
  2. Press key UP
  3. Press key DOWN
  4. Go to step 6.

Expected behavior

switch from

echo 🔥🌡

to

THIS PROBLEM CANNOT BE REPRODUCED ON OTHER TERMINAL EMULATORS

Actual behavior

character 'e' inserted at the beginning

Originally created by @francogp on GitHub (Jun 18, 2020). Originally assigned to: @lhecker on GitHub. _Notes from https://github.com/microsoft/terminal/issues/6555#issuecomment-646206984_ ``` [0x4] OpenConsoleFullPDB!IsGlyphFullWidth + 0x9 [0x5] OpenConsoleFullPDB!WriteCharsLegacy + 0x2a3 <<<<<<<<<<<<<<<<< [0x6] OpenConsoleFullPDB!WriteBuffer::_DefaultStringCase + 0xaf [0x7] OpenConsoleFullPDB!WriteBuffer::PrintString + 0x16 [0x8] OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString + 0x61 [0x9] OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString + 0x42 [0xa] OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString + 0x197 ``` We're calling `IsGlyphFullWidth` once for each half of the surrogate pair. You can well imagine how nicely that works out. --- # Environment ```none Windows build number: Win32NT 10.0.19041.0 Microsoft Windows NT 10.0.19041.0 Windows Terminal version (if applicable): 1.0.1402.0 Using WSL Ubuntu LTS version 20.04 ``` # Steps to reproduce 1. Install Microsoft Terminal from Windows Store. Keep the default settings. 2. Enable WSL and install Ubuntu 20.04 from Windows Store. 3. Start Ubuntu 20.04 via WSL2. 4. When bash prompt appears, run exactly this (from the first character until the end): ```bash echo 🔥🌡 ``` the utf-16 equivalent (just to see the exact characters) ```none \u0065\u0063\u0068\u006f\u0020\ud83d\udd25\ud83c\udf21\u0020\u0020 ``` 5. Press enter to see the result. 6. Press key UP 7. Press key DOWN 8. Go to step 6. # Expected behavior switch from ```bash echo 🔥🌡 ``` to ```none ``` THIS PROBLEM CANNOT BE REPRODUCED ON OTHER TERMINAL EMULATORS # Actual behavior character 'e' inserted at the beginning ![](http://g.recordit.co/nrmLGQqtYF.gif)
claunia commented 2026-01-31 01:45:59 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jun 18, 2020):

Ho boy, is the thermometer supposed to be wide, and we're treating it as narrow?

This problem repros in conhost as well as Terminal, so it's not necessarily a conpty miscommunication.

@zadjii-msft commented on GitHub (Jun 18, 2020): Ho boy, is the thermometer supposed to be wide, and we're treating it as narrow? This problem repros in conhost as well as Terminal, so it's not necessarily a conpty miscommunication.
claunia commented 2026-01-31 01:45:59 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jun 18, 2020):

I think it's the other way around - it's supposed to be narrow but we're treating it as wide. This could perhaps be because conhost uses different width calculations to WT. My understanding was that this was by design, for backwards compatibility reasons (see issue #4604). But that might result in the conhost cursor being in a different position to whether WT expects it to be. Perhaps we need a way to tell conhost to use the standard width calculations when in a conpty session (assuming that is the problem).

@j4james commented on GitHub (Jun 18, 2020): I think it's the other way around - it's supposed to be narrow but we're treating it as wide. This could perhaps be because conhost uses different width calculations to WT. My understanding was that this was by design, for backwards compatibility reasons (see issue #4604). But that might result in the conhost cursor being in a different position to whether WT expects it to be. Perhaps we need a way to tell conhost to use the standard width calculations when in a conpty session (assuming that is the problem).
claunia commented 2026-01-31 01:45:59 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 18, 2020):

I think this is more likely a width disagreement between bash and conhost, actually, coupled with a somewhat “recent” regression where BS can move halfway into a double-width cell.

@DHowett commented on GitHub (Jun 18, 2020): I think this is more likely a width disagreement between _bash and conhost_, actually, coupled with a somewhat “recent” regression where BS can move halfway into a double-width cell.
claunia commented 2026-01-31 01:45:59 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jun 18, 2020):

I think this is more likely a width disagreement between bash and conhost

Yeah, but that's to be expected if conhost doesn't use a standard width measurement. You can get the same result with the U+FFFD character from my original bug report - you just need to be using a font that renders it as double-width (e.g. MS Gothic).

widechar

So unless conpty is triggering different behaviour, conhost's view of character widths is assumedly going to be dependent on whatever font is used by default, which is not necessarily in sync with bash or the Windows Terminal renderer.

@j4james commented on GitHub (Jun 18, 2020): > I think this is more likely a width disagreement between _bash and conhost_ Yeah, but that's to be expected if conhost doesn't use a standard width measurement. You can get the same result with the `U+FFFD` character from my original bug report - you just need to be using a font that renders it as double-width (e.g. MS Gothic). ![widechar](https://user-images.githubusercontent.com/4181424/85048842-cc0a0280-b18b-11ea-8f42-e4a40b184800.gif) So unless conpty is triggering different behaviour, conhost's view of character widths is assumedly going to be dependent on whatever font is used by default, which is not necessarily in sync with bash or the Windows Terminal renderer.
claunia commented 2026-01-31 01:45:59 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jun 18, 2020):

Here's another example of the problem that doesn't involve bash. Try executing these two statements:

printf ".X\e[AX\n\n"
printf "🌡X\e[AX\n\n"

In both cases WT renders the first X in the second column, because it thinks both the . and the thermometer are single cell width. But in the case of the thermometer, conhost then "corrects" the cursor position when it is moved, so the second X ends up in the fourth column instead of the third.

image

@j4james commented on GitHub (Jun 18, 2020): Here's another example of the problem that doesn't involve bash. Try executing these two statements: printf ".X\e[AX\n\n" printf "🌡X\e[AX\n\n" In both cases WT renders the first X in the second column, because it thinks both the `.` and the thermometer are single cell width. But in the case of the thermometer, conhost then "corrects" the cursor position when it is moved, so the second X ends up in the fourth column instead of the third. ![image](https://user-images.githubusercontent.com/4181424/85051141-075a0080-b18f-11ea-86cc-2bc98efe2daa.png)
claunia commented 2026-01-31 01:46:00 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 18, 2020):

Huh, we did work to make sure conpty and WT treat ambiguous width glyphs as narrow always. cf #2066 for more info on that choice (which I believe is the correct one). Perhaps we regressed that and they disagree once more? Hmm

@DHowett commented on GitHub (Jun 18, 2020): Huh, we did work to make sure conpty and WT treat ambiguous width glyphs as narrow _always_. cf #2066 for more info on that choice (which I believe is the correct one). Perhaps we regressed that and they disagree once more? Hmm
claunia commented 2026-01-31 01:46:00 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 18, 2020):

(Conpty was actually circumstantial: the ambiguous width lookup fell through to the VT renderer, which threw up its hands and said “friend you don’t have a font, so it’s narrow.”)

@DHowett commented on GitHub (Jun 18, 2020): (Conpty was actually circumstantial: the ambiguous width lookup fell through to the VT renderer, which threw up its hands and said “friend you don’t have a font, so it’s narrow.”)
claunia commented 2026-01-31 01:46:01 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 18, 2020):

Ah shit.

[0x4]   OpenConsoleFullPDB!IsGlyphFullWidth + 0x9   
[0x5]   OpenConsoleFullPDB!WriteCharsLegacy + 0x2a3                  <<<<<<<<<<<<<<<<<
[0x6]   OpenConsoleFullPDB!WriteBuffer::_DefaultStringCase + 0xaf   
[0x7]   OpenConsoleFullPDB!WriteBuffer::PrintString + 0x16   
[0x8]   OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString + 0x61   
[0x9]   OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString + 0x42   
[0xa]   OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString + 0x197

We're calling IsGlyphFullWidth once for each half of the surrogate pair. You can well imagine how nicely that works out.

@DHowett commented on GitHub (Jun 18, 2020): Ah shit. ``` [0x4] OpenConsoleFullPDB!IsGlyphFullWidth + 0x9 [0x5] OpenConsoleFullPDB!WriteCharsLegacy + 0x2a3 <<<<<<<<<<<<<<<<< [0x6] OpenConsoleFullPDB!WriteBuffer::_DefaultStringCase + 0xaf [0x7] OpenConsoleFullPDB!WriteBuffer::PrintString + 0x16 [0x8] OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString + 0x61 [0x9] OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString + 0x42 [0xa] OpenConsoleFullPDB!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString + 0x197 ``` We're calling `IsGlyphFullWidth` once for each half of the surrogate pair. You can well imagine how nicely that works out.
claunia commented 2026-01-31 01:46:01 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jun 18, 2020):

I was just going to say I thought it might have something to do with surrogate pairs. The fix I was proposing in #4604 didn't help in this case (although it does fix U+FFFD).

@j4james commented on GitHub (Jun 18, 2020): I was just going to say I thought it might have something to do with surrogate pairs. The fix I was proposing in #4604 didn't help in this case (although it does fix `U+FFFD`).
claunia commented 2026-01-31 01:46:01 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 18, 2020):

how did we ever expect this to work 😦

@DHowett commented on GitHub (Jun 18, 2020): _how did we ever expect this to work 😦_
claunia commented 2026-01-31 01:46:02 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 19, 2020):

I'm going to make this the master issue for "narrow emoji that are also surrogate pairs make life very sad"

@DHowett commented on GitHub (Jun 19, 2020): I'm going to make this the master issue for "narrow emoji that are also surrogate pairs make life very sad"
claunia commented 2026-01-31 01:46:02 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 20, 2020):

@francogp this isn't quite the same issue; that glyph is specified as being "narrow" (only taking up one cell), but it's also considered an emoji. We could render it better, but that's more likely #1472.

@DHowett commented on GitHub (Jun 20, 2020): @francogp this isn't quite the same issue; that glyph is specified as being "narrow" (only taking up one cell), but it's also considered an emoji. We could render it better, but that's more likely #1472.
claunia commented 2026-01-31 01:46:02 +00:00
Author
Owner
Copy Link

@ghost commented on GitHub (Jan 24, 2023):

:tada:This issue was addressed in #14640, which has now been successfully released as Windows Terminal Preview v1.17.1023.🎉

Handy links:

@ghost commented on GitHub (Jan 24, 2023): :tada:This issue was addressed in #14640, which has now been successfully released as `Windows Terminal Preview v1.17.1023`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.17.1023) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Output Issue-Bug Priority-2 Product-Conhost Resolution-Fix-Committed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#9097
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?