starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-03 21:25:34 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

Missing glyph substitution #14285

New Issue
Closed
opened 2026-01-31 04:06:10 +00:00 by claunia · 8 comments
claunia commented 2026-01-31 04:06:10 +00:00
Owner
Copy Link

Originally created by @alabuzhev on GitHub (Jun 21, 2021).

Why not let Windows substitute the glyphs that are missing in the current font?

  • I'm sure that this should've been discussed already somewhere due to the obviousness of the topic, but I can't find anything similar.

Currently the GDI renderer of conhost/OpenConsole uses PolyTextOutW for drawing.
It looks like PolyTextOutW doesn't try to substitute any missing glyphs, so the only way to see, say, Hiragana is to change the whole font to something like MS Gothic (which is eye-bleeding, to be honest).

However, there are other APIs that perform such substitution, e.g. ExtTextOutW.
Are there any reasons not to use them?

Proposed technical implementation details (optional)

A trivial replace PolyTextOutW -> ExtTextOutW already does the trick (and probably there are alternative / better methods as well):

diff --git a/src/renderer/gdi/paint.cpp b/src/renderer/gdi/paint.cpp
index 9295d3d6..550727d3 100644
--- a/src/renderer/gdi/paint.cpp
+++ b/src/renderer/gdi/paint.cpp
@@ -436,10 +436,22 @@ using namespace Microsoft::Console::Render;
 
     if (_cPolyText > 0)
     {
+#if 1
+        for (size_t i = 0; i != _cPolyText; ++i)
+        {
+            const auto& t = _pPolyText[i];
+            if (!ExtTextOutW(_hdcMemoryContext, t.x, t.y, t.uiFlags, &t.rcl, t.lpstr, t.n, t.pdx))
+            {
+                hr = E_FAIL;
+                break;
+            }
+        }
+#else
         if (!PolyTextOutW(_hdcMemoryContext, _pPolyText, (UINT)_cPolyText))
         {
             hr = E_FAIL;
         }
+#endif
 
         _polyStrings.clear();
         _polyWidths.clear();

Before the change:

image

After the change:

image

It works as expected in other apps, including positioning, clipping, size, etc.:

image

So, any reason not to do that or something similar?

P.S. I'm aware that there's Windows Terminal, where it already works as expected. This is about conhost/OpenConsole.

Originally created by @alabuzhev on GitHub (Jun 21, 2021). # Why not let Windows substitute the glyphs that are missing in the current font? * I'm sure that this should've been discussed already somewhere due to the obviousness of the topic, but I can't find anything similar. Currently the GDI renderer of conhost/OpenConsole [uses `PolyTextOutW` for drawing](https://github.com/microsoft/terminal/blob/c90de692509b074bfde191910d67154cfe389911/src/renderer/gdi/paint.cpp#L439). It looks like `PolyTextOutW` doesn't try to substitute any missing glyphs, so the only way to see, say, Hiragana is to change the *whole font* to something like MS Gothic (which is eye-bleeding, to be honest). However, there are other APIs that perform such substitution, e.g. `ExtTextOutW`. Are there any reasons not to use them? # Proposed technical implementation details (optional) A trivial replace `PolyTextOutW` -> `ExtTextOutW` already does the trick (and probably there are alternative / better methods as well): ```DIFF diff --git a/src/renderer/gdi/paint.cpp b/src/renderer/gdi/paint.cpp index 9295d3d6..550727d3 100644 --- a/src/renderer/gdi/paint.cpp +++ b/src/renderer/gdi/paint.cpp @@ -436,10 +436,22 @@ using namespace Microsoft::Console::Render; if (_cPolyText > 0) { +#if 1 + for (size_t i = 0; i != _cPolyText; ++i) + { + const auto& t = _pPolyText[i]; + if (!ExtTextOutW(_hdcMemoryContext, t.x, t.y, t.uiFlags, &t.rcl, t.lpstr, t.n, t.pdx)) + { + hr = E_FAIL; + break; + } + } +#else if (!PolyTextOutW(_hdcMemoryContext, _pPolyText, (UINT)_cPolyText)) { hr = E_FAIL; } +#endif _polyStrings.clear(); _polyWidths.clear(); ``` # Before the change: ![image](https://user-images.githubusercontent.com/11453922/122759189-93ff3900-d291-11eb-89a9-b1d47aa22ed9.png) # After the change: ![image](https://user-images.githubusercontent.com/11453922/122759316-b4c78e80-d291-11eb-87aa-7cdc2979ae28.png) It works as expected in other apps, including positioning, clipping, size, etc.: ![image](https://user-images.githubusercontent.com/11453922/122760023-8d24f600-d292-11eb-90f5-ce732fb58c8a.png) So, any reason not to do that or something similar? P.S. I'm aware that there's Windows Terminal, where it already works as expected. This is about conhost/OpenConsole.
claunia commented 2026-01-31 04:06:12 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 21, 2021):

That is an excellent question. I believe we went with PolyTextOutW to reduce the number of times we needed to context-switch into kernel mode to one per buffer instead of one per line (?). @miniksa can keep me honest here, but he's out for a couple days.

If it doesn't impact performance as much as it used to, this seems like a pretty easy win for a problem that's existed since time immemorial. 😄

@DHowett commented on GitHub (Jun 21, 2021): That is an excellent question. I believe we went with PolyTextOutW to reduce the number of times we needed to context-switch into kernel mode to one per buffer instead of one per line (?). @miniksa can keep me honest here, but he's out for a couple days. If it doesn't impact performance as much as it used to, this seems like a pretty easy win for a problem that's existed since time immemorial. :smile:
claunia commented 2026-01-31 04:06:12 +00:00
Author
Owner
Copy Link

@alabuzhev commented on GitHub (Jun 21, 2021):

I believe we went with PolyTextOutW...

As far as I can see, conhost.exe was already using PolyTextOutW back in Windows 7 (2009):

dumpbin /imports:gdi32.dll conhost.exe 
Dump of file conhost.exe

File Type: EXECUTABLE IMAGE

  Section contains the following imports:

    GDI32.dll
             10003F000 Import Address Table
             100041958 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          4F CreateRectRgn
                          22 CombineRgn
                         231 InvertRgn
                          E6 DeleteObject
                          30 CreateCompatibleDC
                          2F CreateCompatibleBitmap
                         277 SelectObject
                         278 SelectPalette
                         2B4 StretchDIBits
                          E3 DeleteDC
                          36 CreateDIBitmap
                         1FD GetObjectW
                          13 BitBlt
                         1CA GetDIBits
                         20D GetStockObject
                         251 PolyPatBlt
                         175 GdiFlush
                         1F6 GetNearestColor
                         285 SetDCBrushColor
                         2A6 SetTextColor
                         27E SetBkColor
                          40 CreateFontIndirectW
                         246 PatBlt
                         21E GetTextExtentPoint32W
                          32 CreateDCW
                         125 EnumFontFamiliesExW
                         28C SetFontEnumeration
                         224 GetTextFaceW
                         1CB GetDeviceCaps
                         27F SetBkMode
                         1C4 GetCurrentObject
                         20A GetRegionData
                         20C GetRgnBox
                    ---> 255 PolyTextOutW
                         2A3 SetSystemPaletteUse
                         25C RealizePalette
                         210 GetStringBitmapW
                          54 CreateSolidBrush
                         1B7 GetCharWidth32W
                          29 CreateBitmap
                         2BA TranslateCharsetInfo
                         27C SetBitmapBits
                         2B3 StretchBlt
                         1A7 GetBitmapBits
                         226 GetTextMetricsW
                         289 SetDIBitsToDevice

...to reduce the number of times we needed to context-switch into kernel mode to one per buffer instead of one per line

I've added this just before the loop:

assert(_cPolyText == 1);

and tried again, with cmd and other apps, outputting quite a few text in large and small blocks, using WriteConsoleW / WriteConsoleOutputW, but that assert never failed. It looks like there's not much to reduce in the first place, if anything :)

I've also tried erasing and filling the whole viewport in a loop and haven't noticed any difference with the stock conhost - it flickers with visually the same frequency. Even the debug version performs acceptably.

@alabuzhev commented on GitHub (Jun 21, 2021): > I believe we went with PolyTextOutW... As far as I can see, conhost.exe was already using PolyTextOutW back in Windows 7 (2009): <details> <summary>dumpbin /imports:gdi32.dll conhost.exe</summary> ``` Dump of file conhost.exe File Type: EXECUTABLE IMAGE Section contains the following imports: GDI32.dll 10003F000 Import Address Table 100041958 Import Name Table 0 time date stamp 0 Index of first forwarder reference 4F CreateRectRgn 22 CombineRgn 231 InvertRgn E6 DeleteObject 30 CreateCompatibleDC 2F CreateCompatibleBitmap 277 SelectObject 278 SelectPalette 2B4 StretchDIBits E3 DeleteDC 36 CreateDIBitmap 1FD GetObjectW 13 BitBlt 1CA GetDIBits 20D GetStockObject 251 PolyPatBlt 175 GdiFlush 1F6 GetNearestColor 285 SetDCBrushColor 2A6 SetTextColor 27E SetBkColor 40 CreateFontIndirectW 246 PatBlt 21E GetTextExtentPoint32W 32 CreateDCW 125 EnumFontFamiliesExW 28C SetFontEnumeration 224 GetTextFaceW 1CB GetDeviceCaps 27F SetBkMode 1C4 GetCurrentObject 20A GetRegionData 20C GetRgnBox ---> 255 PolyTextOutW 2A3 SetSystemPaletteUse 25C RealizePalette 210 GetStringBitmapW 54 CreateSolidBrush 1B7 GetCharWidth32W 29 CreateBitmap 2BA TranslateCharsetInfo 27C SetBitmapBits 2B3 StretchBlt 1A7 GetBitmapBits 226 GetTextMetricsW 289 SetDIBitsToDevice ``` </details> > ...to reduce the number of times we needed to context-switch into kernel mode to one per buffer instead of one per line I've added this just before the loop: ```C++ assert(_cPolyText == 1); ``` and tried again, with cmd and other apps, outputting quite a few text in large and small blocks, using WriteConsoleW / WriteConsoleOutputW, but that assert *never* failed. It looks like there's not much to reduce in the first place, if anything :) I've also tried erasing and filling the whole viewport in a loop and haven't noticed any difference with the stock conhost - it flickers with visually the same frequency. Even the debug version performs acceptably.
claunia commented 2026-01-31 04:06:13 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 21, 2021):

Wow, I found the actual changeset in Win7 that switched us to PolyTextOutW. Unfortunately, it switched us from something called NtGdiConsoleTextOut which took a POLYTEXTW (😅)

NtGdiConsoleTextOut dates to 1998. It calls GreConsoleTextOut (1992!), which is documented as:

/******************************Public*Routine******************************\
* BOOL GreConsoleTextOut
*
* Write text with no spacing and alignment options, thereby saving a ton
* of time.
*
* History:
*  Fri 12-Nov-1993 -by- Redacted [redacted]
* Smaller and Faster
*
*  Wed 16-Sep-1992 17:36:17 -by- Redacted [redacted]
* Duplicated GrePolyTextOut, and then deleted the unneeded code.
\**************************************************************************/

It looks like the decision to have a "fast path" for the console text pipeline to get out to GDI was made back in or before NT 3.5. Back then, and actually probably up until Windows 10, we did just send the entire console buffer to GDI every time it changed. Dirty region tracking (and therefore, not having cPolyText > 1) came about only relatively recently in the console's history.

@DHowett commented on GitHub (Jun 21, 2021): Wow, I found the actual changeset in Win7 that switched us to PolyTextOutW. Unfortunately, it switched us from something called `NtGdiConsoleTextOut` _which took a `POLYTEXTW` (😅)_ `NtGdiConsoleTextOut` dates to 1998. It calls `GreConsoleTextOut` (1992!), which is documented as: ``` /******************************Public*Routine******************************\ * BOOL GreConsoleTextOut * * Write text with no spacing and alignment options, thereby saving a ton * of time. * * History: * Fri 12-Nov-1993 -by- Redacted [redacted] * Smaller and Faster * * Wed 16-Sep-1992 17:36:17 -by- Redacted [redacted] * Duplicated GrePolyTextOut, and then deleted the unneeded code. \**************************************************************************/ ``` It looks like the decision to have a "fast path" for the console text pipeline to get out to GDI was made back in or before NT 3.5. Back then, and actually probably up until Windows 10, we did just send the entire console buffer to GDI every time it changed. Dirty region tracking (and therefore, not having `cPolyText > 1`) came about only relatively recently in the console's history.
claunia commented 2026-01-31 04:06:13 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 21, 2021):

This is probably one of those decisions we never revisited after being told that it was true when we inherited the code. 😄

@DHowett commented on GitHub (Jun 21, 2021): This is probably one of those decisions we never revisited after being told that it was true when we inherited the code. :smile:
claunia commented 2026-01-31 04:06:13 +00:00
Author
Owner
Copy Link

@alabuzhev commented on GitHub (Jun 21, 2021):

I found the actual changeset in Win7 that switched us to PolyTextOutW. Unfortunately, it switched us from something called NtGdiConsoleTextOut

And, surprisingly, glyph substitution worked before that changeset :)

image

Apparently, even NtGdiConsoleTextOut had some extra magic which PolyTextOutW doesn't have.

@alabuzhev commented on GitHub (Jun 21, 2021): > I found the actual changeset in Win7 that switched us to PolyTextOutW. Unfortunately, it switched us from something called NtGdiConsoleTextOut And, surprisingly, glyph substitution worked *before* that changeset :) ![image](https://user-images.githubusercontent.com/11453922/122807614-b27e2800-d2c3-11eb-9284-35d5f836294a.png) Apparently, even NtGdiConsoleTextOut had some extra magic which PolyTextOutW doesn't have.
claunia commented 2026-01-31 04:06:14 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 21, 2021):

worked before that changeset

...

shit.

@DHowett commented on GitHub (Jun 21, 2021): > worked before that changeset ... shit.
claunia commented 2026-01-31 04:06:14 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jun 21, 2021):

Yeah, we should definitely do this. Thanks @alabuzhev!

@DHowett commented on GitHub (Jun 21, 2021): Yeah, we should definitely do this. Thanks @alabuzhev!
claunia commented 2026-01-31 04:06:14 +00:00
Author
Owner
Copy Link

@ghost commented on GitHub (Jul 14, 2021):

:tada:This issue was addressed in #10478, which has now been successfully released as Windows Terminal Preview v1.10.1933.0.🎉

Handy links:

@ghost commented on GitHub (Jul 14, 2021): :tada:This issue was addressed in #10478, which has now been successfully released as `Windows Terminal Preview v1.10.1933.0`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.10.1933.0) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Rendering Impact-Compatibility Impact-Correctness Issue-Task Needs-Triage Priority-2 Product-Conhost Resolution-Fix-Committed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#14285
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?