starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-07-08 18:16:28 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

[PR #14745] [MERGED] Remove TranslateUnicodeToOem and all related code #30235

New Issue
Open
opened 2026-01-31 09:39:29 +00:00 by claunia · 0 comments
claunia commented 2026-01-31 09:39:29 +00:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/microsoft/terminal/pull/14745
Author: @lhecker
Created: 1/27/2023
Status: ✅ Merged
Merged: 2/28/2023
Merged by: @DHowett

Base: main ← Head: dev/lhecker/8000-TranslateUnicodeToOem

📝 Commits (10+)

  • 7b5b96d Excise TranslateUnicodeToOem and everything it touched
  • 819febe Fix build
  • ac1f371 Merge remote-tracking branch 'origin/main' into dev/lhecker/8000-TranslateUnicodeToOem-wip
  • caba56d Rewrite half InputBuffer
  • eac0aeb Remove previously added code
  • 005bc21 Address feedback
  • 738ce5f Simplify InputBuffer::Read and fix KeyEvent repeats
  • a0f1637 Fix zero initialization
  • d3ca747 Fix CRLF
  • 3d14766 Merge remote-tracking branch 'origin/main' into dev/lhecker/8000-TranslateUnicodeToOem

📊 Changes

23 files changed (+603 additions, -1224 deletions)

View changed files

📝 src/host/dbcs.cpp (+0 -74)
📝 src/host/dbcs.h (+0 -7)
📝 src/host/directio.cpp (+7 -61)
📝 src/host/inputBuffer.cpp (+268 -193)
📝 src/host/inputBuffer.hpp (+25 -13)
📝 src/host/misc.cpp (+3 -45)
📝 src/host/misc.h (+0 -2)
📝 src/host/readDataCooked.cpp (+23 -179)
📝 src/host/readDataCooked.hpp (+2 -2)
📝 src/host/readDataDirect.cpp (+39 -96)
📝 src/host/readDataDirect.hpp (+0 -3)
📝 src/host/readDataRaw.cpp (+17 -114)
📝 src/host/stream.cpp (+54 -253)
📝 src/host/stream.h (+6 -0)
📝 src/host/ut_host/CommandLineTests.cpp (+1 -1)
📝 src/host/ut_host/Host.UnitTests.vcxproj (+0 -1)
📝 src/host/ut_host/Host.UnitTests.vcxproj.filters (+0 -3)
📝 src/host/ut_host/InputBufferTests.cpp (+50 -42)
➖ src/host/ut_host/ReadWaitTests.cpp (+0 -129)
➕ src/inc/til/bytes.h (+57 -0)

...and 3 more files

📄 Description

The overarching intention of this PR is to improve our Unicode support. Most
of our APIs still don't support anything beyond UCS-2 and DBCS sequences.
This commit doesn't fix the UTF-16 support (by supporting surrogate pairs),
but it does improve support for UTF-8 by allowing longer char sequences.

It does so by removing TranslateUnicodeToOem which seems to have had an almost
viral effect on code quality wherever it was used. It made the assumption that
all narrow glyphs encode to 1 char and most wide glyphs to 2 chars.
It also didn't bother to check whether WideCharToMultiByte failed or returned
a different amount of chars. So up until now it was easily possible to read
uninitialized stack memory from conhost. Any code that used this function was
forced to do the same "measurement" of narrow/wide glyphs, because of course
it didn't had any way to indicate to the caller how much memory it needs to
store the result. Instead all callers were forced to sorta replicate how it
worked to calculate the required storage ahead of time.
Unsurprisingly, none of the callers used the same algorithm...

Without it the code is much leaner and easier to understand now. The best
example is COOKED_READ_DATA::_handlePostCharInputLoop which used to contain 3
blocks of almost identical code, but with ever so subtle differences. After
reading the old code for hours I still don't know if they were relevant or not.
It used to be 200 lines of code lacking any documentation and it's now 50 lines
with descriptive function names. I hope this doesn't break anything, but to
be honest I can't imagine anyone having relied on this mess in the first place.

I needed some helpers to handle byte slices (std::span<char>), which is why
a new til/bytes.h header was added. Initially I wrote a buf_writer class
but felt like such a wrapper around a slice/span was annoying to use.
As such I've opted for freestanding functions which take slices as mutable
references and "advance" them (offset the start) whenever they're read from
or written to. I'm not particularly happy with the design but they do the job.

Related to #8000
Fixes #4551
Fixes #7589
Fixes #8663

Validation Steps Performed

  • Unit and feature tests ✅
  • Far Manager ✅
  • Fixes test cases in #4551, #7589 and #8663 ✅

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/microsoft/terminal/pull/14745 **Author:** [@lhecker](https://github.com/lhecker) **Created:** 1/27/2023 **Status:** ✅ Merged **Merged:** 2/28/2023 **Merged by:** [@DHowett](https://github.com/DHowett) **Base:** `main` ← **Head:** `dev/lhecker/8000-TranslateUnicodeToOem` --- ### 📝 Commits (10+) - [`7b5b96d`](https://github.com/microsoft/terminal/commit/7b5b96dbf97be9b2e5579fe837f5926c7a18c4d9) Excise TranslateUnicodeToOem and everything it touched - [`819febe`](https://github.com/microsoft/terminal/commit/819febe0ca8b96bb1d9dc94048afc3489358fcfb) Fix build - [`ac1f371`](https://github.com/microsoft/terminal/commit/ac1f3711d133f65ee45370b93c35e2a50bf9791c) Merge remote-tracking branch 'origin/main' into dev/lhecker/8000-TranslateUnicodeToOem-wip - [`caba56d`](https://github.com/microsoft/terminal/commit/caba56d4cb2cab7ca8bbb3890235a5f9035005fa) Rewrite half InputBuffer - [`eac0aeb`](https://github.com/microsoft/terminal/commit/eac0aeb2350c1dac0c3726d4135b03aca7108842) Remove previously added code - [`005bc21`](https://github.com/microsoft/terminal/commit/005bc2154c08dcb6559547c61d479af519a0f0f8) Address feedback - [`738ce5f`](https://github.com/microsoft/terminal/commit/738ce5f0c17ed4aac6b216617c56128be27162da) Simplify InputBuffer::Read and fix KeyEvent repeats - [`a0f1637`](https://github.com/microsoft/terminal/commit/a0f16372646cbeb9f0c62b815adfc476db4700ab) Fix zero initialization - [`d3ca747`](https://github.com/microsoft/terminal/commit/d3ca7470a89b42aea88b02ee497147f4ab4b3f56) Fix CRLF - [`3d14766`](https://github.com/microsoft/terminal/commit/3d14766d8696698752fa1fe1e9047547ed3fe2d2) Merge remote-tracking branch 'origin/main' into dev/lhecker/8000-TranslateUnicodeToOem ### 📊 Changes **23 files changed** (+603 additions, -1224 deletions) <details> <summary>View changed files</summary> 📝 `src/host/dbcs.cpp` (+0 -74) 📝 `src/host/dbcs.h` (+0 -7) 📝 `src/host/directio.cpp` (+7 -61) 📝 `src/host/inputBuffer.cpp` (+268 -193) 📝 `src/host/inputBuffer.hpp` (+25 -13) 📝 `src/host/misc.cpp` (+3 -45) 📝 `src/host/misc.h` (+0 -2) 📝 `src/host/readDataCooked.cpp` (+23 -179) 📝 `src/host/readDataCooked.hpp` (+2 -2) 📝 `src/host/readDataDirect.cpp` (+39 -96) 📝 `src/host/readDataDirect.hpp` (+0 -3) 📝 `src/host/readDataRaw.cpp` (+17 -114) 📝 `src/host/stream.cpp` (+54 -253) 📝 `src/host/stream.h` (+6 -0) 📝 `src/host/ut_host/CommandLineTests.cpp` (+1 -1) 📝 `src/host/ut_host/Host.UnitTests.vcxproj` (+0 -1) 📝 `src/host/ut_host/Host.UnitTests.vcxproj.filters` (+0 -3) 📝 `src/host/ut_host/InputBufferTests.cpp` (+50 -42) ➖ `src/host/ut_host/ReadWaitTests.cpp` (+0 -129) ➕ `src/inc/til/bytes.h` (+57 -0) _...and 3 more files_ </details> ### 📄 Description The overarching intention of this PR is to improve our Unicode support. Most of our APIs still don't support anything beyond UCS-2 and DBCS sequences. This commit doesn't fix the UTF-16 support (by supporting surrogate pairs), but it does improve support for UTF-8 by allowing longer `char` sequences. It does so by removing `TranslateUnicodeToOem` which seems to have had an almost viral effect on code quality wherever it was used. It made the assumption that _all_ narrow glyphs encode to 1 `char` and most wide glyphs to 2 `char`s. It also didn't bother to check whether `WideCharToMultiByte` failed or returned a different amount of `char`s. So up until now it was easily possible to read uninitialized stack memory from conhost. Any code that used this function was forced to do the same "measurement" of narrow/wide glyphs, because _of course_ it didn't had any way to indicate to the caller how much memory it needs to store the result. Instead all callers were forced to sorta replicate how it worked to calculate the required storage ahead of time. Unsurprisingly, none of the callers used the same algorithm... Without it the code is much leaner and easier to understand now. The best example is `COOKED_READ_DATA::_handlePostCharInputLoop` which used to contain 3 blocks of _almost_ identical code, but with ever so subtle differences. After reading the old code for hours I still don't know if they were relevant or not. It used to be 200 lines of code lacking any documentation and it's now 50 lines with descriptive function names. I hope this doesn't break anything, but to be honest I can't imagine anyone having relied on this mess in the first place. I needed some helpers to handle byte slices (`std::span<char>`), which is why a new `til/bytes.h` header was added. Initially I wrote a `buf_writer` class but felt like such a wrapper around a slice/span was annoying to use. As such I've opted for freestanding functions which take slices as mutable references and "advance" them (offset the start) whenever they're read from or written to. I'm not particularly happy with the design but they do the job. Related to #8000 Fixes #4551 Fixes #7589 Fixes #8663 ## Validation Steps Performed * Unit and feature tests ✅ * Far Manager ✅ * Fixes test cases in #4551, #7589 and #8663 ✅ --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
claunia added the pull-request label 2026-01-31 09:39:29 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#30235
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?