starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-09-25 00:15:10 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

[PR #4422] [MERGED] Unify UTF-8 handling using til::u8u16 & revise WriteConsoleAImpl #25754

New Issue
Open
opened 2026-01-31 09:11:35 +00:00 by claunia · 0 comments
claunia commented 2026-01-31 09:11:35 +00:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/microsoft/terminal/pull/4422
Author: @german-one
Created: 1/31/2020
Status: ✅ Merged
Merged: 2/4/2020
Merged by: @DHowett-MSFT

Base: master ← Head: master

📝 Commits (4)

  • 0b59d22 replace utf8Parser with til::u8u16 in VtInputThread
  • 0e57d30 replace utf8Parser with til::u8u16 in WriteConsoleAImpl, get rid of a couple of warnings in _stream.cpp
  • ab4c720 update WriteConsoleAImpl: remove new, delete, unused variables, hungarian notation; unify camel case
  • dd12bd6 try harder to clean up WriteConsoleAImpl

📊 Changes

4 files changed (+86 additions, -148 deletions)

View changed files

📝 src/host/VtInputThread.cpp (+11 -14)
📝 src/host/VtInputThread.hpp (+2 -3)
📝 src/host/_stream.cpp (+72 -130)
📝 src/host/_stream.h (+1 -1)

📄 Description

Summary of the Pull Request

Replace utf8Parser with til::u8u16 in order to have the same conversion algorithms used in terminal and conhost.

References

This PR is a follow up of #4093

PR Checklist

Detailed Description of the Pull Request / Additional comments

This PR addresses item 2 in this list:

  1. ✉ Implement til::u8u16 and til::u16u8 (done in PR #4093)

  2. ✔ Unify UTF-8 handling using til::u8u16 (this PR)
    2.1. ✔ Update VtInputThread::_HandleRunInput()
    2.2. ✔ Update ApiRoutines::WriteConsoleAImpl()
    2.3. ❌ (optional / ask the core team) Remove Utf8ToWideCharParser from the code base to avoid further use

  3. ❌ Enable BOM discarding (follow up)
    3.1. ❌ extend til::u8u16 and til::u16u8 with a 3rd parameter to enable discarding the BOM
    3.2. ❌ Make use of the 3rd parameter to discard the BOM in all current function callers, or (optional / ask the core team) make it the default for til::u8u16 and til::u16u8

  4. ❌ Find UTF-16 to UTF-8 conversions and examine if they can be unified, too (follow up)

@miniksa @DHowett-MSFT

  • Please check if this PR, along with the investigations done in #4093, does really close #3378
  • Please advice if I should remove Utf8ToWideCharParser now that it isn't used anymore

Validation Steps Performed

  • long UTF-8 files outputted to the console
  • printf tested as shown in #4086

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/microsoft/terminal/pull/4422 **Author:** [@german-one](https://github.com/german-one) **Created:** 1/31/2020 **Status:** ✅ Merged **Merged:** 2/4/2020 **Merged by:** [@DHowett-MSFT](https://github.com/DHowett-MSFT) **Base:** `master` ← **Head:** `master` --- ### 📝 Commits (4) - [`0b59d22`](https://github.com/microsoft/terminal/commit/0b59d2285016c816c93d416ee7166463959e7806) replace `utf8Parser` with `til::u8u16` in `VtInputThread` - [`0e57d30`](https://github.com/microsoft/terminal/commit/0e57d307c65bb5028153a9a035ab74f127accf69) replace `utf8Parser` with `til::u8u16` in `WriteConsoleAImpl`, get rid of a couple of warnings in `_stream.cpp` - [`ab4c720`](https://github.com/microsoft/terminal/commit/ab4c720b29826c4b92cc5555360cfdcefbb4489d) update `WriteConsoleAImpl`: remove new, delete, unused variables, hungarian notation; unify camel case - [`dd12bd6`](https://github.com/microsoft/terminal/commit/dd12bd65d18effe11f8b4da5df68dbbb38383119) try harder to clean up `WriteConsoleAImpl` ### 📊 Changes **4 files changed** (+86 additions, -148 deletions) <details> <summary>View changed files</summary> 📝 `src/host/VtInputThread.cpp` (+11 -14) 📝 `src/host/VtInputThread.hpp` (+2 -3) 📝 `src/host/_stream.cpp` (+72 -130) 📝 `src/host/_stream.h` (+1 -1) </details> ### 📄 Description <!-- Enter a brief description/summary of your PR here. What does it fix/what does it change/how was it tested (even manually, if necessary)? --> ## Summary of the Pull Request Replace `utf8Parser` with `til::u8u16` in order to have the same conversion algorithms used in terminal and conhost. <!-- Other than the issue solved, is this relevant to any other issues/existing PRs? --> ## References This PR is a follow up of #4093 <!-- Please review the items on the PR checklist before submitting--> ## PR Checklist * [x] Closes #4086 , Closes #3378 * [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA * [x] Tests added/passed * [ ] Requires documentation to be updated * [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx <!-- Provide a more detailed description of the PR, other things fixed or any additional comments/features here --> ## Detailed Description of the Pull Request / Additional comments This PR addresses item 2 in this list: 1. ✉ Implement `til::u8u16` and `til::u16u8` (done in PR #4093) 2. ✔ **Unify UTF-8 handling using `til::u8u16` (this PR)** 2.1. ✔ **Update VtInputThread::_HandleRunInput()** 2.2. ✔ **Update ApiRoutines::WriteConsoleAImpl()** 2.3. ❌ (optional / ask the core team) Remove Utf8ToWideCharParser from the code base to avoid further use 3. ❌ Enable BOM discarding (follow up) 3.1. ❌ extend `til::u8u16` and `til::u16u8` with a 3rd parameter to enable discarding the BOM 3.2. ❌ Make use of the 3rd parameter to discard the BOM in all current function callers, or (optional / ask the core team) make it the default for `til::u8u16` and `til::u16u8` 4. ❌ Find UTF-16 to UTF-8 conversions and examine if they can be unified, too (follow up) @miniksa @DHowett-MSFT - Please check if this PR, along with the investigations done in #4093, does really close #3378 - Please advice if I should remove `Utf8ToWideCharParser` now that it isn't used anymore <!-- Describe how you validated the behavior. Add automated tests wherever possible, but list manual validation steps taken as well --> ## Validation Steps Performed - long UTF-8 files outputted to the console - `printf` tested as shown in #4086 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
claunia added the pull-request label 2026-01-31 09:11:35 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#25754
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?