starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-03 21:25:34 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

ReadConsole does not work with utf-8 codepage #20527

New Issue
Closed
opened 2026-01-31 07:16:35 +00:00 by claunia · 9 comments
claunia commented 2026-01-31 07:16:35 +00:00
Owner
Copy Link

Originally created by @tholp on GitHub (Sep 22, 2023).

Windows Terminal version

No response

Windows build number

No response

Other Software

No response

Steps to reproduce

Use SetConsoleCP to use utf-8 (or if in utf8 beta-mode in windows).
Read using ReadConsole

Inputting "abcæøå" the input buffer will contain 0x61 0x62 0x63 0x00 0x00 0x00 0xD 0x0A

Expected Behavior

utf-8 characters for the letters æøå.

Actual Behavior

nulls

Originally created by @tholp on GitHub (Sep 22, 2023). ### Windows Terminal version _No response_ ### Windows build number _No response_ ### Other Software _No response_ ### Steps to reproduce Use **SetConsoleCP** to use utf-8 (or if in utf8 beta-mode in windows). Read using **ReadConsole** Inputting "abcæøå" the input buffer will contain 0x61 0x62 0x63 **0x00 0x00 0x00** 0xD 0x0A ### Expected Behavior utf-8 characters for the letters æøå. ### Actual Behavior nulls
claunia commented 2026-01-31 07:16:37 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Sep 22, 2023):

Are you using ReadConsoleA or ReadConsoleW/? (does your main take chars or wchar_ts?)

I'd bet this is the thing that @lhecker just re-wrote the entire input buffer to fix ☺️

@zadjii-msft commented on GitHub (Sep 22, 2023): Are you using `ReadConsoleA` or `ReadConsoleW`/? (does your `main` take `char`s or `wchar_t`s?) I'd bet this is the thing that @lhecker just re-wrote the entire input buffer to fix ☺️
claunia commented 2026-01-31 07:16:37 +00:00
Author
Owner
Copy Link

@tholp commented on GitHub (Sep 22, 2023):

I use ReadConsoleW and actually I have no problems using utf-16 or Codepage 1252. ReadConsoleW reads into to a buffer and I have written what bytes the buffer contains, i.e. without interpretating anything as characters. So the types char and wchar_t are irrelevant.

@tholp commented on GitHub (Sep 22, 2023): I use `ReadConsoleW` and actually I have no problems using utf-16 or Codepage 1252. `ReadConsoleW` reads into to a buffer and I have written what _bytes_ the buffer contains, i.e. without interpretating anything as characters. So the types `char` and `wchar_t` are irrelevant.
claunia commented 2026-01-31 07:16:37 +00:00
Author
Owner
Copy Link

@tholp commented on GitHub (Sep 22, 2023):

By the way, the problem is the same both for the Windows Terminal and the Windows Console Host.

@tholp commented on GitHub (Sep 22, 2023): By the way, the problem is the same both for the **Windows Terminal** and the **Windows Console Host**.
claunia commented 2026-01-31 07:16:38 +00:00
Author
Owner
Copy Link

@lhecker commented on GitHub (Sep 22, 2023):

@tholp Some parts of your comments must be incorrect. You write that your buffer contains:

0x61 0x62 0x63 0x00 0x00 0x00 0xD 0x0A

And you said (emphasis mine):

ReadConsoleW reads into to a buffer and I have written what bytes the buffer contains

If you had truly used the W variant it would've filled your buffer with 16-bit characters which would result in a byte-wise buffer like this:

0x61 0x00, 0x62 0x00, 0x63 0x00, 0x00 0x00, 0x00 0x00, 0x00 0x00, 0x0D 0x00, 0x0A 0x00

= There should be 0x00 high bytes for each 16-bit integer.

So either you used the A variant or you wrote what 16-bit integers your buffer contains. Do you know which one is it?

Furthermore, when you say:

Inputting "abcæøå"

how did you enter that string? Did you enter it regularly with your keyboard or did you use WriteConsoleInput? Depending on your answer I think I know which PR fixed your issue. 🙂

@lhecker commented on GitHub (Sep 22, 2023): @tholp Some parts of your comments must be incorrect. You write that your buffer contains: ``` 0x61 0x62 0x63 0x00 0x00 0x00 0xD 0x0A ``` And you said (emphasis mine): > **`ReadConsoleW`** reads into to a buffer and I have written what **_bytes_** the buffer contains If you had truly used the `W` variant it would've filled your buffer with 16-bit characters which would result in a byte-wise buffer like this: ``` 0x61 0x00, 0x62 0x00, 0x63 0x00, 0x00 0x00, 0x00 0x00, 0x00 0x00, 0x0D 0x00, 0x0A 0x00 ``` = There should be 0x00 high bytes for each 16-bit integer. So either you used the `A` variant or you wrote what 16-bit integers your buffer contains. Do you know which one is it? --- Furthermore, when you say: > Inputting "abcæøå" how did you enter that string? Did you enter it regularly with your keyboard or did you use `WriteConsoleInput`? Depending on your answer I think I know which PR fixed your issue. 🙂
claunia commented 2026-01-31 07:16:38 +00:00
Author
Owner
Copy Link

@tholp commented on GitHub (Sep 25, 2023):

Sorry, you are right. My code actually obtains the (bad) bytes using ReadFile.
I don't know if that problem should be reported here or somewhere else?

Running echo abcæøå | myProgram gives the correct buffer contents, but inputting it using the keyboard gives the null-bytes.

@tholp commented on GitHub (Sep 25, 2023): Sorry, you are right. My code actually obtains the (bad) bytes using `ReadFile`. I don't know if that problem should be reported here or somewhere else? Running `echo abcæøå | myProgram` gives the correct buffer contents, but inputting it using the keyboard gives the null-bytes.
claunia commented 2026-01-31 07:16:38 +00:00
Author
Owner
Copy Link

@lhecker commented on GitHub (Sep 25, 2023):

Ah I see... In that case your issue is fixed by the combination of #14745 and #15783. The former fixes UTF-8 not working via ReadFile and is fixed in 1.18 and later. The latter fixes interactive Unicode input and is fixed in 1.19 and later. 1.18 is currently available via Windows Terminal Preview. (It'll be stable in a few weeks.)

I'll mark it as a /duplicate of #14745, because that one affects you more than the other.

@lhecker commented on GitHub (Sep 25, 2023): Ah I see... In that case your issue is fixed by the combination of #14745 and #15783. The former fixes UTF-8 not working via `ReadFile` and is fixed in 1.18 and later. The latter fixes interactive Unicode input and is fixed in 1.19 and later. 1.18 is currently available via Windows Terminal Preview. (It'll be stable in a few weeks.) I'll mark it as a /duplicate of #14745, because that one affects you more than the other.
claunia commented 2026-01-31 07:16:38 +00:00
Author
Owner
Copy Link

@microsoft-github-policy-service[bot] commented on GitHub (Sep 25, 2023):

Hi! We've identified this issue as a duplicate of another one that already exists on this Issue Tracker. This specific instance is being closed in favor of tracking the concern over on the referenced thread. Thanks for your report!

@microsoft-github-policy-service[bot] commented on GitHub (Sep 25, 2023): Hi! We've identified this issue as a duplicate of another one that already exists on this Issue Tracker. This specific instance is being closed in favor of tracking the concern over on the referenced thread. Thanks for your report!
claunia commented 2026-01-31 07:16:39 +00:00
Author
Owner
Copy Link

@lhecker commented on GitHub (Sep 25, 2023):

Oh and I should say: It's quite likely that Windows Terminal Preview 1.18 already fixes your issue. If you get a chance, please try it out! 🙂 (If you compare your issue description with #4551 for instance, you'll see that your description is very similar, and #4551 has been fixed in 1.18.)

@lhecker commented on GitHub (Sep 25, 2023): Oh and I should say: It's quite likely that Windows Terminal Preview 1.18 already fixes your issue. If you get a chance, please try it out! 🙂 (If you compare your issue description with #4551 for instance, you'll see that your description is very similar, and #4551 has been fixed in 1.18.)
claunia commented 2026-01-31 07:16:39 +00:00
Author
Owner
Copy Link

@tholp commented on GitHub (Sep 26, 2023):

Yes, that sounds exactly like my issue. Thank you, and once more: sorry for the confusion with ReadConsole (I apparently didn't understand our own (old) code and should have made a pure test).

Not sure I can test it until the change is in the public version (I have in any case updated our software to get the right result).

@tholp commented on GitHub (Sep 26, 2023): Yes, that sounds exactly like my issue. Thank you, and once more: sorry for the confusion with `ReadConsole` (I apparently didn't understand our own (old) code and should have made a pure test). Not sure I can test it until the change is in the public version (I have in any case updated our software to get the right result).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Input Area-Output Issue-Bug Product-Conhost Resolution-Duplicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#20527
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?