starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-03 21:25:34 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

WriteConsoleA in CP_UTF8 cannot handle receiving lead, continuation and trail in separate operations #6723

New Issue
Closed
opened 2026-01-31 00:45:41 +00:00 by claunia · 4 comments
claunia commented 2026-01-31 00:45:41 +00:00
Owner
Copy Link

Originally created by @DHowett-MSFT on GitHub (Mar 3, 2020).

In til::u8state, when we decrement backIter, we get a checked iterator exception (in debug mode). In release mode, we just throw away the partial because it fails the "if lead byte found" check.

Throw on --backiter:

2d707f102b/src/inc/til/u8u16convert.h (L88)

Here is a test that demonstrates the failing behavior:

void Utf8Utf16ConvertTests::TestU8ToU16OneByOne()
{
    const std::string u8String1_1{ '\xF0' }; // U+1F4F7 CAMERA (4 bytes)
    const std::string u8String1_2{ '\x9F' };
    const std::string u8String1_3{ '\x93' };
    const std::string u8String1_4{ '\xB7' };

    const std::wstring u16StringComp1{
        gsl::narrow_cast<wchar_t>(0xD83D), // U+1F4F7 CAMERA (surrogate pair)
        gsl::narrow_cast<wchar_t>(0xDCF7)
    };

    til::u8state state{};

    std::wstring u16Out1{};
    VERIFY_SUCCEEDED(til::u8u16(u8String1_1, u16Out1, state));
    VERIFY_ARE_EQUAL(L"", u16Out1); // There should be no output for the first three bytes
    VERIFY_SUCCEEDED(til::u8u16(u8String1_2, u16Out1, state));
    VERIFY_ARE_EQUAL(L"", u16Out1);
    VERIFY_SUCCEEDED(til::u8u16(u8String1_3, u16Out1, state));
    VERIFY_ARE_EQUAL(L"", u16Out1);
    VERIFY_SUCCEEDED(til::u8u16(u8String1_4, u16Out1, state));
    VERIFY_ARE_EQUAL(u16StringComp1, u16Out1);
}

In debug builds:

StartGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne
Verify: SUCCEEDED(til::u8u16(u8String1_1, u16Out1, state))
Verify: AreEqual(L"", u16Out1)
Error: TAEF: [HRESULT 0x800706BE] A failure occurred while running a test operation: 'Utf8Utf16ConvertTests::TestU8ToU16OneByOne'. (The process hosting the test code was unexpectedly terminated with exit code 0x00000003 while invoking a test operation.)
EndGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne [Failed]

In release builds:

StartGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne
Verify: SUCCEEDED(til::u8u16(u8String1_1, u16Out1, state))
Verify: AreEqual(L"", u16Out1)
Verify: SUCCEEDED(til::u8u16(u8String1_2, u16Out1, state))
Error: Verify: AreEqual(L"", u16Out1) - Values (, �) [File: E:\src\OpenConsole\src\til\ut_til\u8u16convertTests.cpp, Function: Utf8Utf16ConvertTests::TestU8ToU16OneByOne, Line: 188]
EndGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne [Failed]

/cc @german-one

Originally created by @DHowett-MSFT on GitHub (Mar 3, 2020). In `til::u8state`, when we decrement backIter, we get a checked iterator exception (in debug mode). In release mode, we just throw away the partial because it fails the "if lead byte found" check. Throw on `--backiter`: https://github.com/microsoft/terminal/blob/2d707f102bf27d455e15dcc6001337a8da6869c2/src/inc/til/u8u16convert.h#L88 Here is a test that demonstrates the failing behavior: ```cpp void Utf8Utf16ConvertTests::TestU8ToU16OneByOne() { const std::string u8String1_1{ '\xF0' }; // U+1F4F7 CAMERA (4 bytes) const std::string u8String1_2{ '\x9F' }; const std::string u8String1_3{ '\x93' }; const std::string u8String1_4{ '\xB7' }; const std::wstring u16StringComp1{ gsl::narrow_cast<wchar_t>(0xD83D), // U+1F4F7 CAMERA (surrogate pair) gsl::narrow_cast<wchar_t>(0xDCF7) }; til::u8state state{}; std::wstring u16Out1{}; VERIFY_SUCCEEDED(til::u8u16(u8String1_1, u16Out1, state)); VERIFY_ARE_EQUAL(L"", u16Out1); // There should be no output for the first three bytes VERIFY_SUCCEEDED(til::u8u16(u8String1_2, u16Out1, state)); VERIFY_ARE_EQUAL(L"", u16Out1); VERIFY_SUCCEEDED(til::u8u16(u8String1_3, u16Out1, state)); VERIFY_ARE_EQUAL(L"", u16Out1); VERIFY_SUCCEEDED(til::u8u16(u8String1_4, u16Out1, state)); VERIFY_ARE_EQUAL(u16StringComp1, u16Out1); } ``` In debug builds: ``` StartGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne Verify: SUCCEEDED(til::u8u16(u8String1_1, u16Out1, state)) Verify: AreEqual(L"", u16Out1) Error: TAEF: [HRESULT 0x800706BE] A failure occurred while running a test operation: 'Utf8Utf16ConvertTests::TestU8ToU16OneByOne'. (The process hosting the test code was unexpectedly terminated with exit code 0x00000003 while invoking a test operation.) EndGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne [Failed] ``` In release builds: ``` StartGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne Verify: SUCCEEDED(til::u8u16(u8String1_1, u16Out1, state)) Verify: AreEqual(L"", u16Out1) Verify: SUCCEEDED(til::u8u16(u8String1_2, u16Out1, state)) Error: Verify: AreEqual(L"", u16Out1) - Values (, �) [File: E:\src\OpenConsole\src\til\ut_til\u8u16convertTests.cpp, Function: Utf8Utf16ConvertTests::TestU8ToU16OneByOne, Line: 188] EndGroup: Utf8Utf16ConvertTests::TestU8ToU16OneByOne [Failed] ``` /cc @german-one
claunia commented 2026-01-31 00:45:43 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Mar 3, 2020):

Oh no, do we need to take _buffer into account when we're backing up over the string to find lead bytes? Because we have a lead from a previous call, and we might only be getting a partial.

We could move _buffer.append(in, 0u, remainingLength) up above the check, and then do all our work on _buffer instead of in. Then we chop the end off _buffer when we're done.

We'll still need to fix the bad iterator decrement, because somebody could still just send a continuation byte.

@DHowett-MSFT commented on GitHub (Mar 3, 2020): Oh no, do we need to take `_buffer` into account when we're backing up over the string to find lead bytes? Because we have a lead from a previous call, and we might only be getting a partial. We could move `_buffer.append(in, 0u, remainingLength)` up above the check, and then do all our work on `_buffer` instead of `in`. Then we chop the end off `_buffer` when we're done. We'll still need to fix the bad iterator decrement, because somebody _could_ still just send a continuation byte.
claunia commented 2026-01-31 00:45:43 +00:00
Author
Owner
Copy Link

@german-one commented on GitHub (Mar 3, 2020):

Oh, that's a nasty bug. I guess it will work with initializing auto backIter = in.end(); and --backiter at the beginning of the loop body. And yes I agree, moving the appending up is also necessary.

@german-one commented on GitHub (Mar 3, 2020): Oh, that's a nasty bug. I guess it will work with initializing `auto backIter = in.end();` and `--backiter` at the beginning of the loop body. And yes I agree, moving the appending up is also necessary.
claunia commented 2026-01-31 00:45:43 +00:00
Author
Owner
Copy Link

@german-one commented on GitHub (Mar 4, 2020):

@DHowett-MSFT Sorry I've stolen your unit test method 😉

@german-one commented on GitHub (Mar 4, 2020): @DHowett-MSFT Sorry I've stolen your unit test method 😉
claunia commented 2026-01-31 00:45:43 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Mar 4, 2020):

I'm happy you've stolen it! :D

@DHowett-MSFT commented on GitHub (Mar 4, 2020): I'm happy you've stolen it! :D
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Output Area-Server Issue-Bug Needs-Tag-Fix Priority-1 Product-Conhost Resolution-Fix-Committed Severity-Blocking
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#6723
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?