Bad characters occasionally displayed when writing lots of identical UTF-8 lines #580

@german-one commented on GitHub (Mar 14, 2019):

I updated the test code above. Please see my comments related to the OUT_PROC macro.
It seems to prove that the application is the culprit rather than the console. As long as the application takes care of character boundaries the console does process UTF-8 correctly. I get the same results using both the new console and the legacy console.

@german-one commented on GitHub (Mar 14, 2019): I updated the test code above. Please see my comments related to the OUT_PROC macro. It seems to prove that the application is the culprit rather than the console. As long as the application takes care of character boundaries the console does process UTF-8 correctly. I get the same results using both the new console and the legacy console.

claunia commented

@DHowett-MSFT commented on GitHub (May 18, 2019):

@miniksa This is a lot like what we saw with CMD and WSL bringing up emoji for the build demo. Marking as a bug.

@DHowett-MSFT commented on GitHub (May 18, 2019): @miniksa This is a lot like what we saw with CMD and WSL bringing up emoji for the build demo. Marking as a bug.

claunia commented

@german-one commented on GitHub (May 18, 2019):

@DHowett-MSFT May I ask you - In case of emojis split UTF-16 surrogates, I guess?

@german-one commented on GitHub (May 18, 2019): @DHowett-MSFT May I ask you - In case of emojis split UTF-16 surrogates, I guess?

claunia commented

@DHowett-MSFT commented on GitHub (May 21, 2019):

That's possible. There's also a chance that we are getting the 3-byte UTF-8 encoded emoji in two parts from the application.

@DHowett-MSFT commented on GitHub (May 21, 2019): That's possible. There's also a chance that we are getting the 3-byte UTF-8 encoded emoji in two parts from the application.

claunia commented

@german-one commented on GitHub (May 21, 2019):

I'm convinced you guys from the terminal/console team are nice and will work around that kind of faulty input from an application, right? 😃

@german-one commented on GitHub (May 21, 2019): I'm convinced you guys from the terminal/console team are nice and will work around that kind of faulty input from an application, right? 😃

claunia commented

@DHowett-MSFT commented on GitHub (May 21, 2019):

As best we can!

@DHowett-MSFT commented on GitHub (May 21, 2019): As best we can!

claunia commented

@miniksa commented on GitHub (May 21, 2019):

I'm taking this because it feels similar to some of the other bugs I have assigned to me talking about U+FFFD.

@miniksa commented on GitHub (May 21, 2019): I'm taking this because it feels similar to some of the other bugs I have assigned to me talking about U+FFFD.

claunia commented

4c47631bf4/src/cascadia/TerminalConnection/ConhostConnection.cpp (L188-L190)

@DHowett-MSFT commented on GitHub (May 25, 2019):

Even if there's an issue in conhost, I'm convinced there's an issue in Terminal as well.

We are absolutely cutting UTF-8 sequences in half right there.

We can do fun games and cache partial sequences (partial code below), but it does add complexity.
It's probably necessary complexity!

if ((buffer[iLastGoodByte] & 0b1'0000000) == 0b1'0000000)
{
    // does the encoding fit?
    const auto leadByte = buffer[iLastGoodByte];
    auto sequenceLen = dwRead - iLastGoodByte;
    if ((((leadByte & 0b111'00000) == 0b110'00000) && sequenceLen < 2) ||  // lead reqs 2 bytes
        (((leadByte & 0b1111'0000) == 0b1110'0000) && sequenceLen < 3) ||  // lead reqs 3 bytes
        (((leadByte & 0b11111'000) == 0b11110'000) && sequenceLen < 4))    // lead reqs 4 bytes
    {
        ::memmove_s(utf8Partials, std::extent<decltype(utf8Partials)>::value, buffer + iLastGoodByte, dwRead - iLastGoodByte);
        // ... do the rest ...

I hacked it up to write 0 1 2 3 over the lead bytes where we cut off the ends of partial sequences, and A B C D over the trail bytes where we cut off the beginnings of partial sequences, and we get this:

(you can see where one ReadFile ended and the next began where we transitioned from numbers to letters.)

It wouldn't be strictly easier to just ReadFile until we got a complete UTF-8 codepoint because we would still need to figure out when we didn't have a complete codepoint.

@DHowett-MSFT commented on GitHub (May 25, 2019): Even if there's an issue in conhost, I'm convinced there's an issue in Terminal as well. https://github.com/microsoft/terminal/blob/4c47631bf4aa907aad4f7088bc7edc7e5cde11b9/src/cascadia/TerminalConnection/ConhostConnection.cpp#L188-L190 We are absolutely cutting UTF-8 sequences in half right there. We can do fun games and cache partial sequences (partial code below), but it does add complexity. It's probably necessary complexity! ```c++ if ((buffer[iLastGoodByte] & 0b1'0000000) == 0b1'0000000) { // does the encoding fit? const auto leadByte = buffer[iLastGoodByte]; auto sequenceLen = dwRead - iLastGoodByte; if ((((leadByte & 0b111'00000) == 0b110'00000) && sequenceLen < 2) || // lead reqs 2 bytes (((leadByte & 0b1111'0000) == 0b1110'0000) && sequenceLen < 3) || // lead reqs 3 bytes (((leadByte & 0b11111'000) == 0b11110'000) && sequenceLen < 4)) // lead reqs 4 bytes { ::memmove_s(utf8Partials, std::extent<decltype(utf8Partials)>::value, buffer + iLastGoodByte, dwRead - iLastGoodByte); // ... do the rest ... ``` I hacked it up to write `0` `1` `2` `3` over the lead bytes where we cut off the _ends_ of partial sequences, and `A` `B` `C` `D` over the trail bytes where we cut off the _beginnings_ of partial sequences, and we get this: ![image](https://user-images.githubusercontent.com/14316954/58362693-cd64e080-7e4e-11e9-9fc8-4dd455f8638d.png) (you can see where one `ReadFile` ended and the next began where we transitioned from numbers to letters.) It wouldn't be strictly easier to just `ReadFile` until we got a complete UTF-8 codepoint because we would still need to figure out when we didn't _have_ a complete codepoint.

claunia commented

@german-one commented on GitHub (May 25, 2019):

@DHowett-MSFT

It wouldn't be strictly easier to just ReadFile until we got a complete UTF-8 codepoint

Sorry for my ignorance but I don't understand this statement. Correct me if I'm wrong but passing a string that contains an incomplete codepoint to any other function causes an additional risk (just like to_hstring() irreversibly corrupts the string in the current code). Wouldn't it be always better to make sure you forward only complete codepoints from the early beginning?

@german-one commented on GitHub (May 25, 2019): @DHowett-MSFT > It wouldn't be strictly easier to just ReadFile until we got a complete UTF-8 codepoint Sorry for my ignorance but I don't understand this statement. Correct me if I'm wrong but passing a string that contains an incomplete codepoint to any other function causes an additional risk (just like `to_hstring()` irreversibly corrupts the string in the current code). Wouldn't it be always better to make sure you forward only complete codepoints from the early beginning?

claunia commented

@DHowett-MSFT commented on GitHub (May 25, 2019):

Sorry, that’s what this proposal is! When we encounter an incomplete codepoint we need to wait for the next trip around the I/O loop to complete it. We can still send everything up to but not including the incomplete codepoint.

My statement about ReadFile was moreso that it is safer/easier to cache the partial data and wait for the next read loop instead of trying to expand the buffer and read a couple more bytes immediately.

@DHowett-MSFT commented on GitHub (May 25, 2019): Sorry, that’s what this proposal is! When we encounter an incomplete codepoint we need to wait for the next trip around the I/O loop to complete it. We can still send everything up to but not including the incomplete codepoint. My statement about `ReadFile` was moreso that it is safer/easier to cache the partial data and wait for the next read loop instead of trying to expand the buffer and read a couple more bytes immediately.

claunia commented

4c47631bf4/src/host/utf8ToWideCharParser.hpp (L5-L14)

@DHowett-MSFT commented on GitHub (May 26, 2019):

This is almost certainly the right place to use @adiviness's Utf8ToWideCharParser!

To be done:

move it from host/ to types/ (check dependencies first)
consider: we trust that conpty will send us well-formed sequences, so we do not need the expensive full-buffer check (O(N)) that U8tWCP does; we only need to check the last 3 bytes of the string for longer-than-will-fit sequences
U8tWCP doesn't do null termination, but an HSTRING reference requires null termination on construction; we can work around this with a moderately expensive copy (O(N))

@DHowett-MSFT commented on GitHub (May 26, 2019): This is almost certainly the right place to use @adiviness's `Utf8ToWideCharParser`! https://github.com/microsoft/terminal/blob/4c47631bf4aa907aad4f7088bc7edc7e5cde11b9/src/host/utf8ToWideCharParser.hpp#L5-L14 To be done: * move it from `host/` to `types/` (check dependencies first) * consider: we trust that conpty will send us well-formed sequences, so we do not need the expensive full-buffer check (`O(N)`) that U8tWCP does; we only need to check the last 3 bytes of the string for longer-than-will-fit sequences * U8tWCP doesn't do null termination, but an HSTRING reference requires null termination on construction; we can work around this with a moderately expensive copy (`O(N)`)

claunia commented

@german-one commented on GitHub (May 26, 2019):

I was about to write a piece of code when I saw you already have a parser. But now that you added some more notes I finished it in the hope it is a little helpful.

// NormalizeUTF8.cpp

#include "pch.h"
#include <string>
#include <algorithm>
#include <type_traits>
#include <windows.h>

bool GetConsoleVTHandles(HANDLE& refHIn, HANDLE& refHOut, HANDLE& refHErr)
{
  HANDLE hIn{ GetStdHandle(STD_INPUT_HANDLE) };
  HANDLE hOut{ GetStdHandle(STD_OUTPUT_HANDLE) };
  HANDLE hErr{ GetStdHandle(STD_ERROR_HANDLE) };
  if (hIn == INVALID_HANDLE_VALUE || hOut == INVALID_HANDLE_VALUE || hErr == INVALID_HANDLE_VALUE)
    return false;

  DWORD dwConsoleMode{};

  if (GetFileType(hIn) == FILE_TYPE_CHAR &&
      (!GetConsoleMode(hIn, &dwConsoleMode) || !SetConsoleMode(hIn, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_INPUT)))
    return false;

  if (GetFileType(hOut) == FILE_TYPE_CHAR &&
      (!GetConsoleMode(hOut, &dwConsoleMode) || !SetConsoleMode(hOut, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN)))
    return false;

  if (GetFileType(hErr) == FILE_TYPE_CHAR &&
      (!GetConsoleMode(hErr, &dwConsoleMode) || !SetConsoleMode(hErr, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN)))
    return false;

  refHIn = hIn;
  refHOut = hOut;
  refHErr = hErr;
  return true;
}

int main()
{
  HANDLE hIn{ INVALID_HANDLE_VALUE };
  HANDLE hOut{ INVALID_HANDLE_VALUE };
  HANDLE hErr{ INVALID_HANDLE_VALUE };
  if (!GetConsoleVTHandles(hIn, hOut, hErr))
    return 1;

  BYTE buffer[256]{ 0 };     // buffer for the chunk read (small size for debugging)
  BYTE utf8Partials[4]{ 0 }; // buffer for code units of a partial UTF-8 code point that have to be cached
  DWORD dwRead{};            // length of a chunk read
  DWORD dwPartialsLen{};     // number of cached UTF-8 code units
  bool fSuccess{};
  bool fStreamBegin{};       // indicates whether text is left in utf8Partials after the BOM check
  const BYTE bitmasks[]{ 0, 0b11000000, 0b11100000, 0b11110000 }; // for comparisons after the Lead Byte was found

  // *** check whether an UTF-8 input stream has a Byte Order Mark ***
  fSuccess = !!ReadFile(hIn, utf8Partials, 3UL, &dwPartialsLen, nullptr);
  if (!fSuccess)
    return 1;

  if (utf8Partials[0] == 0xEF && utf8Partials[1] == 0xBB && utf8Partials[2] == 0xBF)
    dwPartialsLen = 0; // discard the BOM
  else
    fStreamBegin = true;


  // process the data of the standard input in a loop
  while (true)
  {
    // copy UTF-8 code units that were remaining from the previously read chunk (if any)
    if (dwPartialsLen != 0)
      std::move(utf8Partials, utf8Partials + dwPartialsLen, buffer);

    // try to read data from standard input
    fSuccess = !!ReadFile(hIn, &buffer[dwPartialsLen], std::extent<decltype(buffer)>::value - dwPartialsLen, &dwRead, nullptr);

    dwRead += dwPartialsLen;
    dwPartialsLen = 0;

    if (dwRead == 0) // quit the iteration if no data has been read and no cached data was left
      break;
    else if (!fSuccess && !fStreamBegin) // cached data without bytes for completion in the buffer
      return 1;

    const BYTE* const endPtr{ buffer + dwRead };
    const BYTE* backIter{ endPtr - 1 };
    // *** if necessary put up to three bytes in the cache: ***
    if ((*backIter & 0b10000000) == 0b10000000) // last byte in the buffer was a byte belonging to a UTF-8 multi-byte character
    {
      for (DWORD dwSequenceLen{ 1UL }, stop{ dwRead < 4UL ? dwRead : 4UL }; dwSequenceLen < stop; ++dwSequenceLen, --backIter) // check only up to 3 last bytes, if no Lead Byte was found then the byte before must be the Lead Byte and no partials are in the buffer
      {
        if ((*backIter & 0b11000000) == 0b11000000) // Lead Byte found
        {
          if ((*backIter & bitmasks[dwSequenceLen]) != bitmasks[dwSequenceLen - 1]) // if the Lead Byte indicates that the last bytes in the buffer is a partial UTF-8 code point then cache them
          {
            std::move(backIter, endPtr, utf8Partials);
            dwRead -= dwSequenceLen;
            dwPartialsLen = dwSequenceLen;
          }

          break;
        }
      }
    }

    const std::string_view strView{ reinterpret_cast<char*>(buffer), dwRead };
    auto hstr{ winrt::to_hstring(strView) };

    // write the chunk to the console window buffer (as replacement for _outputHandlers() here)
    DWORD  dwWritten{};
    fSuccess = !!WriteConsoleW(hOut, hstr.c_str(), hstr.size(), &dwWritten, nullptr);
    if (!fSuccess)
      return 1;

    fStreamBegin = false;
  }

  return 0;
}

It also still addresses the BOM issue (blank that appears in the first screen shot). Please tell me if you want me to report it separately.

Attached you'll find a testing environment with the compiled app.
NormalizeUTF8.zip

EDIT: Code and attachment updated.
EDIT 2: VT Processing enabled for additional tests

@german-one commented on GitHub (May 26, 2019): I was about to write a piece of code when I saw you already have a parser. But now that you added some more notes I finished it in the hope it is a little helpful. ``` // NormalizeUTF8.cpp #include "pch.h" #include <string> #include <algorithm> #include <type_traits> #include <windows.h> bool GetConsoleVTHandles(HANDLE& refHIn, HANDLE& refHOut, HANDLE& refHErr) { HANDLE hIn{ GetStdHandle(STD_INPUT_HANDLE) }; HANDLE hOut{ GetStdHandle(STD_OUTPUT_HANDLE) }; HANDLE hErr{ GetStdHandle(STD_ERROR_HANDLE) }; if (hIn == INVALID_HANDLE_VALUE || hOut == INVALID_HANDLE_VALUE || hErr == INVALID_HANDLE_VALUE) return false; DWORD dwConsoleMode{}; if (GetFileType(hIn) == FILE_TYPE_CHAR && (!GetConsoleMode(hIn, &dwConsoleMode) || !SetConsoleMode(hIn, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_INPUT))) return false; if (GetFileType(hOut) == FILE_TYPE_CHAR && (!GetConsoleMode(hOut, &dwConsoleMode) || !SetConsoleMode(hOut, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN))) return false; if (GetFileType(hErr) == FILE_TYPE_CHAR && (!GetConsoleMode(hErr, &dwConsoleMode) || !SetConsoleMode(hErr, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN))) return false; refHIn = hIn; refHOut = hOut; refHErr = hErr; return true; } int main() { HANDLE hIn{ INVALID_HANDLE_VALUE }; HANDLE hOut{ INVALID_HANDLE_VALUE }; HANDLE hErr{ INVALID_HANDLE_VALUE }; if (!GetConsoleVTHandles(hIn, hOut, hErr)) return 1; BYTE buffer[256]{ 0 }; // buffer for the chunk read (small size for debugging) BYTE utf8Partials[4]{ 0 }; // buffer for code units of a partial UTF-8 code point that have to be cached DWORD dwRead{}; // length of a chunk read DWORD dwPartialsLen{}; // number of cached UTF-8 code units bool fSuccess{}; bool fStreamBegin{}; // indicates whether text is left in utf8Partials after the BOM check const BYTE bitmasks[]{ 0, 0b11000000, 0b11100000, 0b11110000 }; // for comparisons after the Lead Byte was found // *** check whether an UTF-8 input stream has a Byte Order Mark *** fSuccess = !!ReadFile(hIn, utf8Partials, 3UL, &dwPartialsLen, nullptr); if (!fSuccess) return 1; if (utf8Partials[0] == 0xEF && utf8Partials[1] == 0xBB && utf8Partials[2] == 0xBF) dwPartialsLen = 0; // discard the BOM else fStreamBegin = true; // process the data of the standard input in a loop while (true) { // copy UTF-8 code units that were remaining from the previously read chunk (if any) if (dwPartialsLen != 0) std::move(utf8Partials, utf8Partials + dwPartialsLen, buffer); // try to read data from standard input fSuccess = !!ReadFile(hIn, &buffer[dwPartialsLen], std::extent<decltype(buffer)>::value - dwPartialsLen, &dwRead, nullptr); dwRead += dwPartialsLen; dwPartialsLen = 0; if (dwRead == 0) // quit the iteration if no data has been read and no cached data was left break; else if (!fSuccess && !fStreamBegin) // cached data without bytes for completion in the buffer return 1; const BYTE* const endPtr{ buffer + dwRead }; const BYTE* backIter{ endPtr - 1 }; // *** if necessary put up to three bytes in the cache: *** if ((*backIter & 0b10000000) == 0b10000000) // last byte in the buffer was a byte belonging to a UTF-8 multi-byte character { for (DWORD dwSequenceLen{ 1UL }, stop{ dwRead < 4UL ? dwRead : 4UL }; dwSequenceLen < stop; ++dwSequenceLen, --backIter) // check only up to 3 last bytes, if no Lead Byte was found then the byte before must be the Lead Byte and no partials are in the buffer { if ((*backIter & 0b11000000) == 0b11000000) // Lead Byte found { if ((*backIter & bitmasks[dwSequenceLen]) != bitmasks[dwSequenceLen - 1]) // if the Lead Byte indicates that the last bytes in the buffer is a partial UTF-8 code point then cache them { std::move(backIter, endPtr, utf8Partials); dwRead -= dwSequenceLen; dwPartialsLen = dwSequenceLen; } break; } } } const std::string_view strView{ reinterpret_cast<char*>(buffer), dwRead }; auto hstr{ winrt::to_hstring(strView) }; // write the chunk to the console window buffer (as replacement for _outputHandlers() here) DWORD dwWritten{}; fSuccess = !!WriteConsoleW(hOut, hstr.c_str(), hstr.size(), &dwWritten, nullptr); if (!fSuccess) return 1; fStreamBegin = false; } return 0; } ``` It also still addresses the BOM issue (blank that appears in the first screen shot). _Please tell me if you want me to report it separately._ ![image](https://user-images.githubusercontent.com/46659645/58381279-99f39480-7fbb-11e9-939c-48a724d21d07.png) ![image](https://user-images.githubusercontent.com/46659645/58381290-c3acbb80-7fbb-11e9-994e-4d833d59b9b5.png) Attached you'll find a testing environment with the compiled app. [NormalizeUTF8.zip](https://github.com/microsoft/terminal/files/3268670/NormalizeUTF8.zip) EDIT: Code and attachment updated. EDIT 2: VT Processing enabled for additional tests

claunia commented

@german-one commented on GitHub (Jun 8, 2019):

I updated the latest test code again (with VT Processing enabled) in order to see how it might be related to the occurrence of � (Unicode Replacement Character U+FFFD) as they appear in #455 and #666.
For that reason I've stolen @PhMajerus ' test pattern, converted it to UTF-8, and attached it to the above ZIP archive.
TYPE only:

Piped to the test code:

The replacement characters don't appear at the same positions as in the linked issues. But I guess this is due to different sizes of character buffers in the Console and Terminal source codes.

@german-one commented on GitHub (Jun 8, 2019): I updated the latest test code again (with VT Processing enabled) in order to see how it might be related to the occurrence of � (Unicode Replacement Character U+FFFD) as they appear in #455 and #666. For that reason I've stolen @PhMajerus ' test pattern, converted it to UTF-8, and attached it to the above ZIP archive. TYPE only: ![image](https://user-images.githubusercontent.com/46659645/59149131-15b8fc80-8a12-11e9-8a83-5e338726a8f0.png) Piped to the test code: ![image](https://user-images.githubusercontent.com/46659645/59149138-35502500-8a12-11e9-8e02-35b62fb69231.png) The replacement characters don't appear at the same positions as in the linked issues. But I guess this is due to different sizes of character buffers in the Console and Terminal source codes.

claunia commented

@german-one commented on GitHub (Jun 30, 2019):

It was not too hard to update the function that @DHowett-MSFT quoted in https://github.com/Microsoft/terminal/issues/386#issuecomment-495822626
Unfortunately this doesn't seem to be the only function that has to be updated because it isn't sufficient to close this issue yet.
Could you guys point me to more source files that are worth to review? I'd like to contribute but for me it's a mess to figure out where I have a chance to find the culprit.

@german-one commented on GitHub (Jun 30, 2019): It was not too hard to update the function that @DHowett-MSFT quoted in https://github.com/Microsoft/terminal/issues/386#issuecomment-495822626 Unfortunately this doesn't seem to be the only function that has to be updated because it isn't sufficient to close this issue yet. Could you guys point me to more source files that are worth to review? I'd like to contribute but for me it's a mess to figure out where I have a chance to find the culprit.

claunia commented

@miniksa commented on GitHub (Jul 2, 2019):

Uhhhh, what's still wrong after your proposed test change? That would help me better narrow down which files might still be suspect.

If you're looking inside the console host for problems, I'd personally breakpoint on the WriteConsole* methods inside _stream.cpp and chase them around the code to see how the segments of text received off of the APIs are being handled.

(I think you're trying to fix the console host side given the screenshots above of just CMD.exe in a plain conhost window.)

@miniksa commented on GitHub (Jul 2, 2019): Uhhhh, what's still wrong after your proposed test change? That would help me better narrow down which files might still be suspect. If you're looking inside the console host for problems, I'd personally breakpoint on the `WriteConsole*` methods inside `_stream.cpp` and chase them around the code to see how the segments of text received off of the APIs are being handled. (I think you're trying to fix the console host side given the screenshots above of just CMD.exe in a plain conhost window.)

claunia commented

@german-one commented on GitHub (Jul 2, 2019):

Thanks you @miniksa

The screenshots above have been made before I forked the source code. So, yes that was of course for conhost only. The function that @DHowett-MSFT quoted and that I tried to update is the interface between terminal and conhost (at least that's my poor understanding of how it is linked together, correct me if I'm wrong). You can find it in this commit:
ea46579794

what's still wrong after your proposed test change?

Well, nothing changed yet. I didn't expect that this would already close this issue because the buffer size in this function is 4096 bytes but the U+FFFD character appears more often. I suspect there is a much smaller buffer involved somewhere else. However, the update also includes the BOM removal. And at least that is what I expected to see.

That would help me better narrow down which files might still be suspect.

Some more screenshots might be helpful.

PowerShell did never show that behavior. Neither Windows PowerShell ...

... nor PowerShell 6.

Neither as stand-alone nor in the terminal.

Ubuntu bash has a problem with the BOM (as Linux has in general).
CMD fails entirely. For completeness again in the default conhost ...

... and with the updated code.

At least in the last screenshot I would have expected that the BOM doesn't appear anymore (blank on the top left corner of the pattern).
FWIW the misaligned text ".majerus.net/" is not related to my update. Probably another issue.

I'd personally breakpoint on the WriteConsole* methods inside _stream.cpp and chase them around the code to see how the segments of text received off of the APIs are being handled.

I'll try my best.

Steffen

@german-one commented on GitHub (Jul 2, 2019): Thanks you @miniksa The screenshots above have been made before I forked the source code. So, yes that was of course for conhost only. The function that @DHowett-MSFT quoted and that I tried to update is the interface between terminal and conhost (at least that's my poor understanding of how it is linked together, correct me if I'm wrong). You can find it in this commit: https://github.com/german-one/terminal/commit/ea46579794aa984d3141c2de10e9363fd9db6331 > what's still wrong after your proposed test change? Well, nothing changed yet. I didn't expect that this would already close this issue because the buffer size in this function is 4096 bytes but the U+FFFD character appears more often. I suspect there is a much smaller buffer involved somewhere else. However, the update also includes the BOM removal. And at least that is what I expected to see. > That would help me better narrow down which files might still be suspect. Some more screenshots might be helpful. 1. PowerShell did never show that behavior. Neither Windows PowerShell ... ![pswin](https://user-images.githubusercontent.com/46659645/60530614-6d891180-9cf9-11e9-8170-e5c3eee68eab.png) ... nor PowerShell 6. ![ps6](https://user-images.githubusercontent.com/46659645/60530767-cfe21200-9cf9-11e9-9b52-0d2deccaf134.png) Neither as stand-alone nor in the terminal. ![terminal_ps6](https://user-images.githubusercontent.com/46659645/60530782-db353d80-9cf9-11e9-960a-ae3cf835c0e2.png) 2. Ubuntu bash has a problem with the BOM (as Linux has in general). ![bash](https://user-images.githubusercontent.com/46659645/60530893-120b5380-9cfa-11e9-97b3-16884714483e.png) ![terminal_ubuntu](https://user-images.githubusercontent.com/46659645/60530909-1b94bb80-9cfa-11e9-86dc-c8d46283825c.png) 3. CMD fails entirely. For completeness again in the default conhost ... ![conwin](https://user-images.githubusercontent.com/46659645/60531325-09ffe380-9cfb-11e9-9702-d1917357f929.png) ... and with the updated code. ![conupdate](https://user-images.githubusercontent.com/46659645/60531377-269c1b80-9cfb-11e9-8e12-c213e894255a.png) ![terminal_cmd](https://user-images.githubusercontent.com/46659645/60531396-36b3fb00-9cfb-11e9-9a5c-a57139a93c29.png) At least in the last screenshot I would have expected that the BOM doesn't appear anymore (blank on the top left corner of the pattern). FWIW the misaligned text ".majerus.net/" is not related to my update. Probably another issue. > I'd personally breakpoint on the WriteConsole* methods inside _stream.cpp and chase them around the code to see how the segments of text received off of the APIs are being handled. I'll try my best. Steffen

claunia commented

@miniksa commented on GitHub (Jul 2, 2019):

Oh. The difference between Powershell/PWSH, CMD, and WSL is that they're likely all using different Write* method entrypoints.

I forget which is which, but they're probably using a selection of WriteConsoleA, WriteConsoleW, WriteConsoleOutputA, WriteConsoleOutputW, WriteFile (which should be serviced the same as WriteConsoleA).

The post-API-decoding-calls should be findable by searching the code for ApiRoutines::WriteConsoleAImpl and friends.

It's probably that CMD is using one and the others are using others and the one that CMD is using is broken somehow.

The other thing to watch out for is whether they're calling SetConsoleCP or SetConsoleOutputCP. It's likely/probable that Powershell, PWS, and WSL are either setting the codepage to 65001 (UTF-8) and using the A versions of the APIs OR using the W versions of the APIs with UTF-16 text directly.

But CMD.exe doesn't usually do any of that. It's probably using whatever codepage the system is in and writing everything into the A API without thinking about it.

@miniksa commented on GitHub (Jul 2, 2019): Oh. The difference between Powershell/PWSH, CMD, and WSL is that they're likely all using different Write* method entrypoints. I forget which is which, but they're probably using a selection of `WriteConsoleA`, `WriteConsoleW`, `WriteConsoleOutputA`, `WriteConsoleOutputW`, `WriteFile` (which should be serviced the same as `WriteConsoleA`). The post-API-decoding-calls should be findable by searching the code for `ApiRoutines::WriteConsoleAImpl` and friends. It's probably that CMD is using one and the others are using others and the one that CMD is using is broken somehow. The other thing to watch out for is whether they're calling `SetConsoleCP` or `SetConsoleOutputCP`. It's likely/probable that Powershell, PWS, and WSL are either setting the codepage to 65001 (UTF-8) and using the A versions of the APIs OR using the W versions of the APIs with UTF-16 text directly. But CMD.exe doesn't usually do any of that. It's probably using whatever codepage the system is in and writing everything into the A API without thinking about it.

claunia commented

@german-one commented on GitHub (Jul 2, 2019):

But CMD.exe doesn't usually do any of that. It's probably using whatever codepage the system is in and writing everything into the A API without thinking about it.

Yes I'm aware of that fact. If you look at the prompts in my screenshots you'll see that I always have to use chcp 65001 in the cmd.exe instances. Otherwise you would see the character representation of every single byte in the default OEM codepage.

WriteConsoleAImpl is certainly a good point to look at. It involves the Utf8ToWideCharParser class where I don't expect to find the culprit. The member functions of the class use MultiByteToWideChar to check if all code points are valid and to convert the UTF-8 buffer to UTF-16. All good so far. I expect that the culprit is the caller of WriteConsoleAImpl. My test code proves that UTF-8 itself is not a real problem. But if the caller pass chunks containing partial code points (which is what I correct in my test code) then it might be already too late for the API to work around that issue. Imo we would need a static cache for the partials in WriteConsoleAImpl. But I'm not sure if that would cause more issues. If we have a static cache, would this influence concurrent calls from other processes? Or owns every process an independent instance of the function?

@german-one commented on GitHub (Jul 2, 2019): > But CMD.exe doesn't usually do any of that. It's probably using whatever codepage the system is in and writing everything into the A API without thinking about it. Yes I'm aware of that fact. If you look at the prompts in my screenshots you'll see that I always have to use `chcp 65001` in the cmd.exe instances. Otherwise you would see the character representation of every single byte in the default OEM codepage. `WriteConsoleAImpl` is certainly a good point to look at. It involves the `Utf8ToWideCharParser` class where I don't expect to find the culprit. The member functions of the class use `MultiByteToWideChar` to check if all code points are valid and to convert the UTF-8 buffer to UTF-16. All good so far. I expect that the culprit is the caller of `WriteConsoleAImpl`. My test code proves that UTF-8 itself is not a real problem. But if the caller pass chunks containing partial code points (which is what I correct in my test code) then it might be already too late for the API to work around that issue. Imo we would need a static cache for the partials in `WriteConsoleAImpl`. But I'm not sure if that would cause more issues. If we have a static cache, would this influence concurrent calls from other processes? Or owns every process an independent instance of the function?

claunia commented

@german-one commented on GitHub (Jul 2, 2019):

Um ... wait ... we already have a static instance of Utf8ToWideCharParser in WriteConsoleAImpl. That answers my question and I need to have a closer look at Utf8ToWideCharParser.

@german-one commented on GitHub (Jul 2, 2019): Um ... wait ... we already have a `static` instance of `Utf8ToWideCharParser` in `WriteConsoleAImpl`. That answers my question and I need to have a closer look at `Utf8ToWideCharParser`.

claunia commented

@miniksa commented on GitHub (Jul 2, 2019):

There cannot be concurrent calls in that we cannot service two calls at the exact same time because they're queued and the console is locked.

However, there can be concurrent calls in that Process A can write a partial fragment, then Process B's request is the next in queue and can write some other unrelated fragment, then Process A is next to write its remainder. So a static cache will leave some sort of hole in the quest for a perfect solution.

However, I'm pretty sure Utf8ToWideCharParser right now ignores this problem and effectively IS a static cache for the partials when the codepage is set to 65001.

There is also another static cache for other codepages called WriteConsoleDbcsLeadByte hanging off of the SCREEN_INFORMATION class that is supposed to hold the first half of a split double-byte sequence (for codepages like 932).

That's been that way for decades and is probably why we ignored the concurrence problem when writing Utf8ToWideCharParser in the more recent era.

The caller of WriteConsoleAImpl can be found in the server library, but that library is basically just decoding the entire API packet as it receives it. It will call WriteConsoleAImpl once with the entire contents of the packet that it received from the calling process. So if anyone's tearing bytes in half, it should be the caller on the other end.

But Utf8ToWideCharParser and WriteConsoleDbcsLeadByte are supposed to handle that torn state for 65001 and other code pages respectively already.

@miniksa commented on GitHub (Jul 2, 2019): There cannot be concurrent calls in that we cannot service two calls at the exact same time because they're queued and the console is locked. However, there can be concurrent calls in that Process A can write a partial fragment, then Process B's request is the next in queue and can write some other unrelated fragment, then Process A is next to write its remainder. So a static cache will leave some sort of hole in the quest for a perfect solution. However, I'm pretty sure `Utf8ToWideCharParser` right now ignores this problem and effectively IS a static cache for the partials when the codepage is set to 65001. There is also another static cache for other codepages called `WriteConsoleDbcsLeadByte` hanging off of the `SCREEN_INFORMATION` class that is supposed to hold the first half of a split double-byte sequence (for codepages like 932). That's been that way for decades and is probably why we ignored the concurrence problem when writing `Utf8ToWideCharParser` in the more recent era. The caller of `WriteConsoleAImpl` can be found in the `server` library, but that library is basically just decoding the entire API packet as it receives it. It will call `WriteConsoleAImpl` once with the entire contents of the packet that it received from the calling process. So if anyone's tearing bytes in half, it should be the caller on the other end. But `Utf8ToWideCharParser` and `WriteConsoleDbcsLeadByte` are supposed to handle that torn state for 65001 and other code pages respectively already.

claunia commented

@german-one commented on GitHub (Jul 2, 2019):

We posted at the same time 😀 I didn't see the static declaration in the first place. However, thanks for the explanation and insight.

So if anyone's tearing bytes in half, it should be the caller on the other end.

Yes, that's exactly what I believe is the reason for this problem. See what I did in my test code. If conhost gets the the data directly from cmd.exe (using its internal TYPE command) then we are facing this issue. If the same data is piped to my test app which takes care of incoming partials, and conhost gets chunks of data from the test app that don't contain partials anymore, then you don't have this problems anymore. So, the actual question is whether or not conhost is supposed to work around the deficiencies of applications like cmd.exe 😉 It would be nice, no doubt.

@german-one commented on GitHub (Jul 2, 2019): We posted at the same time 😀 I didn't see the static declaration in the first place. However, thanks for the explanation and insight. >So if anyone's tearing bytes in half, it should be the caller on the other end. Yes, that's exactly what I believe is the reason for this problem. See what I did in my test code. If conhost gets the the data directly from cmd.exe (using its internal TYPE command) then we are facing this issue. If the same data is piped to my test app which takes care of incoming partials, and conhost gets chunks of data from the test app that don't contain partials anymore, then you don't have this problems anymore. So, the actual question is whether or not conhost is supposed to work around the deficiencies of applications like cmd.exe 😉 It would be nice, no doubt.

claunia commented

@german-one commented on GitHub (Jul 3, 2019):

Update:
I was not able to find anything wrong in Utf8ToWideCharParser. Thus, I banned it entirely from WriteConsoleAImpl and implemented my own quick and dirty UTF-8 handling. Result was that nothing changed for the CMD output. But the BOM is getting removed in the WSL output.
@miniksa What do you think? Is the BOM removal something that should be included into the Utf8ToWideCharParser class? My expectation is that the BOM should not show up in the output. But on the other hand \EF \BB \BF shall be treated as BOM only if it is the beginning of the stream.

Next step for me is to review WriteConsoleOutputCharacterAImpl in _output.cpp. There is a real good chance to find the culprit 😀😀 It calls ConvertToW in convert.cpp. This function converts the incoming chunk to UTF-16 without any check for partial code points. I'll let you know ...

@german-one commented on GitHub (Jul 3, 2019): Update: I was not able to find anything wrong in `Utf8ToWideCharParser`. Thus, I banned it entirely from `WriteConsoleAImpl` and implemented my own quick and dirty UTF-8 handling. Result was that nothing changed for the CMD output. But the BOM is getting removed in the WSL output. @miniksa What do you think? Is the BOM removal something that should be included into the `Utf8ToWideCharParser` class? My expectation is that the BOM should not show up in the output. But on the other hand \EF \BB \BF shall be treated as BOM only if it is the beginning of the stream. Next step for me is to review `WriteConsoleOutputCharacterAImpl` in _output.cpp. There is a real good chance to find the culprit 😀😀 It calls `ConvertToW` in convert.cpp. This function converts the incoming chunk to UTF-16 without any check for partial code points. I'll let you know ...

claunia commented

@miniksa commented on GitHub (Jul 3, 2019):

I'm of two minds on the BOM thing.

The first is that it's impossible to tell really what is the beginning of stream without some heuristics or further state on when a process connected and if it's the first thing that it has said. Probably more complicated than we want to deal with.

The second is that I can't think of a valid use for echoing a BOM to a Console/Terminal in the slightest and we should probably just wholesale discard them even though we should technically only do that as the "start of stream" rule. But instead of discarding them here, I think I'd rather pass it along further in the Utf8ToWideCharParser portion of the class and let the TextBuffer treat the BOM as a control character that "does nothing" than to strip it out in the parser layer. I have an agenda item on me to effectively add a WriteStream method to the TextBuffer to attempt to supercede both WriteCharsLegacy in conhost.exe and the WriteCharsLegacy2ElectricBoogaloo method in windowsterminal.exe. Those methods are basically responsible for stripping controls and performing an action anyway so it seems like an apt location to put something like this.

In short, leave the BOM alone for now and we'll revisit.

For the other part, GOOD CALL. You're right, the WriteConsoleOutputCharacterA method is likely doing things quite differently with ConvertToW. I always wanted to reconcile how the cell-write methods (OutputCharacter/OutputAttribute) worked versus the stream-write methods (Output) in terms of codepage conversion, but we never got to that. I forgot about that one when I was writing this yesterday. Hope you find something fruitful there and thank you for continuing to dig into this!

@miniksa commented on GitHub (Jul 3, 2019): I'm of two minds on the BOM thing. The first is that it's impossible to tell really what is the beginning of stream without some heuristics or further state on when a process connected and if it's the first thing that it has said. Probably more complicated than we want to deal with. The second is that I can't think of a valid use for echoing a BOM to a Console/Terminal in the slightest and we should probably just wholesale discard them even though we should technically only do that as the "start of stream" rule. But instead of discarding them here, I think I'd rather pass it along further in the `Utf8ToWideCharParser` portion of the class and let the `TextBuffer` treat the BOM as a control character that "does nothing" than to strip it out in the parser layer. I have an agenda item on me to effectively add a `WriteStream` method to the `TextBuffer` to attempt to supercede both `WriteCharsLegacy` in conhost.exe and the `WriteCharsLegacy2ElectricBoogaloo` method in windowsterminal.exe. Those methods are basically responsible for stripping controls and performing an action anyway so it seems like an apt location to put something like this. In short, leave the BOM alone for now and we'll revisit. For the other part, GOOD CALL. You're right, the `WriteConsoleOutputCharacterA` method is likely doing things quite differently with `ConvertToW`. I always wanted to reconcile how the cell-write methods (`OutputCharacter`/`OutputAttribute`) worked versus the stream-write methods (`Output`) in terms of codepage conversion, but we never got to that. I forgot about that one when I was writing this yesterday. Hope you find something fruitful there and thank you for continuing to dig into this!

claunia commented

@german-one commented on GitHub (Jul 3, 2019):

Thanks for your advice regarding the BOM.

Unfortunately updating the WriteConsoleOutputCharacterA function still didn't resolve the CMD issue. Meanwhile I'm wondering if CMD.exe determines the file type of the standard output handle, and in case it is conhost, it converts the stream to UTF-16 internally. In this case I'm afraid we are out of luck.
I think I should write another small test app that does nothing but forwarding the piped stream from cmd.exe in order to investigate how it behaves.

@german-one commented on GitHub (Jul 3, 2019): Thanks for your advice regarding the BOM. Unfortunately updating the `WriteConsoleOutputCharacterA` function still didn't resolve the CMD issue. Meanwhile I'm wondering if CMD.exe determines the file type of the standard output handle, and in case it is conhost, it converts the stream to UTF-16 internally. In this case I'm afraid we are out of luck. I think I should write another small test app that does nothing but forwarding the piped stream from cmd.exe in order to investigate how it behaves.

claunia commented

@german-one commented on GitHub (Jul 3, 2019):

CMD.exe is an evil beast 👹

another test code:

#include "pch.h"
#include <string>
#include <algorithm>
#include <type_traits>
#include <windows.h>

bool GetConsoleVTHandles(HANDLE& refHIn, HANDLE& refHOut, HANDLE& refHErr)
{
  HANDLE hIn{ GetStdHandle(STD_INPUT_HANDLE) };
  HANDLE hOut{ GetStdHandle(STD_OUTPUT_HANDLE) };
  HANDLE hErr{ GetStdHandle(STD_ERROR_HANDLE) };
  if (hIn == INVALID_HANDLE_VALUE || hOut == INVALID_HANDLE_VALUE || hErr == INVALID_HANDLE_VALUE)
    return false;

  DWORD dwConsoleMode{};

  if (GetFileType(hIn) == FILE_TYPE_CHAR &&
      (!GetConsoleMode(hIn, &dwConsoleMode) || !SetConsoleMode(hIn, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_INPUT)))
    return false;

  if (GetFileType(hOut) == FILE_TYPE_CHAR &&
      (!GetConsoleMode(hOut, &dwConsoleMode) || !SetConsoleMode(hOut, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN)))
    return false;

  if (GetFileType(hErr) == FILE_TYPE_CHAR &&
      (!GetConsoleMode(hErr, &dwConsoleMode) || !SetConsoleMode(hErr, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN)))
    return false;

  refHIn = hIn;
  refHOut = hOut;
  refHErr = hErr;
  return true;
}

int main()
{
  HANDLE hIn{ INVALID_HANDLE_VALUE };
  HANDLE hOut{ INVALID_HANDLE_VALUE };
  HANDLE hErr{ INVALID_HANDLE_VALUE };
  if (!GetConsoleVTHandles(hIn, hOut, hErr))
    return -1;

  BYTE buffer[8191]{ 0 };     // buffer for the chunk read
  DWORD dwRead{};            // length of a chunk read
  bool fSuccess{};
  int iterations{};

  // process the data of the standard input in a loop
  while (true)
  {
    // try to read data from standard input
    fSuccess = !!ReadFile(hIn, buffer, std::extent<decltype(buffer)>::value, &dwRead, nullptr);

    if (dwRead == 0)
      break;
    else if (!fSuccess)
      return -1;

	DWORD  dwWritten{};

	//fSuccess = !!WriteConsoleOutputCharacterA(hOut, (char*)buffer, dwRead, COORD{ 0, 10 }, &dwWritten);
	//SetConsoleCursorPosition(hOut, COORD{ 0, 35 });

	//fSuccess = !!WriteConsoleA(hOut, buffer, dwRead, &dwWritten, nullptr);

	fSuccess = !!WriteFile(hOut, buffer, dwRead, &dwWritten, nullptr);

	if (!fSuccess)
      return -1;

	++iterations;
  }

  return iterations;
}

It just reads from standard input and uses WriteConsoleOutputCharacterA, WriteConsoleA, or WriteFile to write to standard output (I compiled 3 versions). The tool returns the number of write operations.
Note that I used a buffer of 8192 bytes while the test pattern has a size of only 6642 bytes. I tested using the default conhost (no updated functions involved).

This screenshot shows that ConsoleOutputCharacterA can't be the reason for the faulty output. It is not even able to process Line Feed characters to wrap the lines. VT escape sequences are ignored, too. (I still believe that the function should be updated for UTF-8 partials though.)
It was written in 2 steps . No idea if CMD forwards the stream in two steps, if the involved pipe can't process all of the 6642 bytes at once, or if ReadFile has a limitation.

WriteConsoleA successfully writes the buffer read. Still with BOM as expected.

WriteFile behaves the same.

@miniksa My conclusion is that CMD.exe detects the file type (likely using GetFileType). If the type is FILE_TYPE_CHAR then it internally converts UTF-8 to UTF-16 and uses one of the ...W functions to write the output. Unless you have any further idea I'm afraid you can close this issue without having a solution, since CMD.exe is out of scope and as you guys explained several times you won't update it anymore.

However, apparently there are some functions that should be updated even if these updates are not able to resolve the CMD.exe issues. Any advice on how to proceed?

@german-one commented on GitHub (Jul 3, 2019): CMD.exe is an evil beast 👹 another test code: ``` #include "pch.h" #include <string> #include <algorithm> #include <type_traits> #include <windows.h> bool GetConsoleVTHandles(HANDLE& refHIn, HANDLE& refHOut, HANDLE& refHErr) { HANDLE hIn{ GetStdHandle(STD_INPUT_HANDLE) }; HANDLE hOut{ GetStdHandle(STD_OUTPUT_HANDLE) }; HANDLE hErr{ GetStdHandle(STD_ERROR_HANDLE) }; if (hIn == INVALID_HANDLE_VALUE || hOut == INVALID_HANDLE_VALUE || hErr == INVALID_HANDLE_VALUE) return false; DWORD dwConsoleMode{}; if (GetFileType(hIn) == FILE_TYPE_CHAR && (!GetConsoleMode(hIn, &dwConsoleMode) || !SetConsoleMode(hIn, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_INPUT))) return false; if (GetFileType(hOut) == FILE_TYPE_CHAR && (!GetConsoleMode(hOut, &dwConsoleMode) || !SetConsoleMode(hOut, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN))) return false; if (GetFileType(hErr) == FILE_TYPE_CHAR && (!GetConsoleMode(hErr, &dwConsoleMode) || !SetConsoleMode(hErr, dwConsoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING | DISABLE_NEWLINE_AUTO_RETURN))) return false; refHIn = hIn; refHOut = hOut; refHErr = hErr; return true; } int main() { HANDLE hIn{ INVALID_HANDLE_VALUE }; HANDLE hOut{ INVALID_HANDLE_VALUE }; HANDLE hErr{ INVALID_HANDLE_VALUE }; if (!GetConsoleVTHandles(hIn, hOut, hErr)) return -1; BYTE buffer[8191]{ 0 }; // buffer for the chunk read DWORD dwRead{}; // length of a chunk read bool fSuccess{}; int iterations{}; // process the data of the standard input in a loop while (true) { // try to read data from standard input fSuccess = !!ReadFile(hIn, buffer, std::extent<decltype(buffer)>::value, &dwRead, nullptr); if (dwRead == 0) break; else if (!fSuccess) return -1; DWORD dwWritten{}; //fSuccess = !!WriteConsoleOutputCharacterA(hOut, (char*)buffer, dwRead, COORD{ 0, 10 }, &dwWritten); //SetConsoleCursorPosition(hOut, COORD{ 0, 35 }); //fSuccess = !!WriteConsoleA(hOut, buffer, dwRead, &dwWritten, nullptr); fSuccess = !!WriteFile(hOut, buffer, dwRead, &dwWritten, nullptr); if (!fSuccess) return -1; ++iterations; } return iterations; } ``` It just reads from standard input and uses `WriteConsoleOutputCharacterA`, `WriteConsoleA`, or `WriteFile` to write to standard output (I compiled 3 versions). The tool returns the number of write operations. Note that I used a buffer of 8192 bytes while the test pattern has a size of only 6642 bytes. I tested using the default conhost (no updated functions involved). --- 1. ![image](https://user-images.githubusercontent.com/46659645/60626565-76f2a680-9deb-11e9-902a-e4ed851a92c0.png) This screenshot shows that `ConsoleOutputCharacterA` can't be the reason for the faulty output. It is not even able to process Line Feed characters to wrap the lines. VT escape sequences are ignored, too. (I still believe that the function should be updated for UTF-8 partials though.) It was written in 2 steps . No idea if CMD forwards the stream in two steps, if the involved pipe can't process all of the 6642 bytes at once, or if ReadFile has a limitation. --- 2. ![image](https://user-images.githubusercontent.com/46659645/60627076-398f1880-9ded-11e9-8433-e724d8136920.png) `WriteConsoleA` successfully writes the buffer read. Still with BOM as expected. --- 3. ![image](https://user-images.githubusercontent.com/46659645/60627359-1d3fab80-9dee-11e9-90aa-7f534bf43ab6.png) `WriteFile` behaves the same. --- @miniksa My conclusion is that CMD.exe detects the file type (likely using `GetFileType`). If the type is `FILE_TYPE_CHAR` then it internally converts UTF-8 to UTF-16 and uses one of the ...W functions to write the output. Unless you have any further idea I'm afraid you can close this issue without having a solution, since CMD.exe is out of scope and as you guys explained several times you won't update it anymore. However, apparently there are some functions that should be updated even if these updates are not able to resolve the CMD.exe issues. Any advice on how to proceed?

claunia commented

@miniksa commented on GitHub (Jul 5, 2019):

CMD.exe is an evil beast 👹

Yes. Very yes.

@miniksa My conclusion is that CMD.exe detects the file type (likely using GetFileType). If the type is FILE_TYPE_CHAR then it internally converts UTF-8 to UTF-16 and uses one of the ...W functions to write the output. Unless you have any further idea I'm afraid you can close this issue without having a solution, since CMD.exe is out of scope and as you guys explained several times you won't update it anymore.

I wouldn't be surprised if it was converting. I can try to spend like 5 minutes on checking if this is true by reading the internal source for CMD.exe on Monday when I get back into work, but it's not worth spending much time on because we can't really fix anything in CMD without breaking the world.

However, apparently there are some functions that should be updated even if these updates are not able to resolve the CMD.exe issues. Any advice on how to proceed?

Start a PR anyway with the proposed fixes to the other functions referencing this issue. It's OK if we only fix part of it.

@miniksa commented on GitHub (Jul 5, 2019): > CMD.exe is an evil beast 👹 Yes. Very yes. > @miniksa My conclusion is that CMD.exe detects the file type (likely using `GetFileType`). If the type is `FILE_TYPE_CHAR` then it internally converts UTF-8 to UTF-16 and uses one of the ...W functions to write the output. Unless you have any further idea I'm afraid you can close this issue without having a solution, since CMD.exe is out of scope and as you guys explained several times you won't update it anymore. I wouldn't be surprised if it was converting. I can try to spend like 5 minutes on checking if this is true by reading the internal source for CMD.exe on Monday when I get back into work, but it's not worth spending much time on because we can't really fix anything in CMD without breaking the world. > However, apparently there are some functions that should be updated even if these updates are not able to resolve the CMD.exe issues. Any advice on how to proceed? Start a PR anyway with the proposed fixes to the other functions referencing this issue. It's OK if we only fix part of it.

claunia commented