starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

ReadFile: Ctrl+Z sends EOF even if console is in 'raw' mode #6921

New Issue
Open
opened 2026-01-31 00:50:32 +00:00 by claunia · 5 comments
claunia commented 2026-01-31 00:50:32 +00:00
Owner
Copy Link

Originally created by @alexrp on GitHub (Mar 17, 2020).

Originally assigned to: @lhecker on GitHub.

Environment

Windows build number: Microsoft Windows [Version 10.0.19041.153]
.NET Core 3.1.200

Steps to reproduce

C# program:

using System;
using System.Diagnostics;
using System.Text;
using static Vanara.PInvoke.Kernel32;

namespace Test
{
    static class Program
    {
        unsafe static void Main()
        {
            var stdin = GetStdHandle(StdHandleType.STD_INPUT_HANDLE);
            var stdout = GetStdHandle(StdHandleType.STD_OUTPUT_HANDLE);

            Debug.Assert(GetConsoleMode(stdin, out CONSOLE_INPUT_MODE inMode));

            inMode |= CONSOLE_INPUT_MODE.ENABLE_VIRTUAL_TERMINAL_INPUT;
            inMode &= ~(CONSOLE_INPUT_MODE.ENABLE_PROCESSED_INPUT |
                        CONSOLE_INPUT_MODE.ENABLE_LINE_INPUT |
                        CONSOLE_INPUT_MODE.ENABLE_ECHO_INPUT);

            Debug.Assert(SetConsoleMode(stdin, inMode));

            Debug.Assert(GetConsoleMode(stdout, out CONSOLE_OUTPUT_MODE outMode));

            outMode |= CONSOLE_OUTPUT_MODE.ENABLE_VIRTUAL_TERMINAL_PROCESSING |
                       CONSOLE_OUTPUT_MODE.DISABLE_NEWLINE_AUTO_RETURN;
            outMode &= ~(CONSOLE_OUTPUT_MODE.ENABLE_PROCESSED_OUTPUT |
                         CONSOLE_OUTPUT_MODE.ENABLE_WRAP_AT_EOL_OUTPUT);

            Debug.Assert(SetConsoleMode(stdout, outMode));

            byte b = 0;

            Debug.Assert(ReadFile(stdin, (IntPtr)(&b), 1, out var r, IntPtr.Zero));

            Console.Write("Read {0} bytes; value = 0x{1:x2}", r, b);
        }
    }
}

(Uses the Vanara.PInvoke.Kernel32 NuGet package for convenience.)

Run the program in any console host / shell (CMD, PowerShell, Windows Terminal, etc).

Expected behavior

Output: Read 1 bytes; value = 0x1a (behavior on Linux)

Actual behavior

Output: Read 0 bytes; value = 0x00

I understand that Ctrl+Z is (roughly) Windows's equivalent to Ctrl+D to send EOF. However, I would expect it not to send EOF when the console has been put in 'raw' mode as described here. This is the behavior on Unix systems for Ctrl+D when you unset ICANON (see this manual page).

Perhaps there's an additional step that I'm missing?

Originally created by @alexrp on GitHub (Mar 17, 2020). Originally assigned to: @lhecker on GitHub. # Environment ```none Windows build number: Microsoft Windows [Version 10.0.19041.153] .NET Core 3.1.200 ``` # Steps to reproduce C# program: ```csharp using System; using System.Diagnostics; using System.Text; using static Vanara.PInvoke.Kernel32; namespace Test { static class Program { unsafe static void Main() { var stdin = GetStdHandle(StdHandleType.STD_INPUT_HANDLE); var stdout = GetStdHandle(StdHandleType.STD_OUTPUT_HANDLE); Debug.Assert(GetConsoleMode(stdin, out CONSOLE_INPUT_MODE inMode)); inMode |= CONSOLE_INPUT_MODE.ENABLE_VIRTUAL_TERMINAL_INPUT; inMode &= ~(CONSOLE_INPUT_MODE.ENABLE_PROCESSED_INPUT | CONSOLE_INPUT_MODE.ENABLE_LINE_INPUT | CONSOLE_INPUT_MODE.ENABLE_ECHO_INPUT); Debug.Assert(SetConsoleMode(stdin, inMode)); Debug.Assert(GetConsoleMode(stdout, out CONSOLE_OUTPUT_MODE outMode)); outMode |= CONSOLE_OUTPUT_MODE.ENABLE_VIRTUAL_TERMINAL_PROCESSING | CONSOLE_OUTPUT_MODE.DISABLE_NEWLINE_AUTO_RETURN; outMode &= ~(CONSOLE_OUTPUT_MODE.ENABLE_PROCESSED_OUTPUT | CONSOLE_OUTPUT_MODE.ENABLE_WRAP_AT_EOL_OUTPUT); Debug.Assert(SetConsoleMode(stdout, outMode)); byte b = 0; Debug.Assert(ReadFile(stdin, (IntPtr)(&b), 1, out var r, IntPtr.Zero)); Console.Write("Read {0} bytes; value = 0x{1:x2}", r, b); } } } ``` (Uses the Vanara.PInvoke.Kernel32 NuGet package for convenience.) Run the program in any console host / shell (CMD, PowerShell, Windows Terminal, etc). # Expected behavior Output: `Read 1 bytes; value = 0x1a` (behavior on Linux) # Actual behavior Output: `Read 0 bytes; value = 0x00` --- I understand that Ctrl+Z is (roughly) Windows's equivalent to Ctrl+D to send EOF. However, I would expect it not to send EOF when the console has been put in 'raw' mode as described [here](https://docs.microsoft.com/en-us/windows/console/high-level-console-modes). This is the behavior on Unix systems for Ctrl+D when you unset `ICANON` (see [this](https://linux.die.net/man/3/tcsetattr) manual page). Perhaps there's an additional step that I'm missing?
claunia commented 2026-01-31 00:50:33 +00:00
Author
Owner
Copy Link

@eryksun commented on GitHub (Mar 17, 2020):

With ReadFile, Ctrl+Z at index 0 is processed by setting the number of bytes read to 0, which is normally interpreted as EOF. (There is no EOF character that gets 'sent'.) ReadConsole, on the other hand, never processes Ctrl+Z, so you could simply switch to calling ReadConsoleW (preferred) or ReadConsoleA (no UTF-8 support, so legacy codepages only).

That said, this issue is a bonafied bug. If ENABLE_PROCESSED_INPUT is not set, then ReadFile should not process Ctrl+Z at index 0. At least that's how it used to work in the original console implementation. This bug was probably introduced several years ago in Windows 8, but I don't currently have a Windows 8 system to confirm this.

Currently, the server apparently receives a ReadFile request as a CONSOLE_IO_RAW_READ, which unconditionally enables ProcessControlZ. In ServerReadConsole, if ProcessControlZ is set and the buffer starts wih Ctrl+Z, it sets the number of bytes read to 0, regardless of whether ENABLE_PROCESSED_INPUT is set. The latter needs to be fixed.

Previously, ReadFile on a console handle was implemented by routing the call to ReadConsoleA, with additional post-processing on the client side to handle Ctrl+Z at index 0, but only if processed-input was enabled.

When the ConDrv driver was added in Windows 8, the designer decided to no longer flag console handles for special routing, so now ReadFile on a console handle calls NtReadFile instead of routing to ReadConsoleA. This redesign broke existing client-side code that supported Ctrl+Z and Ctrl+C (reported in another issue). I guess it slipped under the radar because there were no unit tests?!?

@eryksun commented on GitHub (Mar 17, 2020): With `ReadFile`, Ctrl+Z at index 0 is processed by setting the number of bytes read to 0, which is normally interpreted as EOF. (There is no EOF character that gets 'sent'.) `ReadConsole`, on the other hand, never processes Ctrl+Z, so you could simply switch to calling `ReadConsoleW` (preferred) or `ReadConsoleA` (no UTF-8 support, so legacy codepages only). That said, this issue is a bonafied bug. If `ENABLE_PROCESSED_INPUT` is not set, then `ReadFile` should not process Ctrl+Z at index 0. At least that's how it used to work in the original console implementation. This bug was probably introduced several years ago in Windows 8, but I don't currently have a Windows 8 system to confirm this. Currently, the server apparently receives a `ReadFile` request as a [`CONSOLE_IO_RAW_READ`](https://github.com/microsoft/terminal/blob/9b92986b49bed8cc41fde4d6ef080921c41e6d9e/src/server/IoSorter.cpp#L69), which unconditionally enables `ProcessControlZ`. In [`ServerReadConsole`](https://github.com/microsoft/terminal/blob/master/src/server/ApiDispatchers.cpp#L252), if `ProcessControlZ` is set and the buffer starts wih Ctrl+Z, it sets the number of bytes read to 0, regardless of whether `ENABLE_PROCESSED_INPUT` is set. The latter needs to be fixed. Previously, `ReadFile` on a console handle was implemented by routing the call to `ReadConsoleA`, with additional post-processing on the client side to handle Ctrl+Z at index 0, but only if processed-input was enabled. When the ConDrv driver was added in Windows 8, the designer decided to no longer flag console handles for special routing, so now `ReadFile` on a console handle calls `NtReadFile` instead of routing to `ReadConsoleA`. This redesign broke existing client-side code that supported Ctrl+Z and Ctrl+C (reported in another issue). I guess it slipped under the radar because there were no unit tests?!?
claunia commented 2026-01-31 00:50:34 +00:00
Author
Owner
Copy Link

@alexrp commented on GitHub (Mar 17, 2020):

Good to hear that it's an actual bug and I'm not going crazy. 😄

Indeed, I only care about this in the !ENABLE_PROCESSED_INPUT case.

Regarding ReadFile vs ReadConsole, my concern is that the latter won't play well with binary data. The library I'm writing exposes the standard I/O streams as binary streams (since this is the way things are on Unix) and as far as I'm aware, I can't just use ReadConsole/WriteConsole with arbitrary binary data.

@alexrp commented on GitHub (Mar 17, 2020): Good to hear that it's an actual bug and I'm not going crazy. 😄 Indeed, I only care about this in the `!ENABLE_PROCESSED_INPUT` case. Regarding `ReadFile` vs `ReadConsole`, my concern is that the latter won't play well with binary data. The library I'm writing exposes the standard I/O streams as binary streams (since this is the way things are on Unix) and as far as I'm aware, I can't just use `ReadConsole`/`WriteConsole` with arbitrary binary data.
claunia commented 2026-01-31 00:50:34 +00:00
Author
Owner
Copy Link

@eryksun commented on GitHub (Mar 17, 2020):

Regarding ReadFile vs ReadConsole, my concern is that the latter won't play well with binary data. The library I'm writing exposes the standard I/O streams as binary streams (since this is the way things are on Unix) and as far as I'm aware, I can't just use ReadConsole/WriteConsole with arbitrary binary data.

ReadFile, when reading from a console, and ReadConsoleA both encode the native wide-character input buffer as 8-bit characters, according to the console's active input codepage. (The input codepage can be set to UTF-8, i.e. 65001, but it doesn't work for non-ASCII characters, so it's pointless.) ReadConsoleW returns the native wide-character input buffer as UTF-16 encoded text, which you can subsequently translate to UTF-8.

If stdin is redirected to a pipe or disk file, GetConsoleMode will fail, which signals the program to switch to ReadFile instead of ReadConsole.

@eryksun commented on GitHub (Mar 17, 2020): > Regarding `ReadFile` vs `ReadConsole`, my concern is that the latter won't play well with binary data. The library I'm writing exposes the standard I/O streams as binary streams (since this is the way things are on Unix) and as far as I'm aware, I can't just use `ReadConsole`/`WriteConsole` with arbitrary binary data. `ReadFile`, when reading from a console, and `ReadConsoleA` both encode the native wide-character input buffer as 8-bit characters, according to the console's active input codepage. (The input codepage can be set to UTF-8, i.e. 65001, but it doesn't work for non-ASCII characters, so it's pointless.) `ReadConsoleW` returns the native wide-character input buffer as UTF-16 encoded text, which you can subsequently translate to UTF-8. If stdin is redirected to a pipe or disk file, `GetConsoleMode` will fail, which signals the program to switch to `ReadFile` instead of `ReadConsole`.
claunia commented 2026-01-31 00:50:34 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Mar 19, 2020):

@eryksun thanks for doing the digging here. I'm tagging this one up and clearing triage.

now ReadFile on a console handle calls NtReadFile instead of routing to ReadConsoleA.

But given that we do receive it in the server as CONSOLE_IO_RAW_READ, and since we're routing it to the backing implementation of ReadConsole*, we gained the agility to fix this on the serverside without having to rev kernelbase/kernel32. 😄

@DHowett-MSFT commented on GitHub (Mar 19, 2020): @eryksun thanks for doing the digging here. I'm tagging this one up and clearing triage. > now `ReadFile` on a console handle calls `NtReadFile` instead of routing to `ReadConsoleA`. But given that we _do_ receive it in the server as `CONSOLE_IO_RAW_READ`, and since we're routing it to the backing implementation of `ReadConsole*`, we gained the agility to fix this on the serverside without having to rev kernelbase/kernel32. :smile:
claunia commented 2026-01-31 00:50:34 +00:00
Author
Owner
Copy Link

@eryksun commented on GitHub (Mar 21, 2020):

But given that we do receive it in the server as CONSOLE_IO_RAW_READ, and since we're routing it to the backing implementation of ReadConsole*, we gained the agility to fix this on the serverside without having to rev kernelbase/kernel32

Yes, fixing Ctrl+Z handling with ReadFile should be possible in the server.

OTOH, on the related issue of processed Ctrl+C handling in a console ReadFile, restoring the old behavior would require modifying kernelbase and the reintroduction of tagged console handles. Ideally, instead of restoring the old behavior, the console server would be changed to return the correct status code STATUS_CANCELLED, which automatically maps to ERROR_OPERATION_ABORTED, instead of the peculiar use of the success code STATUS_ALERTED (intended as a wait status for an alerted thread). However this change would cause a console ReadFile or ReadConsole call to fail when interrupted by Ctrl+C (not in itself bad, IMO, as I'd naturally expect it to work that way), which goes against the documentation and expectations of existing code.

@eryksun commented on GitHub (Mar 21, 2020): > But given that we _do_ receive it in the server as `CONSOLE_IO_RAW_READ`, and since we're routing it to the backing implementation of `ReadConsole*`, we gained the agility to fix this on the serverside without having to rev kernelbase/kernel32 Yes, fixing Ctrl+Z handling with `ReadFile` should be possible in the server. --- OTOH, on the related issue of processed Ctrl+C handling in a console `ReadFile`, restoring the old behavior would require modifying kernelbase and the reintroduction of tagged console handles. Ideally, instead of restoring the old behavior, the console server would be changed to return the correct status code `STATUS_CANCELLED`, which automatically maps to `ERROR_OPERATION_ABORTED`, instead of the peculiar use of the success code `STATUS_ALERTED` (intended as a wait status for an alerted thread). However this change would cause a console `ReadFile` or `ReadConsole` call to fail when interrupted by Ctrl+C (not in itself bad, IMO, as I'd naturally expect it to work that way), which goes against the documentation and expectations of existing code.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Server Help Wanted Impact-Compatibility Issue-Bug Priority-2 Product-Conhost
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#6921
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?