starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

.NET Console.WriteLine throws 'The handle is invalid.' on large output when using ENABLE_VIRTUAL_TERMINAL_PROCESSING #5283

New Issue
Closed
opened 2026-01-31 00:09:35 +00:00 by claunia · 13 comments
claunia commented 2026-01-31 00:09:35 +00:00
Owner
Copy Link

Originally created by @gerardog on GitHub (Nov 28, 2019).

Environment

Windows build number: 10.0.18362.356
Windows Terminal version (if applicable): Using windows conHost or Terminal 0.7.3291.0
Visual Studio 2019 Professional 16.3.7
Net Framework 4.6, 4.6.1 or 4.7.2

Steps to reproduce

Create a simple C# Process that set console mode ENABLE_VIRTUAL_TERMINAL_PROCESSING, and then writes unlimited output to StdOut.
Eventually StdOut gets broken and throws System.IO.IOException: The handle is invalid.

C# Repro code:

using Microsoft.Win32.SafeHandles;
using System;
using System.Runtime.InteropServices;

class Program
{
    static void Main(string[] args)
    {
        if (SetConsoleModeVT())
        {
            for (int i = 0; ; i++)
            {
                try
                {
                    Console.WriteLine(i);
                }
                catch (Exception ex)
                {
                    Console.Error.WriteLine(ex.ToString());
                    Console.ReadLine();
                }
            }
        }
    }

    private static bool SetConsoleModeVT()
    {
        var hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
        if (!GetConsoleMode(hStdOut, out uint outConsoleMode))
        {
            return false;
        }

        outConsoleMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
        if (!SetConsoleMode(hStdOut, outConsoleMode))
        {
            return false;
        }

        return true;
    }

    internal const int STD_OUTPUT_HANDLE = -11;
    internal const uint ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004;

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern SafeFileHandle GetStdHandle(int nStdHandle);

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern bool SetConsoleMode(SafeFileHandle hConsoleHandle, uint mode);

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern bool GetConsoleMode(SafeFileHandle handle, out uint mode);
}

Expected behavior

The app should not throw IOException.

Actual behavior

This is the output:

1
2
(...)
163543
163544
163545
System.IO.IOException: The handle is invalid.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.__ConsoleStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at System.IO.TextWriter.Write(UInt64 value)
   at System.IO.TextWriter.WriteLine(UInt64 value)
   at System.IO.TextWriter.SyncTextWriter.WriteLine(UInt64 value)
   at System.Console.WriteLine(UInt64 value)
   at ConsoleApp5.Program.Main(String[] args) in C:\Users\Me\source\repos\ConsoleApp5\Program.cs:line 17

The exception is only thrown with ENABLE_VIRTUAL_TERMINAL_PROCESSING console mode.
Either launching the app from a Windows CMD/ConHost, or the new Windows Terminal/PowerShell.

Thanks!

Originally created by @gerardog on GitHub (Nov 28, 2019). <!-- 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨 I ACKNOWLEDGE THE FOLLOWING BEFORE PROCEEDING: 1. If I delete this entire template and go my own path, the core team may close my issue without further explanation or engagement. 2. If I list multiple bugs/concerns in this one issue, the core team may close my issue without further explanation or engagement. 3. If I write an issue that has many duplicates, the core team may close my issue without further explanation or engagement (and without necessarily spending time to find the exact duplicate ID number). 4. If I leave the title incomplete when filing the issue, the core team may close my issue without further explanation or engagement. 5. If I file something completely blank in the body, the core team may close my issue without further explanation or engagement. All good? Then proceed! --> <!-- This bug tracker is monitored by Windows Terminal development team and other technical folks. **Important: When reporting BSODs or security issues, DO NOT attach memory dumps, logs, or traces to Github issues**. Instead, send dumps/traces to secure@microsoft.com, referencing this GitHub issue. If this is an application crash, please also provide a Feedback Hub submission link so we can find your diagnostic data on the backend. Use the category "Apps > Windows Terminal (Preview)" and choose "Share My Feedback" after submission to get the link. Please use this form and describe your issue, concisely but precisely, with as much detail as possible. --> # Environment ```none Windows build number: 10.0.18362.356 Windows Terminal version (if applicable): Using windows conHost or Terminal 0.7.3291.0 Visual Studio 2019 Professional 16.3.7 Net Framework 4.6, 4.6.1 or 4.7.2 ``` # Steps to reproduce Create a simple C# Process that set console mode ENABLE_VIRTUAL_TERMINAL_PROCESSING, and then writes unlimited output to StdOut. Eventually StdOut gets broken and throws System.IO.IOException: The handle is invalid. C# Repro code: ``` csharp using Microsoft.Win32.SafeHandles; using System; using System.Runtime.InteropServices; class Program { static void Main(string[] args) { if (SetConsoleModeVT()) { for (int i = 0; ; i++) { try { Console.WriteLine(i); } catch (Exception ex) { Console.Error.WriteLine(ex.ToString()); Console.ReadLine(); } } } } private static bool SetConsoleModeVT() { var hStdOut = GetStdHandle(STD_OUTPUT_HANDLE); if (!GetConsoleMode(hStdOut, out uint outConsoleMode)) { return false; } outConsoleMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING; if (!SetConsoleMode(hStdOut, outConsoleMode)) { return false; } return true; } internal const int STD_OUTPUT_HANDLE = -11; internal const uint ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004; [DllImport("kernel32.dll", SetLastError = true)] internal static extern SafeFileHandle GetStdHandle(int nStdHandle); [DllImport("kernel32.dll", SetLastError = true)] internal static extern bool SetConsoleMode(SafeFileHandle hConsoleHandle, uint mode); [DllImport("kernel32.dll", SetLastError = true)] internal static extern bool GetConsoleMode(SafeFileHandle handle, out uint mode); } ``` # Expected behavior The app should not throw IOException. # Actual behavior This is the output: ``` 1 2 (...) 163543 163544 163545 System.IO.IOException: The handle is invalid. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.__ConsoleStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.IO.StreamWriter.Write(String value) at System.IO.TextWriter.Write(UInt64 value) at System.IO.TextWriter.WriteLine(UInt64 value) at System.IO.TextWriter.SyncTextWriter.WriteLine(UInt64 value) at System.Console.WriteLine(UInt64 value) at ConsoleApp5.Program.Main(String[] args) in C:\Users\Me\source\repos\ConsoleApp5\Program.cs:line 17 ``` The exception is only thrown with ENABLE_VIRTUAL_TERMINAL_PROCESSING console mode. Either launching the app from a Windows CMD/ConHost, or the new Windows Terminal/PowerShell. Thanks!
claunia commented 2026-01-31 00:09:37 +00:00
Author
Owner
Copy Link

@gerardog commented on GitHub (Dec 10, 2019):

If I add a Thread.Sleep(60 seconds) inside the for loop, it also throws after just 5 minutes. So this does not seems to be related to the amount of output.
I was also able to reproduce it using Windows Preview version 2004 (build 19033.1) from the fast ring, and/or .Net Core 3.0.

@gerardog commented on GitHub (Dec 10, 2019): If I add a `Thread.Sleep(60 seconds)` inside the for loop, it also throws after just 5 minutes. So this does not seems to be related to the amount of output. I was also able to reproduce it using Windows Preview version 2004 (build 19033.1) from the fast ring, and/or .Net Core 3.0.
claunia commented 2026-01-31 00:09:37 +00:00
Author
Owner
Copy Link

@amd4dnow commented on GitHub (Dec 17, 2019):

i've the same issue, but have no idea how to solve it.

@amd4dnow commented on GitHub (Dec 17, 2019): i've the same issue, but have no idea how to solve it.
claunia commented 2026-01-31 00:09:37 +00:00
Author
Owner
Copy Link

@gerardog commented on GitHub (Dec 17, 2019):

@amd4dnow I have no idea either, but for what it's worth:

(A non-recommended way to fix this issue that will cause long-term application stability issues follows.)

What I found out is that when using Windows Terminal, Cmder or ConEmu, I can send VT sequences and they get processes even without setting ENABLE_VIRTUAL_TERMINAL_PROCESSING flag. So my current code detects those terminals (so far, reading WT_SESSION or ConEmuANSI environment variables) and only then send VT sequences.

@gerardog commented on GitHub (Dec 17, 2019): @amd4dnow I have no idea either, but for what it's worth: <details> <summary>(A non-recommended way to fix this issue that will cause long-term application stability issues follows.)</summary> What I found out is that when using Windows Terminal, Cmder or ConEmu, I can send VT sequences and they get processes even without setting ENABLE_VIRTUAL_TERMINAL_PROCESSING flag. So my current code detects those terminals (so far, reading WT_SESSION or ConEmuANSI environment variables) and only then send VT sequences. </details>
claunia commented 2026-01-31 00:09:37 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Dec 17, 2019):

@gerardog please, whatever you do, don't do that. You'll run right into https://github.com/microsoft/terminal/issues/1965 which will cause terribly unusual behavior when you reach the edge of the screen.

@DHowett-MSFT commented on GitHub (Dec 17, 2019): @gerardog _please, whatever you do, don't do that._ You'll run right into https://github.com/microsoft/terminal/issues/1965 which will cause terribly unusual behavior when you reach the edge of the screen.
claunia commented 2026-01-31 00:09:37 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Dec 17, 2019):

ConPTY can't pass raw escape sequences in a way that the terminal can differentiate from escape sequences that are in the text buffer, and if they're in the text buffer they're subject to wrapping and splitting.

@DHowett-MSFT commented on GitHub (Dec 17, 2019): ConPTY can't pass raw escape sequences in a way that the terminal can differentiate from escape sequences that are in the text buffer, and if they're in the text buffer they're subject to wrapping and splitting.
claunia commented 2026-01-31 00:09:37 +00:00
Author
Owner
Copy Link

@gerardog commented on GitHub (Dec 17, 2019):

Thanks for the information @DHowett-MSFT . Do you have any insight about how to workaround the The handle is invalid exception in the meantime? Is this bug rooted on ConHost? or Kernel32.dll? Does plain C apps work fine with this flag?

@gerardog commented on GitHub (Dec 17, 2019): Thanks for the information @DHowett-MSFT . Do you have any insight about how to workaround the `The handle is invalid` exception in the meantime? Is this bug rooted on ConHost? or [Kernel32.dll](https://github.com/dotnet/corefx/blob/master/src/System.Console/src/System/ConsolePal.Windows.cs#L1228)? Does plain C apps work fine with this flag?
claunia commented 2026-01-31 00:09:38 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Dec 17, 2019):

We'll have to have a look-- sorry, things have been slow on account of the impending holidays 😄

@DHowett-MSFT commented on GitHub (Dec 17, 2019): We'll have to have a look-- sorry, things have been slow on account of the impending holidays :smile:
claunia commented 2026-01-31 00:09:38 +00:00
Author
Owner
Copy Link

@nszeitli commented on GitHub (Jan 26, 2020):

I am getting the same error when writing to console in an ASP.NET core 3.0 app after a long running process. It appears to be related to high memory usage but not sure.

@nszeitli commented on GitHub (Jan 26, 2020): I am getting the same error when writing to console in an ASP.NET core 3.0 app after a long running process. It appears to be related to high memory usage but not sure.
claunia commented 2026-01-31 00:09:38 +00:00
Author
Owner
Copy Link

@ogamespec commented on GitHub (Feb 20, 2020):

Happens to me when I try to interact with console in native DLL:

DLL code init:

	AllocConsole();
	if (freopen("CONOUT$", "w", stdout) == 0)
		return;

DLL code shutdown:

	fclose(stdout);
	FreeConsole();
@ogamespec commented on GitHub (Feb 20, 2020): Happens to me when I try to interact with console in native DLL: DLL code init: ``` AllocConsole(); if (freopen("CONOUT$", "w", stdout) == 0) return; ``` DLL code shutdown: ``` fclose(stdout); FreeConsole(); ```
claunia commented 2026-01-31 00:09:38 +00:00
Author
Owner
Copy Link

@ogamespec commented on GitHub (Feb 20, 2020):

For your C # application to work properly, you need to make sure that no one else touches the console.

There "somewhere else" you can use GetConsoleWindow function.

@ogamespec commented on GitHub (Feb 20, 2020): For your C # application to work properly, you need to make sure that no one else touches the console. There "somewhere else" you can use GetConsoleWindow function.
claunia commented 2026-01-31 00:09:38 +00:00
Author
Owner
Copy Link

@gerardog commented on GitHub (May 6, 2020):

@DHowett Can a .Net app render its content sending VT sequences? maybe with ENABLE_VIRTUAL_TERMINAL_PROCESSING? or with a passthrough mode #1173...
Is ENABLE_VIRTUAL_TERMINAL_PROCESSING completely broken or am I doing something wrong?

@gerardog commented on GitHub (May 6, 2020): @DHowett Can a .Net app render its content sending VT sequences? maybe with ENABLE_VIRTUAL_TERMINAL_PROCESSING? or with a passthrough mode #1173... Is ENABLE_VIRTUAL_TERMINAL_PROCESSING completely broken or am I doing something wrong?
claunia commented 2026-01-31 00:09:38 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (May 7, 2020):

@gerardog this should definitely work.

Setting the console output handle (retrieved from GetStdHandle(STD_OUTPUT_HANDLE) or CreateFile("CONOUT$") (please only use this if you absolutely require a console handle to function. It isn't good to do this as a default for writing a console application that might be used in non-console scenarios) into ENABLE_VIRTUAL_TERMINAL_PROCESSING mode should be sufficient for sending all types of VT.

You may need to fiddle with the ..NEWLINE_AUTO_RETURN mode and related modes, but at its core this should work fine.

I just got your reproducer under a debugger, and the strangest thing is happening.

  1. Your Main() is still running, and still writing to the console handle.
  2. The standard output handle is being closed by a ".NET Finalizer" thread.

It looks like the SafeHandle that you used to set the output mode is being finalized when its scope ends on line 40. It's closing the only output handle when it does so.

If you replace the use of SafeHandle in GetStdHandle with IntPtr, it works perfectly fine forever as the handle isn't prematurely slain in its sleep.

Chalking this one up to user error :)

@DHowett-MSFT commented on GitHub (May 7, 2020): @gerardog this should definitely work. Setting the console output handle (retrieved from `GetStdHandle(STD_OUTPUT_HANDLE)` or `CreateFile("CONOUT$")` (please only use this if you absolutely require a console handle to function. It isn't good to do this as a default for writing a console application that might be used in non-console scenarios) into `ENABLE_VIRTUAL_TERMINAL_PROCESSING` mode should be sufficient for sending all types of VT. You may need to fiddle with the `..NEWLINE_AUTO_RETURN` mode and related modes, but at its core this should work fine. I just got your reproducer under a debugger, and the strangest thing is happening. 1. Your Main() is still running, and still writing to the console handle. 2. The standard output handle is being closed by a ".NET Finalizer" thread. It looks like the SafeHandle that you used to set the output mode is being finalized when its scope ends on line 40. It's closing the only output handle when it does so. If you replace the use of `SafeHandle` in `GetStdHandle` with `IntPtr`, it works perfectly fine _forever_ as the handle isn't prematurely slain in its sleep. Chalking this one up to user error :)
claunia commented 2026-01-31 00:09:39 +00:00
Author
Owner
Copy Link

@gerardog commented on GitHub (May 22, 2020):

Many thanks @DHowett-MSFT. You really nailed this one!

@gerardog commented on GitHub (May 22, 2020): Many thanks @DHowett-MSFT. You really nailed this one!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Output Issue-Bug Priority-2 Product-Conhost Resolution-External Resolution-Won't-Fix
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#5283
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?