starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

App output different in Terminal than conhost with null characters #11917

New Issue
Open
opened 2026-01-31 03:01:01 +00:00 by claunia · 15 comments
claunia commented 2026-01-31 03:01:01 +00:00
Owner
Copy Link

Originally created by @asklar on GitHub (Dec 30, 2020).

Environment

Windows build number: 21280
Windows Terminal version (if applicable): 1.5.3242.0 (Preview)

Any other software?

Steps to reproduce

Run the project in https://github.com/asklar/textlogo
The program initializes a text buffer, draws something in it, then prints it to stdout.
It looks like Terminal treats Console.WriteLine('\0') as a no-op when it should be interpreting it as "write a space".
If I initialize my buffer to spaces, then the bug goes away, but this should not be necessary and it's a breaking behavior between conhost and terminal.

Expected behavior

In conhost, this prints:
image

Actual behavior

In Terminal, this prints:
image

Originally created by @asklar on GitHub (Dec 30, 2020). <!-- 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨 I ACKNOWLEDGE THE FOLLOWING BEFORE PROCEEDING: 1. If I delete this entire template and go my own path, the core team may close my issue without further explanation or engagement. 2. If I list multiple bugs/concerns in this one issue, the core team may close my issue without further explanation or engagement. 3. If I write an issue that has many duplicates, the core team may close my issue without further explanation or engagement (and without necessarily spending time to find the exact duplicate ID number). 4. If I leave the title incomplete when filing the issue, the core team may close my issue without further explanation or engagement. 5. If I file something completely blank in the body, the core team may close my issue without further explanation or engagement. All good? Then proceed! --> <!-- This bug tracker is monitored by Windows Terminal development team and other technical folks. **Important: When reporting BSODs or security issues, DO NOT attach memory dumps, logs, or traces to Github issues**. Instead, send dumps/traces to secure@microsoft.com, referencing this GitHub issue. If this is an application crash, please also provide a Feedback Hub submission link so we can find your diagnostic data on the backend. Use the category "Apps > Windows Terminal (Preview)" and choose "Share My Feedback" after submission to get the link. Please use this form and describe your issue, concisely but precisely, with as much detail as possible. --> # Environment ```none Windows build number: 21280 Windows Terminal version (if applicable): 1.5.3242.0 (Preview) Any other software? ``` # Steps to reproduce Run the project in https://github.com/asklar/textlogo The program initializes a text buffer, draws something in it, then prints it to stdout. It looks like Terminal treats `Console.WriteLine('\0')` as a no-op when it should be interpreting it as "write a space". If I initialize my buffer to spaces, then the bug goes away, but this should not be necessary and it's a breaking behavior between conhost and terminal. # Expected behavior In conhost, this prints: ![image](https://user-images.githubusercontent.com/22989529/103350829-1a129380-4a56-11eb-9bc7-a8a97f50e894.png) <!-- A description of what you're expecting, possibly containing screenshots or reference material. --> # Actual behavior In Terminal, this prints: ![image](https://user-images.githubusercontent.com/22989529/103350854-30b8ea80-4a56-11eb-9f13-0d8ad75f0a49.png) <!-- What's actually happening? -->
claunia added the Issue-QuestionArea-OutputArea-VTProduct-Conpty labels 2026-01-31 03:01:02 +00:00
claunia commented 2026-01-31 03:01:03 +00:00
Author
Owner
Copy Link

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020):

AFAIK, output from a computer to a terminal has historically used null characters as padding, to give the terminal time to process a command that the computer sent earlier. The terminal is not expected to display the null characters as spaces.

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020): AFAIK, output from a computer to a terminal has historically used null characters as padding, to give the terminal time to process a command that the computer sent earlier. The terminal is not expected to display the null characters as spaces.
claunia commented 2026-01-31 03:01:03 +00:00
Author
Owner
Copy Link

@asklar commented on GitHub (Dec 30, 2020):

The ASCII standard allows NUL to affect layout: https://web.archive.org/web/20140512221203/http://kikaku.itscj.ipsj.or.jp/ISO-IR/001.pdf
image

Wikipedia says "some terminals, however, incorrectly display it as space". Regardless of whether it is correct for conhost to do so, that's where we are and have been for a long time. If Windows Terminal ever wants to be able to replace conhost, it must behave in a backwards compatible way and not break a behavior that's been around for 30-40+ years.

@asklar commented on GitHub (Dec 30, 2020): The ASCII standard allows NUL to affect layout: https://web.archive.org/web/20140512221203/http://kikaku.itscj.ipsj.or.jp/ISO-IR/001.pdf ![image](https://user-images.githubusercontent.com/22989529/103369649-89e94400-4a7f-11eb-8d00-50c5b91e2df0.png) Wikipedia says "some terminals, however, incorrectly display it as space". Regardless of whether it is correct for conhost to do so, that's where we are and have been for a long time. If _Windows_ Terminal ever wants to be able to replace conhost, it must behave in a backwards compatible way and not break a behavior that's been around for 30-40+ years.
claunia commented 2026-01-31 03:01:03 +00:00
Author
Owner
Copy Link

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020):

If you clear the ENABLE_VIRTUAL_TERMINAL_PROCESSING output mode first, then the display is what you wanted.

        static void DisableOutputVirtualTerminalProcessing()
        {
            IntPtr handle = GetStdHandle(STD_OUTPUT_HANDLE);
            if (GetConsoleMode(handle, out uint mode))
            {
                Console.WriteLine($"Original output modes: {mode:X}");
                mode &= ~ENABLE_VIRTUAL_TERMINAL_PROCESSING;
                Console.WriteLine($"New output modes: {mode:X}");
                if (SetConsoleMode(handle, mode))
                {
                    Console.WriteLine("Changed the output modes OK.");
                }
            }
        }

        [System.Runtime.InteropServices.DllImport("kernel32")]
        static extern IntPtr GetStdHandle(int nStdHandle);

        [System.Runtime.InteropServices.DllImport("kernel32")]
        [return: System.Runtime.InteropServices.MarshalAs(System.Runtime.InteropServices.UnmanagedType.Bool)]
        static extern bool GetConsoleMode(IntPtr hConsoleHandle, out uint lpMode);

        [System.Runtime.InteropServices.DllImport("kernel32")]
        [return: System.Runtime.InteropServices.MarshalAs(System.Runtime.InteropServices.UnmanagedType.Bool)]
        static extern bool SetConsoleMode(IntPtr hConsoleHandle, uint dwMode);

        const int STD_OUTPUT_HANDLE = -11;
        const uint ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4;
@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020): If you clear the ENABLE_VIRTUAL_TERMINAL_PROCESSING output mode first, then the display is what you wanted. ```C# static void DisableOutputVirtualTerminalProcessing() { IntPtr handle = GetStdHandle(STD_OUTPUT_HANDLE); if (GetConsoleMode(handle, out uint mode)) { Console.WriteLine($"Original output modes: {mode:X}"); mode &= ~ENABLE_VIRTUAL_TERMINAL_PROCESSING; Console.WriteLine($"New output modes: {mode:X}"); if (SetConsoleMode(handle, mode)) { Console.WriteLine("Changed the output modes OK."); } } } [System.Runtime.InteropServices.DllImport("kernel32")] static extern IntPtr GetStdHandle(int nStdHandle); [System.Runtime.InteropServices.DllImport("kernel32")] [return: System.Runtime.InteropServices.MarshalAs(System.Runtime.InteropServices.UnmanagedType.Bool)] static extern bool GetConsoleMode(IntPtr hConsoleHandle, out uint lpMode); [System.Runtime.InteropServices.DllImport("kernel32")] [return: System.Runtime.InteropServices.MarshalAs(System.Runtime.InteropServices.UnmanagedType.Bool)] static extern bool SetConsoleMode(IntPtr hConsoleHandle, uint dwMode); const int STD_OUTPUT_HANDLE = -11; const uint ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4; ```
claunia commented 2026-01-31 03:01:04 +00:00
Author
Owner
Copy Link

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020):

If you set the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode first, then the output loses the spaces even in conhost.

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020): If you set the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode first, then the output loses the spaces even in conhost.
claunia commented 2026-01-31 03:01:04 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Dec 30, 2020):

I believe this is a duplicate of issue #4363. See also #6265.

@j4james commented on GitHub (Dec 30, 2020): I believe this is a duplicate of issue #4363. See also #6265.
claunia commented 2026-01-31 03:01:06 +00:00
Author
Owner
Copy Link

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020):

I believe this is a duplicate of issue #4363.

I don't think so. The program here uses Console.Write, which goes to WriteFile or maybe WriteConsole, not to WriteConsoleOutputCharacter or other functions that would take coordinates as parameters. I believe that, when ENABLE_PROCESSED_OUTPUT is set but ENABLE_VIRTUAL_TERMINAL_PROCESSING is not, conhost converts the null character to a space in the screen buffer so that https://github.com/microsoft/terminal/issues/4363 is avoided: https://github.com/microsoft/terminal/blob/0b0161d537006ca397e680ac07a071a0cd70e02d/src/host/_stream.cpp#L487-L490

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020): > I believe this is a duplicate of issue #4363. I don't think so. The program here uses Console.Write, which goes to WriteFile or maybe WriteConsole, not to WriteConsoleOutputCharacter or other functions that would take coordinates as parameters. I believe that, when ENABLE_PROCESSED_OUTPUT is set but ENABLE_VIRTUAL_TERMINAL_PROCESSING is not, conhost converts the null character to a space in the screen buffer so that <https://github.com/microsoft/terminal/issues/4363> is avoided: <https://github.com/microsoft/terminal/blob/0b0161d537006ca397e680ac07a071a0cd70e02d/src/host/_stream.cpp#L487-L490>
claunia commented 2026-01-31 03:01:07 +00:00
Author
Owner
Copy Link

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020):

Anyway, the behavior of null characters without ENABLE_VIRTUAL_TERMINAL_PROCESSING is compatible with the old Windows console, and the behavior with ENABLE_VIRTUAL_TERMINAL_PROCESSING is compatible with serial terminals, so neither behavior should be changed. Rather, one could think about how to give each application the behavior it expects.

@KalleOlaviNiemitalo commented on GitHub (Dec 30, 2020): Anyway, the behavior of null characters without ENABLE_VIRTUAL_TERMINAL_PROCESSING is compatible with the old Windows console, and the behavior with ENABLE_VIRTUAL_TERMINAL_PROCESSING is compatible with serial terminals, so neither behavior should be changed. Rather, one could think about how to give each application the behavior it expects.
claunia commented 2026-01-31 03:01:07 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Dec 30, 2020):

@KalleOlaviNiemitalo Yeah, sorry, you're absolutely right. Disabling the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode does fix the issue.

@j4james commented on GitHub (Dec 30, 2020): @KalleOlaviNiemitalo Yeah, sorry, you're absolutely right. Disabling the `ENABLE_VIRTUAL_TERMINAL_PROCESSING` mode does fix the issue.
claunia commented 2026-01-31 03:01:08 +00:00
Author
Owner
Copy Link

@KalleOlaviNiemitalo commented on GitHub (Dec 31, 2020):

one could think about how to give each application the behavior it expects

https://github.com/microsoft/terminal/issues/4954 might help with that.

@KalleOlaviNiemitalo commented on GitHub (Dec 31, 2020): > one could think about how to give each application the behavior it expects <https://github.com/microsoft/terminal/issues/4954> might help with that.
claunia commented 2026-01-31 03:01:08 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jan 8, 2021):

Alright so I'm a little confused on what the resolution was here. Is this a dupe of #4363, because conpty is re-rendering the NULs as NULs, and the Terminal is ignoring them? Or is this "You need to disable ENABLE_VIRTUAL_TERMINAL_PROCESSING manually?"

If it's the second, then maybe the difference is because in conhost, you're running the script from cmd, which is going to disable VT output before launching your process. I'm out on a limb here, but I'm guessing that pwsh doesn't, which is why the output seems different in the Terminal.

@zadjii-msft commented on GitHub (Jan 8, 2021): Alright so I'm a little confused on what the resolution was here. Is this a dupe of #4363, because conpty is re-rendering the NULs as NULs, and the Terminal is ignoring them? Or is this "You need to disable `ENABLE_VIRTUAL_TERMINAL_PROCESSING` manually?" If it's the second, then maybe the difference is because in conhost, you're running the script from `cmd`, which is going to disable VT output before launching your process. I'm out on a limb here, but I'm guessing that `pwsh` _doesn't_, which is why the output seems different in the Terminal.
claunia commented 2026-01-31 03:01:09 +00:00
Author
Owner
Copy Link

@asklar commented on GitHub (Jan 8, 2021):

@zadjii-msft the same problem happens when running the app in cmd.exe inside Terminal. I'm not sure whether this is a dupe of 4363 or not

@asklar commented on GitHub (Jan 8, 2021): @zadjii-msft the same problem happens when running the app in cmd.exe inside Terminal. I'm not sure whether this is a dupe of 4363 or not
claunia commented 2026-01-31 03:01:09 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jan 8, 2021):

Sorry for the confusion. It's not a dup of #4368 - that what just me being an idiot. As @KalleOlaviNiemitalo said, when you're not in VT mode (which is typically the default case in the old conhost shell), then most of the low ascii characters are "printable", and NUL shows as a blank. When you are in VT mode (as it the default case in Windows Terminal), then NUL is interpreted as a padding control character which won't be seen.

So if you want your app to work exactly the way it does in the old conhost, then you have to explicitly disable the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode.

I should also clarify that it's not going to make a difference whether you're using cmd or pwsh. Both should start apps with VT mode enabled in Window Terminal, and both should start apps with VT mode disabled when in conhost (at least by default).

@j4james commented on GitHub (Jan 8, 2021): Sorry for the confusion. It's not a dup of #4368 - that what just me being an idiot. As @KalleOlaviNiemitalo said, when you're _not_ in VT mode (which is typically the default case in the old conhost shell), then most of the low ascii characters are "printable", and `NUL` shows as a blank. When you _are_ in VT mode (as it the default case in Windows Terminal), then `NUL` is interpreted as a padding control character which won't be seen. So if you want your app to work exactly the way it does in the old conhost, then you have to explicitly disable the `ENABLE_VIRTUAL_TERMINAL_PROCESSING` mode. I should also clarify that it's not going to make a difference whether you're using `cmd` or `pwsh`. Both should start apps with VT mode _enabled_ in Window Terminal, and both should start apps with VT mode _disabled_ when in conhost (at least by default).
claunia commented 2026-01-31 03:01:09 +00:00
Author
Owner
Copy Link

@asklar commented on GitHub (Jan 8, 2021):

So if you want your app to work exactly the way it does in the old conhost, then you have to explicitly disable the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode.

Which is not going to be possible for many legacy apps that need to continue to work properly without being recompiled. There needs to be a quirk or Windows shim for this.

@asklar commented on GitHub (Jan 8, 2021): > So if you want your app to work exactly the way it does in the old conhost, then you have to explicitly disable the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode. Which is not going to be possible for many legacy apps that need to continue to work properly without being recompiled. There needs to be a quirk or Windows shim for this.
claunia commented 2026-01-31 03:01:10 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jan 8, 2021):

Which is not going to be possible for many legacy apps that need to continue to work properly without being recompiled. There needs to be a quirk or Windows shim for this.

As @KalleOlaviNiemitalo pointed out, #4954 may be able to help with. See #6973 (which was duped to #4954) for a similar requirement.

@j4james commented on GitHub (Jan 8, 2021): > Which is not going to be possible for many legacy apps that need to continue to work properly without being recompiled. There needs to be a quirk or Windows shim for this. As @KalleOlaviNiemitalo pointed out, #4954 may be able to help with. See #6973 (which was duped to #4954) for a similar requirement.
claunia commented 2026-01-31 03:01:10 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Jul 2, 2021):

Marking this one up for the console backlog -- we need an answer to it, but at the same time we have to be able to move the ecosystem forward. \0 is important to VT applications. We may need a way to tell ConPTY that it should not start in VT enabled mode.

(Sorry for the delay!)

@DHowett commented on GitHub (Jul 2, 2021): Marking this one up for the console backlog -- we need an answer to it, but at the same time we have to be able to move the ecosystem forward. `\0` is important to VT applications. We may need a way to tell _ConPTY_ that it should not start in VT enabled mode. (Sorry for the delay!)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Output Area-VT Issue-Question Product-Conpty
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#11917
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?