starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

Bug Report: What is reordering VT escape sequences in my string? #2822

New Issue
Closed
opened 2026-01-30 23:06:16 +00:00 by claunia · 4 comments
claunia commented 2026-01-30 23:06:16 +00:00
Owner
Copy Link

Originally created by @jazzdelightsme on GitHub (Jul 17, 2019).

Originally assigned to: @zadjii-msft on GitHub.

Environment

Windows build number: Microsoft Windows [Version 10.0.18362.239]
Windows Terminal version (if applicable): (built from latest source as of July 17th, 2019, morning)

I am working on implementing some new VT escape sequences (PR here), but I've run into a strange problem. What the sequences do (or are supposed to do) is not important here. The problem is that in certain circumstances, something is reordering VT escape sequences in my content (which is very surprising to me).

The control sequences in question are XTPUSHSGR and XTPOPSGR: [CSI]#p and [CSI]#q, respectively. Note in the string below, how there is an XTPUSHSGR sequence, then a normal SGR sequence to set the color to red, then an XTPOPSGR sequence (and then a normal SGR sequence to reset the color).

When I open OpenConsole.sln in Visual Studio 2019, with the CascadiaPackage set as the default project to debug, and then F5 to start debugging it, it launches a terminal running Windows PowerShell. When I send my string to the console, something is moving the XTPUSHSGR and XTPOPSGR sequences to the very beginning of the string!

Steps to reproduce

  1. Start OpenConsole.sln in Visual Studio 2019. Make sure CascadiaPackage is the default project, and platform is set to x64, and build the solution if necessary.
  2. F5 to start debugging a new instance. You should now have a PowerShell instance in the terminal.
  3. In the PowerShell instance in the terminal, run the following commands:
 $esc = [char] 0x1b
 $s = "Hello. Push:${esc}[#p ${esc}[91m this text is RED Pop:${esc}[#q FG should " +
      "no longer be red (unless it was red before) courtesy reset: ${esc}[0m"
  1. Set a breakpoint at TerminalControl.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString (the two-parameter one).
  2. In the PowerShell instance in the terminal, run the command $s to output the string.
  3. When you hit the breakpoint, observe the value of rgwch.

Expected behavior

The content of the string should be just as it was set in PowerShell.

Actual behavior

The XTPUSHSGR and XTPOPSGR sequences have been moved to the beginning of the string!

My first thought was that PowerShell must be doing something funny, but I attached windbg to the PowerShell and set a breakpoint on kernel32!WriteConsoleW, and the string passed in there is correct. So something is happening after the string has left PowerShell's hands.

Originally created by @jazzdelightsme on GitHub (Jul 17, 2019). Originally assigned to: @zadjii-msft on GitHub. <!-- 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨 I ACKNOWLEDGE THE FOLLOWING BEFORE PROCEEDING: 1. If I delete this entire template and go my own path, the core team may close my issue without further explanation or engagement. 2. If I list multiple bugs/concerns in this one issue, the core team may close my issue without further explanation or engagement. 3. If I write an issue that has many duplicates, the core team may close my issue without further explanation or engagement (and without necessarily spending time to find the exact duplicate ID number). 4. If I leave the title incomplete when filing the issue, the core team may close my issue without further explanation or engagement. 5. If I file something completely blank in the body, the core team may close my issue without further explanation or engagement. All good? Then proceed! --> <!-- This bug tracker is monitored by Windows Terminal development team and other technical folks. **Important: When reporting BSODs or security issues, DO NOT attach memory dumps, logs, or traces to Github issues**. Instead, send dumps/traces to secure@microsoft.com, referencing this GitHub issue. Please use this form and describe your issue, concisely but precisely, with as much detail as possible. --> # Environment ```none Windows build number: Microsoft Windows [Version 10.0.18362.239] Windows Terminal version (if applicable): (built from latest source as of July 17th, 2019, morning) ``` I am working on implementing some new VT escape sequences (PR [here](https://github.com/microsoft/terminal/pull/1978)), but I've run into a strange problem. What the sequences do (or are supposed to do) is not important here. The problem is that in certain circumstances, something is *reordering* VT escape sequences in my content (which is very surprising to me). The control sequences in question are XTPUSHSGR and XTPOPSGR: `[CSI]#p` and `[CSI]#q`, respectively. Note in the string below, how there is an XTPUSHSGR sequence, then a normal SGR sequence to set the color to red, then an XTPOPSGR sequence (and then a normal SGR sequence to reset the color). When I open OpenConsole.sln in Visual Studio 2019, with the CascadiaPackage set as the default project to debug, and then <kbd>F5</kbd> to start debugging it, it launches a terminal running Windows PowerShell. When I send my string to the console, something is *moving* the XTPUSHSGR and XTPOPSGR sequences to the very beginning of the string! # Steps to reproduce <!-- A description of how to trigger this bug. --> 1. Start OpenConsole.sln in Visual Studio 2019. Make sure CascadiaPackage is the default project, and platform is set to x64, and build the solution if necessary. 1. <kbd>F5</kbd> to start debugging a new instance. You should now have a PowerShell instance in the terminal. 1. In the PowerShell instance in the terminal, run the following commands: ```powershell $esc = [char] 0x1b $s = "Hello. Push:${esc}[#p ${esc}[91m this text is RED Pop:${esc}[#q FG should " + "no longer be red (unless it was red before) courtesy reset: ${esc}[0m" ``` 4. Set a breakpoint at `TerminalControl.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString` (the two-parameter one). 1. In the PowerShell instance in the terminal, run the command `$s` to output the string. 1. When you hit the breakpoint, observe the value of `rgwch`. # Expected behavior <!-- A description of what you're expecting, possibly containing screenshots or reference material. --> The content of the string should be just as it was set in PowerShell. # Actual behavior <!-- What's actually happening? --> The XTPUSHSGR and XTPOPSGR sequences have been moved to the beginning of the string! My first thought was that PowerShell must be doing something funny, but I attached windbg to the PowerShell and set a breakpoint on kernel32!WriteConsoleW, and the string passed in there is correct. So something is happening after the string has left PowerShell's hands.
claunia commented 2026-01-30 23:06:18 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jul 17, 2019):

I'm willing to bet serious money that when conhost in conpty mode processes a long string with VT sequences in it that conhost doesn't understand, we render the failing VT sequences before we repaint for the rest of the string.

So for your example string: "Hello. Push:${esc}[#p ${esc}[91m this text is RED Pop:${esc}[#q FG should no longer be red (unless it was red before) courtesy reset: ${esc}[0m"

Conhost is going to try and process that entire string in one frame. Conpty is only going to render the output at the end of a frame, once every 60th of a second. However, as soon as the adapter (adaptDispatch) sees a sequence it doesn't understand, it will dispatch it to the terminal immediately. So the push/pops will appear in the stream before the rest of the frame is rendered.

Possible solutions:

  • A workaround might just be to split the push/pops into separate strings to write to the terminal. It might trigger a frame in between them and the actual text, and it'll appear correctly.
$push = "${esc}[#p"
$red = "${esc}[91m"
$pop = "${esc}[#q"
$reset = "${esc}[0m"
$s1 = "Hello. Push:"
$s2 = "this text is RED Pop:${esc}[#q FG should no longer be red (unless it was red before) courtesy reset: ${esc}[0m"
$s3 = "FG should no longer be red (unless it was red before) courtesy reset: "
  • A more correct solution would be to have conpty actually be able to process those sequences, though this issue will remain for any sequences conhost doesn't understand
  • Maybe the most correct solution would be to have conpty force a frame when it encounters a sequence it doesn't recognize. That might really hit perf though, so I'd be cautious with that change.
@zadjii-msft commented on GitHub (Jul 17, 2019): I'm willing to bet serious money that when conhost in conpty mode processes a long string with VT sequences in it that conhost doesn't understand, we render the failing VT sequences _before_ we repaint for the rest of the string. So for your example string: `"Hello. Push:${esc}[#p ${esc}[91m this text is RED Pop:${esc}[#q FG should no longer be red (unless it was red before) courtesy reset: ${esc}[0m"` Conhost is going to try and process that entire string in one frame. Conpty is only going to render the output at the end of a frame, once every 60th of a second. However, as soon as the adapter (`adaptDispatch`) sees a sequence it doesn't understand, it will dispatch it to the terminal _immediately_. So the push/pops will appear in the stream _before_ the rest of the frame is rendered. Possible solutions: * A workaround might just be to split the push/pops into separate strings to write to the terminal. It _might_ trigger a frame in between them and the actual text, and it'll appear correctly. ```powershell $push = "${esc}[#p" $red = "${esc}[91m" $pop = "${esc}[#q" $reset = "${esc}[0m" $s1 = "Hello. Push:" $s2 = "this text is RED Pop:${esc}[#q FG should no longer be red (unless it was red before) courtesy reset: ${esc}[0m" $s3 = "FG should no longer be red (unless it was red before) courtesy reset: " ``` * A more correct solution would be to have conpty actually be able to process those sequences, though this issue will remain for any sequences conhost _doesn't_ understand * Maybe the most correct solution would be to have conpty force a frame when it encounters a sequence it doesn't recognize. That might really hit perf though, so I'd be cautious with that change.
claunia commented 2026-01-30 23:06:18 +00:00
Author
Owner
Copy Link

@jazzdelightsme commented on GitHub (Jul 17, 2019):

Whoa. Interesting. Thanks for the explanation, @zadjii-msft ; I would never have figured that out.

Part of my problem seems to be a debugging problem: in my branch with some code changes to handle those sequences, I implemented the push/pop commands in the Terminal class, but in the adaptDispatch stuff, I just returned false. I had set breakpoints in the adaptDispatch functions... but when debugging the CascadiaPackage, those breakpoints are disabled, claiming that no symbols have been loaded for the document, so I thought that code was not even running.

But the code definitely IS running... if I change the adaptDispatch layer to just return true (without actually doing anything), the behavior changes!

Although I don't understand the changed behavior... now the push/pop control sequences are simply gone. Should the adaptDispatch layer be doing something to send the sequences on down the line?

@jazzdelightsme commented on GitHub (Jul 17, 2019): Whoa. Interesting. Thanks for the explanation, @zadjii-msft ; I would never have figured that out. Part of my problem seems to be a debugging problem: in my branch with some code changes to handle those sequences, I implemented the push/pop commands in the `Terminal` class, but in the `adaptDispatch` stuff, I just returned `false`. I had set breakpoints in the `adaptDispatch` functions... but when debugging the CascadiaPackage, those breakpoints are disabled, claiming that no symbols have been loaded for the document, so I thought that code was not even running. But the code definitely IS running... if I change the `adaptDispatch` layer to just return `true` (without actually doing anything), the behavior changes! Although I don't understand the changed behavior... now the push/pop control sequences are simply *gone*. Should the `adaptDispatch` layer be doing something to send the sequences on down the line?
claunia commented 2026-01-30 23:06:18 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jul 17, 2019):

Oh that's because debugging the Terminal's conhost is a pain. Basically, Windows Terminal is running attached to another conhost process, and that conhost process is where adaptDispatch is running. To be able to debug it with VS, you need to manually attach VS to the child conhost process that the Terminal started.

@zadjii-msft commented on GitHub (Jul 17, 2019): Oh that's because debugging the Terminal's conhost is a _pain_. Basically, Windows Terminal is running attached to another conhost process, and that conhost process is where adaptDispatch is running. To be able to debug it with VS, you need to manually attach VS to the child conhost process that the Terminal started.
claunia commented 2026-01-30 23:06:18 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Jul 20, 2019):

Marking this one triaged; it's definitely something we should look at, but we may not have the bandwidth to do so.

@DHowett-MSFT commented on GitHub (Jul 20, 2019): Marking this one triaged; it's definitely something we should look at, but we may not have the bandwidth to do so.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Rendering Issue-Bug Needs-Tag-Fix Priority-1 Product-Conpty Resolution-Fix-Committed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#2822
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?