Escape sequences like DA responses can end up fragmented across multiple TCP/IP packets #23549

Closed
opened 2026-01-31 08:45:37 +00:00 by claunia · 5 comments
Owner

Originally created by @primeos-work on GitHub (Aug 26, 2025).

Windows Terminal version

1.22.12111.0

Windows build number

10.0.26100.0

Other Software

OpenSSH_for_Windows_9.5p1 (less likely with OpenSSH_for_Windows_9.8p2)
tmux 3.5a

Steps to reproduce

By repeatedly (re)attaching to a tmux session after connecting to a Linux system via SSH. The chances of the bug occurring depend on multiple factors like the RTT and the tmux escape-time. The bug is easier to trigger with an OpenSSH for Windows version that isn't newer than 9.5p1 (still the most recent stable version; this is important since https://github.com/PowerShell/openssh-portable/pull/771 improves things on the SSH side and is released in v9.8.2.0+). It also helps if the RTT to the SSH host is higher and if the tmux escape-time is very low (I use 0/1 but 10 should be fine as well and I even get it rarely with 50 ms). The tmux version should also be at most 3.5a (currently still the most recent version) since 817b621d20 should be a reliable workaround for this Windows Terminal issue on the tmux side.

Expected Behavior

Everything works normally like with many other Terminal emulators. No "strange characters" (parts of the device-attributes reply escape sequence) end up in the shell prompt (I don't remember ever hitting this issue from my Linux systems).

Actual Behavior

When attaching to a tmux session, it's possible that part of the device-attributes (DA) escape sequence ends up in the active shell prompt instead of being received/interpreted by tmux.

An example:

mweiss@vm-debian-1 ~ $ 0;10;1c

Another example:

[michael@groot ~]$ 61;6;7;21;22;23;24;28;32;42c

Additional information and examples can be found here: https://github.com/PowerShell/Win32-OpenSSH/issues/2275

The issue seems to be that the escape sequence can end up being split across multiple (two) TCP/IP packets and doesn't get interpreted correctly by tmux if the next TCP/IP packet exceeds the escape-time setting. I couldn't check this in the Windows Terminal source-code but the issue is likely that Windows Terminal doesn't ensure that the whole escape sequence is sent "synchronously" via a single write syscall over the SSH/TCP/IP connection. Could someone confirm this and is there a way to implement this (my Linux terminal emulators (e.g. Alacritty) do this so I never hit this bug there while it regularly (daily) occurs when using Windows Terminal). I also wasn't able to trigger this bug with Alacritty on Windows so it should be possible to ensure that the whole DA reply escape sequence ends up in a single network packet. The terminal emulator Foot seems to use the following code to ensure this: ed7652db50/terminal.c (L115-L138)

Originally created by @primeos-work on GitHub (Aug 26, 2025). ### Windows Terminal version 1.22.12111.0 ### Windows build number 10.0.26100.0 ### Other Software `OpenSSH_for_Windows_9.5p1` (less likely with `OpenSSH_for_Windows_9.8p2`) `tmux 3.5a` ### Steps to reproduce By repeatedly (re)attaching to a tmux session after connecting to a Linux system via SSH. The chances of the bug occurring depend on multiple factors like the RTT and the tmux `escape-time`. The bug is easier to trigger with an OpenSSH for Windows version that isn't newer than 9.5p1 (still the most recent stable version; this is important since https://github.com/PowerShell/openssh-portable/pull/771 improves things on the SSH side and is released in [v9.8.2.0](https://github.com/PowerShell/openssh-portable/releases/tag/v9.8.2.0)+). It also helps if the RTT to the SSH host is higher and if the tmux `escape-time` is very low (I use `0`/`1` but `10` should be fine as well and I even get it rarely with `50` ms). The tmux version should also be at most 3.5a (currently still the most recent version) since https://github.com/tmux/tmux/commit/817b621d2078137b4ddea78835f609a9d7bac339 should be a reliable workaround for this Windows Terminal issue on the tmux side. ### Expected Behavior Everything works normally like with many other Terminal emulators. No "strange characters" (parts of the device-attributes reply escape sequence) end up in the shell prompt (I don't remember ever hitting this issue from my Linux systems). ### Actual Behavior When attaching to a tmux session, it's possible that part of the device-attributes (DA) escape sequence ends up in the active shell prompt instead of being received/interpreted by tmux. An example: ```console mweiss@vm-debian-1 ~ $ 0;10;1c ``` Another example: ```console [michael@groot ~]$ 61;6;7;21;22;23;24;28;32;42c ``` Additional information and examples can be found here: https://github.com/PowerShell/Win32-OpenSSH/issues/2275 The issue seems to be that the escape sequence can end up being split across multiple (two) TCP/IP packets and doesn't get interpreted correctly by tmux if the next TCP/IP packet exceeds the `escape-time` setting. I couldn't check this in the Windows Terminal source-code but the issue is likely that Windows Terminal doesn't ensure that the whole escape sequence is sent "synchronously" via a single `write` syscall over the SSH/TCP/IP connection. Could someone confirm this and is there a way to implement this (my Linux terminal emulators (e.g. Alacritty) do this so I never hit this bug there while it regularly (daily) occurs when using Windows Terminal). I also wasn't able to trigger this bug with Alacritty on Windows so it should be possible to ensure that the whole DA reply escape sequence ends up in a single network packet. The terminal emulator Foot seems to use the following code to ensure this: https://codeberg.org/dnkl/foot/src/commit/ed7652db5056c3658afbc11c04e164e77da18650/terminal.c#L115-L138
claunia added the Needs-TriageIssue-Bug labels 2026-01-31 08:45:37 +00:00
Author
Owner

@similar-issues-ai[bot] commented on GitHub (Aug 26, 2025):

We've found some similar issues:

  • #17439 , similarity score: 84%

If any of the above are duplicates, please consider closing this issue out and adding additional context in the original issue.

Note: You can give me feedback by 👍 or 👎 this comment.

@similar-issues-ai[bot] commented on GitHub (Aug 26, 2025): We've found some similar issues: - #17439 , similarity score: 84% If any of the above are duplicates, please consider closing this issue out and adding additional context in the original issue. > Note: You can give me feedback by 👍 or 👎 this comment.
Author
Owner

@primeos-work commented on GitHub (Aug 27, 2025):

I originally reported this issue as https://github.com/PowerShell/Win32-OpenSSH/issues/2275 but will close that issue in favor of this one as we noticed that the source of this issue should be found in the Windows Terminal source code instead (that analysis/discussion starts at https://github.com/PowerShell/Win32-OpenSSH/issues/2275#issuecomment-2462365617) and with https://github.com/PowerShell/openssh-portable/pull/771 this can probably be considered fixed on the Win32-OpenSSH side.

There are quite a few duplicate issues here (and in other repos, forums, etc.) but they've been closed without correctly identifying the source of this issue:

@primeos-work commented on GitHub (Aug 27, 2025): I originally reported this issue as https://github.com/PowerShell/Win32-OpenSSH/issues/2275 but will close that issue in favor of this one as we noticed that the source of this issue should be found in the Windows Terminal source code instead (that analysis/discussion starts at https://github.com/PowerShell/Win32-OpenSSH/issues/2275#issuecomment-2462365617) and with https://github.com/PowerShell/openssh-portable/pull/771 this can probably be considered fixed on the Win32-OpenSSH side. There are quite a few duplicate issues here (and in other repos, forums, etc.) but they've been closed without correctly identifying the source of this issue: - https://github.com/microsoft/terminal/issues/7185 - https://github.com/microsoft/terminal/issues/16384 - https://github.com/microsoft/terminal/issues/17439 - https://github.com/microsoft/terminal/issues/19120 (might also be a result of the timing/fragmentation)
Author
Owner

@lhecker commented on GitHub (Aug 27, 2025):

Referencing the analysis you linked:

One thing Windows Terminal and ConPTY folks might want to check is if their device-attributes response is really being sent out via a single write() system call (as ESC sequences always should be).

This doesn't make sense to me. First, we have been doing that for a long time.
More importantly though, the PTY that underlies a terminal is essentially a binary ring buffer which has no idea about the boundaries of escape sequences in it. This is true on both Windows and Linux (and certainly on any other NIX). This means that all sorts of reasons can cause escape sequences to be chunked up, including into broken UTF8 sequences. Applications should handle it by implementing a proper finite state machine for parsing them. Many applications don't do this properly, including application you may assume to be of high quality. That's their fault though, because this aspect of PTYs is very old.

If not, splitting such a response [...] that do not arrive within the 10 ms patience time that tmux-3.5 allows before it concludes that this might have been a manual press of the ESC key and not a response sequence.

To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH. As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so.

It is one wide string L"?61;4;6;7;14;21;22;23;24;28;32;42c", except for the initial ESC [, which all may have to go through a UTF-8 encoder, and thus might end up in separate write() system calls and separate TCP packets?

That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here:
52262b05fa/src/cascadia/TerminalControl/ControlCore.cpp (L2174-L2178)


To my understanding there are two issues involved here:

So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as 0;10;1c.

@lhecker commented on GitHub (Aug 27, 2025): Referencing the analysis you linked: > One thing [Windows Terminal](https://github.com/microsoft/terminal?rgh-link-date=2024-11-07T14%3A23%3A13.000Z) and ConPTY folks might want to check is if their device-attributes response is really being sent out via a _single_ `write()` system call (as ESC sequences always should be). This doesn't make sense to me. First, we have been doing that for a long time. More importantly though, the PTY that underlies a terminal is essentially a binary ring buffer which has no idea about the boundaries of escape sequences in it. This is true on both Windows and Linux (and certainly on any other NIX). This means that all sorts of reasons can cause escape sequences to be chunked up, including into broken UTF8 sequences. Applications should handle it by implementing a proper finite state machine for parsing them. Many applications don't do this properly, including application you may assume to be of high quality. That's their fault though, because this aspect of PTYs is very old. > If not, splitting such a response [...] that do not arrive within the 10 ms patience time that tmux-3.5 allows before it concludes that this might have been a manual press of the ESC key and not a response sequence. To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH. As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so. > It is one wide string `L"?61;4;6;7;14;21;22;23;24;28;32;42c"`, except for the initial `ESC [`, which all may have to go through a UTF-8 encoder, and thus might end up in separate `write()` system calls and separate TCP packets? That's not how our system works. Responses get accumulated into the `ControlCore::_pendingResponses` buffer and then flushed out all at once here: https://github.com/microsoft/terminal/blob/52262b05fa0a97d2d3a0fce0990840ffc0fa53f1/src/cascadia/TerminalControl/ControlCore.cpp#L2174-L2178 ---- To my understanding there are two issues involved here: * OpenSSH used to do a bad job at reading stdin, which was "fixed" by https://github.com/PowerShell/openssh-portable/pull/771 (Buffer size should be even higher, but the architecture is not amendable to such changes) * ConPTY (= our PTY) had a race condition: https://github.com/microsoft/terminal/pull/18816 So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as `0;10;1c`.
Author
Owner

@primeos-work commented on GitHub (Aug 27, 2025):

Huge thanks @lhecker for your reply/analysis! ❤️

Many applications don't do this properly, including application you may assume to be of high quality.

Yeah, I'm unfortunately not familiar with the standards but I'd also consider it a tmux issue as well. It should luckily be fixed with 817b621d20 now but it'll take a long time until that's available on most SSH hosts.

To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH.

Ok, so you've also experienced that issue on Linux systems?
I don't remember ever running into that issue from my Linux laptops but it's certainly possible - I guess it depends on the terminal emulator (even though it's not necessarily a terminal emulator bug but it looks like terminal emulators can at least do a better or worse job at preventing it by ensuring the response ends up in a single network packet or at least using large enough buffer sizes - I'm not familiar at all with the involved details though).

As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so.

Ok, I was hoping there would be a way to significantly improve the chances of the response ending up in a single network packet but we can close this issue then if there's no reasonable improvement left.

That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here:

Ok, thanks. If I followed the calls correctly it'll end up here?:

91c9a14a71/src/terminal/adapter/InteractDispatch.cpp (L30-L42)

I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences (or some alternative means to ensure the whole response ends up in a single 1024 bytes buffer) but I have no idea how well that would fit into the code and it would obviously have the drawback of slowing things down by not utilizing the full buffer size in those cases and (significantly) increasing complexity so it's most likely not worth it...

I guess in the end it really is a tmux issue/problem (I also couldn't reproduce it with GNU Screen when I tried it just now but I didn't check if they also issue a DA request but I guess it's very likely the case) as Windows Terminal probably doesn't violate any standards here and I just incorrectly blamed it on Windows Terminal / Win32-OpenSSH as I didn't hit it with other setups (Linux systems/terminals). Sorry about that! (There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue. I guess we at least have it documented/analyzed quite extensively now...)

(Buffer size should be even higher, but the architecture is not amendable to such changes)

Ok, that's a bit of a bummer but fair. With a higher buffer size the issue might already be gone almost entirely.

So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as 0;10;1c.

I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and ea0ba567fd is in version 1.22.12111.0 that I used for the fresh tests.

@primeos-work commented on GitHub (Aug 27, 2025): Huge thanks @lhecker for your reply/analysis! ❤️ > Many applications don't do this properly, including application you may assume to be of high quality. Yeah, I'm unfortunately not familiar with the standards but I'd also consider it a tmux issue as well. It should luckily be fixed with https://github.com/tmux/tmux/commit/817b621d2078137b4ddea78835f609a9d7bac339 now but it'll take a long time until that's available on most SSH hosts. > To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH. Ok, so you've also experienced that issue on Linux systems? I don't remember ever running into that issue from my Linux laptops but it's certainly possible - I guess it depends on the terminal emulator (even though it's not necessarily a terminal emulator bug but it looks like terminal emulators can at least do a better or worse job at preventing it by ensuring the response ends up in a single network packet or at least using large enough buffer sizes - I'm not familiar at all with the involved details though). > As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so. Ok, I was hoping there would be a way to significantly improve the chances of the response ending up in a single network packet but we can close this issue then if there's no reasonable improvement left. > That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here: Ok, thanks. If I followed the calls correctly it'll end up here?: https://github.com/microsoft/terminal/blob/91c9a14a710af7048d7d8c3d74b1d2bc2744bc97/src/terminal/adapter/InteractDispatch.cpp#L30-L42 I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences (or some alternative means to ensure the whole response ends up in a single 1024 bytes buffer) but I have no idea how well that would fit into the code and it would obviously have the drawback of slowing things down by not utilizing the full buffer size in those cases and (significantly) increasing complexity so it's most likely not worth it... I guess in the end it really is a tmux issue/problem (I also couldn't reproduce it with GNU Screen when I tried it just now but I didn't check if they also issue a DA request but I guess it's very likely the case) as Windows Terminal probably doesn't violate any standards here and I just incorrectly blamed it on Windows Terminal / Win32-OpenSSH as I didn't hit it with other setups (Linux systems/terminals). Sorry about that! (There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue. I guess we at least have it documented/analyzed quite extensively now...) > (Buffer size should be even higher, but the architecture is not amendable to such changes) Ok, that's a bit of a bummer but fair. With a higher buffer size the issue might already be gone almost entirely. > So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as `0;10;1c`. I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and https://github.com/microsoft/terminal/commit/ea0ba567fd0edbf92db4fde9ddcf3359f1537426 is in version 1.22.12111.0 that I used for the fresh tests.
Author
Owner

@lhecker commented on GitHub (Aug 29, 2025):

Ok, so you've also experienced that issue on Linux systems?

Not really. I've generally only administered servers that are either in a local DC or in my home. But I know it's an issue, which is why the escape-time setting exists at all.

Ok, thanks. If I followed the calls correctly it'll end up here?:

Sort of. That's the PTY side of things where input is written into the aforementioned ring buffer. The architecture is bad, so don't take the following as "it should be like this" but more like "it's currently like this":

  • Your app write DA1 into stdout (a pipe)
  • The PTY (aka ConPTY aka OpenConsole.exe aka conhost.exe) reads from stdout and
    • before v1.22:
      • Parses stdout into VT sequences
      • Reassembles them back into VT output
      • Sends it to the terminal (another pipe)
    • since v1.22:
      • Sends it to the terminal (another pipe)
  • The terminal reads from the PTY and
    • Parses the VT sequences
    • Sees the DA1
    • Writes the response into _pendingResponses
  • Now it writes _pendingResponses back to the PTY (a pipe)
  • The PTY reads from stdin (terminal) and
    • Parses stdin into VT sequences (like above before v1.22 == bad, needs a refactor)
    • Writes the input events into a "ring buffer" / queue
    • Signals availability of input to your app's stdin pipe
  • Your app gets woken up because stdin is available and
    • Reads stdin from the PTY ring buffer

I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences

I'd be against that. Applications should implement better VT parsers (which are robust against this), and terminals should implement the kitty keyboard protocol (including us; it turns Esc keys into the unmistakable sequence \x1b[27u). This fixes the issue from both sides IMO. Particularly the latter is a good fix, because your suggestion and any other current workaround rely on heuristics which is not great.

There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue.

Windows Terminal has had (and still has) various major bugs with its VT handling, so the attribution is totally fair IMO. Most notably, before #17510 (= before v1.22) we would mangle VT sequences in stdout. We were the only PTY and terminal to do this, and it resulted in lots of issues. We also still somewhat mangle stdin but that's unrelated to this issue. I'm also intending to fix that in the near term.

I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and ea0ba567fd is in version 1.22.12111.0 that I used for the fresh tests.

Hmm that's weird then... Your second example looks like just the Esc character was mistaken for an Esc key so it printed the whole DA1 response to your prompt. But the first example looked exactly like how #18816 presented itself. We may have to investigate this, although I'm not sure how. (It doesn't happen to me so I can't debug it. I also could only figure out #18816 after I was able to reproduce it in wezterm.)

@lhecker commented on GitHub (Aug 29, 2025): > Ok, so you've also experienced that issue on Linux systems? Not really. I've generally only administered servers that are either in a local DC or in my home. But I know it's an issue, which is why the `escape-time` setting exists at all. > Ok, thanks. If I followed the calls correctly it'll end up here?: Sort of. That's the PTY side of things where input is written into the aforementioned ring buffer. The architecture is bad, so don't take the following as "it should be like this" but more like "it's currently like this": * Your app write DA1 into stdout (a pipe) * The PTY (aka ConPTY aka OpenConsole.exe aka conhost.exe) reads from stdout and * before v1.22: * Parses stdout into VT sequences * Reassembles them back into VT output * Sends it to the terminal (another pipe) * since v1.22: * Sends it to the terminal (another pipe) * The terminal reads from the PTY and * Parses the VT sequences * Sees the DA1 * Writes the response into `_pendingResponses` * Now it writes `_pendingResponses` back to the PTY (a pipe) * The PTY reads from stdin (terminal) and * Parses stdin into VT sequences (like above before v1.22 == bad, needs a refactor) * Writes the input events into a "ring buffer" / queue * Signals availability of input to your app's `stdin` pipe * This is where https://github.com/microsoft/terminal/pull/18816 happened * Your app gets woken up because stdin is available and * Reads stdin from the PTY ring buffer > I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences I'd be against that. Applications should implement better VT parsers (which are robust against this), and terminals should implement the kitty keyboard protocol (including us; it turns Esc keys into the unmistakable sequence `\x1b[27u`). This fixes the issue from both sides IMO. Particularly the latter is a good fix, because your suggestion and any other current workaround rely on heuristics which is not great. > There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue. Windows Terminal has had (and still has) various major bugs with its VT handling, so the attribution is totally fair IMO. Most notably, before #17510 (= before v1.22) we would mangle VT sequences in stdout. We were the only PTY and terminal to do this, and it resulted in lots of issues. We also still _somewhat_ mangle stdin but that's unrelated to this issue. I'm also intending to fix that in the near term. > I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and https://github.com/microsoft/terminal/commit/ea0ba567fd0edbf92db4fde9ddcf3359f1537426 is in version 1.22.12111.0 that I used for the fresh tests. Hmm that's weird then... Your second example looks like just the Esc character was mistaken for an Esc key so it printed the whole DA1 response to your prompt. But the first example looked exactly like how #18816 presented itself. We may have to investigate this, although I'm not sure how. (It doesn't happen to me so I can't debug it. I also could only figure out #18816 after I was able to reproduce it in wezterm.)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#23549