Duplicating tab is *significantly* slower than new tab #22673

@jaybosamiya-ms commented on GitHub (Dec 16, 2024):

Do let me know if I can enable some tracing/logging that might help provide more details the next time that I hit upon the slowdown, to help reproduce the issue.

@jaybosamiya-ms commented on GitHub (Dec 16, 2024): Do let me know if I can enable some tracing/logging that might help provide more details the next time that I hit upon the slowdown, to help reproduce the issue.

claunia commented

@lhecker commented on GitHub (Dec 17, 2024):

This is one of those times again where https://superluminal.eu/ would be super helpful...

I think as a basic triage you could try and see how fast Windows Terminal launches wsl.exe and OpenConsole.exe, by watching the process tree via https://systeminformer.sourceforge.io/ (or alternatively https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer). Does it happen instantly? If so, then it's at least quite a bit less likely to be a bug in Windows Terminal.

@lhecker commented on GitHub (Dec 17, 2024): This is one of those times again where https://superluminal.eu/ would be super helpful... I think as a basic triage you could try and see how fast Windows Terminal launches wsl.exe and OpenConsole.exe, by watching the process tree via https://systeminformer.sourceforge.io/ (or alternatively https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer). Does it happen instantly? If so, then it's at least quite a bit less likely to be a bug in Windows Terminal.

claunia commented

@zadjii-msft commented on GitHub (Dec 17, 2024):

My guess is that passing a path to wsl -cd is what's tripping us up here. I'd bet wsl is evaluating the path slowly

@zadjii-msft commented on GitHub (Dec 17, 2024): My guess is that passing a path to `wsl -cd` is what's tripping us up here. I'd bet wsl is evaluating the path slowly

claunia commented

@jaybosamiya-ms commented on GitHub (Jan 13, 2025):

I do semi-regularly run into the issue but haven't had it happen while I have the process tree viewer open. I will ping here again if I am able to hit the issue while the viewer is open.

At least wrt wsl --cd I don't think that is the issue (at least not on its own):

$ hyperfine 'wsl.exe exit' 'wsl.exe --cd ~ exit' 'wsl.exe --cd / exit' 'wsl.exe --cd /tmp exit' 'wsl.exe --cd /not_exist exit' 'wsl.exe --cd ~/not_exist exit'
Benchmark 1: wsl.exe exit
  Time (mean ± σ):     262.8 ms ±   7.1 ms    [User: 2.5 ms, System: 0.2 ms]
  Range (min … max):   254.5 ms … 278.2 ms    10 runs

Benchmark 2: wsl.exe --cd ~ exit
  Time (mean ± σ):     204.3 ms ±  26.0 ms    [User: 2.0 ms, System: 0.1 ms]
  Range (min … max):   183.1 ms … 261.6 ms    11 runs

Benchmark 3: wsl.exe --cd / exit
  Time (mean ± σ):     179.7 ms ±   5.5 ms    [User: 1.5 ms, System: 0.5 ms]
  Range (min … max):   172.4 ms … 195.1 ms    15 runs

Benchmark 4: wsl.exe --cd /tmp exit
  Time (mean ± σ):     176.8 ms ±   4.0 ms    [User: 1.7 ms, System: 0.2 ms]
  Range (min … max):   171.2 ms … 186.2 ms    16 runs

Benchmark 5: wsl.exe --cd /not_exist exit
  Time (mean ± σ):     177.9 ms ±   3.0 ms    [User: 1.6 ms, System: 0.1 ms]
  Range (min … max):   173.3 ms … 182.6 ms    16 runs

Benchmark 6: wsl.exe --cd ~/not_exist exit
  Time (mean ± σ):     177.0 ms ±   3.9 ms    [User: 1.6 ms, System: 0.1 ms]
  Range (min … max):   172.2 ms … 185.8 ms    16 runs

Summary
  wsl.exe --cd /tmp exit ran
    1.00 ± 0.03 times faster than wsl.exe --cd ~/not_exist exit
    1.01 ± 0.03 times faster than wsl.exe --cd /not_exist exit
    1.02 ± 0.04 times faster than wsl.exe --cd / exit
    1.16 ± 0.15 times faster than wsl.exe --cd ~ exit
    1.49 ± 0.05 times faster than wsl.exe exit

These are all within a small factor of each other (more than I would expect due to noise, but not enough to worry about). The behavior I have seen wrt slow-duplicating-tab is literally over a minute of waiting time.

@jaybosamiya-ms commented on GitHub (Jan 13, 2025): I do semi-regularly run into the issue but haven't had it happen while I have the process tree viewer open. I will ping here again if I am able to hit the issue _while_ the viewer is open. At least wrt `wsl --cd` I don't _think_ that is the issue (at least not on its own): ```console $ hyperfine 'wsl.exe exit' 'wsl.exe --cd ~ exit' 'wsl.exe --cd / exit' 'wsl.exe --cd /tmp exit' 'wsl.exe --cd /not_exist exit' 'wsl.exe --cd ~/not_exist exit' Benchmark 1: wsl.exe exit Time (mean ± σ): 262.8 ms ± 7.1 ms [User: 2.5 ms, System: 0.2 ms] Range (min … max): 254.5 ms … 278.2 ms 10 runs Benchmark 2: wsl.exe --cd ~ exit Time (mean ± σ): 204.3 ms ± 26.0 ms [User: 2.0 ms, System: 0.1 ms] Range (min … max): 183.1 ms … 261.6 ms 11 runs Benchmark 3: wsl.exe --cd / exit Time (mean ± σ): 179.7 ms ± 5.5 ms [User: 1.5 ms, System: 0.5 ms] Range (min … max): 172.4 ms … 195.1 ms 15 runs Benchmark 4: wsl.exe --cd /tmp exit Time (mean ± σ): 176.8 ms ± 4.0 ms [User: 1.7 ms, System: 0.2 ms] Range (min … max): 171.2 ms … 186.2 ms 16 runs Benchmark 5: wsl.exe --cd /not_exist exit Time (mean ± σ): 177.9 ms ± 3.0 ms [User: 1.6 ms, System: 0.1 ms] Range (min … max): 173.3 ms … 182.6 ms 16 runs Benchmark 6: wsl.exe --cd ~/not_exist exit Time (mean ± σ): 177.0 ms ± 3.9 ms [User: 1.6 ms, System: 0.1 ms] Range (min … max): 172.2 ms … 185.8 ms 16 runs Summary wsl.exe --cd /tmp exit ran 1.00 ± 0.03 times faster than wsl.exe --cd ~/not_exist exit 1.01 ± 0.03 times faster than wsl.exe --cd /not_exist exit 1.02 ± 0.04 times faster than wsl.exe --cd / exit 1.16 ± 0.15 times faster than wsl.exe --cd ~ exit 1.49 ± 0.05 times faster than wsl.exe exit ``` These are all within a small factor of each other (more than I would expect due to noise, but not enough to worry about). The behavior I have seen wrt slow-duplicating-tab is literally over a minute of waiting time.

claunia commented

@carlos-zamora commented on GitHub (Jan 13, 2025):

Is the rest of the UI still interactive while "duplicate tab" is executing /? This'll help us figure out where the issue may be occurring.

Also, what other shell integrations do you have set up, if any?

@carlos-zamora commented on GitHub (Jan 13, 2025): Is the rest of the UI still interactive while "duplicate tab" is executing /? This'll help us figure out where the issue may be occurring. Also, what other shell integrations do you have set up, if any?

claunia commented

@jaybosamiya-ms commented on GitHub (Jan 13, 2025):

Yes, rest of the UI is indeed still interactive. More details:

While "duplicate tab" appears to be stuck (i.e., waiting for 1~2 minutes before that tab responds)
- Overall Windows Terminal UI is still responsive
- Can use "new tab" and that will work instantly
- Running another "duplicate tab" leads to a new tab that is similarly stuck
- Running Ctrl-C in the "stuck" tab causes the tab to quit trying to run WSL in it (I don't have the message available right now to confirm, but unless I'm mistaken, it is the standard "process exited" one)
- Waiting for the 1~2 minutes causes it to get unstuck, and behave with regular performance going forward
After the "duplicate tab" gets unstuck / when not in the occasional "duplicate tab is stuck" case
- UI is responsive as usual
- Running "duplicate tab" again no longer is slow, is roughly same speed as "new tab"
- Running Ctrl-C immediately after opening a new tab (but before shell is fully initialized) drops me to a basic zsh shell (i.e., without the fancy setup) but never goes into the "process exited" state mentioned above

Shell integrations: the Windows Terminal integration is here: 3a17d8c8bd/chezmoi/dot_zshrc (L567-L580) ; I know my zshrc is fairly sizable, but most of it (everything other than the WT-specific code that I've linked) should have no impact on tab duplication (it is all highly tested code, used on many machines, over many different terminals, and operating systems, across many years, some including Windows Terminal; by process of elimination, the linked code is the narrowest root cause).

@jaybosamiya-ms commented on GitHub (Jan 13, 2025): Yes, rest of the UI is indeed still interactive. More details: * While "duplicate tab" appears to be stuck (i.e., waiting for 1~2 minutes before that tab responds) - Overall Windows Terminal UI is still responsive - Can use "new tab" and that will work instantly - Running _another_ "duplicate tab" leads to a new tab that is similarly stuck - Running Ctrl-C in the "stuck" tab causes the tab to quit trying to run WSL in it (I don't have the message available right now to confirm, but unless I'm mistaken, it is the standard "process exited" one) - Waiting for the 1~2 minutes causes it to get unstuck, and behave with regular performance going forward * After the "duplicate tab" gets unstuck / when not in the occasional "duplicate tab is stuck" case - UI is responsive as usual - Running "duplicate tab" again no longer is slow, is roughly same speed as "new tab" - Running Ctrl-C _immediately_ after opening a new tab (but before shell is fully initialized) drops me to a basic zsh shell (i.e., without the fancy setup) but never goes into the "process exited" state mentioned above Shell integrations: the Windows Terminal integration is here: https://github.com/jaybosamiya-ms/dotfiles/blob/3a17d8c8bdbcfd3c2cfe743ec185bf46371d50a1/chezmoi/dot_zshrc#L567-L580 ; I know my zshrc is fairly sizable, but most of it (everything other than the WT-specific code that I've linked) should have no impact on tab duplication (it is all highly tested code, used on many machines, over many different terminals, and operating systems, across many years, some _including_ Windows Terminal; by process of elimination, the linked code is the narrowest root cause).

claunia commented

@DHowett commented on GitHub (Jan 15, 2025):

Fascinating.

As an experiment... can you put set -x at the top of your .zshrc? I'd love to know what, if anything, it's doing while it's waiting. Throw set +x in at the end to shutter the verbose logs once you're prompt-ready.

I know you can't reproduce this all the time, and that it will generate a lot of noise in the working case... but I'm hoping that it gives us enough info in the broken case to get to a narrower diagnosis. 🙂

@DHowett commented on GitHub (Jan 15, 2025): _Fascinating._ As an experiment... can you put `set -x` at the top of your `.zshrc`? I'd love to know what, if anything, it's doing while it's waiting. Throw `set +x` in at the end to shutter the verbose logs once you're prompt-ready. I know you can't reproduce this all the time, and that it will generate a lot of noise in the working case... but I'm hoping that it gives us enough info in the broken case to get to a narrower diagnosis. 🙂

claunia commented

@jaybosamiya-ms commented on GitHub (Jan 15, 2025):

Ah, good idea with the set -x. I've added this to my zshrc now, will report back next time I'm able to catch a hold of it happening. I still also have System Informer running with wsl|OpenConsole as the regex filter, to try to catch this issue when it manifests again. Heisenbugs are both so fascinating and frustrating at the same time :)

@jaybosamiya-ms commented on GitHub (Jan 15, 2025): Ah, good idea with the `set -x`. I've added this to my zshrc now, will report back next time I'm able to catch a hold of it happening. I still also have System Informer running with `wsl|OpenConsole` as the regex filter, to try to catch this issue when it manifests again. Heisenbugs are both so fascinating and frustrating at the same time :)

claunia commented

@jaybosamiya-ms commented on GitHub (Jan 27, 2025):

Fascinatingly, I have not gotten a single instance of the duplication-slowdown ever since I enabled the set -x.

I am still on the same version of Windows Terminal (1.21.3231.0), and zsh and such. The only updates are Windows (now on 10.0.26100.2894, rather than 10.0.26100.0 during the first report). I don't expect the Windows update to have changed anything (the bug is very Windows Terminal specific; no other oddities in the other things I run), but I can't rule out that a Windows update fixed (or suppressed, by tweaking some underlying conditions enough) the issue.

I'm going to continue keeping set -x enabled (and the System Informer running), since it is not too disruptive to keep those up (waiting for the bug to trigger again), but I think if it doesn't trigger in the next one-or-two weeks, I might remove one/both of those, to try to see if it is one of the Heisenbug effects at play (observation leading to changes in bug behavior), and if I can get it triggering again.

If y'all don't mind, I'd like to keep the bug open (despite the bug technically being "stale"), in hopes of triggering it again to try to resolve it. However, I would also understand if you'd like to close it, given that it is no longer repro'ing under any better-observation conditions. Sorry I don't have a very helpful repro case for this rare bug.

@jaybosamiya-ms commented on GitHub (Jan 27, 2025): _Fascinatingly_, I have not gotten a single instance of the duplication-slowdown ever since I enabled the `set -x`. I am _still_ on the same version of Windows Terminal (1.21.3231.0), and zsh and such. The only updates are Windows (now on 10.0.26100.2894, rather than 10.0.26100.0 during the first report). I don't expect the Windows update to have changed anything (the bug is _very_ Windows Terminal specific; no other oddities in the other things I run), but I can't rule out that a Windows update fixed (or suppressed, by tweaking some underlying conditions enough) the issue. I'm going to continue keeping `set -x` enabled (and the System Informer running), since it is not _too_ disruptive to keep those up (waiting for the bug to trigger again), but I think if it doesn't trigger in the next one-or-two weeks, I might remove one/both of those, to try to see if it is one of the Heisenbug effects at play (observation leading to changes in bug behavior), and if I can get it triggering again. If y'all don't mind, I'd like to keep the bug open (despite the bug technically being "stale"), in hopes of triggering it again to try to resolve it. However, I would also understand if you'd like to close it, given that it is no longer repro'ing under any better-observation conditions. Sorry I don't have a very helpful repro case for this rare bug.

claunia commented

@DHowett commented on GitHub (Feb 3, 2025):

I'll unmark this one as requiring feedback so the bot stops trying to close it, but I will leave it in "Needs Attention" state so we come back to it.

@DHowett commented on GitHub (Feb 3, 2025): I'll unmark this one as requiring feedback so the bot stops trying to close it, but I will leave it in "Needs Attention" state so we come back to it.

claunia commented

@jaybosamiya-ms commented on GitHub (Mar 7, 2025):

Latest update: it has been over a month since I enabled set -x. I don't seem to have hit the slowdown-for-tab-duplication in that entire time. Attempting to observe it seems to have made the bug go away. It used to be a relatively frequent annoyance (once or twice a day, it'd eat up 30s to multiple minutes) and somehow it literally hasn't triggered at all in the past month.

I am going to remove the set -x to see if it returns but given that I haven't significantly changed anything about the setup, modulo the introduction of set -x, so yeah, I seem to still be in the dark as to the cause. If it returns upon removing set -x, I guess the "fix" is "do some console spew" (which would be fairly bizarre, tbqh).

@jaybosamiya-ms commented on GitHub (Mar 7, 2025): Latest update: it has been over a month since I enabled `set -x`. I don't seem to have hit the slowdown-for-tab-duplication in that entire time. Attempting to observe it seems to have made the bug go away. It used to be a relatively frequent annoyance (once or twice a day, it'd eat up 30s to multiple minutes) and somehow it literally hasn't triggered at all in the past month. I am going to remove the `set -x` to see if it returns but given that I haven't significantly changed anything about the setup, modulo the introduction of `set -x`, so yeah, I seem to still be in the dark as to the cause. If it returns upon removing `set -x`, I guess the "fix" is "do some console spew" (which would be fairly bizarre, tbqh).

claunia commented

@zadjii-msft commented on GitHub (Mar 27, 2025):

Attempting to observe it seems to have made the bug go away

Classic heisenbug. I'll let the bot give it another week, but I'm glad that it went away?

@zadjii-msft commented on GitHub (Mar 27, 2025): > Attempting to observe it seems to have made the bug go away Classic heisenbug. I'll let the bot give it another week, but I'm glad that it went away?

claunia commented

@jaybosamiya-ms commented on GitHub (Mar 31, 2025):

Removing the console spew via set -x did bring back the issue exactly once over the past ~month, and unfortunately, I couldn't catch it in System Informer by the time it finished the duplicate tab. Bizarrely difficult to catch this bug.

Anyways, the bug is rare enough, and again, I have a "solution" (wrapping with set -x/set +x to force console spew during initialization), so even though it is suboptimal, I am going to close this issue.

I will re-open if I find a consistently reproducible way to trigger it.

Thank everyone for your help in attempting to catch this heisenbug

@jaybosamiya-ms commented on GitHub (Mar 31, 2025): Removing the console spew via `set -x` _did_ bring back the issue exactly once over the past ~month, and unfortunately, I couldn't catch it in System Informer by the time it finished the duplicate tab. Bizarrely difficult to catch this bug. Anyways, the bug is rare enough, and again, I have a "solution" (wrapping with `set -x`/`set +x` to force console spew during initialization), so even though it is suboptimal, I am going to close this issue. I will re-open if I find a consistently reproducible way to trigger it. Thank everyone for your help in attempting to catch this heisenbug

claunia commented