Terminal crashes when exiting a bash shell #18343

Closed
opened 2026-01-31 06:10:59 +00:00 by claunia · 5 comments
Owner

Originally created by @j4james on GitHub (Aug 30, 2022).

Windows Terminal version

1.15.1862.0

Windows build number

10.0.19044.1889

Other Software

WSL with Ubuntu 20.04.4 LTS

Steps to reproduce

  1. Start Windows Terminal
  2. Open two tabs, one of which is a bash shell.
  3. Exit the bash shell (with exit).

Expected Behavior

The terminal should remain open after the bash shell exits, with the other tab still working.

Actual Behavior

The terminal crashes.

I was able to reproduce this in the debugger with a build derived from commit c12987af41, and the stack trace for the crash looked like this:

ntdll.dll!TppRaiseInvalidParameter()	Unknown
ntdll.dll!TppTimerpValidateTimer()	Unknown
ntdll.dll!TpSetTimerEx()	Unknown
Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_terminalScrollPositionChanged(const int viewTop, const int viewHeight, const int bufferSize) Line 1308	C++
[External Code]	
Microsoft.Terminal.Control.dll!Microsoft::Terminal::Core::Terminal::_NotifyScrollEvent() Line 1305	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::_EraseScrollback() Line 1952	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::EraseInDisplay(const Microsoft::Console::VirtualTerminal::DispatchTypes::EraseType eraseType) Line 614	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionCsiDispatch(const Microsoft::Console::VirtualTerminal::VTID id, const Microsoft::Console::VirtualTerminal::VTParameters parameters) Line 492	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch'::`2'::<lambda_1>>(Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch::__l2::<lambda_1> && lambda) Line 2039	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch(const wchar_t wch) Line 480	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessCharacter(const wchar_t wch) Line 1778	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString(const std::basic_string_view<wchar_t,std::char_traits<wchar_t>> string) Line 1853	C++
Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler(const winrt::hstring & hstr) Line 1706	C++
Microsoft.Terminal.Control.dll!winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_218_>>::Invoke(void * output) Line 174	C++
[External Code]	

The problem, as far as I can make out, is that the ControlCore instance is deleted at this point in time. If you look at ControlCore::Close method, we're closing the connection asynchronously, which I think means we're still receiving VT sequences (in this case \e[3J) after ControlCore is gone. Assuming I've understood that correctly, that would probably explain the crash.

Originally created by @j4james on GitHub (Aug 30, 2022). ### Windows Terminal version 1.15.1862.0 ### Windows build number 10.0.19044.1889 ### Other Software WSL with Ubuntu 20.04.4 LTS ### Steps to reproduce 1. Start Windows Terminal 2. Open two tabs, one of which is a bash shell. 3. Exit the bash shell (with `exit`). ### Expected Behavior The terminal should remain open after the bash shell exits, with the other tab still working. ### Actual Behavior The terminal crashes. I was able to reproduce this in the debugger with a build derived from commit c12987af415c5e0911d7a0a81b8494fbe6307328, and the stack trace for the crash looked like this: ntdll.dll!TppRaiseInvalidParameter() Unknown ntdll.dll!TppTimerpValidateTimer() Unknown ntdll.dll!TpSetTimerEx() Unknown Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_terminalScrollPositionChanged(const int viewTop, const int viewHeight, const int bufferSize) Line 1308 C++ [External Code] Microsoft.Terminal.Control.dll!Microsoft::Terminal::Core::Terminal::_NotifyScrollEvent() Line 1305 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::_EraseScrollback() Line 1952 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::EraseInDisplay(const Microsoft::Console::VirtualTerminal::DispatchTypes::EraseType eraseType) Line 614 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionCsiDispatch(const Microsoft::Console::VirtualTerminal::VTID id, const Microsoft::Console::VirtualTerminal::VTParameters parameters) Line 492 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch'::`2'::<lambda_1>>(Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch::__l2::<lambda_1> && lambda) Line 2039 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch(const wchar_t wch) Line 480 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessCharacter(const wchar_t wch) Line 1778 C++ Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString(const std::basic_string_view<wchar_t,std::char_traits<wchar_t>> string) Line 1853 C++ Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler(const winrt::hstring & hstr) Line 1706 C++ Microsoft.Terminal.Control.dll!winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_218_>>::Invoke(void * output) Line 174 C++ [External Code] The problem, as far as I can make out, is that the `ControlCore` instance is deleted at this point in time. If you look at `ControlCore::Close` method, we're closing the connection asynchronously, which I think means we're still receiving VT sequences (in this case `\e[3J`) after `ControlCore` is gone. Assuming I've understood that correctly, that would probably explain the crash.
Author
Owner

@j4james commented on GitHub (Aug 30, 2022):

This might be a duplicate of #12068. Possibly also #13337 (they initially said they weren't doing anything when the terminal crashed, but later crashes occurred when "attempting to exit restored tabs" which sounds similar to what I'm seeing).

I'm surprised we don't have had more reports like this, though, because it's really easy for me to produce, but I guess there's an element of timing involved. I've also only noticed this recently, so I'm wondering if there's something we've changed that has made it easier to trigger.

@j4james commented on GitHub (Aug 30, 2022): This might be a duplicate of #12068. Possibly also #13337 (they initially said they weren't doing anything when the terminal crashed, but later crashes occurred when "attempting to exit restored tabs" which sounds similar to what I'm seeing). I'm surprised we don't have had more reports like this, though, because it's really easy for me to produce, but I guess there's an element of timing involved. I've also only noticed this recently, so I'm wondering if there's something we've changed that has made it easier to trigger.
Author
Owner

@lhecker commented on GitHub (Aug 30, 2022):

This commit seems to be related (although I don't immediately see why that would be): 23e4d313d5

It tries to work around a deadlock issue by calling UpdatePatternLocations from a background thread. Even if I don't see how it might cause this, it does seem to be related because:

  • It's a recent change (included in c12987af41)
  • It's in ControlCore
  • _updatePatternLocations is called in _terminalScrollPositionChanged which calls TpSetTimerEx

This PR is my attempt to fix one bug caused by that commit: https://github.com/microsoft/terminal/pull/13859
But yours seems to be another: The TppRaiseInvalidParameter call seems like an issue that might be caused if the til::throttled_func has been destroyed, making the wil::unique_threadpool_timer handle invalid. But why would this have worked before? The only difference I can think of is that the previous ThrottledFunc uses a shared_ptr to keep itself alive until the scheduled callback returns, but I don't see how that would be related here since no such ThrottledFunc is involved in _connectionOutputHandler. Did we just coincidentally get lucky up until now?

In either case I believe the underlying issue is this: a85d9e69ed/src/cascadia/TerminalControl/ControlCore.cpp (L99-L100)

The comment is wrong. Just because an event has been revoked, doesn't mean it's not currently being called by another thread, right? TerminalOutput is called from a background thread and so the callback's execution might outlive the main thread's strong reference to the ControlCore instance. The fix can't be to just use get_weak() instead of this, because ControlCore is WinUI-land and thus inherently thread-unsafe and can't be destroyed on a background thread. If I'm correct we have to block in ~ControlCore until the ITerminalConnection can promise that all calls to TerminalOutput have returned.

@lhecker commented on GitHub (Aug 30, 2022): This commit seems to be related (although I don't immediately see why that would be): https://github.com/microsoft/terminal/commit/23e4d313d5dae913ed16db3cfdb318a05b63ba6a It tries to work around a deadlock issue by calling `UpdatePatternLocations` from a background thread. Even if I don't see how it might cause this, it does seem to be related because: * It's a recent change (included in https://github.com/microsoft/terminal/commit/c12987af415c5e0911d7a0a81b8494fbe6307328) * It's in `ControlCore` * `_updatePatternLocations` is called in `_terminalScrollPositionChanged` which calls `TpSetTimerEx` This PR is my attempt to fix one bug caused by that commit: https://github.com/microsoft/terminal/pull/13859 But yours seems to be another: The `TppRaiseInvalidParameter` call seems like an issue that might be caused if the `til::throttled_func` has been destroyed, making the `wil::unique_threadpool_timer` handle invalid. But why would this have worked before? The only difference I can think of is that the previous `ThrottledFunc` uses a `shared_ptr` to keep itself alive until the scheduled callback returns, but I don't see how that would be related here since no such `ThrottledFunc` is involved in `_connectionOutputHandler`. Did we just coincidentally get lucky up until now? In either case _I believe_ the underlying issue is this: https://github.com/microsoft/terminal/blob/a85d9e69ed80bdce6abb3366a0d7e842731e60e7/src/cascadia/TerminalControl/ControlCore.cpp#L99-L100 The comment is wrong. Just because an event has been revoked, doesn't mean it's not currently being called by another thread, right? `TerminalOutput` is called from a background thread and so the callback's execution might outlive the main thread's strong reference to the `ControlCore` instance. The fix can't be to just use `get_weak()` instead of `this`, because `ControlCore` is WinUI-land and thus inherently thread-unsafe and can't be destroyed on a background thread. If I'm correct we have to block in `~ControlCore` until the `ITerminalConnection` can promise that all calls to `TerminalOutput` have returned.
Author
Owner

@j4james commented on GitHub (Aug 30, 2022):

This commit seems to be related (although I don't immediately see why that would be): 23e4d31

I don't think that's to blame, because I'm also getting this crash in preview release 1.15.1862.0, which I think predates that commit.

Did we just coincidentally get lucky up until now?

I was wondering that too. I don't fully understand what's going on in that code, but my initial impression was that it shouldn't ever have worked.

If I'm correct we have to block in ~ControlCore until the ITerminalConnection can promise that all calls to TerminalOutput have returned.

Yeah, I was thinking the same thing.

@j4james commented on GitHub (Aug 30, 2022): > This commit seems to be related (although I don't immediately see why that would be): [23e4d31](https://github.com/microsoft/terminal/commit/23e4d313d5dae913ed16db3cfdb318a05b63ba6a) I don't think that's to blame, because I'm also getting this crash in preview release 1.15.1862.0, which I think predates that commit. > Did we just coincidentally get lucky up until now? I was wondering that too. I don't fully understand what's going on in that code, but my initial impression was that it shouldn't ever have worked. > If I'm correct we have to block in `~ControlCore` until the `ITerminalConnection` can promise that all calls to `TerminalOutput` have returned. Yeah, I was thinking the same thing.
Author
Owner

@ghost commented on GitHub (Sep 13, 2022):

:tada:This issue was addressed in #13882, which has now been successfully released as Windows Terminal Preview v1.16.252.🎉

Handy links:

@ghost commented on GitHub (Sep 13, 2022): :tada:This issue was addressed in #13882, which has now been successfully released as `Windows Terminal Preview v1.16.252`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.16.252) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Author
Owner

@ghost commented on GitHub (Oct 18, 2022):

:tada:This issue was addressed in #13882, which has now been successfully released as Windows Terminal v1.15.2874.🎉

Handy links:

@ghost commented on GitHub (Oct 18, 2022): :tada:This issue was addressed in #13882, which has now been successfully released as `Windows Terminal v1.15.2874`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.15.2874) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#18343