quite old WT crashed with access violation on a machine with slow disk trying to connect to WSLv1 #16628

Closed
opened 2026-01-31 05:17:48 +00:00 by claunia · 10 comments
Owner

Originally created by @vadimkantorov on GitHub (Feb 3, 2022).

Windows Terminal version

1.11.3471.0

Windows build number

Windows 10, 21H1 19043.1466

Other Software

No response

Steps to reproduce

I have no reproducer, this problem has never occurred before. Another terminal instance launched successfully. Probably related to connection to WSLv1 service somehow.

image

image

image

image

I also saved the crash dump from within Visual Studio, just in case

Expected Behavior

No response

Actual Behavior

no crash

Originally created by @vadimkantorov on GitHub (Feb 3, 2022). ### Windows Terminal version 1.11.3471.0 ### Windows build number Windows 10, 21H1 19043.1466 ### Other Software _No response_ ### Steps to reproduce I have no reproducer, this problem has never occurred before. Another terminal instance launched successfully. Probably related to connection to WSLv1 service somehow. ![image](https://user-images.githubusercontent.com/1041752/152447634-0668c972-8ef4-42f5-8053-f58bf31265ae.png) ![image](https://user-images.githubusercontent.com/1041752/152448533-e031cdc2-07c6-427b-aa1c-cbc168f63e06.png) ![image](https://user-images.githubusercontent.com/1041752/152448606-76264828-fe17-447a-9994-73fe5273192e.png) ![image](https://user-images.githubusercontent.com/1041752/152448828-7f517f15-7108-4e06-bcbd-3be2891ca572.png) I also saved the crash dump from within Visual Studio, just in case ### Expected Behavior _No response_ ### Actual Behavior no crash
Author
Owner

@zadjii-msft commented on GitHub (Feb 4, 2022):

Yea I'm gonna need to see the dump here. The closest we've ever gotten was #11142, but that couldn't be repro'd for OP.

@zadjii-msft commented on GitHub (Feb 4, 2022): Yea I'm gonna need to see the dump here. The closest we've ever gotten was #11142, but that couldn't be repro'd for OP.
Author
Owner

@vadimkantorov commented on GitHub (Feb 4, 2022):

I think it's something related to potentially crashed WSLv1 service, but segfault is still not good. Here's the dump:

https://1drv.ms/u/s!Apx8USiTtrYmqpB_bMve6NE2ArogXg?e=kUhypV

@vadimkantorov commented on GitHub (Feb 4, 2022): I think it's something related to potentially crashed WSLv1 service, but segfault is still not good. Here's the dump: https://1drv.ms/u/s!Apx8USiTtrYmqpB_bMve6NE2ArogXg?e=kUhypV
Author
Owner

@zadjii-msft commented on GitHub (Feb 4, 2022):

f2a03016-0886-7581-fb91-71db599304e4 is the failure hash. There's actually a ton of hits here...


STACK_TEXT:  
000000e7`da1ff3e0 00007ffa`164c0b88     : 00000249`76afd0c8 00000000`00002774 00000000`00000000 00000000`00000002 : TerminalApp!winrt::impl::consume_Microsoft_Terminal_Settings_Model_ICascadiaSettings<winrt::Microsoft::Terminal::Settings::Model::ICascadiaSettings>::ActiveProfiles+0x2b
000000e7`da1ff420 00007ffa`164b2160     : 00000249`76a890f0 00000249`761d09b0 00000249`761d0a78 00000000`00000000 : TerminalApp!Jumplist::UpdateJumplist$_ResumeCoro$1+0x1b8
000000e7`da1ff4c0 00007ffa`69940bf9     : 00000249`761d09b0 00000000`7ffe0386 000000e7`da1ff6b8 00000249`761d0a78 : TerminalApp!winrt::impl::resume_background_callback+0x10
000000e7`da1ff4f0 00007ffa`6992315a     : 00000000`00000000 00000249`76a85630 00000000`00000000 00000249`6d402358 : ntdll!TppSimplepExecuteCallback+0x99
000000e7`da1ff540 00007ffa`68137034     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x68a
000000e7`da1ff840 00007ffa`69922651     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
000000e7`da1ff870 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


STACK_COMMAND:  ~6s; .ecxr ; kb

FAULTING_SOURCE_LINE:  C:\a\_work\1\s\src\cascadia\TerminalApp\Generated Files\winrt\Microsoft.Terminal.Settings.Model.h

FAULTING_SOURCE_FILE:  C:\a\_work\1\s\src\cascadia\TerminalApp\Generated Files\winrt\Microsoft.Terminal.Settings.Model.h

FAULTING_SOURCE_LINE_NUMBER:  597

SYMBOL_NAME:  TerminalApp!winrt::impl::consume_Microsoft_Terminal_Settings_Model_ICascadiaSettings<winrt::Microsoft::Terminal::Settings::Model::ICascadiaSettings>::ActiveProfiles+2b

MODULE_NAME: TerminalApp

IMAGE_NAME:  TerminalApp.dll

WERID. Bug is in Jumplist::UpdateJumplist. strongSettings is null when the coroutine resumes. That would imply that it was null when it was passed into that function.

Yea, the settings are null. Why are they null?

0:000> dx -r1 ((TerminalApp!winrt::TerminalApp::implementation::AppLogic*)0x2497611ed48)
((TerminalApp!winrt::TerminalApp::implementation::AppLogic*)0x2497611ed48)                 : 0x2497611ed48 [Type: winrt::TerminalApp::implementation::AppLogic *]
    [+0x008] vtable           [Type: winrt::impl::produce<winrt::TerminalApp::implementation::AppLogic,winrt::TerminalApp::IAppLogic>]
    [+0x010] vtable           [Type: winrt::impl::produce<winrt::TerminalApp::implementation::AppLogic,winrt::TerminalApp::IDirectKeyListener>]
    [+0x018] vtable           [Type: winrt::impl::produce<winrt::TerminalApp::implementation::AppLogic,winrt::TerminalApp::IDialogPresenter>]
    is_composing     : false [Type: bool]
    [+0x028] m_references     : 0x24976a791b0 [Type: std::atomic<unsigned __int64>]
    [+0x030] _RequestedThemeChangedHandlers [Type: winrt::event<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,enum winrt::Windows::UI::Xaml::ElementTheme> >]
    [+0x048] _SettingsChangedHandlers [Type: winrt::event<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::Windows::Foundation::IInspectable> >]
    [+0x060] _isUwp           : true [Type: bool]
    [+0x061] _isElevated      : true [Type: bool]
    [+0x068] _root            [Type: winrt::com_ptr<winrt::TerminalApp::implementation::TerminalPage>]
    [+0x070] _settings        [Type: winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings]
    [+0x078] _settingsLoadExceptionText [Type: winrt::hstring]
    [+0x080] _settingsLoadedResult : 0x00000000 (The operation completed successfully.) [Type: HRESULT]
    [+0x084] _loadedInitialSettings : false [Type: bool]
    [+0x088] _dialogLock      [Type: std::shared_mutex]
    [+0x090] _dialog          [Type: winrt::Windows::UI::Xaml::Controls::ContentDialog]
    [+0x098] _appArgs         [Type: TerminalApp::AppCommandlineArgs]
    [+0x6a0] _settingsAppArgs [Type: TerminalApp::AppCommandlineArgs]
    [+0xca8] _reloadSettings  : {...} [Type: std::shared_ptr<ThrottledFunc<0> >]
    [+0xcb8] _reloadState     [Type: til::throttled_func<0>]
    [+0xd10] _languageProfileNotifier [Type: winrt::com_ptr<winrt::TerminalApp::implementation::LanguageProfileNotifier>]
    [+0xd18] _reader          [Type: wil::unique_any_t<wil::folder_change_reader_t<wil::details::unique_storage<wil::details::resource_policy<wil::details::folder_change_reader_state *,void (__cdecl*)(wil::details::folder_change_reader_state *),&wil::details::delete_folder_change_reader_state,wistd::integral_constant<unsigned __int64,2>,wil::details::folder_change_reader_state *,wil::details::folder_change_reader_state *,0,std::nullptr_t> >,wil::err_returncode_policy> >]
    [+0xd20] _hasCommandLineArguments : false [Type: bool]
    [+0xd21] _hasSettingsStartupActions : false [Type: bool]
    [+0xd28] _warnings        : { size=0 } [Type: std::vector<enum winrt::Microsoft::Terminal::Settings::Model::SettingsLoadWarnings,std::allocator<enum winrt::Microsoft::Terminal::Settings::Model::SettingsLoadWarnings> >]
0:000> dx -r1 (*((TerminalApp!winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings *)0x2497611edb8))
(*((TerminalApp!winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings *)0x2497611edb8))                 [Type: winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings]
    [+0x000] m_ptr            : 0x0 [Type: winrt::impl::abi<winrt::Windows::Foundation::IUnknown,void>::type *]
    

Wait no that's not right. That has both isUwp and isElevated

@zadjii-msft commented on GitHub (Feb 4, 2022): f2a03016-0886-7581-fb91-71db599304e4 is the failure hash. There's actually a ton of hits here... ``` STACK_TEXT: 000000e7`da1ff3e0 00007ffa`164c0b88 : 00000249`76afd0c8 00000000`00002774 00000000`00000000 00000000`00000002 : TerminalApp!winrt::impl::consume_Microsoft_Terminal_Settings_Model_ICascadiaSettings<winrt::Microsoft::Terminal::Settings::Model::ICascadiaSettings>::ActiveProfiles+0x2b 000000e7`da1ff420 00007ffa`164b2160 : 00000249`76a890f0 00000249`761d09b0 00000249`761d0a78 00000000`00000000 : TerminalApp!Jumplist::UpdateJumplist$_ResumeCoro$1+0x1b8 000000e7`da1ff4c0 00007ffa`69940bf9 : 00000249`761d09b0 00000000`7ffe0386 000000e7`da1ff6b8 00000249`761d0a78 : TerminalApp!winrt::impl::resume_background_callback+0x10 000000e7`da1ff4f0 00007ffa`6992315a : 00000000`00000000 00000249`76a85630 00000000`00000000 00000249`6d402358 : ntdll!TppSimplepExecuteCallback+0x99 000000e7`da1ff540 00007ffa`68137034 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x68a 000000e7`da1ff840 00007ffa`69922651 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 000000e7`da1ff870 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 STACK_COMMAND: ~6s; .ecxr ; kb FAULTING_SOURCE_LINE: C:\a\_work\1\s\src\cascadia\TerminalApp\Generated Files\winrt\Microsoft.Terminal.Settings.Model.h FAULTING_SOURCE_FILE: C:\a\_work\1\s\src\cascadia\TerminalApp\Generated Files\winrt\Microsoft.Terminal.Settings.Model.h FAULTING_SOURCE_LINE_NUMBER: 597 SYMBOL_NAME: TerminalApp!winrt::impl::consume_Microsoft_Terminal_Settings_Model_ICascadiaSettings<winrt::Microsoft::Terminal::Settings::Model::ICascadiaSettings>::ActiveProfiles+2b MODULE_NAME: TerminalApp IMAGE_NAME: TerminalApp.dll ``` WERID. Bug is in `Jumplist::UpdateJumplist`. `strongSettings` is null when the coroutine resumes. That would imply that it was null when it was passed into that function. Yea, the settings are null. Why are they null? ``` 0:000> dx -r1 ((TerminalApp!winrt::TerminalApp::implementation::AppLogic*)0x2497611ed48) ((TerminalApp!winrt::TerminalApp::implementation::AppLogic*)0x2497611ed48) : 0x2497611ed48 [Type: winrt::TerminalApp::implementation::AppLogic *] [+0x008] vtable [Type: winrt::impl::produce<winrt::TerminalApp::implementation::AppLogic,winrt::TerminalApp::IAppLogic>] [+0x010] vtable [Type: winrt::impl::produce<winrt::TerminalApp::implementation::AppLogic,winrt::TerminalApp::IDirectKeyListener>] [+0x018] vtable [Type: winrt::impl::produce<winrt::TerminalApp::implementation::AppLogic,winrt::TerminalApp::IDialogPresenter>] is_composing : false [Type: bool] [+0x028] m_references : 0x24976a791b0 [Type: std::atomic<unsigned __int64>] [+0x030] _RequestedThemeChangedHandlers [Type: winrt::event<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,enum winrt::Windows::UI::Xaml::ElementTheme> >] [+0x048] _SettingsChangedHandlers [Type: winrt::event<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::Windows::Foundation::IInspectable> >] [+0x060] _isUwp : true [Type: bool] [+0x061] _isElevated : true [Type: bool] [+0x068] _root [Type: winrt::com_ptr<winrt::TerminalApp::implementation::TerminalPage>] [+0x070] _settings [Type: winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings] [+0x078] _settingsLoadExceptionText [Type: winrt::hstring] [+0x080] _settingsLoadedResult : 0x00000000 (The operation completed successfully.) [Type: HRESULT] [+0x084] _loadedInitialSettings : false [Type: bool] [+0x088] _dialogLock [Type: std::shared_mutex] [+0x090] _dialog [Type: winrt::Windows::UI::Xaml::Controls::ContentDialog] [+0x098] _appArgs [Type: TerminalApp::AppCommandlineArgs] [+0x6a0] _settingsAppArgs [Type: TerminalApp::AppCommandlineArgs] [+0xca8] _reloadSettings : {...} [Type: std::shared_ptr<ThrottledFunc<0> >] [+0xcb8] _reloadState [Type: til::throttled_func<0>] [+0xd10] _languageProfileNotifier [Type: winrt::com_ptr<winrt::TerminalApp::implementation::LanguageProfileNotifier>] [+0xd18] _reader [Type: wil::unique_any_t<wil::folder_change_reader_t<wil::details::unique_storage<wil::details::resource_policy<wil::details::folder_change_reader_state *,void (__cdecl*)(wil::details::folder_change_reader_state *),&wil::details::delete_folder_change_reader_state,wistd::integral_constant<unsigned __int64,2>,wil::details::folder_change_reader_state *,wil::details::folder_change_reader_state *,0,std::nullptr_t> >,wil::err_returncode_policy> >] [+0xd20] _hasCommandLineArguments : false [Type: bool] [+0xd21] _hasSettingsStartupActions : false [Type: bool] [+0xd28] _warnings : { size=0 } [Type: std::vector<enum winrt::Microsoft::Terminal::Settings::Model::SettingsLoadWarnings,std::allocator<enum winrt::Microsoft::Terminal::Settings::Model::SettingsLoadWarnings> >] 0:000> dx -r1 (*((TerminalApp!winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings *)0x2497611edb8)) (*((TerminalApp!winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings *)0x2497611edb8)) [Type: winrt::Microsoft::Terminal::Settings::Model::CascadiaSettings] [+0x000] m_ptr : 0x0 [Type: winrt::impl::abi<winrt::Windows::Foundation::IUnknown,void>::type *] ``` Wait no that's not right. That has both `isUwp` and `isElevated`
Author
Owner

@vadimkantorov commented on GitHub (Feb 4, 2022):

Maybe settings are null because disk is so faulty / sow that IO reading of settings failed / timed out or asked to retry?

@vadimkantorov commented on GitHub (Feb 4, 2022): Maybe settings are null because disk is so faulty / sow that IO reading of settings failed / timed out or asked to retry?
Author
Owner

@zadjii-msft commented on GitHub (Feb 4, 2022):

So I mean, easy solution is to just guess that the Jumplist::UpdateJumplist A/V is the one source, check null there, and hope it goes away. But the settings should really, really never be null there, so that's more troubling to me. At least all the crashes there are all in the same exact stack.

This might just be a blind whack a mole situation for now. Check this case and hope we don't get another a little farther down the line.

Maybe settings are null because disk is so faulty / sow that IO reading of settings failed / timed out or asked to retry?

I'd doubt that. Loading the settings happens synchronously on the UI thread before the window is even created. Above, you've already got a window, it's loaded its settings, and already determined it should launch Ubuntu. The window doesn't look like it started teardown (from looking at the stack, it's still happily in the wndproc), so it's not like the settings object got destroyed.

I think I'm debugging that pointer wrong though. There's a few layers of indirection with winrt, I don't _think_I can just cast it like that in the debugger. Regardless, the settings is null in UpdateJumplist, so we should fix that.

The alternate solution, possibly better, is to take a weak ref at the top of UpdateJumplist and then check it's still alive.

Crashes started around 12/2. That would be v1.11.3471.0 but not v1.11.2921.0 https://github.com/microsoft/terminal/compare/v1.11.2921.0...v1.11.3471.0 only thing in there that stands out is https://github.com/microsoft/terminal/pull/11852, maybe that just so happened to enlighten the backend about this issue.

@zadjii-msft commented on GitHub (Feb 4, 2022): So I mean, easy solution is to just guess that the `Jumplist::UpdateJumplist` A/V is the one source, check null there, and hope it goes away. But the settings should really, really never be null there, so that's more troubling to me. At least all the crashes there are all in the same exact stack. This might just be a blind whack a mole situation for now. Check this case and hope we don't get another a little farther down the line. > Maybe settings are null because disk is so faulty / sow that IO reading of settings failed / timed out or asked to retry? I'd doubt that. Loading the settings happens synchronously on the UI thread before the window is even created. Above, you've already got a window, it's loaded its settings, and already determined it should launch Ubuntu. The window doesn't look like it started teardown (from looking at the stack, it's still happily in the wndproc), so it's not like the settings object got destroyed. I think I'm debugging that pointer wrong though. There's a few layers of indirection with winrt, I don't _think_I can just cast it like that in the debugger. Regardless, the settings is null in `UpdateJumplist`, so we should fix that. The alternate solution, possibly better, is to take a weak ref at the top of `UpdateJumplist` and then check it's still alive. Crashes started around 12/2. That would be [v1.11.3471.0](https://github.com/microsoft/terminal/releases/tag/v1.11.3471.0) but not [v1.11.2921.0](https://github.com/microsoft/terminal/releases/tag/v1.11.2921.0) https://github.com/microsoft/terminal/compare/v1.11.2921.0...v1.11.3471.0 only thing in there that stands out is https://github.com/microsoft/terminal/pull/11852, maybe that just so happened to enlighten the backend about this issue.
Author
Owner

@vadimkantorov commented on GitHub (Feb 4, 2022):

Also, I tried to load symbols from Microsoft Symbol Server as proposed by VS, but it found nothing :( So maybe worth publishing PDF files to MS Symbol Server for public builds and previews

@vadimkantorov commented on GitHub (Feb 4, 2022): Also, I tried to load symbols from Microsoft Symbol Server as proposed by VS, but it found nothing :( So maybe worth publishing PDF files to MS Symbol Server for public builds and previews
Author
Owner

@vadimkantorov commented on GitHub (Feb 4, 2022):

The dump can now be removed from my OneDrive, right?

@vadimkantorov commented on GitHub (Feb 4, 2022): The dump can now be removed from my OneDrive, right?
Author
Owner

@vadimkantorov commented on GitHub (Feb 5, 2022):

Also, it would be nice for Terminal to add some error code explanations if the error code is known, especially for WSL sessions.

I'm getting [process exited with code 3221225786] of WSLv1 session crashes quite often. It would be nice to have some error message for this cases. Probably WSL team could inform these error messages

@vadimkantorov commented on GitHub (Feb 5, 2022): Also, it would be nice for Terminal to add some error code explanations if the error code is known, especially for WSL sessions. I'm getting `[process exited with code 3221225786]` of WSLv1 session crashes quite often. It would be nice to have some error message for this cases. Probably WSL team could inform these error messages
Author
Owner

@ghost commented on GitHub (Feb 11, 2022):

:tada:This issue was addressed in #12430, which has now been successfully released as Windows Terminal v1.12.10393.0.🎉

Handy links:

@ghost commented on GitHub (Feb 11, 2022): :tada:This issue was addressed in #12430, which has now been successfully released as `Windows Terminal v1.12.10393.0`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.12.10393.0) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Author
Owner

@ghost commented on GitHub (Feb 11, 2022):

:tada:This issue was addressed in #12430, which has now been successfully released as Windows Terminal Preview v1.13.10395.0.🎉

Handy links:

@ghost commented on GitHub (Feb 11, 2022): :tada:This issue was addressed in #12430, which has now been successfully released as `Windows Terminal Preview v1.13.10395.0`.:tada: Handy links: * [Release Notes](https://github.com/microsoft/terminal/releases/tag/v1.13.10395.0) * [Store Download](https://www.microsoft.com/store/apps/9n8g5rfz9xk3?cid=storebadge&ocid=badge)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#16628