Report to users publicly results of: Test Terminal in low-memory; slow hard disk conditions, measure/reduce working set, minimize number of used shared libraries; binary size #8903

New Issue

@vadimkantorov commented on GitHub (Jun 9, 2020):

Well, I understood that you don't have generic UI perf tracing at the moment :) It's quite unfortunate, but I'll create an issue in XAML. Let's see what they say.

I'd much rather use those performance issues to help drive improvements within the framework to benefit all Windows applications.

Me too, it's just that specific app-context is important. And tracing may require cross-linking with app-specific events. E.g. in the linked calc.exe issue one of linked PR was delaying loading some libraries and reducing working set size. Look at advice an author of a performant XAML app gave there - it's quite app-specific. If the problem is with too-large working set and loading too many UI libraries at button click, a solution could be preemptive library loading to memory when Terminal gets focus or maybe load some settings from disk preemptively in the background and not on UI thread.

But anyway, I'll create an issue and discussion can also continue there.

@vadimkantorov commented on GitHub (Jun 9, 2020): Well, I understood that you don't have generic UI perf tracing at the moment :) It's quite unfortunate, but I'll create an issue in XAML. Let's see what they say. > I'd much rather use those performance issues to help drive improvements within the framework to benefit _all_ Windows applications. Me too, it's just that specific app-context is important. And tracing may require cross-linking with app-specific events. E.g. in the linked calc.exe issue one of linked PR was delaying loading some libraries and reducing working set size. Look at advice an author of a performant XAML app gave there - it's quite app-specific. If the problem is with too-large working set and loading too many UI libraries at button click, a solution could be preemptive library loading to memory when Terminal gets focus or maybe load some settings from disk preemptively in the background and not on UI thread. But anyway, I'll create an issue and discussion can also continue there.

claunia commented

@vadimkantorov commented on GitHub (Jun 13, 2020):

So far my current/inconclusive summary (discussion continues in that issue) is that WT has a quite large WorkingSet: 65Mb wtih PrivateBytes: 200Mb (one WSLv1 tab open) according to Procexp, and uses a lot of shared libraries and also some fonts. In low memory conditions they only start loading from disk synchronously when a user interacts with the UI.

So one goal could be to have a WT app with a smaller working set and include used memory resources in key metrics

Another could be starting loading shared libraries/unswapping in background once the user focuses the app. An opposite direction could be somehow making sure that no unnecessary / very few libraries are loaded while basic UI interaction.

One way for prevention could be somehow showing a some message UI libraries are loading (but putting the message may itself be tricky if UI libraries are not loaded), but at least Windows has a way to check if memory pages are not in physical memory yet: https://stackoverflow.com/questions/51342150/how-to-determine-if-a-logical-address-is-in-physical-memory, or at least putting some message after they are loaded indicating that WT lacks memory for smooth operation.

Maybe also one way could be asking to never page WT on disk (or have such option). I'm not sure if this is possible. Right now Google Chrome always eats all memory :)

E.g. WT has Windows.UI.Xaml.Phone.dll, Windows.UI.Xaml.InkControls.dll, Windows.UI.Xaml.Maps.dll in modules list while I use WT on a regular laptop and no intention of using maps :)

Related tricks about hinting Windows to not swap out or to swap in the process in one go:
https://stackoverflow.com/questions/1039017/can-i-tell-windows-not-to-swap-out-a-particular-processes-memory
https://superuser.com/questions/600698/is-it-possible-to-hint-windows-to-bring-a-process-completely-out-of-swap

@vadimkantorov commented on GitHub (Jun 13, 2020): So far my current/inconclusive summary (discussion continues in that issue) is that WT has a quite large WorkingSet: 65Mb wtih PrivateBytes: 200Mb (one WSLv1 tab open) according to Procexp, and uses a lot of shared libraries and also some fonts. In low memory conditions they only start loading from disk synchronously when a user interacts with the UI. So one goal could be to have a WT app with a smaller working set and include used memory resources in key metrics Another could be starting loading shared libraries/unswapping in background once the user focuses the app. An opposite direction could be somehow making sure that no unnecessary / very few libraries are loaded while basic UI interaction. One way for prevention could be somehow showing a some message UI libraries are loading (but putting the message may itself be tricky if UI libraries are not loaded), but at least Windows has a way to check if memory pages are not in physical memory yet: https://stackoverflow.com/questions/51342150/how-to-determine-if-a-logical-address-is-in-physical-memory, or at least putting some message after they are loaded indicating that WT lacks memory for smooth operation. Maybe also one way could be asking to never page WT on disk (or have such option). I'm not sure if this is possible. Right now Google Chrome always eats all memory :) E.g. WT has Windows.UI.Xaml.Phone.dll, Windows.UI.Xaml.InkControls.dll, Windows.UI.Xaml.Maps.dll in modules list while I use WT on a regular laptop and no intention of using maps :) Related tricks about hinting Windows to not swap out or to swap in the process in one go: https://stackoverflow.com/questions/1039017/can-i-tell-windows-not-to-swap-out-a-particular-processes-memory https://superuser.com/questions/600698/is-it-possible-to-hint-windows-to-bring-a-process-completely-out-of-swap

claunia commented

@zadjii-msft commented on GitHub (Jun 15, 2020):

From reading that thread, this doesn't sound like the Terminal is doing anything particularly wrong here. Nothing above sounds like anything we're doing specifically, it sounds like these might be things that's universal to UWP XAML applications. I'd love to see a comparison with a dummy UWP application with just a MenuFlyout, something to isolate WT effects on memory pressure compared to a baseline UWP application.

I'd love to reduce our own memory usage, but this sounds like an issue that would be better served by driving the solution upstream. If there's a way to reduce the number of loaded libraries for the Terminal, it'll probably be useful to reduce that commitment from all UWP applications 😉

@zadjii-msft commented on GitHub (Jun 15, 2020): From reading that thread, this doesn't sound like the Terminal is doing anything particularly wrong here. Nothing above sounds like anything we're doing specifically, it sounds like these might be things that's universal to UWP XAML applications. I'd love to see a comparison with a dummy UWP application with just a MenuFlyout, something to isolate WT effects on memory pressure compared to a baseline UWP application. I'd love to reduce our own memory usage, but this sounds like an issue that would be better served by driving the solution upstream. If there's a way to reduce the number of loaded libraries for the Terminal, it'll probably be useful to reduce that commitment from _all_ UWP applications 😉

claunia commented

@vadimkantorov commented on GitHub (Jun 15, 2020):

Terminal doesn't do anything particularly wrong.

On the other hand, Terminal could do / at least discuss potential perf improvements for Terminal even without waiting for some general Windows / XAML / WinUI level improvements.

E.g. was (IL?) linker considered? (for Terminal assemblies and for WinUI assemblies to maybe eliminate unused methods in controls) Why is XAML.Phone and XAML.Maps being loaded at all? Can Terminal reduce number of used libraries? (it would probably lead to faster startup times as well) What are libraries that Terminal loads to draw the context menu? Could it forbid Windows from paging to disk a few core libraries? What is minimal free memory requirement for Terminal for comfortable use? Can we have private bytes tracked as part of release cycle and have a goal that Terminal app should work in low-memory scenarios?

This issue may be better solved upstream, but pointing out perf problems in concrete scenarios (along with numbers, measurements, some telemetry etc) should come from downstream, isn't it?

@vadimkantorov commented on GitHub (Jun 15, 2020): Terminal doesn't do anything particularly wrong. On the other hand, Terminal could do / at least discuss potential perf improvements for Terminal even without waiting for some general Windows / XAML / WinUI level improvements. E.g. was (IL?) linker considered? (for Terminal assemblies and for WinUI assemblies to maybe eliminate unused methods in controls) Why is XAML.Phone and XAML.Maps being loaded at all? Can Terminal reduce number of used libraries? (it would probably lead to faster startup times as well) What are libraries that Terminal loads to draw the context menu? Could it forbid Windows from paging to disk a few core libraries? What is minimal free memory requirement for Terminal for comfortable use? Can we have private bytes tracked as part of release cycle and have a goal that Terminal app should work in low-memory scenarios? This issue may be better solved upstream, but pointing out perf problems in concrete scenarios (along with numbers, measurements, some telemetry etc) should come from downstream, isn't it?

claunia commented

@zadjii-msft commented on GitHub (Jun 15, 2020):

I'm pretty sure IL Linker isn't really useful for a C++ project 😉

We're just using bog-standard XAML, we're really not doing anything special. I couldn't tell you why the Phone and Maps DLLs are getting loaded, they sure don't seem useful, but I honestly couldn't tell you. Hence why I'm thinking that this is a problem that should be investigated to the root of the issue. The Terminal finding a workaround is fine, but that just hides a potentially more fundamental issue with the platform.

IMO, having smaller, more atomic repros makes root-causing issues easier. It's easier for the platform to fix "I have an app that just does a MenuFlyout, and it loads all these DLLs and has this horrible perf" compared to "the MenuFlyout in the Terminal is slow". The first is easy* to debug and analyze - there's not that many moving pieces in that scenario. The second is much harder - it's easier for someone to say "well it might not be XAML's fault, it might not be the Terminal's fault", etc. Hence why I'd love to have a baseline comparison. To isolate whether this is truly the Terminal or XAML's fault.

@zadjii-msft commented on GitHub (Jun 15, 2020): I'm pretty sure IL Linker isn't really useful for a C++ project 😉 We're just using bog-standard XAML, we're really not doing anything special. I couldn't tell you why the Phone and Maps DLLs are getting loaded, they sure don't _seem_ useful, but I honestly couldn't tell you. Hence why I'm thinking that this is a problem that should be investigated to the root of the issue. The Terminal finding a workaround is fine, but that just hides a potentially more fundamental issue with the platform. IMO, having smaller, more atomic repros makes root-causing issues easier. It's easier for the platform to fix "I have an app that just does a MenuFlyout, and it loads all these DLLs and has this horrible perf" compared to "the MenuFlyout in the Terminal is slow". The first is easy\* to debug and analyze - there's not that many moving pieces in that scenario. The second is much harder - it's easier for someone to say "well it might not be XAML's fault, it might not be the Terminal's fault", etc. Hence why I'd love to have a baseline comparison. To isolate whether this is truly the Terminal or XAML's fault.

claunia commented

@vadimkantorov commented on GitHub (Jun 15, 2020):

About IL linker - you are right :) Is there some other native linker that could strip off unused parts of controls libraries and Link all Terminal / dependencies in one executable. This may or may not help. For that a testing lab setup (with Windows running in low-memory mode) should be done.

About more atomic repos - I guess you are right. Unfortunately, I at the moment do not have resources to contribute in Terminal perf testing at this level.

At the very least I'm proposing to explicitly track and measure Terminal perf / working set sizes / strartup time in low-memory mode and collect some telemetry and number of setups that have these problems (regardless of XAML / UI. this can be done by tracking major page faults and their completion time). Then the ask in XAML / WinRT will gain more priority and have more context.

@vadimkantorov commented on GitHub (Jun 15, 2020): About IL linker - you are right :) Is there some other native linker that could strip off unused parts of controls libraries and Link all Terminal / dependencies in one executable. This may or may not help. For that a testing lab setup (with Windows running in low-memory mode) should be done. About more atomic repos - I guess you are right. Unfortunately, I at the moment do not have resources to contribute in Terminal perf testing at this level. At the very least I'm proposing to explicitly track and measure Terminal perf / working set sizes / strartup time in low-memory mode and collect some telemetry and number of setups that have these problems (regardless of XAML / UI. this can be done by tracking major page faults and their completion time). Then the ask in XAML / WinRT will gain more priority and have more context.

claunia commented

@DHowett commented on GitHub (Jun 26, 2020):

With this new title, and the input in the thread, I'm yanking triage and marking this "help wanted."

@DHowett commented on GitHub (Jun 26, 2020): With this new title, and the input in the thread, I'm yanking triage and marking this "help wanted."

claunia commented

@vadimkantorov commented on GitHub (Jun 26, 2020):

@DHowett i hope help is wanted from performance engineers / Windows perf experts (from MS?)...

@vadimkantorov commented on GitHub (Jun 26, 2020): @DHowett i hope help is wanted from performance engineers / Windows perf experts (from MS?)...

claunia commented

@DHowett commented on GitHub (Jun 26, 2020):

Is it necessary for you to be snarky in most of your responses to us?

@DHowett commented on GitHub (Jun 26, 2020): Is it necessary for you to be snarky in most of your responses to us?

claunia commented

@vadimkantorov commented on GitHub (Jun 27, 2020):

It is not. I am sorry for snarky comments.

@zadjii-msft suggested that performance investigation requests to XAML team should be upon users which is very strange to me (and expectedly there is no substantial response from them). In my opinion, requesting performance investigation from regular users asks for at least some explanation or comment.

It is my genuine question if help is required of users or if help is being requested from real experts (within MS included). My wording may not be right, but I genuinely do not understand why a basic performance question (such as mere measuring working set size) is not a considered a core one and requires help from users.

@vadimkantorov commented on GitHub (Jun 27, 2020): It is not. I am sorry for snarky comments. @zadjii-msft suggested that performance investigation requests to XAML team should be upon users which is very strange to me (and expectedly there is no substantial response from them). In my opinion, requesting performance investigation from regular users asks for at least some explanation or comment. It is my genuine question if help is required of users or if help is being requested from real experts (within MS included). My wording may not be right, but I genuinely do not understand why a basic performance question (such as mere measuring working set size) is not a considered a core one and requires help from users.

claunia commented

@Rob-Wal commented on GitHub (Jul 30, 2020):

I didn't want to create a brand new issue for this.

Are there any user steps for troubleshooting or debugging wt slow startup? On one of my system if I do not run terminal for a while it takes 10-20 seconds to startup. I have gone as far as uninstalling, deleting settings, deleting anything in the install directory, etc and performing a clean install. I have tried changing the default profile to cmd. It does not matter.

The terminal app is just a blank grey box during the slow startup then the tab loads. CPU, RAM and storage seems find when this delay is happening. After a random number of so slow startups it will load instantly until its not used for a while or after system starts.

cmd, powershell and wsl startup instantly when runindependently

Win10 1909 (Version 10.0.18363.959
Terminal 1.1.2021.0 (even running wt --version takes a while for the box to show)
I am usually connected to work vpn on that pc. But issue exist when not connected.

@Rob-Wal commented on GitHub (Jul 30, 2020): I didn't want to create a brand new issue for this. Are there any user steps for troubleshooting or debugging wt slow startup? On one of my system if I do not run terminal for a while it takes 10-20 seconds to startup. I have gone as far as uninstalling, deleting settings, deleting anything in the install directory, etc and performing a clean install. I have tried changing the default profile to cmd. It does not matter. The terminal app is just a blank grey box during the slow startup then the tab loads. CPU, RAM and storage seems find when this delay is happening. After a random number of so slow startups it will load instantly until its not used for a while or after system starts. cmd, powershell and wsl startup instantly when runindependently Win10 1909 (Version 10.0.18363.959 Terminal 1.1.2021.0 (even running wt --version takes a while for the box to show) I am usually connected to work vpn on that pc. But issue exist when not connected.

claunia commented

@DHowett commented on GitHub (Jul 30, 2020):

@Rob-Wal When it's taking a while to start up, would you mind trying to run wsl -l (specifically) from another shell and seeing whether that's unusually slow? Terminal will wait a small handful of seconds for wsl to respond with a list of distributions before moving on during startup.

@DHowett commented on GitHub (Jul 30, 2020): @Rob-Wal When it's taking a while to start up, would you mind trying to run `wsl -l` (specifically) from another shell and seeing whether that's unusually slow? Terminal will wait a small handful of seconds for `wsl` to respond with a list of distributions before moving on during startup.

claunia commented

@Rob-Wal commented on GitHub (Jul 30, 2020):

@Rob-Wal When it's taking a while to start up, would you mind trying to run wsl -l (specifically) from another shell and seeing whether that's unusually slow? Terminal will wait a small handful of seconds for wsl to respond with a list of distributions before moving on during startup.

Yes, if I run wsl -l from powershell or cmd during slow startup its slower than normal. Its about as half as slow as terminal startup. Its instant when Terminal is already open or closed.

Update: I tried commenting out my wsl entry from settings, but terminal recreated it in the config file.

@Rob-Wal commented on GitHub (Jul 30, 2020): > @Rob-Wal When it's taking a while to start up, would you mind trying to run `wsl -l` (specifically) from another shell and seeing whether that's unusually slow? Terminal will wait a small handful of seconds for `wsl` to respond with a list of distributions before moving on during startup. Yes, if I run `wsl -l` from powershell or cmd during slow startup its slower than normal. Its about as half as slow as terminal startup. Its instant when Terminal is already open or closed. Update: I tried commenting out my wsl entry from settings, but terminal recreated it in the config file.

claunia commented

@Rob-Wal commented on GitHub (Aug 4, 2020):

@DHowett I was able to run some ProcMon trace today. I see the below LocalState hanging for 15seconds. The settings.json is the only file in that folder.

Is this the wait for wsl or something else?
Any ideas?

Time of Day | Process Name | PID | Operation | Path | Result | Detail | Duration | Completion Time | Command Line
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
**44:17.9** | WindowsTerminal.exe | 14548 | NotifyChangeDirectory | C:\Users\USERNAME\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState | CANCELLED | Filter: FILE_NOTIFY_CHANGE_FILE_NAME, FILE_NOTIFY_CHANGE_DIR_NAME,   FILE_NOTIFY_CHANGE_ATTRIBUTES, FILE_NOTIFY_CHANGE_SIZE,   FILE_NOTIFY_CHANGE_LAST_WRITE, FILE_NOTIFY_CHANGE_SECURITY | **15.30458** | **44:33.2** | "C:\Program   Files\WindowsApps\Microsoft.WindowsTerminal_1.1.2021.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe"

@Rob-Wal commented on GitHub (Aug 4, 2020): @DHowett I was able to run some ProcMon trace today. I see the below LocalState hanging for 15seconds. The settings.json is the only file in that folder. Is this the wait for wsl or something else? Any ideas? ``` Time of Day | Process Name | PID | Operation | Path | Result | Detail | Duration | Completion Time | Command Line -- | -- | -- | -- | -- | -- | -- | -- | -- | -- **44:17.9** | WindowsTerminal.exe | 14548 | NotifyChangeDirectory | C:\Users\USERNAME\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState | CANCELLED | Filter: FILE_NOTIFY_CHANGE_FILE_NAME, FILE_NOTIFY_CHANGE_DIR_NAME, FILE_NOTIFY_CHANGE_ATTRIBUTES, FILE_NOTIFY_CHANGE_SIZE, FILE_NOTIFY_CHANGE_LAST_WRITE, FILE_NOTIFY_CHANGE_SECURITY | **15.30458** | **44:33.2** | "C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.1.2021.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe" ```

claunia commented

@zadjii-msft commented on GitHub (Aug 4, 2020):

That's probably just the Terminal listening for changes to its LocalState directory so that it can be notified when setting.json changes, and hot-reload the settings.

If WSL is acting slowly for you, I'd recommend adding

"disabledProfileSources": ["Windows.Terminal.Wsl"]

to your settings, which will prevent the WSL profile generator from running at all.

@zadjii-msft commented on GitHub (Aug 4, 2020): That's probably just the Terminal listening for changes to its `LocalState` directory so that it can be notified when `setting.json` changes, and hot-reload the settings. If WSL is acting slowly for you, I'd recommend adding ```json "disabledProfileSources": ["Windows.Terminal.Wsl"] ``` to your settings, which will prevent the WSL profile generator from running at all.

claunia commented

@Rob-Wal commented on GitHub (Aug 5, 2020):

@zadjii-msft Thank you but this did not help. I also did an uninstall/re-install before adding the setting. My WSL instance is not available so I know the setting worked. The native console when running cmd, bash or powershell launch in 1sec. 🤷🏾‍♂️

@Rob-Wal commented on GitHub (Aug 5, 2020): @zadjii-msft Thank you but this did not help. I also did an uninstall/re-install before adding the setting. My WSL instance is not available so I know the setting worked. The native console when running cmd, bash or powershell launch in 1sec. 🤷🏾‍♂️

claunia commented

@Rob-Wal commented on GitHub (Aug 7, 2020):

@zadjii-msft Thank you but this did not help. I also did an uninstall/re-install before adding the setting. My WSL instance is not available so I know the setting worked. The native console when running cmd, bash or powershell launch in 1sec. 🤷🏾‍♂️

After a lot of time troubleshooting, I have finally found the root cause of the slow startup of WT on my system.

So I spun up windows sandbox on the same system and wt had no issues in the sandbox. I unregistered wsl distro and completely removed wsl (I backuped up first), no change. I started to compare processes on my other pc with no issues and AV was a difference.

It was the F&(3&*ing AV!

I was able to reproduce and solve the issue by unloading and starting the AV. I added WindowsTerminal.exe to the AV Trusted Applications list and WT has been consistently starting fast. Thats the solution I have been running with. Nothing in the AV logs at all. Its TrenMicro OCSE 11.x

@Rob-Wal commented on GitHub (Aug 7, 2020): > @zadjii-msft Thank you but this did not help. I also did an uninstall/re-install before adding the setting. My WSL instance is not available so I know the setting worked. The native console when running cmd, bash or powershell launch in 1sec. 🤷🏾‍♂️ After a lot of time troubleshooting, I have finally found the root cause of the slow startup of WT on my system. So I spun up windows sandbox on the same system and wt had no issues in the sandbox. I unregistered wsl distro and completely removed wsl (I backuped up first), no change. I started to compare processes on my other pc with no issues and AV was a difference. It was the F&(3&*ing AV! I was able to reproduce and solve the issue by unloading and starting the AV. I added WindowsTerminal.exe to the AV Trusted Applications list and WT has been consistently starting fast. Thats the solution I have been running with. Nothing in the AV logs at all. Its TrenMicro OCSE 11.x

claunia commented

@vadimkantorov commented on GitHub (Dec 11, 2020):

In https://github.com/microsoft/terminal/pull/8489#issuecomment-743459395 @DHowett hinted that in fact Terminal is already getting tested for working set (and probably for waits on paging in low-memory conditions?). It would be really nice to have these measurements public at some point!

@vadimkantorov commented on GitHub (Dec 11, 2020): In https://github.com/microsoft/terminal/pull/8489#issuecomment-743459395 @DHowett hinted that in fact Terminal is already getting tested for working set (and probably for waits on paging in low-memory conditions?). It would be really nice to have these measurements public at some point!

claunia commented