starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

Some of the unittests are flaky in CI #6119

New Issue
Closed
opened 2026-01-31 00:30:16 +00:00 by claunia · 6 comments
claunia commented 2026-01-31 00:30:16 +00:00
Owner
Copy Link

Originally created by @zadjii-msft on GitHub (Jan 23, 2020).

Originally assigned to: @miniksa on GitHub.

FYI, the current build failure is just the unit tests timing out, which seems to happen quite often in the CI build. I don't think there's actually anything wrong with the code itself.

Originally posted by @j4james in https://github.com/microsoft/terminal/pull/3943#issuecomment-577683540

A case in point build: link to ADO. (EDIT: Okay this is really bad now )

This has been happening for a couple weeks now, where sometimes a test run in CI will hang. This time it happened somewhere around these tests:

  √ TestResize
  √ TestCursorVisibility [15ms]
  √ ConptyOutputTestCanary [15ms]
  √ SimpleWriteOutputTest [15ms]
  √ WriteTwoLinesUsesNewline
  √ WriteAFewSimpleLines
##[error]The operation was canceled.
Finishing: Run Unit Tests

This has happened in the past before and was notoriously hard to pin down - last time it had to do with how VtIo/VtRenderer was getting dtor'd. Hopefully we can figure out what's causing this.

As an additional note - there's vso_ut.cmd in the repo that attempts to run the unittests the same way ADO does, but that script hasn't been updated in a while.

Originally created by @zadjii-msft on GitHub (Jan 23, 2020). Originally assigned to: @miniksa on GitHub. > FYI, the current build failure is just the unit tests timing out, which seems to happen quite often in the CI build. I don't think there's actually anything wrong with the code itself. _Originally posted by @j4james in https://github.com/microsoft/terminal/pull/3943#issuecomment-577683540_ A case in point build: [link to ADO](https://dev.azure.com/ms/Terminal/_build/results?buildId=59723&view=logs&jobId=0f660c0a-4423-5c5d-276b-f98b39c69e13&j=0f660c0a-4423-5c5d-276b-f98b39c69e13&t=ecf341a7-dafd-5ce6-eabd-86286d881b78). (EDIT: Okay this is _[really](https://dev.azure.com/ms/Terminal/_build/results?buildId=59721&view=logs&jobId=0f660c0a-4423-5c5d-276b-f98b39c69e13&j=0f660c0a-4423-5c5d-276b-f98b39c69e13&t=ecf341a7-dafd-5ce6-eabd-86286d881b78) bad now_ ) This has been happening for a couple weeks now, where sometimes a test run in CI will hang. This time it happened somewhere around these tests: ``` √ TestResize √ TestCursorVisibility [15ms] √ ConptyOutputTestCanary [15ms] √ SimpleWriteOutputTest [15ms] √ WriteTwoLinesUsesNewline √ WriteAFewSimpleLines ##[error]The operation was canceled. Finishing: Run Unit Tests ``` This has happened in the past before and was notoriously hard to pin down - last time it had to do with how `VtIo`/`VtRenderer` was getting dtor'd. Hopefully we can figure out what's causing this. As an additional note - there's [vso_ut.cmd](https://github.com/microsoft/terminal/blob/master/tools/vso_ut.cmd) in the repo that attempts to run the unittests the same way ADO does, but that script hasn't been updated in a while.
claunia commented 2026-01-31 00:30:18 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jan 23, 2020):

FYI, PR #4295 caused the unit tests to time out yesterday as well (link to ADO). The exact location where it stops is slightly different, but it looks to me like the test execution order is not guaranteed - I'm guessing because it's running multiple tests in parallel? If that's the case, it may be an interaction between two test running at the same time that is the source of the issue.

@j4james commented on GitHub (Jan 23, 2020): FYI, PR #4295 caused the unit tests to time out yesterday as well ([link to ADO](https://dev.azure.com/ms/Terminal/_build/results?buildId=59699&view=logs&jobId=0f660c0a-4423-5c5d-276b-f98b39c69e13&j=0f660c0a-4423-5c5d-276b-f98b39c69e13&t=ecf341a7-dafd-5ce6-eabd-86286d881b78)). The exact location where it stops is slightly different, but it looks to me like the test execution order is not guaranteed - I'm guessing because it's running multiple tests in parallel? If that's the case, it may be an interaction between two test running at the same time that is the source of the issue.
claunia commented 2026-01-31 00:30:18 +00:00
Author
Owner
Copy Link

@DHowett-MSFT commented on GitHub (Jan 23, 2020):

Yanking triage- we see this phenomenon everywhere, and we should close on why it happens.

@DHowett-MSFT commented on GitHub (Jan 23, 2020): Yanking triage- we see this phenomenon everywhere, and we should close on why it happens.
claunia commented 2026-01-31 00:30:18 +00:00
Author
Owner
Copy Link

@j4james commented on GitHub (Jan 24, 2020):

Anyone wanting to try and reproduce this locally, I think this is what the test_cmd variable needs to be updated to in the vso_ut.cmd script to match the CI server:

set test_cmd=%VSTEST_PATH% ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\ConAdapter.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\ConParser.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Conhost.Interactivity.Win32.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Conhost.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Terminal.Core.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\TextBuffer.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Types.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\UnitTests_TerminalApp\Terminal.App.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\til.Unit.Tests.dll" ^
 /Settings:"%OPENCON%\src\unit.tests.%ARCH%.runsettings" ^
 /EnableCodeCoverage ^
 /logger:trx ^
 /TestAdapterPath:"%OPENCON%" ^
 %*

Also note that you'll likely have to update the VSTEST_PATH to match whatever version of Visual Studio you have installed.

That said, I've run this test probably dozens of times now, sometimes multiple instances at the same time, and I've not yet found a way to make it hang.

@j4james commented on GitHub (Jan 24, 2020): Anyone wanting to try and reproduce this locally, I think this is what the `test_cmd` variable needs to be updated to in the `vso_ut.cmd` script to match the CI server: ```bat set test_cmd=%VSTEST_PATH% ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\ConAdapter.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\ConParser.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Conhost.Interactivity.Win32.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Conhost.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Terminal.Core.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\TextBuffer.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Types.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\UnitTests_TerminalApp\Terminal.App.Unit.Tests.dll" ^ "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\til.Unit.Tests.dll" ^ /Settings:"%OPENCON%\src\unit.tests.%ARCH%.runsettings" ^ /EnableCodeCoverage ^ /logger:trx ^ /TestAdapterPath:"%OPENCON%" ^ %* ``` Also note that you'll likely have to update the `VSTEST_PATH` to match whatever version of Visual Studio you have installed. That said, I've run this test probably dozens of times now, sometimes multiple instances at the same time, and I've not yet found a way to make it hang.
claunia commented 2026-01-31 00:30:18 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jan 29, 2020):

Welp, one of the builds from #4382 still got stuck, so that one wasn't the fix.

@zadjii-msft commented on GitHub (Jan 29, 2020): Welp, one of the builds from #4382 still got stuck, so that one wasn't the fix.
claunia commented 2026-01-31 00:30:19 +00:00
Author
Owner
Copy Link

@miniksa commented on GitHub (Jan 29, 2020):

#4384 is where I've been playing around and trying to fix this.

I've identified that Terminal.Core.Unit.Tests.dll is the one that appears to be causing the problem.

EDIT:

  • I ran it tons of times with just the tests. When I have every test but that one, it's fine. When I reintroduce that one, it hangs.
@miniksa commented on GitHub (Jan 29, 2020): #4384 is where I've been playing around and trying to fix this. I've identified that `Terminal.Core.Unit.Tests.dll` is the one that appears to be causing the problem. EDIT: - I ran it tons of times with just the tests. When I have every test but that one, it's fine. When I reintroduce that one, it hangs.
claunia commented 2026-01-31 00:30:19 +00:00
Author
Owner
Copy Link

@miniksa commented on GitHub (Jan 29, 2020):

OK, I also ran it with only the terminal core tests and it hangs. So it's that guy. I'll try to dig in.

@miniksa commented on GitHub (Jan 29, 2020): OK, I also ran it with only the terminal core tests and it hangs. So it's that guy. I'll try to dig in.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Build Issue-Task Needs-Author-Feedback Needs-Tag-Fix Priority-0 Product-Meta Resolution-No-Repro
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#6119
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?