slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1453239	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128735
1449198	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66796
1449199	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66796
1453213	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67055
1447373	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54513
1453114	dotnet/maui	Controls CollectionView	Log	dotnet/maui#31769
1453169	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#129098
2994349	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSStripSymbols osx x64 perfiphone17 net11.0	Log
1453140	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34637
1453069	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32851
1453066	dotnet/maui	Controls Layout	Log	dotnet/maui#28258
1453131	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129091
2994255	dotnet-maui	RunOniOS_MauiRelease ARM64	Log
2994251	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSStripSymbols osx x64 perfiphone17 net11.0	Log
1453021	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67015
1453015	dotnet/sdk	TestBuild: macOS (x64)	Log
1451449	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#65245
1453009	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66487
2994177	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone17 net11.0	Log
1452940	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129092
1452872	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2994025	dotnet-performance	Performance maui_scenarios_ios Default Mono_Debug osx 26 x64 iPhone17 main	Log
1452758	dotnet/aspnetcore	Test: macOS	Log
2993973	dotnet-aspnetcore	Build: macOS arm64	Log
1452723	dotnet/aspnetcore	Build: macOS x64	Log
1452514	dotnet/maui	Controls CollectionView	Log
1452617	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129070
1452310	dotnet/maui	Controls Entry	Log
2993704	dotnet-efcore	macOS	Log
2993607	dotnet-efcore	macOS	Log
1452386	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1451639	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34600
1452263	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129063
1452256	dotnet/aspnetcore	Build: macOS x64	Log
1452250	dotnet/runtime	osx-x64 Debug PALTests	Log
1451780	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34758
1452198	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129061
1452183	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129060
1452157	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129019
1452147	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129059
1452135	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128524
1451755	dotnet/maui	Controls CollectionView	Log	dotnet/maui#29910
1451909	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54610
1452119	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#127190
1451842	dotnet/maui	Controls CollectionView	Log
1452026	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1451992	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129016
2993419	dotnet-sdk	AoT: macOS (x64)	Log	#61867
2993418	dotnet-sdk	AoT: macOS (x64)	Log	#61742
1450992	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54595
1452155	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#129054
1451889	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#127675
1451968	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129017
1451946	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129053
1452116	dotnet/sdk	TestBuild: macOS (x64)	Log
1451887	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1452106	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#64535
1452100	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54605
1452070	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#129058
1451710	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128728
1452076	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67048
1450548	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128927
1452060	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66859
1451877	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1452051	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66964
1451869	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#124721
1451981	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1451823	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129007
1451911	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#129051
1451910	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#53518
2993360	dotnet-efcore	macOS	Log
1451880	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1451882	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1451876	dotnet/runtime	osx-x64 Debug PALTests	Log
2993278	dotnet-maui	RunOniOS_MauiDebug ARM64	Log
1451847	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#129049
1451775	dotnet/maui	Controls Layout	Log
1451455	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128997
1451274	dotnet/maui	Controls CollectionView	Log
1451833	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66808
1451749	dotnet/maui	Controls Layout	Log	dotnet/maui#32851
1451733	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128479
2993258	dotnet-maui	Controls Shell	Log
1451742	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R_CG2 osx x64 checked	Log
2993180	dotnet-sdk	AoT: macOS (x64)	Log	#61852
1451069	dotnet/maui	Controls TabbedPage,TableView,TimePicker,TitleView,ToolbarItem,Triggers	Log
1451717	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128962
1451777	dotnet/sdk	TestBuild: macOS (x64)	Log
2993233	dotnet-sdk	Darwin	Log	#61746
2993234	dotnet-sdk	Darwin	Log	#61745
2993232	dotnet-efcore	macOS	Log	#61753
1451737	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128479
1451606	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129012
1451642	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128950
1451681	dotnet/aspnetcore	Build: macOS x64	Log
1451674	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54622
1451216	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#125436
2993178	dotnet-sdk	Darwin	Log	#61849
2993177	dotnet-sdk	Darwin_AoT_Tests	Log	#61752
1451645	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128950
Displaying 100 of 2133 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
22	793	2133

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.