The bot currently supports excluding flaky tests to allow merging changes that are timing out:
However, occasionally when using diff testing and touching a lot of tests, the 100 count can easily exceed the default 10m timeout value even if the tests may not be flaky. This is especially the case when running on underpowered CI instances.
It would be quite handy if on a exception basis, an admin could overwrite the count and/or timeout when a change is otherwise sound.
Perhaps something like
/flaketimeout 20m
/flakeattempts 50
Ignoring the flaky test results could potentially be a way to cover for tests that are too slow AND introduce flakyness that will surface later on.
The bot currently supports excluding flaky tests to allow merging changes that are timing out:
However, occasionally when using diff testing and touching a lot of tests, the 100 count can easily exceed the default 10m timeout value even if the tests may not be flaky. This is especially the case when running on underpowered CI instances.
It would be quite handy if on a exception basis, an admin could overwrite the count and/or timeout when a change is otherwise sound.
Perhaps something like
Ignoring the flaky test results could potentially be a way to cover for tests that are too slow AND introduce flakyness that will surface later on.