Ensure all controllers that send failure events to notification-controller send recovery events for all failures

[matheuscscp]

After support for commit status for all Flux APIs was introduced, we noticed that some controllers don't send recovery events in all cases. We need to review all the controllers that send a failure event to notification-controller and make sure that all failures are eventually recovered, i.e. a follow-up recovery event is eventually sent as well.

[Iam-Karan-Suresh]

Hi @matheuscscp! I’d like to work on this issue. If you’re okay with it, please assign it to me. Thanks!

[matheuscscp]

@Iam-Karan-Suresh Done

This needs to be worked on controller by controller, for each one you need to:

1. Find out all the places that send an error event to notification-controller
2. For each call site found out in step 1:
   2.1. Exercise that code path by forcing the associated failure, check if an error event was sent to notification-controller
   2.2. Recover from that failure, check if a success event was sent to notification-controller. If not, we need to fix it