Skip to content

Dehydrate timeout_notifications nightly task#2877

Open
whabanks wants to merge 2 commits into
mainfrom
fix/dehydrate-timeout-notifications-task
Open

Dehydrate timeout_notifications nightly task#2877
whabanks wants to merge 2 commits into
mainfrom
fix/dehydrate-timeout-notifications-task

Conversation

@whabanks

Copy link
Copy Markdown
Contributor

Summary | Résumé

In the same vein as 2876, this PR attempts to memory optimize timeout_notifications by not fully hydrating the list of notifications objects that it timed out. We fetch this list of notifications post-timeout in order to construct API callbacks for services with them enabled.

What we were doing before

timeout_notifications fetched and fully loaded the ORM object graph for all notifications that were timed out. We returned a list of fully loaded notifications in order to queue up notification status callbacks for services who have callbacks set up.

Why it's not great & the change

It's unnecessary to fully load these objects into session as we only need a subset of the notification's properties to construct the API callback. Instead of fully hydrating each notification object graph, we instead only return the subset of columns needed to construct the API callback after the notification is timed out.

Secondary bug fix

This PR also fixes a bug where we were querying the notifications BEFORE timing them out, resulting in stale statuses being used when constructing the API callback

Related Issues | Cartes liées

Test instructions | Instructions pour tester la modification

Release Instructions | Instructions pour le déploiement

None.

Reviewer checklist | Liste de vérification du réviseur

  • This PR does not break existing functionality.
  • This PR does not violate GCNotify's privacy policies.
  • This PR does not raise new security concerns. Refer to our GC Notify Risk Register document on our Google drive.
  • This PR does not significantly alter performance.
  • Additional required documentation resulting of these changes is covered (such as the README, setup instructions, a related ADR or the technical documentation).

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

- Previously `timeout_notifications` fetched and fully loaded the ORM
object graph for all notifications that were timed out. We returned a
list of fully loaded notifications in order to queue up notification
status callbacks for services who have callbacks set up.
- It's unnecessary to fully load these objects into session as we only
  need a subset of the notification's properties to construct the API
callback.
- Instead of fully hydrating each notification object graph, we instead only
  return the subset of columns needed to construct the API callback
after the notification is timed out.
- This PR also fixes a bug where we were querying the notifications
  BEFORE timing them out, resulting in stale statuses being used when
constructing the API callback
@whabanks whabanks marked this pull request as ready for review May 26, 2026 21:51
@whabanks whabanks requested a review from jimleroyer as a code owner May 26, 2026 21:51
Copilot AI review requested due to automatic review settings May 26, 2026 21:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the timeout_notifications nightly task/DAO flow to avoid hydrating full Notification ORM graphs when building delivery-status callbacks, and fixes the callback “stale status” issue by updating statuses before constructing callback payloads.

Changes:

  • Refactors _timeout_notifications to update rows first and return lightweight callback-ready data (instead of ORM Notification objects).
  • Extracts Notification.formatted_status computation into a scalar helper (compute_formatted_status) to support dehydrated callback rows.
  • Updates/extends tests to validate callback payload status formatting and adjusts nightly-task callback expectations.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
app/dao/notifications_dao.py Changes timeout logic to bulk-update then return reduced callback data instead of ORM notifications.
app/models.py Adds compute_formatted_status and reuses it from Notification.formatted_status.
app/types.py Introduces NotificationCallbackData dataclass for callback dispatch payload shaping.
tests/app/dao/notification_dao/test_notification_dao.py Adds assertions for callback formatted_status and new edge-case coverage for formatted status computation.
tests/app/celery/test_nightly_tasks.py Updates callback task argument expectations to match dehydrated return values.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/app/dao/notification_dao/test_notification_dao.py Outdated
Comment thread app/models.py
Comment thread app/dao/notifications_dao.py Outdated
Comment thread app/dao/notifications_dao.py
Comment thread app/dao/notifications_dao.py Outdated
-The previous approach of select-then-update could have still
materialized a lot of objects in memory upon execution and introduced a
potential race condition.
- In the refactored approach we use an update CTE for a single atomic
  transaction that leverages UPDATE ... RETURNING directly via
SQLAlchemy core.
bounce_response = update.get("bounce_response")
provider_response = update.get("provider_response")
feedback_reason = update.get("feedback_reason")
for u in updates:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was colliding with the update import from sqlalchemy.

@@ -554,17 +557,63 @@ def dao_delete_notifications_by_id(notification_id):


def _timeout_notifications(current_statuses, new_status, timeout_start, updated_at):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add batching to this task in the future if we're still seeing issues with performance here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants