Skip to content

Add Amazon ECR repository operators#69886

Open
AlejandroMorgante wants to merge 2 commits into
apache:mainfrom
AlejandroMorgante:add-ecr-repository-operators
Open

Add Amazon ECR repository operators#69886
AlejandroMorgante wants to merge 2 commits into
apache:mainfrom
AlejandroMorgante:add-ecr-repository-operators

Conversation

@AlejandroMorgante

@AlejandroMorgante AlejandroMorgante commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Add operators, unit tests, docs, and a system test for creating, configuring policies for, and deleting Amazon ECR repositories.

This provides reusable repository lifecycle tasks for container-based AWS workflows such as AgentCore. SageMaker system tests currently implement the same setup and cleanup pattern with direct SDK calls, showing that the functionality is useful beyond a single service.

Existing usage

The SageMaker system test currently manages the ECR repository lifecycle with direct Boto3 calls:

These operators make the repository setup, access-policy configuration, and cleanup reusable for future system tests and Dags instead of requiring service-specific helper tasks.

Operators

  • EcrCreateRepositoryOperator - creates an ECR repository, exposes the Boto3 request parameters, and returns the complete service response.
  • EcrSetRepositoryPolicyOperator - applies an ECR repository policy, supports policy replacement, and returns the complete service response.
  • EcrDeleteRepositoryOperator - deletes an ECR repository, supports forced deletion, and returns the complete service response.

All three operators reuse the existing EcrHook for AWS authentication and client creation, then call the Boto3 client directly instead of adding one-to-one hook wrappers. The ECR operations are synchronous, so they do not require waiters, triggers, or deferrable execution.

Changes

  • providers/amazon/src/airflow/providers/amazon/aws/operators/ecr.py - new ECR repository operators.
  • providers/amazon/provider.yaml - operator and documentation registration.
  • providers/amazon/src/airflow/providers/amazon/get_provider_info.py - matching generated provider metadata.
  • providers/amazon/docs/operators/ecr.rst - operator documentation.
  • providers/amazon/tests/system/amazon/aws/example_ecr.py - self-contained create, set-policy, and delete lifecycle system test.
  • Unit tests for all three operators.

Validation

Validated end-to-end against a real AWS environment. The system test created an ECR repository, applied a repository policy, and deleted the repository during teardown.

The system test can be reproduced from the Airflow checkout with AWS credentials forwarded to Breeze, AWS_PROFILE and AWS_REGION configured in files/airflow-breeze-config/environment_variables.env, and a unique environment ID:

SYSTEM_TESTS_ENV_ID=<unique-id> \
breeze testing system-tests \
  --forward-credentials \
  --test-timeout 240 \
  providers/amazon/tests/system/amazon/aws/example_ecr.py \
  -q

Latest result: 1 passed in 11.25s. CloudTrail recorded CreateRepository, SetRepositoryPolicy, and DeleteRepository from their respective Airflow operators in the same Dag run. A post-test lookup confirmed the repository no longer exists. The focused hook and operator unit tests also pass: 21 passed.

Additional validation includes mypy, Ruff and provider static checks, and the Amazon provider documentation build.


Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

@AlejandroMorgante AlejandroMorgante force-pushed the add-ecr-repository-operators branch 2 times, most recently from 0f83ffd to dc2b72f Compare July 15, 2026 00:19

@vincbeck vincbeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty solid!

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 15, 2026

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just a few minor suggestions.

Comment thread providers/amazon/docs/operators/ecr.rst
Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/ecr.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/ecr.py
Comment thread providers/amazon/src/airflow/providers/amazon/aws/operators/ecr.py
Comment thread providers/amazon/tests/system/amazon/aws/example_ecr.py

@o-nikolas o-nikolas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's here looks pretty clean, but it's only adding "thin" Operators. There are no sensors, Triggers, Waiters, etc. Should we expect to have any of those for the ECR usecase? Do we expect it to be very fast in all cases? Even failure paths where waits and retries may happen at the boto level?

@SameerMesiah97

Copy link
Copy Markdown
Contributor

What's here looks pretty clean, but it's only adding "thin" Operators. There are no sensors, Triggers, Waiters, etc. Should we expect to have any of those for the ECR usecase? Do we expect it to be very fast in all cases? Even failure paths where waits and retries may happen at the boto level?

I was thinking the same thing. I briefly considered the suggestion to add deferrable mode to the create and delete repository operators but I was hesistant because I was not confident that they would be long-running processes. If @AlejandroMorgante can do some basic manual tests to see how long these operations take on average, that would be great. We can decide if waiters, triggers etc are worth it.

I do agree the operators are thin, which does call into question their value. But my understanding is that these operators fill a gap in the API at the highest level of abstraction. It would be natural for users to expect them for ECR (as I imagine that this is a fairly popular AWS service).

Container-based AWS workflows need reusable repository setup and cleanup. AgentCore deployments share this lifecycle, and the SageMaker system tests currently implement it with direct SDK calls.
@AlejandroMorgante AlejandroMorgante force-pushed the add-ecr-repository-operators branch from dc2b72f to 9a02caa Compare July 16, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:amazon AWS/Amazon - related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants