Add Amazon ECR repository operators#69886
Conversation
0f83ffd to
dc2b72f
Compare
SameerMesiah97
left a comment
There was a problem hiding this comment.
Looks good. Just a few minor suggestions.
o-nikolas
left a comment
There was a problem hiding this comment.
What's here looks pretty clean, but it's only adding "thin" Operators. There are no sensors, Triggers, Waiters, etc. Should we expect to have any of those for the ECR usecase? Do we expect it to be very fast in all cases? Even failure paths where waits and retries may happen at the boto level?
I was thinking the same thing. I briefly considered the suggestion to add deferrable mode to the create and delete repository operators but I was hesistant because I was not confident that they would be long-running processes. If @AlejandroMorgante can do some basic manual tests to see how long these operations take on average, that would be great. We can decide if waiters, triggers etc are worth it. I do agree the operators are thin, which does call into question their value. But my understanding is that these operators fill a gap in the API at the highest level of abstraction. It would be natural for users to expect them for ECR (as I imagine that this is a fairly popular AWS service). |
Container-based AWS workflows need reusable repository setup and cleanup. AgentCore deployments share this lifecycle, and the SageMaker system tests currently implement it with direct SDK calls.
dc2b72f to
9a02caa
Compare
Add operators, unit tests, docs, and a system test for creating, configuring policies for, and deleting Amazon ECR repositories.
This provides reusable repository lifecycle tasks for container-based AWS workflows such as AgentCore. SageMaker system tests currently implement the same setup and cleanup pattern with direct SDK calls, showing that the functionality is useful beyond a single service.
Existing usage
The SageMaker system test currently manages the ECR repository lifecycle with direct Boto3 calls:
These operators make the repository setup, access-policy configuration, and cleanup reusable for future system tests and Dags instead of requiring service-specific helper tasks.
Operators
EcrCreateRepositoryOperator- creates an ECR repository, exposes the Boto3 request parameters, and returns the complete service response.EcrSetRepositoryPolicyOperator- applies an ECR repository policy, supports policy replacement, and returns the complete service response.EcrDeleteRepositoryOperator- deletes an ECR repository, supports forced deletion, and returns the complete service response.All three operators reuse the existing
EcrHookfor AWS authentication and client creation, then call the Boto3 client directly instead of adding one-to-one hook wrappers. The ECR operations are synchronous, so they do not require waiters, triggers, or deferrable execution.Changes
providers/amazon/src/airflow/providers/amazon/aws/operators/ecr.py- new ECR repository operators.providers/amazon/provider.yaml- operator and documentation registration.providers/amazon/src/airflow/providers/amazon/get_provider_info.py- matching generated provider metadata.providers/amazon/docs/operators/ecr.rst- operator documentation.providers/amazon/tests/system/amazon/aws/example_ecr.py- self-contained create, set-policy, and delete lifecycle system test.Validation
Validated end-to-end against a real AWS environment. The system test created an ECR repository, applied a repository policy, and deleted the repository during teardown.
The system test can be reproduced from the Airflow checkout with AWS credentials forwarded to Breeze,
AWS_PROFILEandAWS_REGIONconfigured infiles/airflow-breeze-config/environment_variables.env, and a unique environment ID:Latest result:
1 passed in 11.25s. CloudTrail recordedCreateRepository,SetRepositoryPolicy, andDeleteRepositoryfrom their respective Airflow operators in the same Dag run. A post-test lookup confirmed the repository no longer exists. The focused hook and operator unit tests also pass:21 passed.Additional validation includes mypy, Ruff and provider static checks, and the Amazon provider documentation build.
Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-5) following the guidelines