AWS Airflow - System test description

System test Requirements Description Notes
example_appflow_run N/A Verify appflow run operator
example_athena N/A Verify athena operator
example_batch
  • IAM role ARN
  • Subnets
  • Security groups
Verify batch operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AWSBatchFullAccess
    • CloudWatchLogsFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_cloudformation N/A Verify cloudformation operator
example_datasync
  • IAM role ARN
Verify datasync operator
example_dms
  • IAM role ARN
Verify dms operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonRDSFullAccess
    • AmazonEC2FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"dms:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_dynamodb N/A Verify dynamodb operator
example_dynamodb_to_s3 N/A Verify dynamodb to s3 operator
example_ec2 N/A Verify ec2 operator
example_ecs
  • Cluster name
  • Subnets
Verify ecs operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonECS_FullAccess
    • service-role/AmazonECSTaskExecutionRolePolicy
    • CloudWatchLogsFullAccess
example_ecs_fargate
  • Subnets
  • Security groups
Verify ecs fargate operator
example_eks_with_fargate_in_one_step
  • Cluster IAM role ARN
  • Fargate pod IAM role ARN
  • Subnets
Verify eks with fargate in one step operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEC2FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"eks:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["iam:GetRole","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_eks_with_fargate_profile
  • Cluster IAM role ARN
  • Fargate pod IAM role ARN
  • Subnets
Verify eks with fargate profile operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEC2FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"eks:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["iam:GetRole","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_eks_with_nodegroup_in_one_step
  • IAM role ARN
  • Subnets
Verify eks with nodegroup in one step operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEKSWorkerNodePolicy
    • AmazonEKSClusterPolicy
    • AmazonEC2ContainerRegistryReadOnly
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["eks:Describe*","eks:List*","eks:CreateNodegroup","eks:DeleteNodegroup","eks:CreateCluster","eks:DeleteCluster","eks:TagResource","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["ec2:CreateLaunchTemplate","ec2:DeleteLaunchTemplate","ec2:DescribeLaunchTemplates","ec2:DescribeLaunchTemplateVersions","ec2:CreateTags","ec2:RunInstances"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_eks_with_nodegroups
  • IAM role ARN
  • Subnets
Verify eks with nodegroups operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEKSWorkerNodePolicy
    • AmazonEKSClusterPolicy
    • AmazonEC2ContainerRegistryReadOnly
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["eks:Describe*","eks:List*","eks:CreateNodegroup","eks:DeleteNodegroup","eks:CreateCluster","eks:DeleteCluster","eks:TagResource","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["ec2:CreateLaunchTemplate","ec2:DeleteLaunchTemplate","ec2:DescribeLaunchTemplates","ec2:DescribeLaunchTemplateVersions","ec2:CreateTags","ec2:RunInstances"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_emr
  • Execution IAM role ARN
  • Bucket name
Verify emr operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonElasticMapReduceFullAccess
    • AmazonSSMReadOnlyAccess
example_emr_eks
  • IAM role ARN
  • Subnets
  • Job IAM role ARN
  • Job IAM role name
Verify emr eks operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEC2FullAccess
    • IAMReadOnlyAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["emr-containers:*","eks:*","iam:GetRole","iam:UpdateAssumeRolePolicy","iam:CreateServiceLinkedRole","iam:CreateOpenIDConnectProvider","iam:DeleteOpenIDConnectProvider","iam:TagOpenIDConnectProvider"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_emr_serverless
  • IAM role ARN
Verify emr serverless operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":["emr-serverless:*","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_eventbridge N/A Verify eventbridge operator
example_glue
  • IAM role ARN
Verify glue operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • Amazons3FullAccess
    • AWSGlueConsoleFullAccess
    • CloudWatchLogsFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_glue_databrew
  • IAM role ARN
Verify glue databrew operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • Amazons3FullAccess
    • AwsGlueDataBrewFullAccessPolicy
example_google_api_youtube_to_s3
  • Secret ARN
Verify google api youtube to s3 operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"secretsmanager:GetSecretValue","Effect":"Allow","Resource":"arn:aws:secretsmanager::*****:secret:<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_http_to_s3 N/A Verify http to s3 operator
example_lambda
  • IAM role ARN
Verify lambda operator
example_local_to_s3 N/A Verify local to s3 operator
example_neptune N/A Verify neptune operator
example_quicksight N/A Verify quicksight operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"quicksight:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_rds_event N/A Verify rds event operator
example_rds_export
  • KMS key id
  • IAM role ARN
Verify rds export operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonRDSFullAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_rds_instance N/A Verify rds instance operator
example_rds_snapshot N/A Verify rds snapshot operator
example_redshift
  • Security group
  • Cluster subnet group
Verify redshift operator
example_redshift_s3_transfers
  • Security group
  • Cluster subnet group
Verify redshift s3 transfers operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonS3FullAccess
    • AmazonVPCFullAccess
example_s3 N/A Verify s3 operator
example_s3_to_sql
  • Security group
  • Cluster subnet group
Verify s3 to sql operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonS3FullAccess
    • AmazonVPCFullAccess
example_sagemaker
  • IAM role ARN
Verify sagemaker operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonSageMakerFullAccess
    • CloudWatchLogsFullAccess
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"ecr:*","Effect":"Allow","Resource":"*"},{"Action":"ecr-public:*","Effect":"Allow","Resource":"*"},{"Action":"sts:GetServiceBearerToken","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_sagemaker_endpoint
  • IAM role ARN
Verify sagemaker endpoint operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonSageMakerFullAccess
    • CloudWatchLogsFullAccess
example_sagemaker_notebook
  • IAM role ARN
Verify sagemaker notebook operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonSageMakerFullAccess
    • CloudWatchLogsFullAccess
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"ecr:*","Effect":"Allow","Resource":"*"},{"Action":"ecr-public:*","Effect":"Allow","Resource":"*"},{"Action":"sts:GetServiceBearerToken","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_sagemaker_pipeline
  • IAM role ARN
Verify sagemaker pipeline operator
example_sns N/A Verify sns operator
example_sql_to_s3
  • Security group
  • Cluster subnet group
Verify sql to s3 operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonS3FullAccess
    • AmazonVPCFullAccess
example_sqs N/A Verify sqs operator
example_step_functions
  • IAM role ARN
Verify step functions operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AWSStepFunctionsFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}