AWS Airflow - System test description

System test Requirements Description Notes
example_appflow_run N/A Verify appflow run operator
example_athena N/A Verify athena operator
example_batch
  • IAM role ARN
  • Subnets
  • Security groups
 Verify batch operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AWSBatchFullAccess
    • AmazonECS_FullAccess
    • CloudWatchLogsFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_bedrock
  • IAM role ARN
 Verify bedrock operator
example_bedrock_batch_inference
  • IAM role ARN
 Verify bedrock batch inference operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonBedrockFullAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_bedrock_retrieve_and_generate
  • IAM role ARN
 Verify bedrock retrieve and generate operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonBedrockFullAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":["aoss:*","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_cloudformation N/A Verify cloudformation operator
example_comprehend
  • IAM role ARN
 Verify comprehend operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
    • ComprehendFullAccess
    • AmazonTextractFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_comprehend_document_classifier
  • IAM role ARN
  • Bucket name
  • Bucket key discharge
  • Bucket key doctors notes
 Verify comprehend document classifier operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
    • ComprehendFullAccess
    • AmazonTextractFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_datasync
  • IAM role ARN
 Verify datasync operator
example_dms_serverless
  • IAM role ARN
 Verify dms serverless operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonRDSFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":["dms:*","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_dynamodb N/A Verify dynamodb operator
example_dynamodb_to_s3 N/A Verify dynamodb to s3 operator
example_ec2 N/A Verify ec2 operator
example_ecs
  • Cluster name
  • Subnets
 Verify ecs operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonECS_FullAccess
    • service-role/AmazonECSTaskExecutionRolePolicy
    • CloudWatchLogsFullAccess
example_ecs_fargate
  • Subnets
  • Security groups
 Verify ecs fargate operator
example_eks_with_fargate_in_one_step
  • Cluster IAM role ARN
  • Fargate pod IAM role ARN
  • Subnets
 Verify eks with fargate in one step operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEC2FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"eks:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["iam:GetRole","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_eks_with_fargate_profile
  • Cluster IAM role ARN
  • Fargate pod IAM role ARN
  • Subnets
 Verify eks with fargate profile operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEC2FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"eks:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["iam:GetRole","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_eks_with_nodegroup_in_one_step
  • IAM role ARN
  • Subnets
 Verify eks with nodegroup in one step operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEKSWorkerNodePolicy
    • AmazonEKSClusterPolicy
    • AmazonEC2ContainerRegistryReadOnly
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["eks:Describe*","eks:List*","eks:CreateNodegroup","eks:DeleteNodegroup","eks:CreateCluster","eks:DeleteCluster","eks:TagResource","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["ec2:CreateLaunchTemplate","ec2:DeleteLaunchTemplate","ec2:DescribeLaunchTemplates","ec2:DescribeLaunchTemplateVersions","ec2:CreateTags","ec2:RunInstances"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_eks_with_nodegroups
  • IAM role ARN
  • Subnets
 Verify eks with nodegroups operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEKSWorkerNodePolicy
    • AmazonEKSClusterPolicy
    • AmazonEC2ContainerRegistryReadOnly
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["eks:Describe*","eks:List*","eks:CreateNodegroup","eks:DeleteNodegroup","eks:CreateCluster","eks:DeleteCluster","eks:TagResource","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["ec2:CreateLaunchTemplate","ec2:DeleteLaunchTemplate","ec2:DescribeLaunchTemplates","ec2:DescribeLaunchTemplateVersions","ec2:CreateTags","ec2:RunInstances"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_emr
  • Execution IAM role ARN
  • Bucket name
 Verify emr operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonElasticMapReduceFullAccess
    • AmazonSSMReadOnlyAccess
example_emr_eks
  • IAM role ARN
  • Subnets
  • Job IAM role ARN
  • Job IAM role name
 Verify emr eks operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonEC2FullAccess
    • IAMReadOnlyAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":["emr-containers:*","eks:*","iam:GetRole","iam:UpdateAssumeRolePolicy","iam:CreateServiceLinkedRole","iam:CreateOpenIDConnectProvider","iam:DeleteOpenIDConnectProvider","iam:TagOpenIDConnectProvider"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_emr_serverless
  • IAM role ARN
 Verify emr serverless operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":["emr-serverless:*","iam:CreateServiceLinkedRole"],"Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_eventbridge N/A Verify eventbridge operator
example_glue
  • IAM role ARN
 Verify glue operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • Amazons3FullAccess
    • AWSGlueConsoleFullAccess
    • CloudWatchLogsFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_glue_data_quality
  • IAM role ARN
 Verify glue data quality operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • Amazons3FullAccess
    • AWSGlueConsoleFullAccess
    • AmazonAthenaFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_glue_data_quality_with_recommendation
  • IAM role ARN
 Verify glue data quality with recommendation operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • Amazons3FullAccess
    • AWSGlueConsoleFullAccess
    • AmazonAthenaFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_glue_databrew
  • IAM role ARN
 Verify glue databrew operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • Amazons3FullAccess
    • AwsGlueDataBrewFullAccessPolicy
example_http_to_s3 N/A Verify http to s3 operator
example_kinesis_analytics
  • IAM role ARN
 Verify kinesis analytics operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonKinesisAnalyticsFullAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_lambda
  • IAM role ARN
 Verify lambda operator
example_local_to_s3 N/A Verify local to s3 operator
example_mwaa
  • IAM role without invoke rest api ARN
  • Environment name
  • Dag id
 Verify mwaa operator
  • System: BUILD_GENERAL1_MEDIUM
  • Inline policies needed
    • {"Statement":[{"Action":"airflow:InvokeRestApi","Effect":"Allow","Resource":"arn:aws:airflow::*****:role/SysTestMWAA_example_mwaa/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_neptune N/A Verify neptune operator
example_quicksight N/A Verify quicksight operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"quicksight:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_rds_event N/A Verify rds event operator
example_rds_export
  • KMS key id
  • IAM role ARN
 Verify rds export operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonRDSFullAccess
    • AmazonS3FullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
example_rds_instance N/A Verify rds instance operator
example_rds_snapshot N/A Verify rds snapshot operator
example_redshift
  • Security group
  • Cluster subnet group
 Verify redshift operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonVPCFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"redshift-data:BatchExecuteStatement","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_redshift_s3_transfers
  • Security group
  • Cluster subnet group
 Verify redshift s3 transfers operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonS3FullAccess
    • AmazonVPCFullAccess
example_s3 N/A Verify s3 operator
example_s3_to_dynamodb N/A Verify s3 to dynamodb operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonDynamoDBFullAccess
    • CloudWatchLogsFullAccess
example_s3_to_sql
  • Security group
  • Cluster subnet group
 Verify s3 to sql operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonS3FullAccess
    • AmazonVPCFullAccess
example_sagemaker
  • IAM role ARN
 Verify sagemaker operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonSageMakerFullAccess
    • CloudWatchLogsFullAccess
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"ecr:*","Effect":"Allow","Resource":"*"},{"Action":"ecr-public:*","Effect":"Allow","Resource":"*"},{"Action":"sts:GetServiceBearerToken","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_sagemaker_endpoint
  • IAM role ARN
 Verify sagemaker endpoint operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonSageMakerFullAccess
    • CloudWatchLogsFullAccess
example_sagemaker_notebook
  • IAM role ARN
 Verify sagemaker notebook operator
  • System: BUILD_GENERAL1_2XLARGE
  • Managed policies needed
    • AmazonS3FullAccess
    • AmazonSageMakerFullAccess
    • CloudWatchLogsFullAccess
    • IAMReadOnlyAccess
  • Inline policies needed
    • {"Statement":[{"Action":"ecr:*","Effect":"Allow","Resource":"*"},{"Action":"ecr-public:*","Effect":"Allow","Resource":"*"},{"Action":"sts:GetServiceBearerToken","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}
example_sagemaker_pipeline
  • IAM role ARN
 Verify sagemaker pipeline operator
example_sns N/A Verify sns operator
example_sql_to_s3
  • Security group
  • Cluster subnet group
 Verify sql to s3 operator
  • System: BUILD_GENERAL1_LARGE
  • Managed policies needed
    • AmazonRedshiftFullAccess
    • AmazonS3FullAccess
    • AmazonVPCFullAccess
example_sqs N/A Verify sqs operator
example_step_functions
  • IAM role ARN
 Verify step functions operator
  • System: BUILD_GENERAL1_MEDIUM
  • Managed policies needed
    • AWSStepFunctionsFullAccess
  • Inline policies needed
    • {"Statement":[{"Action":"iam:PassRole","Effect":"Allow","Resource":"arn:aws:iam::*****:role/<resource_provided_as_requirement>"}],"Version":"2012-10-17"}
test_aws_auth_manager N/A Verify aws auth manager operator
  • System: BUILD_GENERAL1_MEDIUM
  • Inline policies needed
    • {"Statement":[{"Action":"verifiedpermissions:*","Effect":"Allow","Resource":"*"}],"Version":"2012-10-17"}