########################################################################### # Author: Sudhir Gupta # Date: 15-Aug-2019 # Version: 1.4 ########################################################################### AWSTemplateFormatVersion: '2010-09-09' Description: 'CloudFormation Template to create Amazon Redshift Cluster' Parameters: ParentVPCStack: Description: 'Provide Stack name of parent VPC stack based on VPC-3AZs yaml template. Refer Cloudformation dashboard in AWS Console to get this.' Type: String MinLength: '1' MaxLength: '128' AllowedPattern: '^[a-zA-Z]+[0-9a-zA-Z\-]*$' ParentSSHBastionStack: Description: 'Optional - Provide Stack name of parent Amazon Linux bastion host stack based on linux-bastion template. Refer Cloudformation dashboard in AWS Console to get this.' Type: String MinLength: '0' MaxLength: '128' DatabaseName: Description: The name of the first database to be created when the cluster is created. Type: String Default: rsdev01 AllowedPattern: '([a-z]|[0-9])+' RedshiftClusterPort: Description: The port number on which the cluster accepts incoming connections. Type: Number Default: '8200' NumberOfNodes: Description: >- The number of compute nodes in the cluster. For multi-node clusters, the NumberOfNodes parameter must be greater than 1. Type: Number Default: '2' NodeType: Description: The type of node to be provisioned Type: String Default: dc2.large AllowedValues: - dc2.large - dc2.8xlarge - ds2.xlarge - ds2.8xlarge MasterUsername: Description: The user name that is associated with the master user account for the cluster that is being created Type: String Default: rsadmin AllowedPattern: '([a-z])([a-z]|[0-9])*' ConstraintDescription: must start with a-z and contain only a-z or 0-9. MasterUserPassword: Description: The password that is associated with the master user account for the cluster that is being created. Type: String NoEcho: 'true' MinLength: '8' MaxLength: '64' AllowedPattern: >- ^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?!._*[@/\\\"']).*$ ConstraintDescription: >- Enter alphanumeric password for the master user. The password must contain 8 to 64 printable ASCII characters, excluding: /, ", \'', \ and @. It must contain one uppercase letter, one lowercase letter, and one number. Maintenancewindow: Description: Maintenance Window for Redshift Cluster Type: String Default: 'sat:05:00-sat:05:30' MaxConcurrentCluster: Description: The maximum number of concurrency scaling Redshift clusters. Type: String Default: '1' EncryptionAtRest: Description: Enables or disables encryption at rest of the Redshift database. Type: String Default: 'false' AllowedValues: - true - false ConstraintDescription: must be true or false. kmskey: Description: The existing KMS key ID for encrypting Redshift database at-rest. Type: String Default: '' SnapshotIdentifier: Description: The Redshift snapshot identifier. Leave this blank for a new cluster. Enter the snapshot identifier, only if you want to restore from a snapshot. Default: '' Type: String SnapshotAccountNumber: Description: The AWS account number where the Redshift snapshot was created. Leave this blank, if the snapshot was created in the current AWS account. Default: '' Type: String NotificationList: Type: String Description: The Email notification list is used to configure a SNS topic for sending cloudwatch alarm and event notifications Default: "ops@company.com" AllowedPattern: '^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$' ConstraintDescription: provide a valid email address. EnableLoggingToS3: Default: 'false' Type: String AllowedValues: - true - false Description: Enables or disables logging to an S3 bucket. To enable logging, select True. S3BucketForRedshiftIAMRole: Type: String Description: The existing Amazon S3 bucket. An IAM role will be created and associated to the Redshift cluster with GET and LIST access to this bucket. Default: '' GlueCatalogDatabase: Type: String Description: The name of your Glue Data Catalog database. AllowedPattern: '([ \t\n\x0B\f\r])*|([a-z])([\-]|[a-z]|[\-]|[0-9])*' ConstraintDescription: must start with a-z and contain only a-z or 0-9 or hyphen (-). TagName: Type: String Description: The unique friendly name as required by your company’s tagging strategy document, and which will be added to the environment tag. Default: 'redshift' TagEnvironment: Type: String AllowedValues: - Development - test - pre-prod - Production Description: The environment tag that is used to designate the environment stage of the associated AWS resource. Default: 'Development' TagTier: Type: String AllowedValues: - data - web - application Description: The functional tier of the associated AWS resource. Default: 'data' TagProjectCostCenter: Type: String Description: The cost center associated with the project of the given AWS resource. AllowedPattern: "^[a-zA-Z0-9]+$" Default: '12345' ConstraintDescription: provide a valid cost center TagOwner: Type: String Description: The email address of business owners who are associated with the given AWS resource, for sending outage or maintenance notifications. Default: 'ops@company.com' TagConfidentiality: Type: String Description: The confidentiality classification of the data that is associated with the resource. AllowedValues: - public - private - confidential - pii/phi Default: 'private' TagCompliance: Type: String Description: The compliance level for the AWS resource. AllowedValues: - hipaa - sox - fips - other - none Default: 'none' Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Environment Parameters: - TagEnvironment - Label: default: Networking Parameters Parameters: - ParentVPCStack - ParentSSHBastionStack - Label: default: Amazon Redshift Parameters Parameters: - NodeType - NumberOfNodes - RedshiftClusterPort - DatabaseName - MasterUsername - MasterUserPassword - Label: default: Amazon Redshift Advanced Parameters Parameters: - EnableLoggingToS3 - MaxConcurrentCluster - LoadTPCDS3TBDataset - EncryptionAtRest - kmskey - SnapshotIdentifier - SnapshotAccountNumber - Maintenancewindow - S3BucketForRedshiftIAMRole - GlueCatalogDatabase - NotificationList - Label: default: Tag Identifiers Parameters: - TagName - TagOwner - TagTier - TagProjectCostCenter - TagConfidentiality - TagCompliance ParameterLabels: DatabaseName: default: Redshift database name RedshiftClusterPort: default: Redshift cluster port NodeType: default: Node type for Redshift cluster NumberOfNodes: default: Number of nodes in Redshift cluster MasterUsername: default: Redshift master user name MasterUserPassword: default: Redshift master user password S3BucketForRedshiftIAMRole: default: Amazon S3 bucket for Redshift IAM role NotificationList: default: Email address for SNS notification EnableLoggingToS3: default: Enable Redshift logging to S3 MaxConcurrentCluster: default: Max. number of concurrent clusters EncryptionAtRest: default: Encryption at rest kmskey: default: KMS key ID SnapshotIdentifier: default: Redshift snapshot identifier SnapshotAccountNumber: default: AWS account-ID of the Redshift snapshot GlueCatalogDatabase: default: Glue catalog database name Maintenancewindow: default: Redshift maintenance window TagEnvironment: default: Environment TagName: default: Unique friendly name TagTier: default: Functional tier TagOwner: default: Designate business owner's email TagCompliance: default: Compliance classifier TagConfidentiality: default: Confidentiality classifier TagProjectCostCenter: default: Project cost center ParentVPCStack: default: Parent VPC Stack ParentSSHBastionStack: default: Parent Bastion Stack (Optional) Mappings: RedshiftLoggingAccountIDRegionMap: us-gov-west-1: RSAccountID: xx us-east-1: RSAccountID: 193672423079 us-east-2: RSAccountID: 391106570357 us-west-1: RSAccountID: 262260360010 us-west-2: RSAccountID: 902366379725 ap-east-1: RSAccountID: 313564881002 ap-south-1: RSAccountID: 865932855811 ap-northeast-3: RSAccountID: 090321488786 ap-northeast-2: RSAccountID: 760740231472 ap-southeast-1: RSAccountID: 361669875840 ap-southeast-2: RSAccountID: 762762565011 ap-northeast-1: RSAccountID: 404641285394 ca-central-1: RSAccountID: 907379612154 cn-north-1: RSAccountID: 111890595117 cn-northwest-1: RSAccountID: 660998842044 eu-west-1: RSAccountID: 210876761215 eu-central-1: RSAccountID: 053454850223 eu-west-2: RSAccountID: 307160386991 eu-west-3: RSAccountID: 915173422425 eu-north-1: RSAccountID: 729911121831 sa-east-1: RSAccountID: 075028567923 Conditions: IsGovCloud: !Equals [!Ref 'AWS::Region', 'us-gov-west-1'] RedshiftSingleNodeClusterCondition: Fn::Equals: - Ref: NumberOfNodes - '1' IsProd: !Equals [!Ref TagEnvironment, 'Production'] IsEncryptionAtRest: !Equals [!Ref EncryptionAtRest, 'true'] IsEnableLoggingToS3: !And - !Equals [!Ref EnableLoggingToS3, 'true'] - !Not [!Condition IsGovCloud] IsParentSSHBastionStack: Fn::Not: - Fn::Equals: - '' - Ref: ParentSSHBastionStack IsNotParentSSHBastionStack: !Equals ['', !Ref ParentSSHBastionStack] IsConfigureRedshiftIAMRole: Fn::Not: - Fn::Equals: - '' - Ref: S3BucketForRedshiftIAMRole IsGlueCatalogName: Fn::Not: - Fn::Equals: - '' - Ref: GlueCatalogDatabase IsSnapshotSpecified: Fn::Not: - Fn::Equals: - '' - Ref: SnapshotIdentifier IsSnapshotAccountSpecified: Fn::Not: - Fn::Equals: - '' - Ref: SnapshotAccountNumber Resources: RedshiftSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: !Join [ " ", [ !Ref 'AWS::StackName', " - Redshift Security Group" ] ] VpcId: {'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'} SecurityGroupIngress: - !If - IsNotParentSSHBastionStack - IpProtocol: tcp FromPort: !Ref RedshiftClusterPort ToPort: !Ref RedshiftClusterPort CidrIp: {'Fn::ImportValue': !Sub '${ParentVPCStack}-CidrBlock'} Description: 'Redshift Access to VPC CIDR' - !Ref 'AWS::NoValue' - !If - IsParentSSHBastionStack - IpProtocol: tcp FromPort: !Ref RedshiftClusterPort ToPort: !Ref RedshiftClusterPort SourceSecurityGroupId: {'Fn::ImportValue': !Sub '${ParentSSHBastionStack}-BastionSecurityGroupID'} Description: 'Access to Bastion Host Security Group' - !Ref 'AWS::NoValue' Tags: - Key: Name Value: !Join - '_' - - !Ref TagName - !Sub '${AWS::StackName}-RedshiftSecurityGroup' - Key: Owner Value: !Ref TagOwner - Key: Environment Value: !Ref TagEnvironment - Key: Tier Value: !Ref TagTier - Key: ProjectCostCenter Value: !Ref TagProjectCostCenter - Key: Confidentiality Value: !Ref TagConfidentiality - Key: Compliance Value: !Ref TagCompliance # --- RedshiftSecurityGroupIngress: Type: 'AWS::EC2::SecurityGroupIngress' Properties: IpProtocol: "-1" SourceSecurityGroupId: !Ref RedshiftSecurityGroup GroupId: !Ref RedshiftSecurityGroup MyGlueIAMRole: Type: 'AWS::IAM::Role' Properties: RoleName: !Sub "AWSGlueServiceRole-${AWS::StackName}" AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "glue.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole Policies: - PolicyName: !Sub "${AWS::StackName}-Glue-Read-S3-Policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - s3:GetBucketLocation - s3:GetObject - s3:ListMultipartUploadParts - s3:ListBucket - s3:ListBucketMultipartUploads Resource: - !Join - "" - - "arn:aws:s3:::" - {'Fn::ImportValue': !Sub '${ParentSSHBastionStack}-BigQueryDataS3BucketName'} - "/*" # --- RedshiftLoggingS3Bucket: Type: 'AWS::S3::Bucket' DeletionPolicy: Retain Condition: IsEnableLoggingToS3 Properties: LifecycleConfiguration: Rules: - Id: RedshiftLogsArchivingToGlacier Status: Enabled ExpirationInDays: '30' Transitions: - TransitionInDays: '14' StorageClass: Glacier Tags: - Key: Name Value: !Join [ "-", [ !Ref TagName, !Ref 'AWS::StackName', "Redshift-Cluster-LoggingBucket" ] ] - Key: Owner Value: !Ref TagOwner - Key: Environment Value: !Ref TagEnvironment - Key: Tier Value: !Ref TagTier - Key: ProjectCostCenter Value: !Ref TagProjectCostCenter - Key: Confidentiality Value: !Ref TagConfidentiality - Key: Compliance Value: !Ref TagCompliance RedshiftLoggingS3BucketPolicy: Type: 'AWS::S3::BucketPolicy' Condition: IsEnableLoggingToS3 Properties: Bucket: !Ref RedshiftLoggingS3Bucket PolicyDocument: Statement: - Principal: AWS: !Join [ '', ['arn:aws:iam::', !FindInMap [RedshiftLoggingAccountIDRegionMap, !Ref 'AWS::Region', RSAccountID ] ,':user/logs']] Effect: Allow Action: 's3:GetBucketAcl' Resource: !Sub '${RedshiftLoggingS3Bucket.Arn}' - Principal: AWS: !Join [ '', ['arn:aws:iam::', !FindInMap [RedshiftLoggingAccountIDRegionMap, !Ref 'AWS::Region', RSAccountID ] ,':user/logs']] Effect: Allow Action: 's3:PutObject' Resource: !Sub '${RedshiftLoggingS3Bucket.Arn}/AWSLogs/*' RedshiftClusterParameterGroup: Type: 'AWS::Redshift::ClusterParameterGroup' Properties: Description: !Join [ " ", [ !Ref 'AWS::StackName', " - Redshift Cluster Parameter group" ]] ParameterGroupFamily: redshift-1.0 Parameters: - ParameterName: enable_user_activity_logging ParameterValue: !If [IsProd, 'true', 'false'] - ParameterName: require_ssl ParameterValue: 'true' - ParameterName: auto_analyze ParameterValue: 'true' - ParameterName: statement_timeout ParameterValue: '43200000' - ParameterName: max_concurrency_scaling_clusters ParameterValue: !Ref MaxConcurrentCluster - ParameterName: "wlm_json_configuration" ParameterValue: "[ {\"query_group\" : [ ], \"query_group_wild_card\" : 0, \"user_group\" : [ ], \"user_group_wild_card\" : 0, \"concurrency_scaling\" : \"auto\", \"rules\" : [ { \"rule_name\" : \"DiskSpilling\", \"predicate\" : [ { \"metric_name\" : \"query_temp_blocks_to_disk\", \"operator\" : \">\", \"value\" : 100000} ], \"action\" : \"log\", \"value\" : \"\"}, { \"rule_name\" : \"RowJoining\", \"predicate\" : [ { \"metric_name\" : \"join_row_count\", \"operator\" : \">\", \"value\" : 1000000000 } ], \"action\" : \"log\", \"value\" : \"\" } ], \"auto_wlm\" : true}, {\"short_query_queue\" : true} ]" Tags: - Key: Name Value: !Join [ "-", [ !Ref TagName, !Ref 'AWS::StackName', "Primary Cluster Parameter group" ] ] - Key: Owner Value: !Ref TagOwner - Key: Environment Value: !Ref TagEnvironment - Key: Tier Value: !Ref TagTier - Key: ProjectCostCenter Value: !Ref TagProjectCostCenter - Key: Confidentiality Value: !Ref TagConfidentiality - Key: Compliance Value: !Ref TagCompliance RedshiftClusterSubnetGroup: Type: 'AWS::Redshift::ClusterSubnetGroup' Properties: Description: Cluster subnet group SubnetIds: !Split [',', {'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetsPrivate'}] Tags: - Key: Name Value: !Join [ "-", [ !Ref TagName, !Ref 'AWS::StackName', "Primary Redshift Cluster Subnet group" ] ] - Key: Owner Value: !Ref TagOwner - Key: Environment Value: !Ref TagEnvironment - Key: Tier Value: !Ref TagTier - Key: ProjectCostCenter Value: !Ref TagProjectCostCenter - Key: Confidentiality Value: !Ref TagConfidentiality - Key: Compliance Value: !Ref TagCompliance RedshiftCluster: Type: 'AWS::Redshift::Cluster' DeletionPolicy: Snapshot UpdateReplacePolicy: Snapshot Properties: ClusterType: !If [RedshiftSingleNodeClusterCondition, "single-node", "multi-node"] ClusterIdentifier: !Join ["-", [!Ref DatabaseName, !Ref 'AWS::StackName']] NumberOfNodes: !If [RedshiftSingleNodeClusterCondition, !Ref "AWS::NoValue", !Ref NumberOfNodes] NodeType: !Ref NodeType DBName: !Ref DatabaseName KmsKeyId: !If [IsEncryptionAtRest, !Ref kmskey, !Ref "AWS::NoValue"] Encrypted: !Ref EncryptionAtRest Port: !Ref RedshiftClusterPort MasterUsername: !Ref MasterUsername MasterUserPassword: !Ref MasterUserPassword ClusterParameterGroupName: !Ref RedshiftClusterParameterGroup SnapshotIdentifier: !If [IsSnapshotSpecified, !Ref SnapshotIdentifier, !Ref "AWS::NoValue"] OwnerAccount: !If [IsSnapshotAccountSpecified, !Ref SnapshotAccountNumber, !Ref "AWS::NoValue"] VpcSecurityGroupIds: - !Ref RedshiftSecurityGroup PreferredMaintenanceWindow: !Ref Maintenancewindow AutomatedSnapshotRetentionPeriod: !If [IsProd, 35, 8] PubliclyAccessible: 'false' ClusterSubnetGroupName: !Ref RedshiftClusterSubnetGroup LoggingProperties: !If - IsEnableLoggingToS3 - BucketName: !Ref RedshiftLoggingS3Bucket S3KeyPrefix: 'AWSLogs' - !Ref 'AWS::NoValue' IamRoles: - !If - IsConfigureRedshiftIAMRole - !GetAtt MyRedshiftIAMRole.Arn - !Ref 'AWS::NoValue' Tags: - Key: Name Value: !Join [ "-", [ !Ref TagName, !Ref 'AWS::StackName', "Redshift-Cluster" ] ] - Key: Owner Value: !Ref TagOwner - Key: Environment Value: !Ref TagEnvironment - Key: Tier Value: !Ref TagTier - Key: ProjectCostCenter Value: !Ref TagProjectCostCenter - Key: Confidentiality Value: !Ref TagConfidentiality - Key: Compliance Value: !Ref TagCompliance GlueCatalogDB: Condition: IsGlueCatalogName Type: 'AWS::Glue::Database' Properties: CatalogId: !Ref AWS::AccountId DatabaseInput: Name: !Ref GlueCatalogDatabase Description: !Join [ " ", ["AWS Glue Catalog database from Stack ", !Ref 'AWS::StackName'] ] MyRedshiftIAMRole: Type: 'AWS::IAM::Role' Condition: IsConfigureRedshiftIAMRole Properties: RoleName: !Join [ "-", [!Ref 'AWS::StackName', "RedshiftSpectrumRole"] ] AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "redshift.amazonaws.com" - "glue.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" Policies: - PolicyName: !Join [ "-", [!Ref 'AWS::StackName', "Spectrum-Glue-Access-Policy"] ] PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - s3:GetBucketLocation - s3:GetObject - s3:ListMultipartUploadParts - s3:ListBucket - s3:ListBucketMultipartUploads Resource: - !Join ['', ["arn:aws:s3:::", !Ref S3BucketForRedshiftIAMRole]] - !Join ['', ["arn:aws:s3:::", !Ref S3BucketForRedshiftIAMRole, "/*"]] - Effect: Allow Action: - glue:CreateDatabase - glue:DeleteDatabase - glue:GetDatabase - glue:GetDatabases - glue:UpdateDatabase - glue:CreateTable - glue:DeleteTable - glue:BatchDeleteTable - glue:UpdateTable - glue:GetTable - glue:GetTables - glue:BatchCreatePartition - glue:CreatePartition - glue:DeletePartition - glue:BatchDeletePartition - glue:UpdatePartition - glue:GetPartition - glue:GetPartitions - glue:BatchGetPartition - logs:* Resource: - "*" SNSTopic: Type: AWS::SNS::Topic Properties: Subscription: - Endpoint: !Ref NotificationList Protocol: email DiskSpacealarmredshift: Type: 'AWS::CloudWatch::Alarm' DependsOn: RedshiftCluster Properties: MetricName: !Join - '' - - !Ref RedshiftCluster - High-PercentageDiskSpaceUsed AlarmDescription: !Join - '' - - DiskSpace Utilization > 85% for - !Ref RedshiftCluster Namespace: AWS/Redshift Statistic: Average Period: '300' EvaluationPeriods: '3' Threshold: '85' AlarmActions: - !Ref SNSTopic Dimensions: - Name: ClusterIdentifier Value: !Ref RedshiftCluster ComparisonOperator: GreaterThanThreshold Unit: Percent HighCPUutilizationalarmredshift: Type: 'AWS::CloudWatch::Alarm' DependsOn: RedshiftCluster Condition: IsProd Properties: MetricName: !Join - '' - - !Ref RedshiftCluster - High-CPUUtilization AlarmDescription: !Join - '' - - CPUUtilization > 95% for last 15 min for cluster - !Ref RedshiftCluster Namespace: AWS/Redshift Statistic: Average Period: '300' EvaluationPeriods: '3' Threshold: '95' AlarmActions: - !Ref SNSTopic Dimensions: - Name: ClusterIdentifier Value: !Ref RedshiftCluster ComparisonOperator: GreaterThanThreshold Unit: Percent Outputs: StackName: Description: 'Stack name' Value: !Sub '${AWS::StackName}' RedshiftClusterEndpoint: Description: Redshift Cluster endpoint Value: !Sub "${RedshiftCluster.Endpoint.Address}:${RedshiftCluster.Endpoint.Port}" Export: Name: !Sub '${AWS::StackName}-RedshiftClusterEndpoint' PSQLCommandLine: Description: PSQL Command Line Value: !Join - '' - - 'psql -h ' - !GetAtt 'RedshiftCluster.Endpoint.Address' - ' -p ' - !GetAtt 'RedshiftCluster.Endpoint.Port' - ' -U ' - !Ref MasterUsername - ' -d ' - !Ref DatabaseName RedshiftParameterGroupName: Description: Name of the Redshift Parameter Group Value: !Ref RedshiftClusterParameterGroup Export: Name: !Sub '${AWS::StackName}-RedshiftParameterGroupName' RedshiftDatabaseName: Description: Name of the Redshift Database Value: !If - IsSnapshotSpecified - !Join [ ' ',['Check name of database from which the Snapshot', !Ref SnapshotIdentifier , ' was originally taken.']] - !Ref DatabaseName Export: Name: !Sub '${AWS::StackName}-RedshiftDatabaseName' RedshiftUsername: Value: !Ref MasterUsername Export: Name: !Sub '${AWS::StackName}-RedshiftUsername' RedshiftLoggingS3Bucket: Description: Amazon S3 bucket created for audit logging Condition: IsEnableLoggingToS3 Value: !Ref RedshiftLoggingS3Bucket Export: Name: !Sub '${AWS::StackName}-RedshiftLoggingS3Bucket' RedshiftClusterIAMRole: Description: IAM Role created and assigned to Redshift cluster Condition: IsConfigureRedshiftIAMRole Value: !GetAtt MyRedshiftIAMRole.Arn Export: Name: !Sub '${AWS::StackName}-RedshiftClusterIAMRole' GlueCatalogDBName: Description: AWS Glue Catalog Database Name Condition: IsGlueCatalogName Value: !Ref GlueCatalogDB Export: Name: !Sub '${AWS::StackName}-GlueCatalogDBName'