bit of a shorter post than usual in terms of the process but sometimes the short and sweet stuff can be one of the most important things you need.. especially in your hour of need!
Ok so hypothetically speaking- lets say disaster has struck- everyone's worst fear: someone has accidentally deleted the cloudformation stack not realizing that the deletionpolicy: retain has not been set and it has deleted the RDS database alongside with it.
So what do we do?
Well hopefully before we have gotten to this point we have the right backups in place. I've heard a great saying somewhere... "A DBA is only as good as their last backup". The saying is quite funny, because it is absolutely 100% true. It is so important to have the correct backups in place because when the worst thing imaginable can happen it's such a vital and important safety net to have! The default deletion policy for AWS::RDS::DBCluster and AWS::RDS::DBInstance resources that don't specify the DBClusterIdentifier property is snapshot. Luckily if your using RDS automatic backups have been enabled by default: you can configure a backup window and also the backup retention period.
It's really important to note that it is recommended that you take a manual snapshot before making any amendments to the cloud formation template. Should anything go wrong you have something you can easily revert to by restoring a snapshot.
It is so important to test your recovery plans.. lets not do the classic mistake of waiting till we hit the problem where we need to restore a backup. As a DBA/System Admin you should know exactly the steps involved- how long these steps could roughly take. These steps should also be documented so the knowledge is not reliant on one person knowing the process. Often when this is the case that one person is on holiday when disaster strikes!
Deploying a cloudformation template to restore a snapshot
You'll need to specify the snapshotidentifier. The snapshotIdentifier is just the name of the snapshot. You can find out this through the RDS console and navigating to snapshots. It'll display all the recent snapshots you have.
Within your cloudformation template You'll need to specify the snapshotidentifier within the properties of the AWS::RDS::DBCluster
Type: 'AWS::RDS::DBCluster'
Properties:
SnapshotIdentifier: 'beckystestsnapshot'
BackupRetentionPeriod: '20'
Engine: aurora-postgresql
EngineVersion: '9.6.17'Remember to save your template to S3 so you can reference this later on when deploying the template!
Special considerations
MasterUsername and MasterUserPassword: If you specify the SnapshotIdentifier property, don't specify these two properties. The value is inherited from the source DB instance or snapshot.
If your Cloudformation template is deleted and you are using a KMS key there's a good chance your KMS key has also been deleted. You may get the following error when creating the stack: "The specified KMS key [arn:aws:kms:eu-west-2:AccountNo:key/KEYID] does not exist, is not enabled or you do not have permissions to access it." When a Cloudformation stack has been deleted the key will sit in a "pending deletion" status. You'll have to find the key within the KMS console (you can get the keyID from the error message). You can search by the keyID and then click on the key > navigate to key actions > cancel key deletion. Once that's done you should be able to re-enable the key by the same process.
!!SUPER IMPORTANT!!
After you restore a DB cluster with a SnapshotIdentifier property, you must specify the same SnapshotIdentifier property for any future updates to the DB cluster. When you specify this property for an update, the DB cluster is not restored from the DB cluster snapshot again, and the data in the database is not changed. However, if you don't specify the SnapshotIdentifier property, an empty DB cluster is created, and the original DB cluster is deleted. If you specify a property that is different from the previous snapshot restore property, a new DB cluster is restored from the specified SnapshotIdentifier property, and the original DB cluster is deleted.
Deploying the cloud formation template
1. As per the usual process to deploy the stack go into cloud formation and select create stack with new resources
2. Fill out all your relevant parameters and stack name etc and then when your happy deploy the template and this should successfully create your stack!
Comments