In September 2018, AWS released CloudFormation Macros, a powerful addition to CloudFormation. Macros allow you to write your own mini-DSLs on top of CloudFormation, giving you the power to enforce organization-wide defaults or allow for more flexible syntax in your templates.
In this post, we'll cover the what, why, and how of CloudFormation macros. This post includes:
What are CloudFormation macros and when should you use them?
An example of writing a CloudFormation template macro to add Python-style string formatting to your CloudFormation template
An example of writing a CloudFormation snippet macro to add default properties for DynamoDB Table resources.
Let's get started.
Background: What are CloudFormation Macros and when should you use them?
First, let's understand what CloudFormation macros are and why they are helpful.
CloudFormation macros are like pre-processors of your CloudFormation templates. After you submit your CloudFormation template, macros are called to transform portions of your template before CloudFormation actually starts provisioning resources.
Under the hood, macros are powered by AWS Lambda functions that you write. They should be functional components that take in some existing CloudFormation and output additional CloudFormation.
In effect, CloudFormation macros allow you to write custom DSLs on top of CloudFormation without, you know, writing an entire custom DSL on top of CloudFormation.
If you've used AWS SAM for deploying serverless applications, you're already using CloudFormation Macros! The AWS::Serverless Transform is a macro hosted by AWS for building serverless applications.
To see why macros can be helpful, first we must learn the two types of CloudFormation macros.
Template-level macros
The first type of macro is a template macro, which has access to your entire CloudFormation template. This macro is specified in the Transform section of your CloudFormation template, as follows:
Description: "My CloudFormation Template"
Resources:
MySNSTopic:
Type: "AWS::SNS::Topic"
...
Transform: # <--- Look here
- VariableSubstitution
- CompanyDefaults
In the example above, look at the top-level Transform property. Our template will invoke two separate macros, VariableSubstitution and CompanyDefaults. The macros will be invoked in the order given, so VariableSubstitution would be invoked first, then CompanyDefaults would be invoked with the results from the VariableSubstitution macro.
The two macro names given above are representative examples of when you may want to use template macros.
One good use of a template macro would be to allow for a more flexible variable syntax language than CloudFormation's clunky Parameter syntax. You could use Python's str.format() syntax or JavaScript template strings to template your CloudFormation. We'll explore writing and using this macro later on in this post.
A second use case could be to pull in company-wide defaults for certain resources. For example, perhaps you want to set all your DynamoDB tables to be using on-demand pricing or you want all of your S3 buckets to have a certain logging configuration. Using a template macro can make it easy to spread these standards across your organization in a more reliable way than word of mouth + copy-pasta.
Snippet macros
The second type of CloudFormation macro is a snippet macro, which has access to the CloudFormation resource to which it's attached and any of the resource's children. This macro is used by adding a Fn::Transform property to your CloudFormation template:
Description: "My CloudFormation Template"
Resources:
MyTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: BigDataTable
KeySchema: ...
Fn::Transform: # <-- Snippet macro
- Name: DynamoDBDefaults
IAMRole:
Type: AWS::IAM::Role
Properties:
Fn::Transform: # <-- Snippet macro
- Name: IAMGenerator
Parameters:
Policies:
ReadWrite: { Ref: MyTable }
In this example, there are two snippet macros. The first one, DynamoDBDefaults, is attached to the MyTable DynamoDB table resource we're creating. Similar to the CompanyDefaults in the previous section, this macro could be used to provide defaults for a particular resource. We use this as an example for writing a template macro later on in this post.
An important note here is that macros are evaluated from most deeply nested outward. Thus, snippet macros would be evaluated before template macros. If you had both the DynamoDBDefaults macro on the DynamoDB resource and the CompanyDefaults macro on your template, the defaults for your specific resouce would be applied first.
The second snippet macro in the above example is on an IAM Role and is called IAMGenerator. You could imagine writing a macro with a more terse syntax for creating IAM statements.
The IAMGenerator macro also shows how to pass parameters into your macro. In this example, we're passing a custom IAM DSL into the macro which will be expanded into a fuller, valid IAM policy statement.
Now that we know about the types of CloudFormation macros, let's see how to set them up.
How to set up a CloudFormation macro
Before you can use a CloudFormation macro in a template, you need to configure a macro in your account. This is a three-step process.
Write the logic for your CloudFormation macro in an AWS Lambda function.
First, you need to actually write your logic.
Remember that your macro logic should be functional in nature -- taking in a CloudFormation template or snippet as input and returning an updated template or snippet as output.
You shouldn't be provisioning additional resources directly in your macro logic. If you need to provision a resource that is not supported by CloudFormation directly, you should use CloudFormation custom resources.
Deploy your Lambda function and register it as a CloudFormation macro.
To register a CloudFormation macro, you need to configure a resource of type
AWS::CloudFormation::Macroin a CloudFormation template.You can use separate templates for your Lambda function and for creating the CloudFormation macro, or you can do it in a single CloudFormation template. I'd recommend doing it in the same template unless you have specific reasons not to.
The
AWS::CloudFormation::Macroresource has two key properties:NameandFunctionName.Nameis the name you'll give the macro to be called by other CloudFormation templates in your account.FunctionNameis the ARN of the Lambda function that will be called when your macro is used.Your
AWS::CloudFormation::Macrowill look something like the following:CompanyDefaultsMacro:
Type: AWS::CloudFormation::Macro
Properties:
Name: CompanyDefaults
FunctionName:
Fn::GetAtt:
- CompanyDefaultsLambdaFunction
- ArnI've named this macro
CompanyDefaults, which is how I'll use it in any subsequent CloudFormation templates.Reference your macro in your application templates.
Now that I've written my Lambda function and registered it as a macro, I need to use it in my application template. To reference your macro, you use the name given in the
Nameproperty of yourAWS::CloudFormation::Macroresource.As mentioned in the previous section, it can be used in the
Transformsection of the CloudFormation template to apply to the entire template, or it can be added as aFn::Transformfunction on a particular resource in the template.
With this knowledge of configuring CloudFormation macros in hand, let's dive into some examples of writing and using macros.
Example: Writing a CloudFormation Template Macro
For our first example, we'll write a template macro that adds Python-style string formatting to our CloudFormation templates.
CloudFormation allows you to use Parameters to templatize your CloudFormation templates. However, using these Parameters in your CloudFormation templates can be awkward as you'll need to make heavy use of Fn::Join, Fn::Sub, or other CloudFormation intrinsic functions to use the Parameters, such as the following:
Parameters:
stage:
Type: String
Default: dev
AllowedValues:
- dev
- staging
- prod
Resources:
MySNSTopic:
Type: AWS::SNS::Topic
Properties:
TopicName:
Fn::Join:
- "-"
- - "MyTopic"
- Ref: stage
With the macro we write in this section, we'll allow team members to use Python's string formatting to add parameters into properties:
Parameters:
stage:
Type: String
Default: dev
AllowedValues:
- dev
- staging
- prod
Resources:
MySNSTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: "MyTopic-{stage}" # <-- Look Ma, Python templating!
Transform:
- VariableSubstitution
You can follow along with code examples here.
Writing your macro logic
First, we need to write our macro's logic.
For this macro, we want to use any provided CloudFormation Parameters to format any string values in our CloudFormation template. The parameters will be provided on our Lambda event object under the templateParameterValues key, and our CloudFormation template will be available in the fragment key.
Your response in your Lambda function should be an object with three keys:
the
requestId, which is provided to you as therequestIdkey on the Lambda event object;a
status, which should besuccessif successful, or any other value if not.a
fragment, which will replace the fragment given to your macro.
We'll use the following logic in our Lambda:
import json
def variable_substitution(event, context):
context = event['templateParameterValues']
fragment = walk(event['fragment'], context)
resp = {
'requestId': event['requestId'],
'status': 'success',
'fragment': fragment
}
return resp
def walk(node, context):
if isinstance(node, dict):
return { k: walk(v, context) for k, v in node.items() }
elif isinstance(node, list):
return [walk(elem, context) for elem in node]
elif isinstance(node, str):
return node.format(**context)
else:
return node
In our function, we're using a walk() function to walk the elements in our templates. Once we get to a leaf node, we use Python's str.format() function to format the string with any variables from the given template parameters.
Pro-tip: Because CloudFormation macros should be functional in nature, it should be pretty easy to write tests for them. Check out the tests for this macro here
Deploying and registering our macro
Now that we've written our function logic, let's deploy our function and register it as a macro.
I'm going to use the Serverless Framework to deploy our functions and register the macros, but you can use any Lambda deployment tool you prefer.
Here's my serverless.yml:
service: variable-substitution
provider:
name: aws
runtime: python3.7
stage: dev
region: us-east-1
functions:
variableSubstitution:
handler: handler.variable_substitution
resources:
Resources:
VariableSubstitutionMacro:
Type: AWS::CloudFormation::Macro
Properties:
Name: VariableSubstitution
FunctionName:
Fn::GetAtt:
- VariableSubstitutionLambdaFunction
- Arn
In the functions block, the Serverless Framework is taking care of deploying my Lambda function for me.
Then, I'm using the resources block to add custom CloudFormation. In that block, I'm registering my AWS::CloudFormation::Macro resource with the name VariableSubstitution.
You can deploy this Serverless service by running serverless deploy in your terminal.
Using your template macro
Now that we have deployed and registered our VariableSubstitution macro, let's use it in a service.
Let's use the template from the beginning of this section. Save the following as template.yaml:
Parameters:
stage:
Type: String
Default: dev
AllowedValues:
- dev
- staging
- prod
Resources:
MySNSTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: "MyTopic-{stage}" # <-- Look Ma, Python templating!
Transform:
- VariableSubstitution
Notice how our SNS Topic name is set to MyTopic-{stage}. Our macro will use Python's string formatting to add the stage parameter from the Parameter section.
Deploy your CloudFormation template with the following command:
aws cloudformation deploy \
--stack-name sns-topic-variables \
--template-file template.yaml
Your CloudFormation stack will deploy in a minute or so.
Head to the CloudFormation home page in the AWS console. You should see a stack with the name sns-topic-variables. Click on it, and scroll to the Template section.
When View original template is selected, you can see the original template with the unformatted TopicName:
If you click View processed template, you'll see your properly-formatted template, with the name filled in:
Cool! We built our first CloudFormation macro to add easier string formatting to our template.
Example: Writing a CloudFormation snippet macro
Now that we've written a macro that operates on the whole template, let's build a snippet macro
Recall that a snippet macro acts on a single resource within a CloudFormation template rather than the whole template. This can be nice for adding defaults or a shorthand syntax to particular resources.
In this walkthrough, we'll create a DynamoDBDefaults macro that will add default properties to our DynamoDB table. In particular, it will:
Use DynamoDB On-Demand pricing if a pricing model isn't specified;
Set the read and write capacity units to 1 if the pricing model is specified as
PROVISIONEDbut the user didn't specify the read and write capacity units.Add a DynamoDB stream if one is not provided.
The code used in this example can be found here.
Let's get started.
Writing the Lambda function
Like the previous example, the first step is to write our Lambda function.
Writing a snippet macro is mostly the same as writing a template macro. The big change is in the fragment property. Rather than containing the entire CloudFormation template, it will only contain sibling or children elements of the macro as specified in the CloudFormation template.
Let's use the following CloudFormation template as an example:
Resources:
MyNewTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: "MyNewTable"
KeySchema:
- AttributeName: key
KeyType: HASH
AttributeDefinitions:
- AttributeName: key
AttributeType: S
Fn::Transform: DynamoDBDefaults
Notice the Fn::Transform that is under the Properties attribute in our MyNewTable resource. The fragment property on our Lambda event will contain the other values in Properties for this resource.
To set our default values, let's use the following Lambda logic:
import json
def dynamodb_defaults(event, context):
fragment = add_defaults(event['fragment'])
return {
'requestId': event['requestId'],
'status': 'success',
'fragment': fragment
}
def add_defaults(fragment):
# Set to On-Demand Billing if not set
if not fragment.get('BillingMode'):
fragment['BillingMode'] = 'PAY_PER_REQUEST'
# Set default provisioned throughput if not provided
if fragment.get('BillingMode') == 'PROVISIONED' and not fragment.get('ProvisionedThroughput'):
fragment['ProvisionedThroughput'] = { 'ReadCapacityUnits': 1, 'WriteCapacityUnits': 1 }
# Add a stream if not set
if not fragment.get('StreamSpecification'):
fragment['StreamSpecification'] = { 'StreamViewType': 'NEW_IMAGE' }
return fragment
In this function, we pass our fragment into an add_defaults() function, which checks for certain properties and adds defaults if they're not present.
Deploying and registering a snippet macro
With our logic written, let's deploy our function and register our macro.
Once again, I'll use the Serverless Framework to deploy our function and register the macro, though you may use other tools.
The serverless.yml file will look as follows:
service: dynamodb-defaults
provider:
name: aws
runtime: python3.7
stage: dev
region: us-east-1
functions:
dynamodbDefaults:
handler: handler.dynamodb_defaults
resources:
Resources:
VariableSubstitutionMacro:
Type: AWS::CloudFormation::Macro
Properties:
Name: DynamoDBDefaults
FunctionName:
Fn::GetAtt:
- DynamodbDefaultsLambdaFunction
- Arn
Like our previous example, we're configuring our function in the functions section. Then, we're using the resources block to provision an AWS::CloudFormation::Macro resource. Our macro will be named DynamoDBDefaults in accordance with our Name property.
Deploy and register your macro by running serverless deploy in your terminal. Once your macro is registered, move on to the next section to use it!
Using your CloudFormation snippet macro with Fn::Transform
Finally, let's use our snippet macro in a CloudFormation template.
Our macro adds default properties to DynamoDB tables, so let's use the following CloudFormation template:
Resources:
MyNewTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: "MyNewTable"
KeySchema:
- AttributeName: key
KeyType: HASH
AttributeDefinitions:
- AttributeName: key
AttributeType: S
Fn::Transform: DynamoDBDefaults
Note that without the Transform, this isn't a valid DynamoDB table resource. The AWS::DynamoDB::Table resource requires you to set the ProvisionedThroughput property unless you set the BillingMode to PAY_PER_REQUEST.
However, our macro will handle this for us 😎.
Let's deploy our template with the following command:
aws cloudformation deploy \
--stack-name dynamodb-table-macro \
--template-file template.yaml
Once our deployment is complete, head to the CloudFormation dashboard in the AWS console. Click on the dynamodb-table-macro stack and scroll to the Template section.
When looking at the original template, you should see the following:
Note that it doesn't have a BillingMode or StreamSpecification property.
Click on View processed template:
You'll see that the BillingMode has been set to PAY_PER_REQUEST and a StreamSpecification has been added. Nice!
Conclusion
CloudFormation macros are powerful ways to extend CloudFormation without going down the rabbit hole of building out an entire DSL. In this post, we learned about the different kinds of CloudFormation macros as well as the process for deploying CloudFormation macros. Finally, we deployed two example macros to see how they can be used.