All Articles

How and Why to Use CloudFormation Macros

In September 2018, AWS released CloudFormation Macros, a powerful addition to CloudFormation. Macros allow you to write your own mini-DSLs on top of CloudFormation, giving you the power to enforce organization-wide defaults or allow for more flexible syntax in your templates.

In this post, we’ll cover the what, why, and how of CloudFormation macros. This post includes:

Let’s get started.

Background: What are CloudFormation Macros and when should you use them?

First, let’s understand what CloudFormation macros are and why they are helpful.

CloudFormation macros are like pre-processors of your CloudFormation templates. After you submit your CloudFormation template, macros are called to transform portions of your template before CloudFormation actually starts provisioning resources.

Under the hood, macros are powered by AWS Lambda functions that you write. They should be functional components that take in some existing CloudFormation and output additional CloudFormation.

In effect, CloudFormation macros allow you to write custom DSLs on top of CloudFormation without, you know, writing an entire custom DSL on top of CloudFormation.

To see why macros can be helpful, first we must learn the two types of CloudFormation macros.

Template-level macros

The first type of macro is a template macro, which has access to your entire CloudFormation template. This macro is specified in the Transform section of your CloudFormation template, as follows:

Description: "My CloudFormation Template"
Resources:
  MySNSTopic:
    Type: "AWS::SNS::Topic"
    ...
Transform: # <--- Look here
  - VariableSubstitution
  - CompanyDefaults

In the example above, look at the top-level Transform property. Our template will invoke two separate macros, VariableSubstitution and CompanyDefaults. The macros will be invoked in the order given, so VariableSubstitution would be invoked first, then CompanyDefaults would be invoked with the results from the VariableSubstitution macro.

The two macro names given above are representative examples of when you may want to use template macros.

One good use of a template macro would be to allow for a more flexible variable syntax language than CloudFormation’s clunky Parameter syntax. You could use Python’s str.format() syntax or JavaScript template strings to template your CloudFormation. We’ll explore writing and using this macro later on in this post.

A second use case could be to pull in company-wide defaults for certain resources. For example, perhaps you want to set all your DynamoDB tables to be using on-demand pricing or you want all of your S3 buckets to have a certain logging configuration. Using a template macro can make it easy to spread these standards across your organization in a more reliable way than word of mouth + copy-pasta.

Snippet macros

The second type of CloudFormation macro is a snippet macro, which has access to the CloudFormation resource to which it’s attached and any of the resource’s children. This macro is used by adding a Fn::Transform property to your CloudFormation template:

Description: "My CloudFormation Template"
Resources:
  MyTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: BigDataTable
      KeySchema: ...
      Fn::Transform: # <-- Snippet macro
        - Name: DynamoDBDefaults
  IAMRole:
    Type: AWS::IAM::Role
    Properties:
      Fn::Transform: # <-- Snippet macro
        - Name: IAMGenerator
          Parameters:
            Policies:
              ReadWrite: { Ref: MyTable }

In this example, there are two snippet macros. The first one, DynamoDBDefaults, is attached to the MyTable DynamoDB table resource we’re creating. Similar to the CompanyDefaults in the previous section, this macro could be used to provide defaults for a particular resource. We use this as an example for writing a template macro later on in this post.

An important note here is that macros are evaluated from most deeply nested outward. Thus, snippet macros would be evaluated before template macros. If you had both the DynamoDBDefaults macro on the DynamoDB resource and the CompanyDefaults macro on your template, the defaults for your specific resouce would be applied first.

The second snippet macro in the above example is on an IAM Role and is called IAMGenerator. You could imagine writing a macro with a more terse syntax for creating IAM statements.

The IAMGenerator macro also shows how to pass parameters into your macro. In this example, we’re passing a custom IAM DSL into the macro which will be expanded into a fuller, valid IAM policy statement.

Now that we know about the types of CloudFormation macros, let’s see how to set them up.

How to set up a CloudFormation macro

Before you can use a CloudFormation macro in a template, you need to configure a macro in your account. This is a three-step process.

CloudFormation Macro Usage

  1. Write the logic for your CloudFormation macro in an AWS Lambda function.

    First, you need to actually write your logic.

    Remember that your macro logic should be functional in nature — taking in a CloudFormation template or snippet as input and returning an updated template or snippet as output.

    You shouldn’t be provisioning additional resources directly in your macro logic. If you need to provision a resource that is not supported by CloudFormation directly, you should use CloudFormation custom resources.

  2. Deploy your Lambda function and register it as a CloudFormation macro.

    To register a CloudFormation macro, you need to configure a resource of type AWS::CloudFormation::Macro in a CloudFormation template.

    You can use separate templates for your Lambda function and for creating the CloudFormation macro, or you can do it in a single CloudFormation template. I’d recommend doing it in the same template unless you have specific reasons not to.

    The AWS::CloudFormation::Macro resource has two key properties: Name and FunctionName.

    Name is the name you’ll give the macro to be called by other CloudFormation templates in your account. FunctionName is the ARN of the Lambda function that will be called when your macro is used.

    Your AWS::CloudFormation::Macro will look something like the following:

    CompanyDefaultsMacro:
     Type: AWS::CloudFormation::Macro
     Properties:
       Name: CompanyDefaults
       FunctionName:
         Fn::GetAtt:
           - CompanyDefaultsLambdaFunction
           - Arn

    I’ve named this macro CompanyDefaults, which is how I’ll use it in any subsequent CloudFormation templates.

  3. Reference your macro in your application templates.

    Now that I’ve written my Lambda function and registered it as a macro, I need to use it in my application template. To reference your macro, you use the name given in the Name property of your AWS::CloudFormation::Macro resource.

    As mentioned in the previous section, it can be used in the Transform section of the CloudFormation template to apply to the entire template, or it can be added as a Fn::Transform function on a particular resource in the template.

With this knowledge of configuring CloudFormation macros in hand, let’s dive into some examples of writing and using macros.

Example: Writing a CloudFormation Template Macro

For our first example, we’ll write a template macro that adds Python-style string formatting to our CloudFormation templates.

CloudFormation allows you to use Parameters to templatize your CloudFormation templates. However, using these Parameters in your CloudFormation templates can be awkward as you’ll need to make heavy use of Fn::Join, Fn::Sub, or other CloudFormation intrinsic functions to use the Parameters, such as the following:

Parameters: 
  stage: 
    Type: String
    Default: dev
    AllowedValues: 
      - dev
      - staging
      - prod
Resources:
  MySNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName:
        Fn::Join:
          - '-'
          - - "MyTopic"
            - Ref: stage 

With the macro we write in this section, we’ll allow team members to use Python’s string formatting to add parameters into properties:

Parameters:
  stage:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - staging
      - prod
Resources:
  MySNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: "MyTopic-{stage}" # <-- Look Ma, Python templating!
Transform:
  - VariableSubstitution

You can follow along with code examples here.

Writing your macro logic

First, we need to write our macro’s logic.

For this macro, we want to use any provided CloudFormation Parameters to format any string values in our CloudFormation template. The parameters will be provided on our Lambda event object under the templateParameterValues key, and our CloudFormation template will be available in the fragment key.

Your response in your Lambda function should be an object with three keys:

  • the requestId, which is provided to you as the requestId key on the Lambda event object;

  • a status, which should be success if successful, or any other value if not.

  • a fragment, which will replace the fragment given to your macro.

We’ll use the following logic in our Lambda:

import json


def variable_substitution(event, context):
    context = event['templateParameterValues']

    fragment = walk(event['fragment'], context)

    resp = {
      'requestId': event['requestId'],
      'status': 'success',
      'fragment': fragment
    }

    return resp


def walk(node, context):
    if isinstance(node, dict):
        return { k: walk(v, context) for k, v in node.items() }
    elif isinstance(node, list):
        return [walk(elem, context) for elem in node]
    elif isinstance(node, str):
        return node.format(**context)
    else:
        return node

In our function, we’re using a walk() function to walk the elements in our templates. Once we get to a leaf node, we use Python’s str.format() function to format the string with any variables from the given template parameters.

Pro-tip: Because CloudFormation macros should be functional in nature, it should be pretty easy to write tests for them. Check out the tests for this macro here

Deploying and registering our macro

Now that we’ve written our function logic, let’s deploy our function and register it as a macro.

I’m going to use the Serverless Framework to deploy our functions and register the macros, but you can use any Lambda deployment tool you prefer.

Here’s my serverless.yml:

service: variable-substitution

provider:
  name: aws
  runtime: python3.7
  stage: dev
  region: us-east-1

functions:
  variableSubstitution:
    handler: handler.variable_substitution

resources:
  Resources:
    VariableSubstitutionMacro:
      Type: AWS::CloudFormation::Macro
      Properties:
        Name: VariableSubstitution
        FunctionName:
          Fn::GetAtt:
            - VariableSubstitutionLambdaFunction
            - Arn

In the functions block, the Serverless Framework is taking care of deploying my Lambda function for me.

Then, I’m using the resources block to add custom CloudFormation. In that block, I’m registering my AWS::CloudFormation::Macro resource with the name VariableSubstitution.

You can deploy this Serverless service by running serverless deploy in your terminal.

Using your template macro

Now that we have deployed and registered our VariableSubstitution macro, let’s use it in a service.

Let’s use the template from the beginning of this section. Save the following as template.yaml:

Parameters:
  stage:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - staging
      - prod
Resources:
  MySNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: "MyTopic-{stage}" # <-- Look Ma, Python templating!
Transform:
  - VariableSubstitution

Notice how our SNS Topic name is set to MyTopic-{stage}. Our macro will use Python’s string formatting to add the stage parameter from the Parameter section.

Deploy your CloudFormation template with the following command:

aws cloudformation deploy \
  --stack-name sns-topic-variables \
  --template-file template.yaml

Your CloudFormation stack will deploy in a minute or so.

Head to the CloudFormation home page in the AWS console. You should see a stack with the name sns-topic-variables. Click on it, and scroll to the Template section.

When View original template is selected, you can see the original template with the unformatted TopicName:

CloudFormation: Before VariableSubstitution macro

If you click View processed template, you’ll see your properly-formatted template, with the name filled in:

CloudFormation: After VariableSubstitution macro

Cool! We built our first CloudFormation macro to add easier string formatting to our template.

Example: Writing a CloudFormation snippet macro

Now that we’ve written a macro that operates on the whole template, let’s build a snippet macro

Recall that a snippet macro acts on a single resource within a CloudFormation template rather than the whole template. This can be nice for adding defaults or a shorthand syntax to particular resources.

In this walkthrough, we’ll create a DynamoDBDefaults macro that will add default properties to our DynamoDB table. In particular, it will:

  • Use DynamoDB On-Demand pricing if a pricing model isn’t specified;

  • Set the read and write capacity units to 1 if the pricing model is specified as PROVISIONED but the user didn’t specify the read and write capacity units.

  • Add a DynamoDB stream if one is not provided.

The code used in this example can be found here.

Let’s get started.

Writing the Lambda function

Like the previous example, the first step is to write our Lambda function.

Writing a snippet macro is mostly the same as writing a template macro. The big change is in the fragment property. Rather than containing the entire CloudFormation template, it will only contain sibling or children elements of the macro as specified in the CloudFormation template.

Let’s use the following CloudFormation template as an example:

Resources:
  MyNewTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: 'MyNewTable'
      KeySchema:
        - AttributeName: key
          KeyType: HASH
      AttributeDefinitions:
        - AttributeName: key
          AttributeType: S
      Fn::Transform: DynamoDBDefaults

Notice the Fn::Transform that is under the Properties attribute in our MyNewTable resource. The fragment property on our Lambda event will contain the other values in Properties for this resource.

To set our default values, let’s use the following Lambda logic:

import json


def dynamodb_defaults(event, context):
    fragment = add_defaults(event['fragment'])

    return {
      'requestId': event['requestId'],
      'status': 'success',
      'fragment': fragment
    }


def add_defaults(fragment):
    # Set to On-Demand Billing if not set
    if not fragment.get('BillingMode'):
        fragment['BillingMode'] = 'PAY_PER_REQUEST'

    # Set default provisioned throughput if not provided
    if fragment.get('BillingMode') == 'PROVISIONED' and not fragment.get('ProvisionedThroughput'):
        fragment['ProvisionedThroughput'] = { 'ReadCapacityUnits': 1, 'WriteCapacityUnits': 1 }

    # Add a stream if not set
    if not fragment.get('StreamSpecification'):
        fragment['StreamSpecification'] = { 'StreamViewType': 'NEW_IMAGE' }

    return fragment

In this function, we pass our fragment into an add_defaults() function, which checks for certain properties and adds defaults if they’re not present.

Deploying and registering a snippet macro

With our logic written, let’s deploy our function and register our macro.

Once again, I’ll use the Serverless Framework to deploy our function and register the macro, though you may use other tools.

The serverless.yml file will look as follows:

service: dynamodb-defaults

provider:
  name: aws
  runtime: python3.7
  stage: dev
  region: us-east-1

functions:
  dynamodbDefaults:
    handler: handler.dynamodb_defaults

resources:
  Resources:
    VariableSubstitutionMacro:
      Type: AWS::CloudFormation::Macro
      Properties:
        Name: DynamoDBDefaults
        FunctionName:
          Fn::GetAtt:
            - DynamodbDefaultsLambdaFunction
            - Arn

Like our previous example, we’re configuring our function in the functions section. Then, we’re using the resources block to provision an AWS::CloudFormation::Macro resource. Our macro will be named DynamoDBDefaults in accordance with our Name property.

Deploy and register your macro by running serverless deploy in your terminal. Once your macro is registered, move on to the next section to use it!

Using your CloudFormation snippet macro with Fn::Transform

Finally, let’s use our snippet macro in a CloudFormation template.

Our macro adds default properties to DynamoDB tables, so let’s use the following CloudFormation template:

Resources:
  MyNewTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: 'MyNewTable'
      KeySchema:
        - AttributeName: key
          KeyType: HASH
      AttributeDefinitions:
        - AttributeName: key
          AttributeType: S
      Fn::Transform: DynamoDBDefaults

Note that without the Transform, this isn’t a valid DynamoDB table resource. The AWS::DynamoDB::Table resource requires you to set the ProvisionedThroughput property unless you set the BillingMode to PAY_PER_REQUEST.

However, our macro will handle this for us 😎.

Let’s deploy our template with the following command:

aws cloudformation deploy \
  --stack-name dynamodb-table-macro \
  --template-file template.yaml

Once our deployment is complete, head to the CloudFormation dashboard in the AWS console. Click on the dynamodb-table-macro stack and scroll to the Template section.

When looking at the original template, you should see the following:

CloudFormation: Before DynamoDBDefaults macro

Note that it doesn’t have a BillingMode or StreamSpecification property.

Click on View processed template:

CloudFormation: After DynamoDBDefaults macro

You’ll see that the BillingMode has been set to PAY_PER_REQUEST and a StreamSpecification has been added. Nice!

Conclusion

CloudFormation macros are powerful ways to extend CloudFormation without going down the rabbit hole of building out an entire DSL. In this post, we learned about the different kinds of CloudFormation macros as well as the process for deploying CloudFormation macros. Finally, we deployed two example macros to see how they can be used.

Published 21 Mar 2019

Working for Serverless, Inc. Infrastructure & data engineer with expertise in AWS, data processing, and serverless technologies. Learning never stops.
Alex DeBrie on Twitter