Go: AWS Lambda Project Structure Using Golang

AWS Lambda is a serverless solution which enables engineers to deploy single functions. AWS Lambda handles orchestrating, executing, scaling the function invocations. It’s important to structure go lambda projects so that the lambda is a simple entry point into the application, equivalent to cmd/. After a project is structured, it important to keep logic outside the lambda, which allows for easy reuse and testing of the application logic. The following are a series of steps which can be used in Go based lambda projects to help keep projects structured and increase the testability of lambda-based projects.

Structure

In Go, it’s common to see a cmd/ directory which contains CLI entry points into an application. Using a test project with 2 separate apps, the layout appears as:

$ tree test-go-lambda/
test-go-lambda/
└── cmd
    ├── app1
    │   └── main.go
    └── app2
        └── main.go

It’s helpful to take the same approach with lambdas:

$ tree test-go-lambda/
test-go-lambda/
├── cmd
│   ├── app1
│   │   └── main.go
│   └── app2
│       └── main.go
└── lambda
    ├── lambda1
    │   └── main.go
    └── lambda2
        └── main.go

All lambdas live in the lambda directly, and each directory within lambda contains a single lambdas main command. In the example above there are two lambdas, lambda1 and lambda2. Each contains a main.go file with a main command which can be executed by AWS lambda's go runtime. The benefits to this convention are the same as the cmd/ convention:

• Makes it easier to inventory entry points into the code base
• Helps to reduce onboarding friction
• Provides a structured convention which makes builds easier

This convention simplifies build tooling by providing a single location for all lambdas. It’s trivial to package all lambdas in the same .zip or generate a .zip per lambda.

“Thin” Lambdas

“Thin” lambdas delegate to other functions. I like to structure it so that a lambda delegates to a single function. AWS Lambda documentation recommends this as a best practice:

Separate the Lambda handler from your core logic.

To achieve this, lambdas can delegate to a single domain logic entry point. Each lambda will:

• Parse the environment and initialize a domain specific config in main
• Initialize domain logic in Handler
• Delegate to the domain logic and return an error

Which looks like:

func Handler(ctx context.Context, e events.CloudWatchEvent) error {
    var conf Config
    // initialize conf
    doer, err := domain.NewDoer(conf)
    if err != nil {
        return err
    }
    return doer.Do(ctx, e)
}

func main() {
    lambda.Start(Handler)
}

“Lambda” Structs for Configuration

AWS Lambda can reuse execution environments for individual lambdas, which means that AWS may keep handlers alive and invoke them multiple times. Resources, like database connections, initialized outside of the handler function can be maintained for multiple handler invocations! I like to call these main scoped resources, opposed to handler scoped resources. I like to setup each lambda main.go file to have a structure:

# lambda/x/main.go

type Lambda struct {
    Conf domain.SpecificConf
}

func (l Lambda) Handler(...) error {
    // handler scoped l.Conf configuration
    doer, err := domain.NewDoer(l.Conf)
    if err != nil {
        return err
    }
    return doer.Do(ctx, e)
}

func main() {
    // initialize resources
    // build conf
    l := Lambda{
        Conf: domain.SpecificConf{
            // main scoped configuration
        }
    }

    lambda.Start(l.Handler)
}

Examples of main scoped configuration are:

• Environmental Variables
• Session i.e. AWS-SDK sessions and clients
• Database connections (redis, mysql, postgres, elastic search, etc.)

Examples of handler scoped configuration:

• Times / Timing
• AWS Lambda Payload / Parameter based configuration

Each lambda is configured through its environment. I like to use envdecode to handle parsing the environmental variables in the main function:

func main() {
    // initialize resources
    // build conf
    conf := domain.SpecificConf{
        // main scoped configuration
        // db connections, aws-sdk sessions, etc
    }

    // pull in environment based config
    if err := envdecode.Decode(&conf); err != nil {
		panic(err)
	}

    l := Lambda{
        Conf: conf,
    }

    lambda.Start(l.Handler)
}

Expose SQS Interface on Cron Lambdas

One great use case for lambda is “cron” based workloads (through cloudwatch events). This is where AWS executes a lambda function on a fixed schedule. Many of these include a dead letter queue for failed messages and a lambda that consumes from dead letter queue. Since the messages in the dead letter queue contain the original message the dead letter queue lambda is usually very similar to the original lambda in terms of configuration and resource dependency. What does change is that the dead letter queue messages have a different structure, events.SQSEvent.

One way to handle this is to expose a new SQSHandler which delegates to the original:

type LambdaConf struct {
	HandlerType string `env:"HANDLER_TYPE,default=cloudwatch_event"`
}

type Lambda struct {}

func (l *Lambda) SQSHandler(ctx context.Context, sqsEvent events.SQSEvent) error {
    var e events.CloudWatchEvent
    for _, record := sqsEvent.Records {
        if err := json.Unmarshal([]byte(record.Body), &e); err != nil {
            log.Printf("event: %+v\n", record)
            return fmt.Errorf("unable to parse event into CloudWatchEvent: %s", err)
        }

        // delegate to l.Handler(ctx, e)
    }
}

func (l *Lambda) Handler(ctx context.Context, e events.CloudWatchEvent) error {
    doer, err := domain.NewDoer()
    if err != nil {
        return err
    }
    return doer.Do(ctx, e)
}

func main() {
    l := Lambda{}

    lambdaConf := LambdaConf{}
    if err := envdecode.Decode(&lambdaConf); err != nil {
		panic(err)
	}

    if lambdaConf.HandlerType == "sqs_event" {
		lambda.Start(l.SQSHandler)
	} else {
		lambda.Start(l.Handler)
	}
}

The SQSHandler delegates to the Handler in order to reduce duplication and keep the handlers "thin".

Conclusion

If careful attention isn’t paid to go-based lambdas, projects risk creating untestable, hard to work with lambdas. Placing lambdas in their own directory makes lambda discovery and builds easy. Minimizing business logic inside of lambdas makes it easier to isolate logic for unit tests. I’ve found that projects that follow the above structure are much easier to understand, test, build and extend. Happy Hacking!