AWS Control Tower and AWS Lambda end of support for Python 3.6

UPDATE (28/5/22): AWS has released AWS Control Tower version 2.9. This updates the Lambda functions used by AWS Control Tower to the Python 3.9 runtime. Simply upgrade your Control Tower (you can find a guide I wrote on how to do this here) to roll out the fix to your accounts.

The Emails

I was checking my emails this morning and saw I had a bunch of emails from AWS about end of support for Python 3.6 on AWS Lambda. Here’s what it looked like:

Overall, I had around 6 of them. Now this was a bit puzzling to me, because some of the accounts in question like my Audit and Log Archive account I knew I hadn’t created any Lambda functions in.

Well, AWS is asking me to action something, so let’s have a look.

The Investigation

The best place to start for these types of notifications (and AWS has given us a nice link at the bottom of the email), is the AWS Health Dashboard.

This dashboard gives you an overall status of AWS services and personalised alerts relating to your resources you have configured in your AWS account.

Under Other Notification, I found the alert in question: Lambda Operational Notification This contained the same message from the email but also provides you’re a list of the affected resources in your account, in my Audit account, it looks like I only have 1 affected resource.

Annoyingly, the resource listed is just my account ID. Surely it would be better just to see a list of all the Lambda functions that are affected instead. After all, the CLI command to find the resource is right there in the body of the alert text.

Clicking the Account ID link shows another no so great interface decision. Turns out, the link is just to the AWS Lambda console (not a specific resource) and because the Health Dashboard is a global service, I now get asked to select a region.

Certainly, some improvements to the console could be made here but at least its pointing me in a direction. Now Australia, Sydney is the only real region I use for the most part, so let’s select Sydney and see if that’s where this resource is living.

Turns out it was a good guess, here we can see an AWS Lambda Function, running Python 3.6. Using the CLI prompt ourselves also provides the same result:

Lets go deeper

Now looking at bit further into this function we can see that it has been deployed from a StackSet called StackSet-AWSControlTowerBP-BASELINE-CLOUDWATCH

Looking at this StackSet in the CloudFormation console, we can see its used for Forwarding notifications from a local SNS topic to the Security Topic.

Here is some of the template code, where we can see it will create the Lambda Function, with a Python 3.6 runtime.

What is it?

Now I’m sure you’ve already figured it out, but this AWS Lambda function was deployed by AWS Control Tower, we can find references to this StackSet in the How AWS Control Tower Works documentation.

Maybe my AWS Control Tower needs an update?

I’ve talked about the need to update AWS Control Tower in a previous blog post, but long story short. Keeping AWS Control Tower up to date is a manual process. Let's check my Landing Zone and see if it's up to date.

Looks like I’m one version behind. Let’s update it and see if this resolves our issue with an outdated Lambda Python runtime. I’m not going to go over this process step by step, but if you’re interested in how to update your AWS Control Tower, have a read of my previous blog post.

A little while later, my AWS Control Tower Landing Zone is up to date.

Looking back into my Audit account, let’s have a look at the function and see if it’s been updated.

Nope, no change at all. Going in to the CloudFormation StackSet, there has been no update to the template either.

What Now?

Well, because this Lambda function is deployed by AWS Control Tower, you don’t want to go messing with this function at this stage.

According to the Lambda Runtime depreciation policy:

• From a security perspective AWS will be continuing to provide security updates to the Python Runtime until the 18th of July 2022.
• From an operations perspective, Python 3.6 functions will be able to be created and updated up until the 17th of Aug 2022.

This means we have a bit of time so AWS can fix this issue.

Looking on Twitter, I’ve seen AWS Support reaching out to some of the individuals asking what’s going on as they have also received these Needs Action emails. They’ve been saying that they have notified the AWS Control Tower team and there is currently no instructions for fixing this issue yet.

As AWS Control Tower is a managed service from AWS, I’m sure we will see some updates and news about this issue soon.

I will update this blog post with more information once its released, if you see anything before I do, let me know in the comments below.