In a world where vulnerabilities appear in the most unlikely places, such as the recent vulnerability within the xz package, as listed here
Problem 1
Development teams may come under closer scrutiny from security teams regarding which package versions are allowed. Package versions can be controlled easily in Python applications with the use of requirements.txt files. Similar methods are applicable for other languages such as Javascript or Ruby. We have a requirements.in file that generates a requirements.txt file using pipreqs and pip-compile. Here’s the requirements.in file (with packages approved by the security team):
black==22.8.0
isort==5.11.4
boto3==1.26.55
botocore==1.29.55
moto==4.1.6
pytest==7.2.1
pyopenssl==23.1.1
The application itself runs on AWS Lambda and Lambda is built using Dockerised configurations. The security team wants to ensure these packages cannot be updated without their approval.
Solution
As the Docker images are created as part of the CI pipeline in Github Actions, these can be tagged in an incremental fashion using tools such as this repository.
This is coupled with notifications from the CI pipeline on updated Docker Image builds to the security team so nothing can be put into production without their knowledge.
Problem 2
The next problem came from the development team. When a new Docker Image was built and put into ECR, the specified Lambda was still using the previous version in the Terraform code.
Solution
When a new Docker Image appears in ECR, Amazon Eventbridge can trigger a Lambda (I refer to these as control Lambdas) that updates the configuration of your application’s Lambda. This is a fully automated solution to only publish approved packages in a Serverless fashion.
Q&A
Can I use this to update ECS Fargate configurations too?
Absolutely!
Terraform is building the Lambda, wouldn’t that just change the image version in the configuration to the known setting in the Terraform code?
Correct, that is the natural action. It would be necessary to add a lifecycle argument to ignore changes to the image version as per Hashicorp’s documentation
In that case, if the Lambda was removed and recreated, which image would it use?
It would use the version listed in the Terraform configuration, provided it still exists in ECR, if it doesn’t exist in ECR Terraform would error at the apply stage. In order to update the Lambda to the use the most recent image, you have 2 options. Either amend the Terraform code to match the image you want it to use, or trigger the CI/CD pipeline to build a new image after the Lambda has been recreated, this would update the configuration to use the image just uploaded to ECR.
