Digital Endeavours was engaged by a consultancy working for The National Archives to provide AWS Infrastructure expertise for their case law data enrichment service, which provides free public access to UK legal judgments, during the Alpha phase. The consultancy held the prime contract, leading a multi-disciplinary team including data scientists and government staff. Digital Endeavours’s role focused on establishing robust deployment processes and infrastructure improvements ahead of handover to The National Archives for ongoing operational management.
As a secondary project with limited time exposure, Digital Endeavours collaborated with the consultancy’s data scientists and government stakeholders. Digital Endeavours’s infrastructure automation enabled reliable deployment of the data science team’s Python development, with clear separation of responsibilities - data scientists owned application logic whilst Digital Endeavours owned infrastructure automation and AWS architecture.
As the Alpha phase project approached handover, the AWS Infrastructure required enhancement to support ongoing operational management by The National Archives. The consultancy engaged Digital Endeavours’s AWS Infrastructure expertise to establish deployment validation processes and environment promotion controls, standardise environment configurations to support automation reliability, and implement network security patterns appropriate for operational sustainability.
Digital Endeavours proposed and implemented a comprehensive CI/CD workflow redesign, establishing mandatory staging validation before production deployment. The new workflow incorporated pull request validation with Terraform linting, formatting checks, and Python testing to ensure code quality before merge. Staging and production deployment pipelines were separated with automated dependencies, ensuring staging success was required before production deployment could proceed. Lambda function and Docker image deployment automation was implemented, complementing the data science team’s Python development work. Environment naming was standardised to resolve the staging/dev confusion that had affected workflow reliability, and the database was migrated from public to private subnets to align with security best practices. Given time constraints, Digital Endeavours produced a MoSCoW prioritisation matrix covering additional infrastructure improvements, enabling stakeholders to assess priorities for each individual phase.
To support operational handover from the consultancy to The National Archives, Digital Endeavours created comprehensive AWS architecture diagrams using Lucidchart. These diagrams illustrated the interconnected services spanning multiple AWS accounts and components delivered by different teams, providing essential visibility for the multi-team project structure.
Results
The redesigned CI/CD workflow successfully enforced staging validation, eliminating production deployments without testing. The standardised naming and improved database security established a more robust foundation for the service. The consultancy successfully handed the service over to The National Archives, with stakeholders acknowledging the robust infrastructure improvements. The MoSCoW matrix provided a roadmap for future infrastructure evolution as the service progressed beyond Alpha phase.
Key Technologies
CI/CD: GitHub Actions
Infrastructure: AWS (Lambda, VPC, RDS, ECR), Terraform
Application: Python
Documentation: Lucidchart
