**
Frozen Package Management: A Serverless AWS Pattern for Air-Gapped Fleets
**As organizations move to the cloud, they face the challenge of managing packages for their air-gapped fleets while ensuring security and compliance. In this post, we'll explore a serverless AWS pattern that separates connected package ingestion from an air-gapped fleet, providing an authorized package baseline and explicit decision points for later package changes or upgrades.
**The Challenge: Managing Packages in Air-Gapped Fleets**
Air-gapped fleets, common in regulated industries such as finance, healthcare, and defense, require a secure and controlled environment for package management. However, this often conflicts with the need for operating system updates and package upgrades. Traditional delivery models assume internet access, which is not feasible for air-gapped environments.
**Introducing the Two-Account Pattern**
Our solution involves creating a two-account pattern, where one account is used for connected package ingestion and the other is air-gapped and read-only. The Distribution account owns the writable control plane and internet path, while the Workload account is air-gapped and read-only. This separation provides an additional layer of security and control over package management.
**The Reference Implementation**
Our reference implementation demonstrates the two-account pattern for RPM-based RHEL-family systems, such as AlmaLinux. It consists of the following components:
1. **Distribution Account**: This account owns the writable control plane, internet path, and frozen S3 repository. It runs Amazon EventBridge, AWS Lambda functions, Amazon DynamoDB, and the AWS Fargate sync task. 2. **Workload Account**: This account is air-gapped and read-only, running the internal HTTPS mirror, EC2 Image Builder, Patch Manager, and the compute fleet. 3. **Frozen Package Repository**: This repository is stored in Amazon S3 and contains the frozen package snapshot. It uses a customer-managed KMS key for confidentiality and cross-account authorization control.
**Package Baseline and Scheduled Upgrade Workflow**
Before the scheduled workflow begins, your organization must authorize and run a full sync to establish the initial repository baseline. This bootstrap does not provide package-by-package approval. After the baseline, the detector runs on a customer-defined schedule, defaulting to monthly.
**Benefits and Design Considerations**
Our design provides the following benefits and considerations:
1. **Human Approval is Not Malware Detection**: Human approval is not a reliable method for detecting malware in packages. Use vulnerability intelligence, scanning, pre-production tests, and staged deployment as additional controls. 2. **Defense in Depth**: The sync task verifies a vendor's signature before content enters the trusted repository, and the system verifies it again at installation through dnf. 3. **Cost and Scalability**: Components include S3 storage and requests, KMS requests, Lambda invocations, DynamoDB, SNS, Fargate tasks, the internal load balancer, Distribution-account internet egress, Amazon VPC endpoints, and AMI snapshots. 4. **Validation and Testing**: Validate the internal package management workflow by running the AlmaLinux EC2 Image Builder pipeline and launching a test instance from the generated AMI.
**Conclusion**
Frozen package management is a critical component of air-gapped fleet security. Our serverless AWS pattern separates connected package ingestion from an air-gapped fleet, providing an authorized package baseline and explicit decision points for later package changes or upgrades. By following this pattern and implementing the reference solution, organizations can ensure secure and compliant package management for their air-gapped fleets.
**Learn More**
To learn more about this pattern and implementation, visit the following resources:
* EC2 Image Builder service page * EC2 Image Builder documentation * Patch Manager documentation * Amazon S3 user guide * Reference implementation on AWS Samples