AWS Config to Elasticsearch
Generates an AWS Config Snapshot and ingests it into ElasticSearch for further analysis using Kibana.
| Category | Configuration & Change Management |
|---|---|
| Community Stars | 71 |
| Last Commit | 4 years ago |
| Last page update | 19 days ago |
| Pricing Details | Free to use under Apache License 2.0, costs may apply for AWS resources used. |
| Target Audience | AWS developers and system administrators looking for enhanced visibility and analysis of AWS resource configurations. |
The integration of AWS Config snapshots into the ElasticSearch/Logstash/Kibana (ELK) stack manages visibility and analysis of AWS resource configurations and changes. This solution automates the ingestion of AWS Config data, enabling comprehensive searching and mapping of AWS environments.
Technically, this setup involves configuring an AWS Elasticsearch Service domain, typically using a t2.small.elasticsearch instance type, and setting up public access with an appropriate access policy.
A Lambda function is triggered by S3 events, specifically when new AWS Config snapshots are uploaded. This Lambda function automates the ingestion of these snapshots into the Elasticsearch domain. The function code is uploaded as a ZIP file, and environment variables are set to include the Elasticsearch endpoint URL.
Key operational considerations include the need for careful management of access policies to avoid security risks, such as allowing open access to the Elasticsearch domain. Additionally, the performance and cost of retaining large volumes of configuration data in Elasticsearch must be monitored, as retention costs can escalate in multi-account setups.
From a technical details perspective, the esingest.py script is used to import AWS Config snapshots into Elasticsearch. This script can be run in verbose mode and supports region-specific imports. It relies on Python packages like boto3 and requests for AWS interactions and data processing.
In Kibana, the index pattern and time filter are configured to enable time-series analysis, with the snapshotTimeIso field often used for time-based queries. This setup allows for detailed searches, such as identifying EC2 instance distributions or security group usage, and visualizing configuration changes over time.