VPC Flow Logs
VPC Flow Logs provide visibility into network traffic within Amazon VPCs, essential for diagnosing security issues and monitoring network activity.
| Category | Security Monitoring & Logging |
|---|---|
| This page updated | a month ago |
| Pricing Details | Costs associated with data ingestion and archival, with potential for significant expenses for large volumes of logs. |
| Target Audience | Network administrators, security professionals, compliance officers. |
VPC Flow Logs address the critical challenge of visibility into network traffic within Amazon VPCs, which is essential for diagnosing security issues, monitoring network activity, and ensuring compliance. The technical architecture of VPC Flow Logs involves capturing IP traffic data from network interfaces, subnets, or entire VPCs, without impacting network throughput or latency since the data collection occurs outside the traffic path.
You can configure flow logs to publish data to various destinations, including Amazon CloudWatch Logs, Amazon S3, or Amazon Data Firehose. Each destination requires specific permissions and configurations; for example, publishing to CloudWatch Logs necessitates an IAM role with the iam:PassRole action, while publishing to S3 or Data Firehose requires appropriate bucket or delivery stream permissions.
Key operational considerations include the type of traffic to log (accepted, rejected, or all), the maximum aggregation interval, and the log record format. You can choose between the default AWS format or a custom format, and optionally include additional metadata from Amazon ECS. Log files can be delivered in plain text or Parquet format, with options for Hive-compatible prefixes and hourly partitions to optimize query performance and storage efficiency.
From a technical standpoint, log files are aggregated and published at 5-minute intervals, and the file structure in S3 can be customized to include Hive-compatible prefixes or hourly partitions. The log file names follow a specific format that includes the AWS account ID, region, flow log ID, and timestamp, ensuring clear identification and organization of the logs.
In terms of limitations, the cost of data ingestion and archival can be significant, especially for large volumes of logs. Applying cost allocation tags can help manage these costs by aggregating usage and costs by business categories. Additionally, query performance can be affected by the chosen log format and storage options, with Parquet format generally offering faster query times but potentially higher storage costs for small log volumes.