Matano

Matano manages managing and analyzing vast amounts of security logs in cloud environments, particularly on AWS, by providing a cloud-native security data lake. This solution normalizes unstructured security logs into a structured, real-time data lake, integrating with over 50 sources for security logs and allowing easy extension with custom sources.

Technically, Matano leverages Apache Iceberg for an open table format and ECS (Elastic Common Schema) for open schema standards, ensuring vendor-neutral data ownership. The log transformation pipeline supports custom VRL (Vector Remap Language) scripting to parse, enrich, normalize, and transform logs as they are ingested, all without the need to manage servers. Detection capabilities are enhanced through "Detection-as-Code" using Python, with support for automatic import of Sigma detections.

Operationally, Matano is deployed and managed via a CLI wizard that initializes the AWS account and sets up the necessary resources. The directory structure is organized to manage log sources, detections, and other configurations efficiently. Alerts can be delivered to external systems such as Email, Slack, and other services via SNS topics.

Key considerations include the scalability of the solution, which is designed to handle petabyte-scale data, and the potential for increased costs associated with data retention and processing in large-scale deployments. Additionally, while the real-time monitoring capabilities are robust, the complexity of custom log transformations and detection rules may require significant technical expertise to fully leverage.