Observability (WIP)

There are three key services we are using to store logs:

API calls Elastic Load Balancing is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Elastic Load Balancing. CloudTrail captures all API calls for Elastic Load Balancing as events by default. You can view the most recent events in the CloudTrail console in Event history.

Using the information collected by CloudTrail, you can determine the request that was made to Elastic Load Balancing, the IP address from which the request was made, who made the request, when it was made, and additional details (more details).

Access logs Elastic Load Balancing optionally provides access and connection logs.

Access logs capture detailed information about requests sent to your load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses (more details).

Connection logs capture detailed information about requests sent to your load balancer. Each log contains information such as the client's IP address and port, listener port, the TLS cipher and protocol used, TLS handshake latency, connection status, and client certificate details (more details).

You can use these logs to analyze request patterns and troubleshoot issues. We enable these logs using Terraform configuration, which stores them in the logs bucket.

Pricing API calls are stored in CloudTrail for free, as for access and connection logs, you are charged storage costs for Amazon S3, but not charged for the bandwidth used by Elastic Load Balancing to send log files to Amazon S3. So you can reduce costs by log expiration days.

API calls By default CloudTrail captures all S3 management events. So in order to review management events you can visit Event history page in AWS CloudTrail. (more details).

Data events Data events provide information about the resource operations performed on or in a resource (for example, reading or writing to an Amazon S3 object). These are also known as data plane operations. Data events are often high-volume activities. By default, CloudTrail doesn’t log data events. The CloudTrail Event history doesn't record data events (more details).

We use CloudTrail and S3 to store data events, as it offers many advantages, most notable among which are speed of log delivery, JSON log format, seachable UI for logs, etc

Pricing Additional charges apply for data events ($0.10 per 100,000 data events delivered). For more information about CloudTrail pricing, see AWS CloudTrail Pricing. In order to reduce costs we filter down the logs to be only related to backup buckets and only to certain events types.

Control Plane logging Amazon EKS control plane logging provides audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs. These logs make it easy for you to secure and run your clusters. We selected the api, authenticator and audit log types you need, and logs are sent as log streams to a group for each Amazon EKS cluster in CloudWatch. more info.

Node and application logging We use CloudWatch Container Insights to capture logs and metrics for Amazon EKS. Container Insights implements cluster, node, and pod-level metrics with the CloudWatch agent for log capture to CloudWatch. Container Insights also provides automatic dashboards with layered views of your captured CloudWatch metrics. Container Insights is deployed as CloudWatch DaemonSetthat runs on every Amazon EKS node (more info).

WIP

Connection logs VPN client connection are enabled to be stored in CloudWatch dedicated log group.

Pricing CloudWatch pricing apply.

Metrics for all the services are aggregated into and can be access via AWS CloudWatch service, within Metrics explorer.

Application Load Balancer Elastic Load Balancing reports metrics to CloudWatch only when requests are flowing through the load balancer. If there are requests flowing through the load balancer, Elastic Load Balancing measures and sends its metrics in 60-second intervals. If there are no requests flowing through the load balancer or no data for a metric, the metric is not reported (more details). No additional setup is needed.

AWS S3 Storage metrics for Amazon S3 are reported once per day and are provided to all customers at no additional cost (more details).

Elastic Kubernetes Service All the metrics for EKS are collected by Amazon CloudWatch Observability EKS add-on, which uses various methods to provide huge range of metrics on all possible levels. For example resource utilisation (cpu and memory) on container, pod and nodes level. All these metrics are aggregated and accessible via ContainerInsights section of CloudWatch. Container Insights has additional cost (more details).

Hosty blueprint includes number of predefined metric alarms, which are triggered when certain conditions are met:

You can receive these notifications either by email, webhook or both. Open main.tf file and edit locals.alarms_email_address to set email to which you want to recieve notifications (you will receive an email with a confirmation link). Edit locals.alarms_webhook_url to push notifications into a webhook.