Observability (WIP)

logging storage services there are three key services we are using to store logs amazon cloudwatch monitoring service for aws cloud resources and the applications you run on amazon web services you can use amazon cloudwatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your aws resources aws cloudtrail aws service that helps you enable operational and risk auditing, governance, and compliance of your aws account actions taken by a user, role, or an aws service are recorded as events in cloudtrail amazon simple storage service (amazon s3) an object storage service that offers industry leading scalability, data availability, security, and performance we use amazon s3 to store and retrieve logs from s3 buckets and alb application load balancer log storage api calls elastic load balancing is integrated with aws cloudtrail, a service that provides a record of actions taken by a user, role, or an aws service in elastic load balancing cloudtrail captures all api calls for elastic load balancing as events by default you can view the most recent events in the cloudtrail console in event history using the information collected by cloudtrail, you can determine the request that was made to elastic load balancing, the ip address from which the request was made, who made the request, when it was made, and additional details ( more details ) access logs elastic load balancing optionally provides access and connection logs access logs capture detailed information about requests sent to your load balancer each log contains information such as the time the request was received, the client's ip address, latencies, request paths, and server responses ( more details ) connection logs capture detailed information about requests sent to your load balancer each log contains information such as the client's ip address and port, listener port, the tls cipher and protocol used, tls handshake latency, connection status, and client certificate details ( more details ) you can use these logs to analyze request patterns and troubleshoot issues we enable these logs using terraform configuration, which stores them in the logs bucket pricing api calls are stored in cloudtrail for free, as for access and connection logs, you are charged storage costs for amazon s3, but not charged for the bandwidth used by elastic load balancing to send log files to amazon s3 so you can reduce costs by log expiration days s3 bucket log storage api calls by default cloudtrail captures all s3 management events so in order to review management events you can visit event history page in aws cloudtrail ( more details ) data events data events provide information about the resource operations performed on or in a resource (for example, reading or writing to an amazon s3 object) these are also known as data plane operations data events are often high volume activities by default, cloudtrail doesn’t log data events the cloudtrail event history doesn't record data events ( more details ) we use cloudtrail and s3 to store data events, as it offers many advantages, most notable among which are speed of log delivery, json log format, seachable ui for logs, etc pricing additional charges apply for data events ($0 10 per 100,000 data events delivered) for more information about cloudtrail pricing, see aws cloudtrail pricing in order to reduce costs we filter down the logs to be only related to backup buckets and only to certain events types elastic kubernetes service (eks) control plane logging amazon eks control plane logging provides audit and diagnostic logs directly from the amazon eks control plane to cloudwatch logs these logs make it easy for you to secure and run your clusters we selected the api, authenticator and audit log types you need, and logs are sent as log streams to a group for each amazon eks cluster in cloudwatch more info node and application logging we use cloudwatch container insights to capture logs and metrics for amazon eks container insights implements cluster, node, and pod level metrics with the cloudwatch agent for log capture to cloudwatch container insights also provides automatic dashboards with layered views of your captured cloudwatch metrics container insights is deployed as cloudwatch daemonsetthat runs on every amazon eks node ( more info ) rds wip vpn connection logs vpn client connection are enabled to be stored in cloudwatch dedicated log group pricing cloudwatch pricing apply monitoring metrics for all the services are aggregated into and can be access via aws cloudwatch service, within metrics explorer metrics application load balancer elastic load balancing reports metrics to cloudwatch only when requests are flowing through the load balancer if there are requests flowing through the load balancer, elastic load balancing measures and sends its metrics in 60 second intervals if there are no requests flowing through the load balancer or no data for a metric, the metric is not reported ( more details ) no additional setup is needed aws s3 storage metrics for amazon s3 are reported once per day and are provided to all customers at no additional cost ( more details ) elastic kubernetes service all the metrics for eks are collected by amazon cloudwatch observability eks add on , which uses various methods to provide huge range of metrics on all possible levels for example resource utilisation (cpu and memory) on container, pod and nodes level all these metrics are aggregated and accessible via containerinsights section of cloudwatch container insights has additional cost ( more details ) alerts hosty blueprint includes number of predefined metric alarms, which are triggered when certain conditions are met space running out on nodes high cpu/memory utilisation of nodes high cpu/memory utilisation of containers high occurance of 5xx http errors on alb missing pods for 10 minutes you can receive these notifications either by email, webhook or both open main tf file and edit locals alarms email address to set email to which you want to recieve notifications (you will receive an email with a confirmation link) edit locals alarms webhook url to push notifications into a webhook auditing