Hi readers! A website could be live, servers could be running smoothly, and your monitoring dashboard may show all-green indications, but users can still be facing slow-loading pages, failed requests, or random errors. So where should you look when none of the standard metrics tell you what is going on?
Most likely in the place where traffic comes in.
A load balancer receives requests before they hit application servers. From its logs, you will know who made the request, which targets received it, how long it took, which requests have failed, and whether the issue began at the load balancer level or elsewhere in the application.
For example, AWS access logs record such things as request time, client IP address, request URI, target information, latency, and response status. According to AWS, these logs can contain requests that never reach the target at all, making them very valuable when it comes to troubleshooting.
Otherwise, you risk solving the wrong server, scaling the wrong element, or wasting hours trying to find an application error that has originated in the traffic layer.
Understanding Load Balancing Logs
A load balancing log is a set of data that describes the traffic passing through the load balancer. Depending on the environment and settings used, it can include information about requests, targets, connections, times, response codes, clients, and errors.
Load balancing logs show the trail of activity that the traffic leaves.
An Application Load Balancer log includes such fields as client and target IP address, target processing time, request processing time, response processing time, load balancer status code, request line, target group, target status code, user-agent, and rule info.
This information becomes relevant in cases when you need more than a simple uptime check to learn what goes wrong.
What Makes Load Balancer Essential?
The load balancer acts as a mediator between the users and the backend systems. That makes it a good tool to use while investigating problems.
For example, suppose users claim that checkout is taking five seconds. While it seems that everything is fine with the availability of the application servers, you don't know whether this latency results from routing, network activity, the target server, or the application itself.
Here, load balancing logs can be useful in differentiating these possibilities.
For instance, AWS gives processing times separately for the load balancer and target. Comparing these two can give engineers a way to detect latency areas.
Why Ignoring Load Balancing Logs is Risky?
This cost may not be visible as a direct cost for infrastructure resources.
Delays in Requests Can Lead to Financial Issues
Consider an e-commerce site where pages are usually returned within 300 milliseconds. Now, some of the pages take three seconds to return.
The lack of traffic logs makes an engineer to scale out servers. This improves the symptoms but wouldn't solve the root cause of the problem.
Such logs would help to identify a single route with unusually long processing time, leading engineers to application code rather than the entire infrastructure.
Knowing about cloud storage, CDN, and auto-scaling is essential.
Failures May Be Concealed Behind the Dashboard
A dashboard displays good health of servers while the users are still receiving failures.
Good health of infrastructure resources doesn't equal successful requests.
The AWS access logs can give you data on the target status, load balancer status, as well as requests that don't reach the target.
What Do Balancer Activity Logs Tell You?
It all depends on the connection between particular fields, rather than an analysis of each record individually.
Data Point | What It Reveals? |
Request path | Frequently problematic routes, or routes accessed |
Client IP | Unusual clients and traffic sources |
Target status | Backend application responses |
Status code | Failed and successful requests |
Target group | Which backend pool handled traffic |
Processing time | Potential latency sources |
User agent | Crawler and client patterns |
Response size | Unusual or large requests |
Request size | Heavy response traffic |
Rule information | How routing decisions influenced requests |
An increase in 5xx errors would warrant additional exploration beyond just calculating the number of errors.
One would be curious about where the errors are originating – the load balancer or the target applications.
The latest iteration of CloudWatch on AWS helps in making this differentiation possible, as one can query ALB and attribute the 5xx errors to either the load balancer or the target applications.
Access Logs for Load Balancers and Performance Monitoring
Access logs generated by load balancers can prove to be very valuable if combined with the performance measures.
Identify the Slow Routes
Average application latency doesn’t always indicate the problem accurately.
Let's assume that an application receives 100,000 requests per hour. The majority of requests process quickly, but one API route is causing significant delays for a certain percentage of users.
Calculating the average response time might fail to reveal any information about the problem.
Exploring higher percentile latency values and categorizing the requests based on the route would give a different picture. AWS currently offers guidance on querying routes with high average, p90, and p99 target processing times.
Instead of asking "Is my application slow?", it's better to rephrase the question as "Which route is more useful?’
Detect Problematic Targets
A load balancer needs healthy backend targets to function properly. When there is a failure of health checks on targets, it could become a problem that will have implications for availability in the future.
AWS currently offers support for the Application Load Balancer access logs, connection logs, and health check logs using CloudWatch Logs. It is possible to examine health check failures along with requests made.
It becomes possible to tie two events together.
The target might experience problems with health checks before the customers encounter issues. Without any historical data, the engineers could see only the last incident. With this information, it becomes easier to examine the process.
In case you wish to know about Amazon CloudWatch logs, it would be beneficial for you to read this blog.
Request Logs from Load Balancers Are Useful for Enhancing Security
Performance is not the only factor to check when analyzing traffic logs.
Traffic logs may assist security personnel in finding unusual patterns in requests.
For example, an unexplained spike in requests from a handful of IP addresses can be considered. Requests to paths that should not exist, malformed request patterns, and unusual user agents may be worth investigating too.
Records in AWS Application Load Balancer include request paths, client information, user agents, status codes, and classifications for some malformed requests.
But do not use log data as the only security measure.
Log data works best when it is integrated into the larger security framework, which comprises authentication measures, access control policies, firewalls, WAF rules, vulnerability management practices, and application monitoring practices.
The bottom line is that log data allows you to base your analysis of suspicious activities on actual facts.
How to Analyze Traffic Data from a Load Balancer?
Having collected logs is just half the battle. A huge number of logs does not mean valuable insights are obtained right away.
Start With a Question
I would recommend approaching analysis with a particular question in mind.
Rather than opening a thousand logs and trying to search through them randomly, one may consider asking whether 5xx errors have increased since the last deployment, which routes experience high latency, which targets receive the most traffic, and which client generates an unusually high amount of requests.
This will simplify log analysis a lot.
Cross-Reference Logs
The best analyses usually do not rest on one source of information.
Correlate the traffic data with application logs, infrastructure data, deployments, database performance, and security data.
If one sees from traffic logs that the time for target processing has increased at 10:15 AM, and from application logs, one sees that the same thing happened with database queries, then the investigation gets a lot stronger.
This is when observability goes beyond mere data collection.
Structured Analysis Approach
Previous workflow methods usually meant storing the compressed access logs in Amazon S3 and using them by means of Athena. The newer CloudWatch Logs approach offered by AWS is another option when dealing with Application Load Balancing logs.
According to AWS, the access, connection, and health check logs for ALB can now be sent to CloudWatch Logs in JSON format, where users can benefit from Log Analytics, Live Tail, metric filters, and other features.
This simplifies the investigation process since it does not require as much custom-built infrastructure for logging.
When Should You Look into Your Logs?
You shouldn't have to wait until the entire system goes down.
Look into your logs if you see unexpected latency, health checks failing, repeated 5xx errors, traffic anomalies, unexpected routing results, or any other discrepancies between users' experience and what you see in the data.
They also come in handy when there is a large deployment.
Comparing performance before and after a deployment could be useful if performance changes right after a release.
It doesn't mean that you have to monitor logs manually all the time. There are ways to automate that.
The current AWS CloudWatch service even allows you to set up alarms based on Log Analytics queries, including alarms for health check failures.
Final Words
Load balancer data is located at a critical point in the application flow. This data allows you to determine what users asked for, how much time was spent at each stage, where requests were routed, and what failed.
Overlooking such a rich source of data will make troubleshooting more difficult and lead you to fix the effects, not the causes of the problems.
I would suggest a clear way: collect valuable logs, set up reasonable retention policies, keep an eye on the relevant fields, look for patterns rather than single events, and join traffic logs with application and infrastructure metrics.
For AWS customers, the newly available ability to use CloudWatch Logs for storing Application Load Balancer access, connection, and health check logs brings structured log analysis closer to the observation process.
But the real power does not lie in having more logs. It lies in asking better questions.
Sources and References
https://aws.amazon.com/blogs/mt/analyze-application-load-balancer-logs-with-amazon-cloudwatch-logs/
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html