Just a quick note about loggly.com and AWS logging from S3 and CloudFront. Previously I had looked into parsing CloudFront logs with AWSTATS, just because I’ve used that in the past. That should be possible, but I gave up on it after a few hours.
Recently I had need to look at S3 and ClouldFront logs in some detail and loggly turned up in my search results so I gave it a try. It was pretty straightforward to setup a trial account and they have tons of options for loading all kinds of logs. I choose to use AWS Lambda functions so I could push log files to loggly. They had sample Node.js scripts with instructions for building them and setting up in AWS Lambda. That worked fine and within a couple of hours logs were streaming into loggly.
The ClouldFront logs are great in loggly. I could see and search for what I needed in the HTTP referer, and I could see other nice stuff like the CloudFront edge locations.
The S3 logs didn’t get parsed so well. I think if I cared, I’d modify the Lambda function to parse it better, or use a different import method. But the raw data was there, and I could search for and find what I needed and confirm that a redirect I had set up in my S3 bucket was working.
So loggly was great for what I needed. Their monthly plans seem kind of expensive, at least for what I’m doing in AWS. But I think I qualify for the free tier so I’ll keep loggly around for those times I need to see what’s happening. The only drawback is the free tier only keeps a week of data, but that’s enough to check on things from time to time.