Learn how to – Analyze Nginx/Apache Log Files using SEO Log File Analyser

Learn how to – Analyze Nginx/Apache Log Files using SEO Log File Analyser.

Normally your web server records every request sent to it in a log file for analysis to reveal access data to your site. Web servers such as Apache and Nginx store their log files under /var/log/access_log or logs/access.log. Getting logs using this method is always challenging in that:

• Clients or other internal teams maybe unwilling to provide log files or access to retrieve them
• There might be huge file sizes
• Log files that only store recent data (based either on a number of days or hits)
• Finding that log files have been disabled by a server admin and are not available

A web server log file contains the following data:

• The HTTP status code of the request
• The URL of the page or resource being requested
• The IP address of the request server
• The method of the request (GET/POST)
• The user agent making the request (e.g., Googlebot)
• Timestamp of the hit (time and date)

Looking at these logs for the first time can be so confusing and you may not be in a position to understand. An SEO Log File Analyser can be used to process, store and give an analysis of log files in a smart database that can then be used to make informed decisions.

An SEO Log File Analyser offers the following features:

• Identify Crawled URLs – analyse the exact bots crawling the URL.
• Discover Crawl Frequency – Gives insights on how frequent the URL has been crawled and the number of bot events.
• Find Broken Links & Errors – Helps discover the broken links and errors encountered by search engine bots.
• Improve Crawl Budget – By giving the analysis of crawled URLs, one is able to make decisions to improve the crawl efficiency.
• Finding frequently crawled redirect chains – You can as well identify orphan or unknown pages or URLs which Googlebot hasn’t crawled.
• Combine & Compare Data – you can import crawls, directives, or external link data for advanced analysis.
• Audit Redirects – It helps identify temporary and permanent redirects encountered by search bots, that might be different to those in a browser or simulated crawl.

In this guide, we will walk through how to install the Screaming Frog SEO Log File Analyser that allows one to upload log files, analyze and give a detailed SEO insight. The Screaming Frog SEO Log File Analyser is available in free and paid versions with improved features and capabilities.

Download the Screaming Frog SEO Log File Analyser

This guide demonstrates how to install the Screaming Frog SEO Log File Analyser on:

• Windows
• MacOS
• Linux

The Screaming Frog SEO Log File analyzer can be downloaded from the official Screaming Frog SEO log file analyzer page.

Alternatively, you can use Wget to pull down the free version as below:

##On Windows
wget https://download.screamingfrog.co.uk/products/log-file-analyser/ScreamingFrogLogFileAnalyser-4.5.exe

##On macOS
wget https://download.screamingfrog.co.uk/products/log-file-analyser/ScreamingFrogLogFileAnalyser-4.5.dmg

##On Linux(Ubuntu/Debian)
wget https://download.screamingfrog.co.uk/products/log-file-analyser/screamingfroglogfileanalyser_4.5_all.deb

Install Screaming Frog SEO Log File Analyser.

Once downloaded as above, proceed and install the tool on your system.

• On Windows

On Windows systems, install the .exe file by right-clicking and selecting install

• On macOS

On macOS, locate the .dmg file, double click to mount it and the contents of the file should appear

• On Linux(Debian/Ubuntu).

To install the .deb package on Debian/Ubuntu proceed as below.

sudo apt install xdg-utils
sudo dpkg -i screamingfroglogfileanalyser_4.5_all.deb

Sample Output:

Selecting previously unselected package screamingfroglogfileanalyser.
(Reading database ... 146779 files and directories currently installed.)
Preparing to unpack screamingfroglogfileanalyser_4.5_all.deb ...
Unpacking screamingfroglogfileanalyser (4.5) ...
Setting up screamingfroglogfileanalyser (4.5) ...
Processing triggers for gnome-menus (3.36.0-1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for mailcap (3.69) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.9.4-2) ...

Analyze Nginx / Apache Log Files using SEO Log File Analyser

Once installed, launch the application from the App Menu.

Accept the license terms.

The application will be launched as below.

Import a log file. For this guide, I will demonstrate using an Nginx access.log file.

Once imported, set the name and timezone of the project.

Set the user agents, here you can add multiple of them.

Click Ok and proceed to import the log file. Enter the URL of the site

You will then be granted an overview of the logs as below.

Now you can see several details about the added log file. You can see the URLs and their details.

The events data can as well be viewed.

You can also view IPs and the number of events associated.

Conclusion.

That was enough learning!

Based on the data presented by this SEO Log File Analyser, you can make better-informed decisions about your site and hence improve the crawl budget. I hope this guide was of great importance to you.

See more on this page:

• Forward Kubernetes Logs to Elasticsearch (ELK) using Fluentbit
• How To Forward Logs to Grafana Loki using Promtail
• Forward Server logs and metrics to Elasticsearch using Beats