Server Administration

Reading Server Access Logs for Security and SEO Insights

Reading Server Access Logs for Security and SEO Insights

When I investigate a hacked WordPress site, the first thing I ask for is the access logs. Analytics tools only show visitors who load JavaScript and accept cookies. Security plugins only see what happens after WordPress loads. Server access logs see everything: every request, from every visitor, bot and attacker, in the order it happened.

Logs are also one of the most underused tools for SEO. They show exactly which pages search engines crawl, how often and what responses they get, something no other tool shows as directly.

Reading logs can look intimidating at first. Thousands of lines of IP addresses and cryptic text. But once you understand the format and a handful of techniques, they become incredibly revealing.

What an access log contains

Most web servers, including Nginx and Apache, use a similar format called the combined log format. A typical line looks like this:

Breaking it down:

  • 203.0.113.25: the IP address that made the request. Behind a CDN, this may be the CDN’s address unless the server is configured to log the real visitor IP.
  • Timestamp: when the request happened, with time zone.
  • Request: the method, like GET or POST, the URL path and the protocol.
  • Status code: 200 for success, 301 or 302 for redirects, 404 for not found, 403 for forbidden, 500 for server errors.
  • Bytes: the size of the response.
  • Referrer: the page that linked to this request, if any.
  • User agent: the browser or bot identifying itself.

Error logs are separate. They record problems, like PHP errors, permission issues and upstream timeouts, rather than every request.

Where to find logs

  • Managed or shared hosting: usually in the control panel, often under a “Logs” or “Statistics” section. Some hosts only keep logs for a short time or charge for access.
  • Your own server: typically in /var/log/nginx/ or /var/log/apache2/, often with a separate log per site if configured that way.
  • CDN logs: if you use a CDN, it may have its own logs showing traffic before it reaches your server.

Check how long your logs are retained. Many servers rotate logs and delete old ones after a week or two. For security investigations, older logs are often exactly what you need, so consider keeping them longer, compressed.

Basic command-line techniques

You don’t need specialised software to start. A few standard commands go a long way on a Linux server.

Show the most recent requests:

Follow requests live as they happen:

Find the IP addresses making the most requests:

<code>awk '{print $1}' access.logsortuniq -csort -rnhead -20</code>

Find the most requested URLs:

<code>awk '{print $7}' access.logsortuniq -csort -rnhead -20</code>

Count requests by status code:

<code>awk '{print $9}' access.logsortuniq -csort -rn</code>

Search for requests to a specific file:

<code>grep "wp-login.php" access.logtail -20</code>

Compressed older logs can be searched with zgrep or zcat without extracting them.

For a friendlier view, tools like GoAccess generate interactive reports from logs in the terminal or as an HTML page. They’re great for getting an overview quickly.

Using logs for security

Spot brute force attacks

Repeated POST requests to wp-login.php or xmlrpc.php from the same IPs are classic brute force patterns:

<code>grep "POST /wp-login.php" access.logawk '{print $1}'sortuniq -csort -rnhead
grep "POST /xmlrpc.php" access.logawk '{print $1}'sortuniq -csort -rnhead</code>

Hundreds or thousands of attempts from a handful of addresses show you where rate limiting and blocking would help.

Find vulnerability scanning

Bots constantly probe for vulnerable plugins, exposed files and old scripts. Look for requests to paths that don’t exist on your site:

<code>grep " 404 " access.logawk '{print $7}'sortuniq -csort -rnhead -30</code>

You’ll typically see requests for files like .env, backup archives, phpMyAdmin, known vulnerable plugin paths and random PHP files. Most fail harmlessly, but they show what attackers are looking for and help you confirm sensitive files aren’t exposed.

Investigate a compromise

During a hack investigation, logs help answer the key question: how did they get in?

  1. Find the timestamp of the earliest malicious file or change, using file modification times.
  2. Look at access log entries around that time.
  3. Look for POST requests to plugin files, upload handlers, admin-ajax.php with unusual actions or direct requests to unfamiliar PHP files.
  4. Note the attacker’s IP addresses and search for all their activity.
<code>grep "14/Nov/2026:03:2" access.loggrep POST

Requests that succeed with a 200 status to strange PHP files in uploads or plugin folders are especially suspicious; they may be backdoors being used.

Keep in mind that timestamps can be manipulated on files, and attackers sometimes hide behind many IP addresses. Logs are evidence to be interpreted carefully, alongside file analysis and other clues.

Watch for unusual traffic

Sudden spikes in requests, heavy traffic to a single URL or large volumes from one country or network can indicate an attack, a misbehaving bot or a viral moment. Logs tell you which.

Using logs for SEO

See how search engines crawl your site

Filter for requests from search engine crawlers to see which pages they visit:

<code>grep "Googlebot" access.logawk '{print $7}'sortuniq -csort -rnhead -30</code>

This shows which pages Google crawls most often, which it rarely visits and whether it’s wasting time on low-value URLs like filters, parameters or old pages.

Verify real Googlebot

Anyone can claim to be Googlebot in their user agent. Many scrapers and bad bots do exactly that. To verify, do a reverse DNS lookup on the IP address. Genuine Googlebot IPs resolve to hostnames ending in googlebot.com or google.com, and a forward lookup of that hostname returns the original IP:

Google also publishes the IP ranges its crawlers use. Bing and other major search engines provide similar verification methods.

Find crawl errors and wasted crawling

Check the status codes search engines receive:

<code>grep "Googlebot" access.logawk '{print $9}'sortuniq -c</code>

Lots of 404s mean crawlers are following broken links or old URLs. Lots of 301s mean internal links or sitemaps point to redirected URLs. 5xx errors mean crawlers hit server problems, which can reduce crawling.

Large numbers of requests to parameter URLs, like sorting or filter combinations, suggest crawl effort is being wasted. That’s especially important for large WooCommerce stores.

Check whether important pages are crawled

If a key page isn’t appearing in search, check the logs to see whether search engines have visited it at all. If they haven’t, improve internal linking and check your sitemap. If they have, the issue is more likely content quality or indexing.

Privacy and retention

Access logs contain IP addresses, which can be personal data under privacy laws like GDPR. Keep logs only as long as you need them for security and operational purposes, restrict access to them and mention log processing in your privacy policy. Many businesses keep detailed logs for a few weeks or months, then delete or anonymise them.

Making logs easier to use

For a single site, command-line tools and occasional reports are enough. For many sites or busy servers, consider:

  • Centralised logging, sending logs from several servers to one place for searching.
  • Log analysis tools that build dashboards and alerts.
  • Fail2ban, which watches logs and automatically blocks IPs that show attack patterns.
  • Alerts for spikes in 5xx errors or suspicious requests.

Build a simple log review habit

You don’t need to read logs every day. A short monthly routine is enough for most sites. Look at the top IP addresses and user agents to spot new aggressive bots. Check the count of 5xx errors for signs of server trouble. Scan the most common 404 URLs for broken internal links or old pages that deserve redirects. Filter for search engine crawlers to confirm they’re reaching your important pages without errors.

Fifteen minutes a month gives you a baseline of what normal looks like on your site. When something unusual happens, like a sudden spike in login attempts or a crawler hammering a single URL, you’ll recognise it immediately instead of trying to work out what normal ever was. And if you ever need to investigate a security incident, you’ll already be comfortable with the tools.

Frequently asked questions

Do I need server access to read logs?

Not always. Many hosts provide logs through their control panel. Full command-line analysis requires SSH access, but you can download logs and analyse them on your own computer.

Why do I see my CDN’s IP instead of visitors’ IPs?

Your server sees requests coming from the CDN. Configure your web server to log the real visitor IP from the header the CDN provides, trusting it only from the CDN’s IP ranges.

Are logs better than Google Analytics?

They’re different. Analytics shows user behaviour and marketing data for visitors who load scripts and consent to tracking. Logs show every request, including bots and attackers, but with less context about user behaviour. Both are valuable.

How long should I keep logs?

Long enough to investigate incidents, which often means several weeks to a few months, while respecting privacy obligations. Compress older logs to save space.

Final thoughts

Access logs are the most complete record of what really happens on your site. Learn the format and a few basic commands, and you can spot attacks, trace how a compromise happened, verify search engine crawlers and find crawl problems that affect your SEO. Keep logs long enough to be useful, protect them as the sensitive data they are and check them whenever something seems off. They’ll tell you the truth when other tools can’t.