Question posted in
Some documentation to get you started can be found here.

Apache – Get log file entries between two dates, only fatal ones

I searched the web, SO and nothing wants to do the job.
Given a simple apache logfile like

[Fri Jun 22 11:46:13 2018] [error] [client xxxxxxxx] PHP Parse error:  syntax error, unexpected end of file in yyyyyyyyyyyyyyyyy on line 3554, referer: wwwwwwwwwwwwwwwwwwwwwwww
[Fri Jun 22 14:09:37 2018] [error] [client xxxxxxxx] PHP Fatal error:  Call to undefined function date_mysql2german() in yyyyyyyyyyyyyyyyy on line 156, referer: wwwwwwwwwwwwwwwwwwwwwwww
[Mon Jun 25 17:03:37 2018] [error] [client xxxxxxxx] PHP Warning:  mysql_num_rows() expects parameter 1 to be resource, null given in yyyyyyyyyyyyyyyyy on line 1409, referer: wwwwwwwwwwwwwwwwwwwwwwww
[Tue Jun 26 11:46:26 2018] [error] [client xxxxxxxx] PHP Warning:  mysql_num_rows() expects parameter 1 to be resource, boolean given in yyyyyyyyyyyyyyyyy on line 9390, referer: wwwwwwwwwwwwwwwwwwwwwwww
[Tue Jun 26 11:46:26 2018] [error] [client xxxxxxxx] PHP Warning:  mysql_num_rows() expects parameter 1 to be resource, boolean given in yyyyyyyyyyyyyyyyy on line 9432, referer: wwwwwwwwwwwwwwwwwwwwwwww

I need to extract all “PHP Warning:” lines (full line) of the last 5 minutes.

Here is what I tried so far

awk 
-v Date="$(date "+[%a %b %d %H:%M:%S %Y")"  
-v Date2="$(date --date="5 minutes ago" "+[%a %b %d %H:%M:%S %Y")" 
'$4 > Date && $4 < Date2' /var/log/apache2/apache2/my_log_file.log

But even that (without grep for “Warning” e.g.) does not return anything. The pasted log entries and their date/times are just examples – in reality I do have Warnings within the last 5 minutes, so it should definitly return something.

Any ideas on how to change this stuff and get it to work?
Thanks in advance
Alex

Tags:

2

Answers


  1. 0

    The problem is that you compare your strings way too literal. The dates are ordered by data, but awk compares lexicographical (“Jan” < “Feb” datewise but string wise it is not). There are various approaches that can be taken here, but I suggest to make the comparisons in UNIX epoch time.

    $ tend=$(date "+%s")
$ tstart=$(date --date="5 minutes ago" "+%s")
$ awk -F '[][]' '!/PHP Warning/{next}
                 { cmd="date --date=""$2"" "+%s""
                   time=((cmd | getline line) > 0 ? line : -1)
                   close(cmd) }
                 (time == -1) { exit 1 }
                 (tend <= time && time <= tstart)
                ' tstart=$tstart tend=$tend <logfile>

    note: this will execute a large number of calls to date if your file is big.

    A different approach might be calling mktime from GNU awk or reformatting your time string as yyyymmddHHMMSS. This allows you to use lexicographical ordering for strings :

    $ tstart=$(date -d="5 minutes ago" "+%Y%m%d%H%M%S"")
$ tend=$(date "+%Y%m%d%H%M%S"")
$ awk 'BEGIN{ month["Jan"]="01"; month["Feb"]="02"; month["Mar"]="03"
            month["Arp"]="04"; month["May"]="05"; month["Jun"]="06"
            month["Jul"]="07"; month["Aug"]="08"; month["Sep"]="09"
            month["Oct"]="10"; month["Nov"]="11"; month["Dec"]="12" }
      !/PHP Warning/{next}
      { time=$4; gsub(/:/,"",time); year=substr($5,1,4);
        date=sprintf(%4s%2s%0.2d%6s,year,month[$2],$3,time) }
      }
      (tstart <= date && tend <= date)
     ' tend=$tend tstart=$tstart  <logfile>

    Or per suggestion of Ed Morton:

    $ awk '!/PHP Warning/{next}
       { year=substr($5,1,4)
         month=(index("JanFebMarAprMayJunJulAugSepOctNovDec",$2)+2)/3
         time=$4; gsub(/:/,"",time);
         date=sprintf(%4s%0.2d%0.2d%6s,year,month,$3,time)
       }
       (tstart <= date && tend <= date)
      ' tend=$end tstart=$tstart <logfile>

    A related post can be found here: Regex to match logfile custom date formats

    Login or Signup to reply.
  2. 0

    This worked in bash + gawk for me:

    #!/bin/bash

LC_ALL=C gawk -v limit=$(date --date="5 minutes ago" "+%s") '
function epoch(month, day, hhmmss, year) {
    gsub(/:/, " ", hhmmss)
    sub(/]/, "", year)
    month = (index("JanFebMarAprMayJunJulAugSepOctNovDec", month) + 2) / 3
    return mktime(year" "month" "day" "hhmmss)
}
/PHP Warning/ && epoch($2, $3, $4, $5) > limit
' "test.log"

    If you want to include fatal errors as well, just remove /PHP Warning/ && or adjust the regex as desired (e.g. /Warning|error/ or similar).

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search