☰

Apache CLF 로그 파일의 일부 데이터 필드 요약

tag-icon • tag-icon shell-script text-processing scripting apache-httpd
Apache CLF 로그 파일의 일부 데이터 필드 요약

원시에서 일부 정보를 수집하고 싶습니다.결합된 로그 형식 아파치통나무:

51.254.56.62 - - [01/Jun/2016:20:49:28 +0500] "GET /vendors/jquery.slimscroll.min.js HTTP/1.1" 404 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:28 +0500] "GET /jquery.fullPage.js HTTP/1.1" 304 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/TweenLite.min.js HTTP/1.1" 304 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/EasePack.min.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/rAF.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/networkconfig.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:29 +0500] "GET /js/rAF.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:29 +0500] "GET /js/networkconfig.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:28 +0500] "GET /vendors/jquery.slimscroll.min.js HTTP/1.1" 404 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:28 +0500] "GET /jquery.fullPage.js HTTP/1.1" 304 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"

이것이 내가 한 일입니다:

  awk '{ print $1,$11}' accesslog | sort | uniq -c | sort -nr | head -n 10

  3 51.254.56.62 "http://networkconfig.net/"
  3 51.254.56.62 "http://networkconfig.com/"
  2 182.180.10.40 "http://networkconfig.net/"
  2 182.180.10.40 "http://networkconfig.com/"

내가 얻고 싶은 것은 :

Domains                     Hits By IP

networkconfig.net           3 hits 51.254.56.62  | 2 hits 182.180.10.40 and so on
networkconfig.com           3 hits 51.254.56.62 | 2 hits 182.180.10.40 and so on

답변1

수정된 버전(3) 중못생긴 쉿:

#!/bin/bash
{ echo "Domains  Hits by IP" ; \
  awk '{ print $1 gsub(/^.*:\/\/|\"|\/.*$/,"",$11) "\t" $11 }' $1 | \
      sort | \
      uniq  -c | \
      sort -k3,3 -k1,1nr | \
      while n="" read a b c; do \
          [ $a = 1 ] && p='' || p=s ; \
          if [ "$n" = "$c" ] ; then \
               echo -n "  |  $a hit$p $b" ; \
          else echo ; \
               echo -n   "$c $a hit$p $b" ; \
          fi  ; n="$c" ; \
      done ; \
      echo ; \
} | \
while read a b ; do \
    printf "%-30s   %s\n" "$a" "$b" ; \
done

출력 ./ugly.sh accesslog:

Domains                          Hits by IP

networkconfig.com                3 hits 51.254.56.62  |  2 hits 182.180.10.40
networkconfig.net                3 hits 51.254.56.62  |  2 hits 182.180.10.40

(OP의 데이터 URL ./ugly.sh log.txt:로그.txt):

Domains                          Hits by IP

-                                1 hit 180.76.15.138  |  1 hit 192.243.55.136
www.google.com.pk                3 hits 122.129.73.92
www.networkconfigorchard.com     2 hits 39.46.59.57  |  8 hits 39.46.6.0

관련 정보