작업 기계

작업 기계

logrotate15분마다 호출됩니다.

logrotate크기 임계값이 1M를 초과한 후에만 회전


작업 기계

[root@machine1 output]# ls -l --block-size=1K
total 296
-rw------- 1 root root  55 Oct 15 10:46 all_devices.log
-rw------- 1 root root 122 Oct 15 10:00 all_devices.log-2018-10-15-1539612001.gz
-rw------- 1 root root 114 Oct 15 10:45 all_devices.log-2018-10-15-1539614701.gz
[root@machine1 output]# cat ../conf/syslog-ng-rotate.conf 
/app/syslog-ng/custom/output/all_devices.log {
size 1M
copytruncate
dateext
dateformat -%Y-%m-%d-%s
rotate 365
sharedscripts
compress
postrotate
     /app/syslog-ng/sbin/syslog-ng-ctl reload
endscript
}
[root@machine1 output]# crontab -l
# 00 23 * * * /TSM/bkup 1>/dev/null 2>&1
0,15,30,45 * * * * /usr/sbin/logrotate /app/syslog-ng/custom/conf/syslog-ng-rotate.conf 
[root@machine1 output]# cat /etc/system-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
[root@machine1 output]# 
[root@machine1 output]# cat /var/lib/logrotate/logrotate.status
logrotate state -- version 2
"/var/log/yum.log" 2018-5-29-10:0:0
"/var/log/sssd/sssd_nss.log" 2018-10-15-3:35:1
"/var/log/boot.log" 2018-10-15-3:35:1
"/var/log/up2date" 2018-5-29-10:0:0
"/var/log/sssd/sssd_pam.log" 2018-10-15-3:35:1
"/var/log/sssd/sssd_LDAPFG.log" 2018-10-15-3:35:1
"/var/log/sssd/sssd_sudo.log" 2018-10-15-3:35:1
"/var/log/wtmp" 2018-5-29-10:0:0
"/var/log/chrony/*.log" 2018-5-29-10:0:0
"/var/log/sssd/sssd.log" 2018-5-29-10:0:0
"/var/log/spooler" 2018-10-15-3:35:1
"/var/log/falcon-sensor.log" 2018-10-15-3:35:1
"/app/syslog-ng/custom/output/all_devices.log" 2018-10-15-10:45:1
"/var/log/btmp" 2018-10-1-3:28:2
"/var/log/rhsm/rhsmcertd.log" 2018-10-15-3:35:1
"/var/log/puppet/*log" 2018-5-29-10:0:0
"/var/log/maillog" 2018-10-15-3:35:1
"/var/log/secure" 2018-10-15-3:35:1
"/var/log/rhsm/rhsm.log" 2018-10-15-3:35:1
"/var/log/messages" 2018-10-15-3:35:1
"/var/log/sssd/ldap_child.log" 2018-5-29-10:0:0
"/var/log/cron" 2018-10-15-3:35:1
[root@machine1 output]# 

문제가 있는 기계

[root@machine2 output]# ls -l --block-size=1K
total 2492
-rw------- 1 root root 2485 Oct 15 10:47 all_devices.log
[root@machine2 output]# 
[root@machine2 output]# 
[root@machine2 output]# 
[root@machine2 output]# cat ../conf/syslog-ng-rotate.conf 
/app/syslog-ng/custom/output/all_devices.log {
size 1M
copytruncate
dateext
dateformat -%Y-%m-%d-%s
rotate 365
sharedscripts
compress
postrotate
      /app/syslog-ng/sbin/syslog-ng-ctl reload
endscript
}
[root@machine2 output]# 
[root@machine2 output]# 
[root@machine2 output]# 
[root@machine2 output]# crontab -l
# 00 23 * * * /TSM/bkup 1>/dev/null 2>&1
0,15,30,45 * * * * /usr/sbin/logrotate /app/syslog-ng/custom/conf/syslog-ng-rotate.conf 
[root@machine2 output]# 
[root@machine2 output]# 
[root@machine2 output]# 
[root@machine2 output]# cat /etc/system-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
[root@machine2 output]# 
[root@machine2 output]# cat /var/lib/logrotate/logrotate.status 
logrotate state -- version 2
"/var/log/yum.log" 2018-6-18-19:0:0
"/var/log/sssd/sssd_nss.log" 2018-7-1-3:18:1
"/var/log/boot.log" 2018-6-19-3:14:1
"/var/log/up2date" 2018-6-18-19:0:0
"/var/log/sssd/sssd_pam.log" 2018-6-24-3:14:1
"/var/log/sssd/sssd_LDAPFG.log" 2018-9-9-3:47:1
"/var/log/sssd/sssd_sudo.log" 2018-6-24-3:14:1
"/var/log/wtmp" 2018-6-18-19:0:0
"/var/log/chrony/*.log" 2018-6-18-19:0:0
"/var/log/sssd/sssd.log" 2018-6-18-19:0:0
"/var/log/spooler" 2018-9-9-3:47:1
"/app/syslog-ng/custom/output/all_devices.log" 2018-10-10-11:0:0
"/var/log/falcon-sensor.log" 2018-9-9-3:47:1
"/var/log/btmp" 2018-9-1-3:14:1
"/var/log/rhsm/rhsmcertd.log" 2018-9-9-3:47:1
"/var/log/puppet/*log" 2018-6-18-19:0:0
"/var/log/maillog" 2018-9-9-3:47:1
"/var/log/secure" 2018-9-9-3:47:1
"/var/log/rhsm/rhsm.log" 2018-9-9-3:47:1
"/var/log/messages" 2018-9-9-3:47:1
"/var/log/sssd/ldap_child.log" 2018-6-18-19:0:0
"/var/log/cron" 2018-9-9-3:47:1
[root@machine2 output]# 

위의 내용을 바탕으로 cronjob이 15분마다 호출되지 않는다는 것을 이해했습니다 logrotate.machine2

/var/lib/logrotate/logrotate.statuson은 15분마다 호출되지 않는다는 machine2의미 이기 때문입니다 .logrotate

문제의 기계에 대한 관찰 내용은 다음과 같습니다 /var/log/cron.

Sep  9 20:00:01 machine2 crond[102124]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Sep  9 20:00:01 machine2 crond[102124]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Sep  9 20:01:01 machine2 crond[102303]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Sep  9 20:01:01 machine2 crond[102303]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Sep  9 20:10:01 machine2 crond[110162]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Sep  9 20:10:01 machine2 crond[110162]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
:
:
Oct  3 10:50:01 machine2 crond[387863]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 10:50:01 machine2 crond[387863]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 10:57:44 machine2 crond[1895]: (CRON) INFO (Shutting down)
Oct  3 10:57:44 machine2 crond[392146]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 56% if used.)
Oct  3 10:57:44 machine2 crond[392146]: (CRON) INFO (running with inotify support)
Oct  3 10:57:44 machine2 crond[392146]: (CRON) INFO (@reboot jobs will be run at computer's startup.)
Oct  3 10:57:46 machine2 crond[392146]: (CRON) INFO (Shutting down)
Oct  3 10:57:46 machine2 crond[392159]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 96% if used.)
Oct  3 10:57:46 machine2 crond[392159]: (CRON) INFO (running with inotify support)
Oct  3 10:57:46 machine2 crond[392159]: (CRON) INFO (@reboot jobs will be run at computer's startup.)
Oct  3 11:00:01 machine2 crond[397340]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:00:01 machine2 crond[397340]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 11:01:01 machine2 crond[397518]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:01:01 machine2 crond[397518]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 11:10:01 machine2 crond[402878]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:10:01 machine2 crond[402878]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 11:20:01 machine2 crond[407579]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:20:01 machine2 crond[407579]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 11:30:01 machine2 crond[412423]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:30:01 machine2 crond[412423]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)

10월 3일보다 오래된 로그가 없습니다./var/log/cron

참고로....다음 cronjob 항목이 10월 12일에 추가되었습니다.

[root@machine2 output]# crontab -l
    # 00 23 * * * /TSM/bkup 1>/dev/null 2>&1
    0,15,30,45 * * * * /usr/sbin/logrotate /app/syslog-ng/custom/conf/syslog-ng-rotate.conf 

프로세스의 시스템 상태 crond:

[root@machine2 log]# tail -5 /var/log/cron
Oct  3 11:10:01 machine2 crond[402878]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 11:20:01 machine2 crond[407579]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:20:01 machine2 crond[407579]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
Oct  3 11:30:01 machine2 crond[412423]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct  3 11:30:01 machine2 crond[412423]: (root) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)
[root@machine2 log]#
[root@machine2 log]#
[root@machine2 log]#
[root@machine2 log]#
[root@machine2 log]# systemctl status crond
â— crond.service - Command Scheduler
   Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-10-03 10:57:46 EDT; 1 weeks 5 days ago
 Main PID: 392159 (crond)
   CGroup: /system.slice/crond.service
           └─392159 /usr/sbin/crond -n

Oct 15 11:10:01 machine2.abc.com crond[58065]: pam_unix(crond:account): expired password for user root (password aged)
Oct 15 11:10:01 machine2.abc.com crond[58065]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct 15 11:15:01 machine2.abc.com crond[58989]: pam_unix(crond:account): expired password for user root (password aged)
Oct 15 11:15:01 machine2.abc.com crond[58989]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct 15 11:20:01 machine2.abc.com crond[60067]: pam_unix(crond:account): expired password for user root (password aged)
Oct 15 11:20:01 machine2.abc.com crond[60067]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct 15 11:30:01 machine2.abc.com crond[62044]: pam_unix(crond:account): expired password for user root (password aged)
Oct 15 11:30:01 machine2.abc.com crond[62044]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
Oct 15 11:30:01 machine2.abc.com crond[62045]: pam_unix(crond:account): expired password for user root (password aged)
Oct 15 11:30:01 machine2.abc.com crond[62045]: (root) PAM ERROR (Authentication token is no longer valid; new one required)
[root@machine2 log]# 

문제를 진단하는 방법은 무엇입니까?

관련 정보