TL;DR - 트래픽이 많을 때(약 900) 웹사이트의 CPU가 100%에 도달하고 속도가 느려지고 충돌이 발생합니다.
소개하다
우리 회사는 WordPress CMS를 사용하여 Digital Ocean에서 호스팅되는 40개 이상의 콘텐츠 웹사이트를 보유하고 있습니다. 지난 3일 동안 매일매일 다른 웹사이트의 CPU가 100%까지 올라가서 결국 웹사이트가 다운되는 이상한 일들이 일어나기 시작했습니다.
세부 사항
섬기는 사람
- 기계 사양- 48GB RAM, 12코어, Centos
- 주인- 디지털 오션
- 환경 구성- nginx & php-fpm & mysql - 모두 하나의 서버에 있습니다.
우리가 아는 것
이건 공격이 아닐 수도 있어- 하나의 웹사이트에서 발생했다면 공격으로 간주하지만, 4개의 웹사이트이므로 다른 공격일 수도 있습니다.
이것은 플러그인이 아닙니다- 서비스를 종료하고 다시 시작해 보았으나 여전히 CPU 사용량이 높습니다. 또한 다른 모든 웹사이트에도 동일한 플러그인이 있습니다.
우리는 아무것도 변경하지 않았습니다- 다른 모든 사이트는 잘 작동하고 있으며 이를 위해 어떠한 변경도 하지 않았습니다.
이는 내부 프로세스입니다.- 캐싱을 제공하기 위해 CDN 작업자를 사용하므로 데이터베이스에서 아무것도 요청하면 안 됩니다.
우리가 시도했지만 도움이 되지 않았던 것들
php-fpm 구성 변경- php-fpm 최대 하위 프로세스 수가 50에서 200으로 증가했습니다. 또한 프로세스가 20초 이내에 종료되도록 제한 시간을 설정합니다.
nginx 구성 변경- ngnix fastcgi_ignore_client_abort 끄기
보고서
최고 보고서
php fpm 오류 로그
Nginx 오류 로그
mysql 프로세스 목록
mysql 오류 로그
2019-04-02 21:13:34 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-04-02 21:13:34 0 [Note] /usr/sbin/mysqld (mysqld 5.6.42) starting as process 14287 ...
2019-04-02 21:13:34 14287 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)
2019-04-02 21:13:34 14287 [Warning] Buffered warning: Changed limits: table_open_cache: 431 (requested 2000)
2019-04-02 21:13:34 14287 [Note] Plugin 'FEDERATED' is disabled.
2019-04-02 21:13:34 14287 [Note] InnoDB: Using atomics to ref count buffer pool pages
2019-04-02 21:13:34 14287 [Note] InnoDB: The InnoDB memory heap is disabled
2019-04-02 21:13:34 14287 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-04-02 21:13:34 14287 [Note] InnoDB: Memory barrier is not used
2019-04-02 21:13:34 14287 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-04-02 21:13:34 14287 [Note] InnoDB: Using Linux native AIO
2019-04-02 21:13:34 14287 [Note] InnoDB: Using CPU crc32 instructions
2019-04-02 21:13:34 14287 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-04-02 21:13:34 14287 [Note] InnoDB: Completed initialization of buffer pool
2019-04-02 21:13:34 14287 [Note] InnoDB: Highest supported file format is Barracuda.
2019-04-02 21:13:34 14287 [Note] InnoDB: 128 rollback segment(s) are active.
2019-04-02 21:13:34 14287 [Note] InnoDB: Waiting for purge to start
2019-04-02 21:13:34 14287 [Note] InnoDB: 5.6.42 started; log sequence number 79628880679
2019-04-02 21:13:34 14287 [Note] Server hostname (bind-address): '*'; port: 3306
2019-04-02 21:13:34 14287 [Note] IPv6 is available.
2019-04-02 21:13:34 14287 [Note] - '::' resolves to '::';
2019-04-02 21:13:34 14287 [Note] Server socket created on IP: '::'.
2019-04-02 21:13:34 14287 [Note] Event Scheduler: Loaded 0 events
2019-04-02 21:13:34 14287 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.6.42' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2019-04-02 21:19:56 14287 [Warning] IP address '85.93.20.38' could not be resolved: Name or service not known
2019-04-02 21:21:20 14287 [Note] /usr/sbin/mysqld: Normal shutdown
2019-04-02 21:21:20 14287 [Note] Giving 2 client threads a chance to die gracefully
2019-04-02 21:21:20 14287 [Note] Event Scheduler: Purging the queue. 0 events
2019-04-02 21:21:20 14287 [Note] Shutting down slave threads
2019-04-02 21:21:22 14287 [Note] Forcefully disconnecting 0 remaining clients
2019-04-02 21:21:22 14287 [Note] Binlog end
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'partition'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_METRICS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMPMEM'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_LOCKS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_TRX'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'InnoDB'
2019-04-02 21:21:22 14287 [Note] InnoDB: FTS optimize thread exiting.
2019-04-02 21:21:22 14287 [Note] InnoDB: Starting shutdown...
2019-04-02 21:21:24 14287 [Note] InnoDB: Shutdown completed; log sequence number 79629654999
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'BLACKHOLE'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'ARCHIVE'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'MRG_MYISAM'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'MyISAM'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'MEMORY'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'CSV'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'sha256_password'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'mysql_old_password'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'mysql_native_password'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'binlog'
2019-04-02 21:21:24 14287 [Note] /usr/sbin/mysqld: Shutdown complete
190402 21:21:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190402 21:21:25 mysqld_safe Logging to '/var/log/mysqld.log'.
190402 21:21:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2019-04-02 21:21:25 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-04-02 21:21:25 0 [Note] /usr/sbin/mysqld (mysqld 5.6.42) starting as process 14764 ...
2019-04-02 21:21:25 14764 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)
2019-04-02 21:21:25 14764 [Warning] Buffered warning: Changed limits: table_open_cache: 431 (requested 2000)