어젯밤에 Ubuntu 서버의 시스템 시계가 5분 앞당겨져 있는 것을 발견하여 "ntpdate pool.ntp.org" 명령을 실행하고 잠자리에 들었습니다.
오늘 아침 나는 SAMBA 주식이 작동하지 않는 것을 발견했습니다. 서버를 보면 권한이 ?로 설정되어 있는 것을 볼 수 있습니다. ? ? 해당 주식이 위치한 곳의 거래량입니다.
서버를 다시 시작했는데 mdadm이 실패한 것을 볼 수 있습니다.
[ 13.920349] sd 3:0:0:0: [sdb]
[ 13.920388] Sense Key : Medium Error [current] [descriptor]
[ 13.920499] Descriptor sense data with sense descriptors (in hex):
[ 13.920559] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[ 13.922059] 00 00 00 00
[ 13.922223] sd 3:0:0:0: [sdb]
[ 13.922255] Add. Sense: Unrecovered read error - auto reallocate failed
[ 13.922316] sd 3:0:0:0: [sdb] CDB:
[ 13.922347] Read(16): 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00
[ 13.922855] end_request: I/O error, dev sdb, sector 0
[ 13.922888] Buffer I/O error on device sdb, logical block 0
[ 13.922927] ata4: EH complete
[ 14.859145] ldm_validate_partition_table(): Disk read failed.
[ 14.859203] Dev sdb: unable to read RDB block 0
[ 14.870317] sdb: unable to read partition table
[ 14.870646] sdb: detected capacity change from 0 to 4000787030016
[ 14.870869] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 14.886265] random: nonblocking pool is initialized
[ 15.510741] md: bind<sdc1>
그래서 여기서 이 문제를 해결하려는 것은 mdadm.conf입니다.
cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
# This file was auto-generated on Mon, 16 Feb 2015 18:24:04 -0500
# by mkconf $Id$
DEVICE /dev/sdb1 /dev/sdc1
ARRAY /dev/md0 level=raid0 devices=/dev/sdb1,/dev/sdc1
다음으로 RAID의 두 디스크 모두에서 smartctl 명령을 실행했는데 둘 다 정상으로 보였습니다.
smartctl -a -s on /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-45-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EZRX-00SPEB0
Serial Number: WD-WCC4E0NLZ6ED
LU WWN Device Id: 5 0014ee 20b74560f
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Oct 2 11:45:31 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Disabled
=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (55740) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 557) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 253 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 184 184 021 Pre-fail Always - 7775
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 15
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 093 093 000 Old_age Always - 5434
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 15
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 6
193 Load_Cycle_Count 0x0032 132 132 000 Old_age Always - 204074
194 Temperature_Celsius 0x0022 119 109 000 Old_age Always - 33
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 96
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 49
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 199 199 000 Old_age Offline - 758
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
smartctl -a -s on /dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-45-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EZRX-00SPEB0
Serial Number: WD-WCC4E0CZDE98
LU WWN Device Id: 5 0014ee 2b6205b40
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Oct 2 11:47:29 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Disabled
=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (52020) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 520) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 253 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 174 174 021 Pre-fail Always - 8258
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 10
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 093 093 000 Old_age Always - 5434
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 10
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 138 138 000 Old_age Always - 188394
194 Temperature_Celsius 0x0022 120 112 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
지금까지는 모든 것이 괜찮아 보이지만 mdadm을 실행하면 다음과 같은 결과가 나타납니다.
mdadm -E /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 933d1825:56122a49:779fbad0:926ab5c9
Name : BAILEYFS01:0 (local to host BAILEYFS01)
Creation Time : Tue Feb 17 17:22:13 2015
Raid Level : raid0
Raid Devices : 2
Avail Dev Size : 7814033392 (3726.02 GiB 4000.79 GB)
Data Offset : 16 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e78061dc:86e60bc0:f4f81839:3816d74a
Update Time : Tue Feb 17 17:22:13 2015
Checksum : 1d8e1dfc - correct
Events : 0
Chunk Size : 512K
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing)
mdadm -E /dev/sdb1
mdadm: cannot open /dev/sdb1: No such file or directory
이것은 어레이의 두 디스크에 대한 fdisk의 출력입니다.
fdisk -l /dev/sdb
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
fdisk -l /dev/sdc
WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdc: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdc1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
다음은 parted의 출력입니다. (이것은 내 모든 드라이브를 나열하는 것 같습니다. 내가 관심 있는 유일한 드라이브는 내 RAID 어레이, sdb 및 sdc를 구성하는 드라이브입니다.)
parted -l /dev/sdb
Model: ATA ST3250318AS (scsi)
Disk /dev/sda: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1049kB 246GB 246GB primary ext4 boot
2 246GB 250GB 3754MB extended
5 246GB 250GB 3754MB logical linux-swap(v1)
Model: ATA WDC WD40EZRX-00S (scsi)
Disk /dev/sdb: 4001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 4001GB 4001GB ext3 primary
Model: ATA WDC WD40EZRX-00S (scsi)
Disk /dev/sdc: 4001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 4001GB 4001GB ext3 primary
Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 3001GB 3001GB ext4 primary msftdata
Model: Seagate Desktop (scsi)
Disk /dev/sde: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 3001GB 3001GB Basic data partition msftdata
이것은 두 디스크 모두에서 gdisk -l의 출력입니다.
gdisk -l /dev/sdb
GPT fdisk (gdisk) version 0.8.8
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 7814037168 sectors, 3.6 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): DA484D62-BB5D-461B-9F96-EAC8A5815C7B
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3693 sectors (1.8 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 7814035455 3.6 TiB 8300 primary
gdisk -l /dev/sdc
GPT fdisk (gdisk) version 0.8.8
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdc: 7814037168 sectors, 3.6 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): BDE33471-BF86-4B5F-9DAF-5D3E67AE7E40
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3693 sectors (1.8 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 7814035455 3.6 TiB 8300 primary
다음에 무엇을 해야 할지 정말 모르겠습니다... 이 배열을 고칠 수 있는 방법이 있나요?
답변1
몇 가지 일반적인 의견.
두 디스크 모두에서 SMART를 비활성화했습니다(또는 적어도 활성화하지 않았습니다). 실행된 테스트가 없으며 이전에 실행된 테스트도 없습니다. 이것이 나에게 알려주는 것은 디스크에 결함이 있는지 알 수 있는 방법이 없다는 것입니다.
커널 오류 메시지는
Add. Sense: Unrecovered read error - auto reallocate failed현재 오류가 발생한 섹터를 교체하는 데 사용할 수 있는 예비 섹터가 더 이상 없기 때문에 디스크가 곧 치명적인 오류를 일으킬 것임을 나타냅니다. 이는 RAID 0 어레이의 디스크에 있어서 정말 나쁜 소식입니다.
전원을 완전히 끄고 재부팅할 수도 있지만, 무슨 일이 있어도 SMART 도구를 설치하고 정기적으로 디스크를 테스트하도록 구성하는 것이 좋습니다.
답변2
의견을 보내주신 @casey에게 감사드립니다!
parted 및 gdisk 명령을 실행한 후에는 디스크가 정상임이 분명하므로 부팅 시 어레이가 실패하는 이유를 알 수 없습니다.
동료가 다음과 같은 달리기를 제안했습니다.
partprobe /dev/sdb
이 작업을 수행하고 mdadm 명령을 다시 실행하면 이제 mdadm이 sdb를 볼 수 있습니다.
mdadm -E /dev/sdb
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm -E /dev/sdc
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
다시 재시작했는데 이번에는 오류도 없이 레이드가 제대로 구축되었습니다.
시작 프로세스가 실행 중인 명령이 무엇인지 파악해야 할 수 있으므로 이런 일이 다시 발생하면 수동으로 명령을 실행할 수 있습니다.
와... 정말 미친 아침이네요!