YATHTFBB - Yet another tutorial how to fix bad blocks

Tue 2015-06-09

There are already many good tutorials on fixing bad block in ext4 filesystems that can be found on the web, but last time I had to do it, did it a bit different. The method presented here does not require so much manual calculation of offsets, but may need more time where the computer with the damaged drive has to run unsupervised.

Disclaimer

I am am not liable if you accidentally wipe all your data.

Sources

The original smartmontools badblocks howto: http://www.smartmontools.org/browser/trunk/www/badblockhowto.xml

How lvm works in linux: https://www.howtoforge.com/linux_lvm

Badblocks Options: http://en.wikipedia.org/wiki/Badblocks

The Problem

How you notice something is wrong

You run smartctl -A /dev/sdX and see a non-nil Current_Pending_Sector count.

user@host:~$ smartctl -A /dev/sda
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.35-std452-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   094   094   062    Pre-fail  Always       -       1310720
  2 Throughput_Performance  0x0005   100   100   040    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   214   214   033    Pre-fail  Always       -       1
  4 Start_Stop_Count        0x0012   097   097   000    Old_age   Always       -       5982
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   040    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   094   094   000    Old_age   Always       -       2915
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1272
191 G-Sense_Error_Rate      0x000a   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       39
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       13718
194 Temperature_Celsius     0x0002   187   187   000    Old_age   Always       -       32 (Min/Max 14/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       88
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
223 Load_Retry_Count        0x000a   100   100   000    Old_age   Always       -       0

This means that some sectors are corrupted and can not be read anymore.

What can you do about this?

The next time a file is to be written to such a sector, the hard drive will internally mark it as "bad" and remap one of the good spare sectors. Until then, the sectors are shown as "Pending" in smartctl's output. The strategy for fixing this is to write zeros to the bad sectors and force a relocate. Of course this will destroy the part of a file which is saved in this block. But as it stands, it is unreadable anyway right now, so you might as well zap it and replace the whole file from backup.

The solution

OK, let's get cracking

To fix the hard drive it must be unmounted. All of the following stuff can safely be done from a Live CD environment, which helpfully also includes all the tools we need.

Grab your favourite system rescue bootable Live CD (for example from http://www.sysresccd.org/) and fire it up. For the remainder of this article I will use systemrescuecd, but the process will be the same anywhere else.

Further I assume the errors are on /dev/sda, so fill in your partition as needed.

Find which partition the damaged sectors are on

First find the LBA (logical block address) of the first error.

root@sysresccd:~ # smartctl -t short /dev/sda

This will show an estimate when the selftest will be finished. When the time is up, run

root@sysresccd:~ # smartctl -l selftest /dev/sda
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.35-std452-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      2864         5742232
# 2  Short offline       Completed without error       00%      2864         -
# 3  Short offline       Completed without error       00%      1314         -

If the short test does not find any errors, run the long test.

Now take a look at your partition table.

root@sysresccd:~ # fdisk -l /dev/sda
Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1D7EC26C-C143-429F-8343-E644947980EE

Device           Start          End   Size Type
/dev/sda1         2048       999423   487M EFI System
/dev/sda2       999424      1499135   244M Microsoft basic data
/dev/sda3      1499136    976771071   465G Linux LVM

Comparing LBA_of_first_error with the Start and End numbers of each partition, locate the partition the corrupt sector is on. In my case it is /dev/sda3.

If your partition is formatted ext2, ext3 or ext4 ("Linux") you can skip the next part. If it has the type "Linux LVM", read on.

Only for logical partitions

It might be the case that the partition containing the errors is a lvm volume. In this case you have to investigate further to find the logical partition the error is on. The tools you will need are pvdisplay, vdisplay and lvdisplay.

In my example case the erroneous partition /dev/sda3 is indeed formatted LVM.

Now we need to do some calculations (for more information on what we are doing here see the original howto).

Write down the difference between the start of the partition and the LBA of the error shown in smartctl's output:

BadPhysicalOffset = LBA_of_first_error - PartitionStart

Let x be the output of

root@sysresccd #  pvdisplay -c /dev/sda3 | awk -F: '{print $8}'

then let pe_size=2*x

Let pe_start be the output of

root@sysresccd # grep pe_start $(grep -l /dev/sda3 /etc/lvm/backup/*)

Let BadPE = BadPhysicalOffset//pe_size where // denotes integer division, i.e. division without remainder.

Next list the PE ranges of your logial volumes and look out for the one in whose extend BadPE falls.

root@sysresccd # lvdisplay --maps |egrep 'Physical|LV Name|Type'

Finally find the device file of this partition

root@sysresccd # lvdisplay | grep "LV Path"

Take a note of the appropriate path and you are good to go.

Here is an example for clarification (cerebellum is the name of the machine with the faulty drive).

BadPhysicalOffset = 5742232 - 1499136 = 4243096

root@sysresccd #  pvdisplay -c /dev/sda3 | awk -F: '{print $8}'
-> pe_size=2*4096=8192

root@sysresccd # grep pe_start $(grep -l /dev/sda3 /etc/lvm/backup/*)
-> pe_start = 2048
-> BadPE = 4243096/8192 = 517

root@sysresccd # lvdisplay --maps |egrep 'Physical|LV Name|Type'
  LV Name                root
    Type             linear
    Physical volume  /dev/sda3
    Physical extents 0 to 2383
  LV Name                swap_1
    Type             linear
    Physical volume  /dev/sda3
    Physical extents 2384 to 4372
  LV Name                home
    Type             linear
    Physical volume  /dev/sda3
    Physical extents 4373 to 119050

root@sysresccd # lvdisplay | grep "LV Path"
  LV Path                /dev/cerebellum/root
  LV Path                /dev/cerebellum/swap_1
  LV Path                /dev/cerebellum/home

Because 0 < 517 < 2383, the faulty logical volume is root. It can be accessed via /dev/cerebellum/root.

Finding all corrupted sectors

Now we know the first damaged sector and on which (logical or physical) partition it is.

We could fix this one now, but then we would have to run the SMART selftest over and over again to find all the other damaged blocks. If these are many, this becomes inconvenient very quickly.

Luckily we can use the program badblocks to generate a list of all bad blocks on a device. After that we use this list to zero out all of the blocks in one go.

IMPORTANT: Badblocks has both a destructive and a non-destructive mode of operation. Choose the right one, or you will wipe your data!

You need to tell badblocks about the filesystems block size. Run on your partition (primary or, in this example, lvm):

root@sysresccd # dumpe2fs /dev/cerebellum/root| grep 'Block size'
dumpe2fs 1.42.12 (29-Aug-2014)
Block size:               4096

root@sysresccd # badblocks -nvs -b 4096 -o bad_blocks_root /dev/cerebellum/root

Badblocks will now run for a long time (multiple hours up to a day) and write the bad block numbers to the file bad_blocks_root.

When badblocks is finished, take a look at this list:

root@sysresccd # cat bad_blocks_root
530131
639448
1052382
1059879
1770719

Find corrupted files

Now it is important to know which files used these blocks for storage so that they can be replaced afterwards. Start the program debugfs.

root@sysresccd # debugfs
debugfs 1.42.12 (29-Aug-2014)
debugfs:

Open your partition

debugfs: open /dev/cerebellum/root

For each bad block in your list, see which inode stored something there.

debugfs:  icheck 639448
Block        Inode number
639448       296528

Next, look up the path associated with this inode.

debugfs:  ncheck 296528
Inode        Pathname
296528       /usr/lib/libwebkitgtk-3.0.so.0.13.2

Take note of the file. It has to be replaced later on.

Do this for every block in bad_blocks_root (or whatever your file is named).

Exit debugfs.

debugfs: quit

Zeroing the bad sectors

Before writing zeros to parts of your partition, make sure you got the right block numbers. You really don't want random zeros in some of your perfectly good files.

For each block number try to read it.

root@sysresccd # dd if=/dev/cerebellum/root of=/dev/null count=1 bs=4096 skip=<BLOCKNUMBER>

This command should fail with an error (after all, the damaged block is unreadable). If it doesn't, check your calculations, especially if you gave the right block size to badblocks.

If all your block numbers are right, you can now finally zero the corresponding blocks.

For each one do

root@sysresccd # dd if=/dev/zero of=/dev/cerebellum/root count=1 bs=4096 seek=<BLOCKNUMBER>

This command should not fail.

Note: If you wonder why the first command uses skip=N and the second uses seek=N, this is because skip denotes an offset in if and seek in of. Compare this to the position of /dev/cerebellum/root. It is important not to mix these up.

Checking the Number of pending sectors again

root@sysresccd # smartctl -A /dev/sda
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.35-std452-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   094   094   062    Pre-fail  Always       -       1310720
  2 Throughput_Performance  0x0005   100   100   040    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   214   214   033    Pre-fail  Always       -       1
  4 Start_Stop_Count        0x0012   097   097   000    Old_age   Always       -       5988
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   040    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   094   094   000    Old_age   Always       -       2916
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1272
191 G-Sense_Error_Rate      0x000a   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       39
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       13724
194 Temperature_Celsius     0x0002   206   206   000    Old_age   Always       -       29 (Min/Max 14/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
223 Load_Retry_Count        0x000a   100   100   000    Old_age   Always       -       0

If Current_Pending_Sector is not 0 yet, other logical partitions have errors too. Go back to the start and repeat the whole process for them.

Replace the damaged files

By now you should have a more or less long list of files that had used the damaged blocks for storage. They loosely fall into three categories:

  1. System files: If they are critical to your core system (e.g. libc), get the right version from the net or from a backup and put it back while still being logged into the live cd. If they are not critical, you might be able to boot and then reinstall the corresponding packages with your distribution's package management software.
  2. Important files in your home directory: Restore them from a Backup. You did make one, right?
  3. Not so important files in your home directory: Browser cache, databases of file indexing software etc. If they are recreated on next login, you can skip restoring them from backup and just delete them.

Conclusion

Congratulations, you have repaired the damaged blocks!

It may be a good idea though to run a SMART selftest now and then in the near future. If more pending sectors develop, you should think of replacing the disc.

Tags:

This text by Ludger Sandig is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.