Andrea Farneti - Wiki

HomeGraph viewStarting ToolsBasic KnowledgeProjectsUtilities
Andrea Farneti - Wiki

Filesystem rebuild: recreate partitions and restore the data

10 min read

Overview

This guide covers the worst-case recovery scenario: the partition table itself is damaged or wrong: not just GRUB, not just a broken config file.

Typical situations:

  • A partition was accidentally deleted (fdisk d, the wrong parted rm, an automated provisioning script gone rogue).
  • /boot is missing entirely (the partition that should be there isn’t, or it’s the wrong size / wrong filesystem).
  • The disk was partly overwritten by another OS installer that didn’t ask the right questions.
  • An unclean disk swap left the partition table in an inconsistent state.

In all these cases, GRUB rescue alone is not enough: you have to rebuild the partition layout before GRUB has anywhere to be installed.

DANGER

This procedure can destroy all data on the disk if you make a single typo.

The commands here (fdisk, mkfs, dd) are immediate and irreversible.

There is no “Are you sure?” safety net for most of them.

IMPORTANT

Try [[#before-you-wipe-anything—try-testdisk-first|testdisk]] before any of this. It can rebuild a destroyed partition table from on-disk signatures, in most cases recovering the original layout without writing a single byte of user data. Don’t skip Step 0.

Decision tree

Use this to choose your path before doing anything destructive:

SituationWhat to do
Partition table is wrong, but all data should still be on disk (you just fdisk d and w-saved without writing anything else)testdisk: non-destructive, recovers the table
Partition table is wrong and a partition was reformatted on topImage the disk first (dd), then testdisk + file carving
Partition table is wrong and data has been overwritten by new partition contentsRestore from external backup (this guide’s Steps 1-7)
/boot is missing but root partition is intactRecreate just /boot (this guide’s Steps 4-7, skip the rsync of root)
Disk is dead at the hardware level (SMART errors, read failures)Stop. Send to data recovery, this guide can’t help

Step 0. Try testdisk first

testdisk is an interactive partition recovery tool that reads the disk surface and reconstructs the partition table from filesystem signatures.

It’s open source, lives in every distro’s repos, and is the first thing to try before reaching for fdisk.

TIP

If testdisk finds most of the partitions but misses one, run Deeper Search before giving up. It’s slower (can take hours on a multi-TB disk) but reads every sector instead of just signature locations.

Step 1: Boot from rescue media

You need to be running on something other than the broken disk.

The classic options:

Once you have a shell on the rescue environment, you control the machine without anything mounted from the broken disk.

Step 2: Image the disk (insurance copy)

This is non-negotiable if the data on the disk has any value at all.

Before any destructive write, take a binary image of the affected disk(s) to a separate location.

If anything goes wrong from Step 3 onwards, this image lets you start over from the exact pre-rescue state.

WARNING

The destination filesystem must support large files (no FAT32 for >4 GB images: use ext4, xfs, or NTFS).

Step 3: Back up the live data (rsync)

Even with the binary image from Step 2, restoring from dd is slow and gives you the broken state back.

A file-level backup of the readable partitions is what you actually want to use during the rebuild.

Mount each readable partition and rsync its content to a backup location:

# Mount the broken root partition (read-only — paranoid mode)
sudo mount -o ro /dev/sdaN /mnt
 
# Start a screen / tmux session — rsync of a full root takes hours
screen -S backup
 
# rsync to an external disk or a remote host
sudo rsync -avxHAX --progress /mnt/ /mnt/external/sda-backup/
 
# Or to a remote host
sudo rsync -avxHAX --progress /mnt/ user@backup-host:/path/to/sda-backup/
 
# Detach from screen with Ctrl+A, then D — comes back with `screen -r backup`

The flags matter:

FlagMeaning
-aarchive mode (preserves permissions, ownership, symlinks, timestamps)
-vverbose
-xdon’t cross filesystem boundaries (so /proc, /sys, etc. don’t get copied)
-Hpreserve hard links
-Apreserve ACLs
-Xpreserve extended attributes
--progressper-file progress (useful for the long copies)

When it finishes, unmount:

sudo umount /mnt

TIP

screen quick reference:

  • screen -S <name> — start a named session
  • Ctrl+A, then D — detach (the command keeps running)
  • screen -ls — list sessions
  • screen -r <name> — reattach

Essential when you’re on an SSH session that might drop mid-copy.

Step 4: Recreate the partition table

Now we get destructive.

Open fdisk and rewrite the table:

sudo fdisk /dev/sda

INFO

Why deleting partitions inside fdisk doesn’t destroy data: fdisk d only removes the entry from the partition table: it doesn’t touch the actual sectors holding the filesystem data.

As long as you don’t mkfs over those sectors afterwards, the data is still there, and testdisk can usually rebuild the entries from the surviving filesystem signatures.

Only mkfs, dd, and the file-level operations after mounting actually overwrite content.

Typical layout for a Linux server (BIOS boot)

PartitionSizeTypePurpose
/dev/sda11 MBBIOS boot (type 4)Reserved for GRUB stage-2 (Legacy BIOS only)
/dev/sda2~1 GBLinux filesystem (type 83)/boot
/dev/sda3remainingLinux filesystem (type 83)/ (root)

Typical layout for a Linux server (UEFI)

PartitionSizeTypePurpose
/dev/sda1500 MBEFI System (type 1)EFI System Partition (ESP) — mounted at /boot/efi
/dev/sda2~1 GBLinux filesystem (type 83)/boot
/dev/sda3remainingLinux filesystem (type 83)/ (root)

TIP

If you have an identical working machine at hand (e.g. a sibling node in a cluster), copy its exact partition layout: start/end sectors, types, alignment.

Eyeballing fdisk boundaries on a fresh table is error-prone, cloning a known-good layout removes that risk.

When the table looks right, press w to write.

There’s no undo.

Step 5: Format the new partitions

# /boot (and / if you didn't preserve its contents)
sudo mkfs.ext4 /dev/sda2
sudo mkfs.ext4 /dev/sda3
 
# EFI System Partition, if applicable
sudo mkfs.vfat -F32 /dev/sda1

IMPORTANT

Don’t reformat partitions whose data you want to keep.

If your root partition (/dev/sda3) wasn’t actually damaged and you intend to restore the backup onto it, formatting wipes it.

Only format the partitions whose data is gone or whose backup you’re confident in.

Step 6: Restore the data from your backup

Mount the new root, then mount /boot (and /boot/efi if UEFI) into the right place:

sudo mount /dev/sda3 /mnt              # / will be here
sudo mkdir -p /mnt/boot
sudo mount /dev/sda2 /mnt/boot         # /boot here
 
# UEFI only — mount the ESP
sudo mkdir -p /mnt/boot/efi
sudo mount /dev/sda1 /mnt/boot/efi

Rsync the backup back into /mnt.

Use the same flags as Step 3, the metadata preservation matters even more during restore:

# From an external disk
sudo rsync -avxHAX --progress /mnt/external/sda-backup/ /mnt/
 
# Or from the remote backup host
sudo rsync -avxHAX --progress user@backup-host:/path/to/sda-backup/ /mnt/

Run this inside screen again: it’s the same multi-hour copy as Step 3, in the other direction.

Step 7: Bind-mount, chroot, and reinstall GRUB

At this point your data is back on the new partitions, but the bootloader hasn’t been installed onto the new partition table yet.

From here on, you follow the same procedure as GRUB rescue, scenario B, picking up from Step 5:

Scenario B: Reinstall GRUB from a Live USB

The canonical procedure: boot a Live USB of any modern Linux distro, chroot into your broken system, and reinstall GRUB from there.

Step 1: Boot from a Live USB

Plug in the Live USB and select it at POST as the boot device.

If the firmware doesn’t pick it up automatically, see BIOS to change the boot order.

TIP

The Live USB version doesn’t have to match the broken install.

A Live USB of Ubuntu 24.04 can chroot into a Debian 11 install and reinstall its GRUB just fine, what matters is having the grub-install binary, which every desktop / server distro ships.

Step 2: Identify your partitions

sudo lsblk -f

Identify:

  • The root partition: typically the biggest, formatted as ext4 / btrfs / xfs. On LVM setups it appears as vg-root (or similar) under /dev/mapper/.
  • The EFI System Partition (ESP): only on UEFI systems. Small (~500 MB), vfat, type EFI.
  • A separate /boot partition, if any. Small, ext4 or ext2.

For UUIDs and filesystem types, use blkid:

sudo blkid

Step 3: Mount the root partition

sudo mount /dev/sdaN /mnt

Replace /dev/sdaN with your actual root partition (e.g. /dev/sda5, /dev/nvme0n1p2, or /dev/mapper/vg-root for LVM).

Step 4: Mount /boot and /boot/efi, if separate

Check what your broken install expects:

cat /mnt/etc/fstab

If /boot is on a separate partition, mount it inside the chroot tree:

sudo mount /dev/sdaM /mnt/boot

If it’s a UEFI system, also mount the EFI System Partition:

sudo mount /dev/sdaK /mnt/boot/efi

Step 5: Bind-mount the kernel pseudo-filesystems

GRUB needs access to a “live” device tree, kernel, and process info when it runs.

Bind-mount them from the Live environment into the chroot:

for d in /sys /proc /run /dev; do
  sudo mount --rbind "$d" "/mnt$d"
done

INFO

The --rbind (recursive bind) flag is important: without it, sub-mounts like /sys/fs/cgroup and /dev/pts aren’t visible inside the chroot and some tools (apt, dpkg, update-initramfs) will fail with cryptic errors.

Step 6: Chroot into your broken install

sudo chroot /mnt

You’re now root inside your actual system, using its own libraries and binaries, not the Live environment’s.

From this point on, run commands as if you were locally logged in as root.

Step 7: Reinstall GRUB

Step 8: Exit cleanly and reboot

exit                    # exit the chroot
sudo umount -R /mnt     # release ALL the bind mounts at once
sudo reboot

Remove the Live USB while the system reboots.

GRUB should now appear (or boot straight to the kernel, depending on your GRUB_TIMEOUT: see How to enter the GRUB boot menu if you want to change that).

WARNING

Skipping umount -R /mnt is a common pitfall: without it, reboot waits for each bind mount to release (one per --rbind you made in Step 5) and adds a ~90-second timeout to the reboot.

Annoying, not fatal.

Link to the full note →

Step 8: Update /etc/fstab with the new UUIDs

This is the most common cause of “I rebuilt everything and it still won’t boot”.

Recreating the partitions changes their UUIDs, but /etc/fstab still references the old ones.

The system gets through GRUB, starts mounting partitions, fails on the UUID mismatch, and drops into emergency mode.

While still inside the chroot:

# Show the new UUIDs
blkid
 
# Show what /etc/fstab expects
cat /etc/fstab

Compare line by line.

Update every UUID in /etc/fstab that doesn’t match the current blkid output:

nano /etc/fstab

A typical fix looks like:

- UUID=old-uuid-here    /          ext4    defaults    0 1
+ UUID=new-uuid-here    /          ext4    defaults    0 1
- UUID=old-boot-uuid    /boot      ext4    defaults    0 2
+ UUID=new-boot-uuid    /boot      ext4    defaults    0 2

TIP

A useful shortcut to spot mismatches at a glance:

diff <(blkid -o export /dev/sda3 | grep ^UUID=) <(grep ' / ' /etc/fstab | awk '{print $1}')

Step 9: Exit cleanly and reboot

exit                       # exit chroot
sudo umount -R /mnt        # release all the bind mounts at once
sudo reboot

Remove the rescue media, then watch the boot carefully.

The first reboot after a filesystem rebuild is the moment of truth: if you see the GRUB menu, the kernel loading, and the login prompt, you’re done.

If it fails at any stage, the most likely culprits, in order:

  1. UUID mismatch in /etc/fstab: boot to the GRUB rescue console, edit fstab from initramfs, retry.
  2. Wrong partition type on the BIOS boot / EFI partition: back to Live USB, fix with fdisk t.
  3. GRUB installed to the wrong device: back to Live USB, redo grub-install against the correct disk.

⚠️ Last Resort: data is gone, no backup

If you reach this point with no usable backup and the data on the broken partitions is genuinely lost (Step 2 image wasn’t taken, no external backup exists, testdisk failed), the honest answer is:

  • For valuable data: stop, power off the disk, send it to a professional data recovery service. Any further write attempts make their job harder. Cost: typically €500-€5000 depending on damage type — only worth it for irreplaceable data.

  • For a recoverable system: reinstall from scratch. Recreate the partition table, install a fresh OS, restore configuration (not data) from your config management (Ansible, Terraform, manual notes, whatever you have). This is fast for a stateless node, painful for a stateful one that didn’t have backups.

The real lesson is that filesystem rebuild without a backup is the same as no rebuild at all.

The whole procedure in this guide assumes Step 2 succeeded: if it didn’t, you’re not rebuilding, you’re installing fresh.

0/00%

Graph View