Linux: recover a rm'd file still open

nospam@example.com (Raymond P. Burkholder) — Sat, 01 Jul 2023 02:45:34 +0000

I had an application running which had an open file it was actively using.

I accidentally performed an rm (remove) on the file rather than another I actually meant to remove.

Due to Linux's method of linking files, even though the directory entry link to the file was removed, the file is still open and has an additional link via the application process pseudo directory.

The process sub-directory is composed of the process id. The process id can be found with something like:

$ pidof BasketTrading
2663042

The deleted file can then be found with:

$ lsof -p 2663042 | grep deleted
BasketTra 2663042  rpb   18u      REG               0,46   2739200 29438986 /home/.../BasketTrading.db (deleted)

The 18u reflects the file descriptor used for the file. This can be used to perform a simple copy of the file to an alternate location. A link does not seem to fix it. You may want to complete any outstanding writes to the file first. But do not close it or the application. If you do, the file will be unrecoverable.

$ ls /proc/2663042/fd/18
/proc/2663042/fd/18

$ cp /proc/2663042/fd/18 /home/.../BasketTrading.db.rescue

Debian Linux chroot

nospam@example.com (Raymond P. Burkholder) — Tue, 13 Aug 2019 03:58:03 +0000

When upgrading kernels, I have an issue with lack of space in /boot, so I need to manually remove files, and the rebuild them. In the latest case, I managed to forget to run 'update-grub' which rebuilds the grub menu.

Upon reboot, the kernel aborted with an error along the lines of 'could not mount device(0,0). This was because there was a missing 'initrd .... ' line in the grub menu.

It didn't solve the problem, but I did learn a new sequence of commands. After booting with a live cd:

sudo mount /dev/nvme0n1p4 /mnt
sudo mount /dev/nvme0n1p2 /mnt/boot
sudo mount --bind /proc /mnt/proc
sudo mount --bind /dev /mnt/dev
sudo mount --bind /sys /mnt/sys
sudo mount --bind /dev/pts /mnt/dev/pts
chroot /mnt

The first mount mounts the root file system (I needed to look in /etc/fstab on the target to find that), and the second mounts my separate /boot partition. The other three mounts mount operating system virtual file systems.

The chroot then provides an environment within the target file system for performing management functions without having actually booted into the drive. This being the fundamental of containers.

Under certain circumstances, after changes, the following may be required:

update-grub2

2020/12/02 - Rather than using /dev/nvme... devices, it is possible to mount via other parameters. By using the command 'blkid', one is able to obtain the device, UUID and PARTLABEL. The mount commands above could then be:

# blkid
/dev/nvme0n1p2: UUID="a6c3ee33-993d-41e8-bc88-df9f7347588c" BLOCK_SIZE="1024" TYPE="ext3" PARTLABEL="boot" PARTUUID="64764385-ccc8-4a8b-9258-3d175f6e956e"
/dev/nvme0n1p4: UUID="0c862707-3c94-4920-b253-83e2533f2813" UUID_SUB="f067826b-d66d-4aa4-b4e5-255aa4bccb08" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="root" PARTUUID="d66c9cd9-0e56-4ee0-b577-742ced18de6c"
/dev/nvme0n1p5: UUID="53289aac-6e1d-4f96-9d7c-cbcea3700c44" UUID_SUB="53bfef2c-6eb0-4113-bc5c-988dd76ab0ee" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="var" PARTUUID="547ef81a-ce36-4fc0-831c-a7acae55161b"

# mount PARTLABEL=root /mnt
# mount PARTLABEL=boot /mnt/boot
# mount PARTLABEL=var /mnt/var

Raymond P. Burkholder - Things I Do - Storage

Linux: recover a rm'd file still open

Debian Linux chroot