It happened after Ubuntu Auto-Remove iproute2
上文配置好 Nix 管理后（还没完全发布），有些洁癖，就用 apt 卸载掉部分基础包，同时执行 apt auto-remove。这时习惯性 reboot 后发现 ssh 连接失败，控制台重启系统也还是不行。无奈，用 KVM 提供的 VNC 上去看看。但并不能复制
备份
救援模式启动！
挂载系统盘后备份 Nix 配置文件。
lsblk 或 fdisk -l 查看硬盘和分区，找出我们的系统盘
root # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 79.9M 1 loop /snap/lxd/22923
loop1 7:1 0 87M 1 loop /snap/lxd/28373
loop2 7:2 0 63.9M 1 loop /snap/core20/2318
loop3 7:3 0 38.8M 1 loop /snap/snapd/21759
loop4 7:4 0 61.9M 1 loop /snap/core20/1434
sr0 11:0 1 376K 0 rom
vda 252:0 0 80G 0 disk
├─vda1 252:1 0 79.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
root # fdisk -l
Disk /dev/loop0: 79.95 MiB, 83832832 bytes, 163736 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop1: 87.03 MiB, 91258880 bytes, 178240 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop2: 63.95 MiB, 67051520 bytes, 130960 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop3: 38.83 MiB, 40714240 bytes, 79520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop4: 61.9 MiB, 64909312 bytes, 126776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/vda: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier:
Device Start End Sectors Size Type
/dev/vda1 227328 167772126 167544799 79.9G Linux filesystem
/dev/vda14 2048 10239 8192 4M BIOS boot
/dev/vda15 10240 227327 217088 106M EFI System
Partition table entries are not in disk order.
Type 与 Szie 都能发现 /dev/vda1 是我们要找的跟分区
创建挂载点并挂载分区
- 创建一个挂载点目录，这里选择
/mnt/rescue
mkdir /mnt/rescue
- 挂载根分区到这个挂载点：
mount /dev/vda1 /mnt/rescue
如果需要，可以用这个方法挂载
/boot分区
这时就可以顺利备份，或者还可以在挂载好必要的文件系统后使用 chroot 进入挂载的环境，方便修复
退出时使用 umount 卸载所有文件系统再 reboot
修复
本来到此就可以告一段落，直接重置系统安装 Nix，恢复配置一条命令结束，但想着在 Linux 下遇到一点问题就重装还是不爽，还是想试试恢复。于是就有了下面的部分
ifconfig 发现只剩下 loopback
ifconfig -a / ip a 看网络配置，网卡还在，只是没激活
demsg 看报错，但在这个 VNC 下，less 会有 bug，用着用着就输入错乱，也不能方便的导出到本地（其实可以导出，输出写入文件即可，但是懒得再次关机进入 rescue system）
lspci 查网卡是否还在，存在 Ethernet controller: Red Hat, Inc. Virtio network device 条目
运行 Ubuntu(17.10 及之后版本使用
Netplan), 于是去/etc/netplan/看配置, 长下面这个样子
network:
version: 2
ethernets:
eth0: # 网卡名称可能与 eth0 不同，根据实际情况(ifconfig -a)调整
dhcp4: no
addresses:
- VPS IP/24 # CIDR 表示法
gateway4: gateway IP # 也可能是 routes 字段中的 via 子项的值
nameservers:
addresses:
- 8.8.8.8
- 8.8.4.4
ifconfig 激活网卡
ifconfig ${网卡名} up
接着用 ifconfig 配置 ipadress 与 mask
ifconfig ${网卡名} ${ipadress}
ifconfig ${网卡名} netmask ${mask, 根据 `/etc/netplan/` 中文件的 CIDR 表示的 mask 转为 x.x.x.x 格式}
后用 route 配置网关
route add default gw ${`/etc/netplan/` 里查到的网关地址}
也可以用 ip 配置，随后尝试 ping 8.8.8.8，没有问题
但 ping www.google.com 失败，ip 能访问，域名访问失败，猜测是本地 nameserver 配置问题。编辑 /etc/resolv.conf，加上一个域名服务器地址即可
至此，SSH 可以链接了，下面来排查当时具体移除了什么 pkg（其实这一步该放到最开始看，但当时没多想就先想着有网连上 SSH 后好用）
回到案发现场（/var/log/apt/history.log）
Start-Date: 2024-07-10 06:08:05
Commandline: /usr/bin/unattended-upgrade
Upgrade: python3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-stdlib:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), python3.10-minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4)
End-Date: 2024-07-10 06:08:08
Start-Date: 2024-07-13 18:21:13
Commandline: apt remove neovim
Remove: neovim:amd64 (0.6.1-3)
End-Date: 2024-07-13 18:21:14
Start-Date: 2024-07-13 18:50:24
Commandline: apt autoremove
Remove: libvterm0:amd64 (0.1.4-1), lua-luv:amd64 (1.36.0-0-1), libluajit-5.1-2:amd64 (2.1.0~beta3+dfsg-6), x11-common:amd64 (1:7.7+23ubuntu2), libtree-sitter0:amd64 (0.20.3-1), libtermkey1:amd64 (0.22-1), neovim-runtime:amd64 (0.6.1-3), libxmu6:amd64 (2:1.1.3-3), python3-pynvim:amd64 (0.4.2-1), libunibilium4:amd64 (2.1.0-1), libmsgpackc2:amd64 (3.3.0-4), libice6:amd64 (2:1.0.10-1build2), xclip:amd64 (0.13-2), python3-msgpack:amd64 (1.0.3-1build1), python3-neovim:amd64 (0.4.2-1), libsm6:amd64 (2:1.2.3-1build2), python3-greenlet:amd64 (1.1.2-3build1), libxt6:amd64 (1:1.2.1-1), libluajit-5.1-common:amd64 (2.1.0~beta3+dfsg-6)
End-Date: 2024-07-13 18:50:25
Start-Date: 2024-07-13 18:52:19
Commandline: apt remove vim
Remove: vim:amd64 (2:8.2.3995-1ubuntu2.16), ubuntu-server:amd64 (1.481)
End-Date: 2024-07-13 18:52:20
Start-Date: 2024-07-13 18:55:12
Commandline: apt remove git
Remove: git:amd64 (1:2.34.1-1ubuntu1.11)
End-Date: 2024-07-13 18:55:12
Start-Date: 2024-07-13 18:55:24
Commandline: apt remove gcc
Remove: gcc:amd64 (4:11.2.0-1ubuntu1)
End-Date: 2024-07-13 18:55:24
Start-Date: 2024-07-13 18:55:59
Commandline: apt autoremove
Remove: libgomp1:amd64 (12.3.0-1ubuntu1~22.04), manpages-dev:amd64 (5.10-1ubuntu1), gcc-11:amd64 (11.4.0-1ubuntu1~22.04), libtsan0:amd64 (11.4.0-1ubuntu1~22.04), cpp:amd64 (4:11.2.0-1ubuntu1), libjpeg-turbo8:amd64 (2.1.2-0ubuntu1), libdeflate0:amd64 (1.10-2), libjpeg8:amd64 (8c-2ubuntu10), libtiff5:amd64 (4.3.0-6ubuntu0.9), libcc1-0:amd64 (12.3.0-1ubuntu1~22.04), libmpc3:amd64 (1.2.1-2build1), libxpm4:amd64 (1:3.5.12-1ubuntu0.22.04.2), libfontconfig1:amd64 (2.13.1-4.2ubuntu5), fontconfig-config:amd64 (2.13.1-4.2ubuntu5), libsodium23:amd64 (1.0.18-1build2), libasan6:amd64 (11.4.0-1ubuntu1~22.04), libnsl-dev:amd64 (1.3.0-2build2), libpython3.10:amd64 (3.10.12-1~22.04.4), rpcsvc-proto:amd64 (1.4.2-0ubuntu6), libwebp7:amd64 (1.2.2-2ubuntu0.22.04.2), libcrypt-dev:amd64 (1:4.4.27-1), vim-runtime:amd64 (2:8.2.3995-1ubuntu2.16), cpp-11:amd64 (11.4.0-1ubuntu1~22.04), liberror-perl:amd64 (0.17029-1), libitm1:amd64 (12.3.0-1ubuntu1~22.04), libjbig0:amd64 (2.1-3.1ubuntu0.22.04.1), libc-dev-bin:amd64 (2.35-0ubuntu3.8), libc-devtools:amd64 (2.35-0ubuntu3.8), libisl23:amd64 (0.24-2build1), libc6-dev:amd64 (2.35-0ubuntu3.8), libquadmath0:amd64 (12.3.0-1ubuntu1~22.04), fonts-dejavu-core:amd64 (2.37-2build1), libubsan1:amd64 (12.3.0-1ubuntu1~22.04), liblsan0:amd64 (12.3.0-1ubuntu1~22.04), gcc-11-base:amd64 (11.4.0-1ubuntu1~22.04), libgd3:amd64 (2.3.0-2ubuntu2), git-man:amd64 (1:2.34.1-1ubuntu1.11), libtirpc-dev:amd64 (1.3.2-2ubuntu0.1), libgcc-11-dev:amd64 (11.4.0-1ubuntu1~22.04), libatomic1:amd64 (12.3.0-1ubuntu1~22.04), linux-libc-dev:amd64 (5.15.0-113.123)
End-Date: 2024-07-13 18:56:00
Start-Date: 2024-07-13 18:59:03
Commandline: apt remove htop
Remove: htop:amd64 (3.0.5-7build2)
End-Date: 2024-07-13 18:59:03
Start-Date: 2024-07-13 18:59:19
Commandline: apt remove nano less
Remove: less:amd64 (590-1ubuntu0.22.04.3), nano:amd64 (6.2-1), ubuntu-minimal:amd64 (1.481)
End-Date: 2024-07-13 18:59:20
Start-Date: 2024-07-13 19:00:11
Commandline: apt autoremove
Remove: libnl-3-200:amd64 (3.5.0-0.1), libnl-genl-3-200:amd64 (3.5.0-0.1)
End-Date: 2024-07-13 19:00:11
Start-Date: 2024-07-13 19:49:04
Commandline: apt remove iproute2
Remove: byobu:amd64 (5.133-1), cloud-init:amd64 (22.1-14-g2e17a0d6-0ubuntu1~22.04.5), isc-dhcp-client:amd64 (4.4.1-2.3ubuntu2.3), open-vm-tools:amd64 (2:12.1.5-3~ubuntu0.22.04.4), netplan.io:amd64 (0.106.1-7ubuntu0.22.04.4), iproute2:amd64 (5.15.0-1ubuntu2)
End-Date: 2024-07-13 19:49:07
Start-Date: 2024-07-13 19:49:26
Commandline: apt autoremove
Remove: pastebinit:amd64 (1.5.1-1ubuntu1), zerofree:amd64 (1.1.1-1build3), libatm1:amd64 (1:2.5.1-4build2), libmspack0:amd64 (0.10.1-2build2), libbpf0:amd64 (1:0.5.0-1ubuntu22.04.1), isc-dhcp-common:amd64 (4.4.1-2.3ubuntu2.3), libisc-export1105:amd64 (1:9.11.19+dfsg-2.1ubuntu3), libxmlsec1:amd64 (1.2.33-1build2), run-one:amd64 (1.17-0ubuntu1), libxslt1.1:amd64 (1.1.34-4ubuntu0.22.04.1), libdns-export1110:amd64 (1:9.11.19+dfsg-2.1ubuntu3), python3-newt:amd64 (0.52.21-5ubuntu2), libnetplan0:amd64 (0.106.1-7ubuntu0.22.04.4), libxmlsec1-openssl:amd64 (1.2.33-1build2)
End-Date: 2024-07-13 19:49:27
发现 13日晚上 删除 iproute2 同时 auto-remove 了 cloud-init 以及 netplan，太难蹦了。删除后 Nix 中 home.nix 也包含了 iproute2，却忘了把自动移除的包也写入 home.nix，导致没有网络。那解决方案就很简单了，把缺少的包写入 home.nix 后执行安装即可。
Start-Date: 2024-07-13 19:49:04
Commandline: apt remove iproute2
Remove: byobu:amd64 (5.133-1), cloud-init:amd64 (22.1-14-g2e17a0d6-0ubuntu1~22.04.5), isc-dhcp-client:amd64 (4.4.1-2.3ubuntu2.3), open-vm-tools:amd64 (2:12.1.5-3~ubuntu0.22.04.4), netplan.io:amd64 (0.106.1-7ubuntu0.22.04.4), iproute2:amd64 (5.15.0-1ubuntu2)
End-Date: 2024-07-13 19:49:07