[SOLVED] slarm64 rebuild-initrd.sh problem

wowbaggerHU · 03-14-2022, 02:36 PM

Hello Everyone,

I ran into a problem, where after a package update, I updated the kernel and related packages on my 8 GB Raspberry PI 4.
After the update, I rebuilt the initrd image, and after a reboot, I was greeted with a kernel panic:

Code:

/bin/ash: error while loading shared libraries: libm.so.6: cannot open shared object file: No such file or directory
Kernel panic - not syncing: Attempted to kill init! exitcode=....

I believe the initrd tree likely doesn't contain that file for some reason.
Is there maybe some reason why this broke? Things were working fine before.

sndwvs · 03-15-2022, 04:10 AM

took the last image slarm64-current-aarch64-server-raspberry_pi_4-5.16.7-build-20220215.img.zst, then updated and recreated uInitrd loaded normally.

wowbaggerHU · 03-15-2022, 05:24 AM

Quote:

Originally Posted by sndwvs

took the last image slarm64-current-aarch64-server-raspberry_pi_4-5.16.7-build-20220215.img.zst, then updated and recreated uInitrd loaded normally.

I will have this topic moved over to the slarm64 forum soon.

On the other hand: I managed to build a working initrd while on another Raspberry PI4, and having chrooted to the problematic machine's SD card.
That image is around 10 MB in size, while the other, that I manage to create locally is only around 6 MB (and doesn't work).
This image I created from a chroot, and works: https://drive.google.com/file/d/1kK3...ew?usp=sharing
This is the image that I create locally and doesn't work: https://drive.google.com/file/d/1ukM...ew?usp=sharing

sndwvs · 03-15-2022, 06:19 AM

I unpacked your broken uInitrd library list:

Code:

-rwxr-xr-x 1 root root 219224 Mar 15 12:14 ld-2.35.so
lrwxrwxrwx 1 root root     10 Mar 15 13:05 ld-linux-aarch64.so.1 -> ld-2.35.so
-r-xr-xr-x 1 root root  10064 Mar 15 12:14 libdevmapper-event-lvm2.so.2.03
-r-xr-xr-x 1 root root  26440 Mar 15 12:14 libdevmapper-event.so.1.02
-r-xr-xr-x 1 root root 384352 Mar 15 12:14 libdevmapper.so.1.02
-rwxr-xr-x 1 root root  42424 Mar 15 12:14 libnss_compat-2.35.so
lrwxrwxrwx 1 root root     21 Mar 15 13:05 libnss_compat.so.2 -> libnss_compat-2.35.so
-rwxr-xr-x 1 root root   7904 Mar 15 12:14 libnss_files-2.35.so
lrwxrwxrwx 1 root root     20 Mar 15 13:05 libnss_files.so.2 -> libnss_files-2.35.so

it's built on my raspberry pi 4

Code:

-rwxr-xr-x 1 root root  219224 Mar 15 11:06 ld-2.35.so
lrwxrwxrwx 1 root root      10 Mar 15 13:10 ld-linux-aarch64.so.1 -> ld-2.35.so
lrwxrwxrwx 1 root root      15 Mar 15 13:10 libaio.so.1 -> libaio.so.1.0.1
-rwxr-xr-x 1 root root    9368 Mar 15 11:06 libaio.so.1.0.1
lrwxrwxrwx 1 root root      17 Mar 15 13:10 libblkid.so.1 -> libblkid.so.1.1.0
-rwxr-xr-x 1 root root  351104 Mar 15 11:06 libblkid.so.1.1.0
-rwxr-xr-x 1 root root 2086776 Mar 15 11:06 libc-2.35.so
lrwxrwxrwx 1 root root      12 Mar 15 13:10 libc.so.6 -> libc-2.35.so
-rwxr-xr-x 1 root root 2866888 Mar 15 11:06 libcrypto.so.1.1
-r-xr-xr-x 1 root root   10064 Mar 15 11:06 libdevmapper-event-lvm2.so.2.03
-r-xr-xr-x 1 root root   26440 Mar 15 11:06 libdevmapper-event.so.1.02
-rwxr-xr-x 1 root root  384352 Mar 15 11:06 libdevmapper.so.1.02
-rwxr-xr-x 1 root root    8056 Mar 15 11:06 libdl-2.35.so
lrwxrwxrwx 1 root root      13 Mar 15 13:10 libdl.so.2 -> libdl-2.35.so
lrwxrwxrwx 1 root root      16 Mar 15 13:10 libkmod.so.2 -> libkmod.so.2.3.7
-rwxr-xr-x 1 root root  100256 Mar 15 11:06 libkmod.so.2.3.7
-r-xr-xr-x 1 root root 2714016 Mar 15 11:06 liblvm2cmd.so.2.03
lrwxrwxrwx 1 root root      16 Mar 15 13:10 liblzma.so.5 -> liblzma.so.5.2.5
-rwxr-xr-x 1 root root  153464 Mar 15 11:06 liblzma.so.5.2.5
-rwxr-xr-x 1 root root  670712 Mar 15 11:06 libm-2.35.so
lrwxrwxrwx 1 root root      12 Mar 15 13:10 libm.so.6 -> libm-2.35.so
-rwxr-xr-x 1 root root   42424 Mar 15 11:06 libnss_compat-2.35.so
lrwxrwxrwx 1 root root      21 Mar 15 13:10 libnss_compat.so.2 -> libnss_compat-2.35.so
-rwxr-xr-x 1 root root    7904 Mar 15 11:06 libnss_files-2.35.so
lrwxrwxrwx 1 root root      20 Mar 15 13:10 libnss_files.so.2 -> libnss_files-2.35.so
-rwxr-xr-x 1 root root    8432 Mar 15 11:06 libpthread-2.35.so
lrwxrwxrwx 1 root root      18 Mar 15 13:10 libpthread.so.0 -> libpthread-2.35.so
-rwxr-xr-x 1 root root   70672 Mar 15 11:06 libresolv-2.35.so
lrwxrwxrwx 1 root root      17 Mar 15 13:10 libresolv.so.2 -> libresolv-2.35.so
lrwxrwxrwx 1 root root      16 Mar 15 13:10 libudev.so.1 -> libudev.so.1.6.3
-rwxr-xr-x 1 root root  174176 Mar 15 11:06 libudev.so.1.6.3
lrwxrwxrwx 1 root root      16 Mar 15 13:10 libuuid.so.1 -> libuuid.so.1.3.0
-rwxr-xr-x 1 root root   26552 Mar 15 11:06 libuuid.so.1.3.0
lrwxrwxrwx 1 root root      14 Mar 15 13:10 libz.so.1 -> libz.so.1.2.11
-rwxr-xr-x 1 root root   92024 Mar 15 11:06 libz.so.1.2.11
lrwxrwxrwx 1 root root      16 Mar 15 13:10 libzstd.so.1 -> libzstd.so.1.5.2
-rwxr-xr-x 1 root root  661864 Mar 15 11:06 libzstd.so.1.5.2

it looks like mkinitrd is failing to find dependencies.

wowbaggerHU · 03-15-2022, 07:23 AM

Quote:

Originally Posted by sndwvs

it looks like mkinitrd is failing to find dependencies.

As far as I could check the ldd command works fine. I just failed to find the /bin/ash binary, so that I could check that. I checked the /bin/sh and /bin/bash binaries, and they both return the same libraries, on both my working machine and the one where I can't build a working initrd.

When creating the initrd, I didn't see anything that would indicate an error:

Code:

root@pi4-build:/boot# bash ./rebuild-initrd.sh 
select kernel: 5.16.7
kernel: 5.16.7
select modules: ext4:xfs:btrfs
modules: ext4:xfs:btrfs
Started build uInitrd-5.16.7
OK: /lib/modules/5.16.7/kernel/fs/xfs/xfs.ko added.
OK: /lib/modules/5.16.7/kernel/lib/zstd/zstd_compress.ko added.
OK: /lib/modules/5.16.7/kernel/lib/raid6/raid6_pq.ko added.
OK: /lib/modules/5.16.7/kernel/arch/arm64/lib/xor-neon.ko added.
OK: /lib/modules/5.16.7/kernel/crypto/xor.ko added.
OK: /lib/modules/5.16.7/kernel/fs/btrfs/btrfs.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-mod.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-mod.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-bufio.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-snapshot.ko added.
41860 blocks
/tmp/initrd.gz created.
Be sure to run lilo again if you use it.
/tmp/initrd-tree /boot
/boot
Image Name:   uInitrd
Created:      Tue Mar 15 13:23:26 2022
Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
Data Size:    6274387 Bytes = 6127.33 KiB = 5.98 MiB
Load Address: 00000000
Entry Point:  00000000

Rebuild uInitrd-5.16.7 complited.

root@pi4-build:/boot#

sndwvs · 03-15-2022, 08:16 AM

Quote:

Originally Posted by wowbaggerHU

As far as I could check the ldd command works fine. I just failed to find the /bin/ash binary, so that I could check that. I checked the /bin/sh and /bin/bash binaries, and they both return the same libraries, on both my working machine and the one where I can't build a working initrd.

When creating the initrd, I didn't see anything that would indicate an error:

Code:

root@pi4-build:/boot# bash ./rebuild-initrd.sh 
select kernel: 5.16.7
kernel: 5.16.7
select modules: ext4:xfs:btrfs
modules: ext4:xfs:btrfs
Started build uInitrd-5.16.7
OK: /lib/modules/5.16.7/kernel/fs/xfs/xfs.ko added.
OK: /lib/modules/5.16.7/kernel/lib/zstd/zstd_compress.ko added.
OK: /lib/modules/5.16.7/kernel/lib/raid6/raid6_pq.ko added.
OK: /lib/modules/5.16.7/kernel/arch/arm64/lib/xor-neon.ko added.
OK: /lib/modules/5.16.7/kernel/crypto/xor.ko added.
OK: /lib/modules/5.16.7/kernel/fs/btrfs/btrfs.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-mod.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-mod.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-bufio.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-snapshot.ko added.
41860 blocks
/tmp/initrd.gz created.
Be sure to run lilo again if you use it.
/tmp/initrd-tree /boot
/boot
Image Name:   uInitrd
Created:      Tue Mar 15 13:23:26 2022
Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
Data Size:    6274387 Bytes = 6127.33 KiB = 5.98 MiB
Load Address: 00000000
Entry Point:  00000000

Rebuild uInitrd-5.16.7 complited.

root@pi4-build:/boot#

I think a lot is not displayed, that's where /bin/ash is located /usr/share/mkinitrd/initrd-tree.tar.gz

Code:

mkinitrd -R -L -u -w 2 -c -k 5.16.7 -m tun:btrfs:xfs -s /tmp/initrd-tree -o /tmp/initrd.gz

you can see what happens /tmp/initrd-tree

wowbaggerHU · 03-15-2022, 11:42 AM

Quote:

Originally Posted by sndwvs

I think a lot is not displayed, that's where /bin/ash is located /usr/share/mkinitrd/initrd-tree.tar.gz

Code:

mkinitrd -R -L -u -w 2 -c -k 5.16.7 -m tun:btrfs:xfs -s /tmp/initrd-tree -o /tmp/initrd.gz

you can see what happens /tmp/initrd-tree

Thanks for pointing that out!
I checked /bin/ash in the initrd-tree archive, and it's a symlink to busybox (no surprises there).
I ran ldd on the busybox binary, and it produced the following output:

Code:

root@pi4-build:~/kernel# ldd busybox 
        linux-vdso.so.1 (0x0000007fb377a000)
        libm.so.6 => /lib64/libm.so.6 (0x0000007fb3690000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x0000007fb3660000)
        libc.so.6 => /lib64/libc.so.6 (0x0000007fb34a0000)
        /lib64/ld-linux-aarch64.so.1 (0x0000007fb3740000)
root@pi4-build:~/kernel#

I also ran the mkinitrd command you mentioned, and it produced the following output:

Code:

root@pi4-build:~/kernel# mkinitrd -R -L -u -w 2 -c -k 5.16.7 -m tun:btrfs:xfs -s /tmp/initrd-tree -o initrd.gz
OK: /lib/modules/5.16.7/kernel/drivers/net/tun.ko added.
OK: /lib/modules/5.16.7/kernel/lib/zstd/zstd_compress.ko added.
OK: /lib/modules/5.16.7/kernel/lib/raid6/raid6_pq.ko added.
OK: /lib/modules/5.16.7/kernel/arch/arm64/lib/xor-neon.ko added.
OK: /lib/modules/5.16.7/kernel/crypto/xor.ko added.
OK: /lib/modules/5.16.7/kernel/fs/btrfs/btrfs.ko added.
OK: /lib/modules/5.16.7/kernel/fs/xfs/xfs.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-mod.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-mod.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-bufio.ko added.
OK: /lib/modules/5.16.7/kernel/drivers/md/dm-snapshot.ko added.
42052 blocks
/root/kernel/initrd.gz created.
Be sure to run lilo again if you use it.
root@pi4-build:~/kernel#

And when checking archive it created, I found that the /lib64/libm.so.6 is still missing.

sndwvs · 03-15-2022, 12:02 PM

find + ldd + awk + cat + sort + uniq + wc + cp to find libraries
in file /sbin/mkinitrd copy_libs function.

you can add the output of variables/commands everywhere to understand where the problem is.

you can try an alternative mkinitramfs-r10.1_560-aarch64-7mara.txz

Code:

mkinitramfs -k 5.16.7 -m xfs:ext4:tun -b /tmp

extract

Code:

zcat /tmp/initrd.img-5.16.7 | cpio -idmv

wowbaggerHU · 03-16-2022, 09:36 AM

Quote:

Originally Posted by sndwvs

find + ldd + awk + cat + sort + uniq + wc + cp to find libraries
in file /sbin/mkinitrd copy_libs function.

you can add the output of variables/commands everywhere to understand where the problem is.

Turns out having your /tmp mounted as noexec, doesn't pair up well with

Code:

find $SOURCE_TREE -type f -exec ldd {} 2>/dev/null \;

Thanks for your help sndwvs! I hope you're well!