[Qemu-discuss] Help with -kernel option (unable to mount root fs)

[Martin Ichilevici de Oliveira]

Hello,

I'm trying to use the -kernel option of QEMU, in order to later debug
Linux with GDB, but I've been unable to boot the system.

My setup: a working CentOS 7 with a manually compilled kernel (3.17-rc5).
If I simply boot the image with qemu-system-x86_64, it works fine. So
I copied the bzImage out of the VM and ran:

$ qemu-system-x86_64 -m 4G -hda image.img -kernel bzImage -append "root=/dev/sda console=ttyS0" -nographic

The system failed to boot (log at the end of this email), but basically
it's complaining about being unable to mount the root filesystem.

I thought I'd used the wrong /dev/sdX on the append option, so I
checked with

# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a2200

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 8e Linux LVM

Disk /dev/mapper/centos-swap: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-root: 10.5 GB, 10485760000 bytes, 20480000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-home: 18.0 GB, 17976786944 bytes, 35110912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Not many options here... I tried with /dev/sda1, /dev/sda2 and /dev/mapper/centos-root,
but all got the same error.

Finally, I checked the grub entry for that kernel and it contains:
set root='hd0,msdos1'

But I'm not really sure how to use this information.

Any help is appreciated.

Thank you,
Martin

Kernel panic log:
(...)
[ 2.278415] List of all partitions:
[ 2.279764] No filesystem could mount root, tried:
[ 2.281594] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.282071] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc6+ #1
[ 2.282071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140617_173321-var-lib-archbuild-testing-x86_64-tobias 04/01/2014
[ 2.282071] 0000000000000000 00000000724949f3 ffff88003daabd60 ffffffff81686bcd
[ 2.282071] ffffffff818ba3c0 ffff88003daabde8 ffffffff81683afb ffffffff00000010
[ 2.282071] ffff88003daabdf8 ffff88003daabd90 00000000724949f3 00000000724949f3
[ 2.282071] Call Trace:
[ 2.282071] [<ffffffff81686bcd>] dump_stack+0x45/0x56
[ 2.282071] [<ffffffff81683afb>] panic+0xd5/0x209
[ 2.282071] [<ffffffff81b3d5ec>] mount_block_root+0x2a4/0x2b3
[ 2.282071] [<ffffffff81b3d64e>] mount_root+0x53/0x56
[ 2.282071] [<ffffffff81b3d78d>] prepare_namespace+0x13c/0x174
[ 2.282071] [<ffffffff81b3d25a>] kernel_init_freeable+0x23d/0x261
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] [<ffffffff8167a3de>] kernel_init+0xe/0xf0
[ 2.282071] [<ffffffff8168ebbc>] ret_from_fork+0x7c/0xb0
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[ 2.282071] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.282071] general protection fault: fff2 [#1] SMP
[ 2.282071] Modules linked in:
[ 2.282071] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc6+ #1
[ 2.282071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140617_173321-var-lib-archbuild-testing-x86_64-tobias 04/01/2014
[ 2.282071] task: ffff88003daa0000 ti: ffff88003daa8000 task.ti: ffff88003daa8000
[ 2.282071] RIP: 0010:[<ffffffff81683beb>] [<ffffffff81683beb>] panic+0x1c5/0x209
[ 2.282071] RSP: 0000:ffff88003daabd70 EFLAGS: 00000282
[ 2.282071] RAX: 0000000000000057 RBX: ffffffff818ba3c0 RCX: 0000000000000057
[ 2.282071] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
[ 2.282071] RBP: ffff88003daabde8 R08: 00000000000067b8 R09: 0000000000000096
[ 2.282071] R10: ffffffff81b177c0 R11: 000000000000018b R12: ffff88003daabdf8
[ 2.282071] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88003d710000
[ 2.282071] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 2.282071] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2.282071] CR2: 0000000000000000 CR3: 00000000019c4000 CR4: 00000000000006f0
[ 2.282071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2.282071] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 2.282071] Stack:
[ 2.282071] ffffffff00000010 ffff88003daabdf8 ffff88003daabd90 00000000724949f3
[ 2.282071] 00000000724949f3 ffff88003daabe08 0000000000000012 0000000000000000
[ 2.282071] 000000000000000a 000000000000fffb ffff88003d710000 0000000000000000
[ 2.282071] Call Trace:
[ 2.282071] [<ffffffff81b3d5ec>] mount_block_root+0x2a4/0x2b3
[ 2.282071] [<ffffffff81b3d64e>] mount_root+0x53/0x56
[ 2.282071] [<ffffffff81b3d78d>] prepare_namespace+0x13c/0x174
[ 2.282071] [<ffffffff81b3d25a>] kernel_init_freeable+0x23d/0x261
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] [<ffffffff8167a3de>] kernel_init+0xe/0xf0
[ 2.282071] [<ffffffff8168ebbc>] ret_from_fork+0x7c/0xb0
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] Code: c3 64 eb b1 83 3d 15 35 70 00 00 74 05 e8 1e 32 a1 ff 48 c7 c6 40 71 d8 81 48 c7 c7 60 be 8c 81 31 c0 e8 64 02 00 00 fb 66 66 90 <66> 66 90 45 31 e4 e8 3a 49 aa ff 4d 39 ec 7c 18 41 83 f6 01 44
[ 2.282071] RIP [<ffffffff81683beb>] panic+0x1c5/0x209
[ 2.282071] RSP <ffff88003daabd70>
[ 2.282071] ---[ end trace 5c54a652a799b4be ]---

[Jakob Bohm]

Post by Martin Ichilevici de Oliveira
Hello,
I'm trying to use the -kernel option of QEMU, in order to later debug
Linux with GDB, but I've been unable to boot the system.
My setup: a working CentOS 7 with a manually compilled kernel (3.17-rc5).
If I simply boot the image with qemu-system-x86_64, it works fine. So
$ qemu-system-x86_64 -m 4G -hda image.img -kernel bzImage -append "root=/dev/sda console=ttyS0" -nographic

root=/dev/sda is clearly wrong, that is the whole unpartitioned disk,
not the root partition inside it. You probably want /dev/sda1. But
morebelow...

Post by Martin Ichilevici de Oliveira
The system failed to boot (log at the end of this email), but basically
it's complaining about being unable to mount the root filesystem.
I thought I'd used the wrong /dev/sdX on the append option, so I
checked with
# fdisk -l

Did you run this command on the guest?

Post by Martin Ichilevici de Oliveira
Disk /dev/sda: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a2200
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 8e Linux LVM

I take that to mean that /dev/sda1 is a small 512MB partition
containing grub, kernels, initrd images etc. Probably mounted as
"/boot" by the guest once it is running.

/dev/sda2 is subpartitioned using LVM and I am not sure the built
in root mounting code in the kernel can handle that without help
from user mode tools in an initrd.

You need to look in the grub config for the file name of that initrd
image, then pass that to qemu along with the kernel image and the
required kernel commandline. The kernel will run the initrd before
it tries to mount root, and the initrd (presumably generated
automatically by CentOS) will then load the needed LVM tools to
find /dev/mapper/centos-root and mount it as root.

Post by Martin Ichilevici de Oliveira
Disk /dev/mapper/centos-swap: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 10.5 GB, 10485760000 bytes, 20480000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-home: 18.0 GB, 17976786944 bytes, 35110912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Not many options here... I tried with /dev/sda1, /dev/sda2 and /dev/mapper/centos-root,
but all got the same error.
set root='hd0,msdos1'

I guess this line is just there to help grub find the kernel and
initrd, then otherparameters will make the initrd tell the kernel
where the real root is.

Note that grub, for unfathomable reasons, has its own weird naming
of partitions. hd0,msdos1probably means /dev/sda1 (the small boot
partition), or it could mean /dev/sda2 (the LVMcontainer, which
obviously won't be used directly as root).

Post by Martin Ichilevici de Oliveira
But I'm not really sure how to use this information.
Any help is appreciated.
Thank you,
Martin
(...)
[ 2.281594] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.282071] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc6+ #1
[ 2.282071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140617_173321-var-lib-archbuild-testing-x86_64-tobias 04/01/2014
[ 2.282071] 0000000000000000 00000000724949f3 ffff88003daabd60 ffffffff81686bcd
[ 2.282071] ffffffff818ba3c0 ffff88003daabde8 ffffffff81683afb ffffffff00000010
[ 2.282071] ffff88003daabdf8 ffff88003daabd90 00000000724949f3 00000000724949f3
[ 2.282071] [<ffffffff81686bcd>] dump_stack+0x45/0x56
[ 2.282071] [<ffffffff81683afb>] panic+0xd5/0x209
[ 2.282071] [<ffffffff81b3d5ec>] mount_block_root+0x2a4/0x2b3
[ 2.282071] [<ffffffff81b3d64e>] mount_root+0x53/0x56
[ 2.282071] [<ffffffff81b3d78d>] prepare_namespace+0x13c/0x174
[ 2.282071] [<ffffffff81b3d25a>] kernel_init_freeable+0x23d/0x261
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] [<ffffffff8167a3de>] kernel_init+0xe/0xf0
[ 2.282071] [<ffffffff8168ebbc>] ret_from_fork+0x7c/0xb0
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[ 2.282071] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.282071] general protection fault: fff2 [#1] SMP
[ 2.282071] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc6+ #1
[ 2.282071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140617_173321-var-lib-archbuild-testing-x86_64-tobias 04/01/2014
[ 2.282071] task: ffff88003daa0000 ti: ffff88003daa8000 task.ti: ffff88003daa8000
[ 2.282071] RIP: 0010:[<ffffffff81683beb>] [<ffffffff81683beb>] panic+0x1c5/0x209
[ 2.282071] RSP: 0000:ffff88003daabd70 EFLAGS: 00000282
[ 2.282071] RAX: 0000000000000057 RBX: ffffffff818ba3c0 RCX: 0000000000000057
[ 2.282071] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
[ 2.282071] RBP: ffff88003daabde8 R08: 00000000000067b8 R09: 0000000000000096
[ 2.282071] R10: ffffffff81b177c0 R11: 000000000000018b R12: ffff88003daabdf8
[ 2.282071] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88003d710000
[ 2.282071] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 2.282071] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2.282071] CR2: 0000000000000000 CR3: 00000000019c4000 CR4: 00000000000006f0
[ 2.282071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2.282071] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 2.282071] ffffffff00000010 ffff88003daabdf8 ffff88003daabd90 00000000724949f3
[ 2.282071] 00000000724949f3 ffff88003daabe08 0000000000000012 0000000000000000
[ 2.282071] 000000000000000a 000000000000fffb ffff88003d710000 0000000000000000
[ 2.282071] [<ffffffff81b3d5ec>] mount_block_root+0x2a4/0x2b3
[ 2.282071] [<ffffffff81b3d64e>] mount_root+0x53/0x56
[ 2.282071] [<ffffffff81b3d78d>] prepare_namespace+0x13c/0x174
[ 2.282071] [<ffffffff81b3d25a>] kernel_init_freeable+0x23d/0x261
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] [<ffffffff8167a3de>] kernel_init+0xe/0xf0
[ 2.282071] [<ffffffff8168ebbc>] ret_from_fork+0x7c/0xb0
[ 2.282071] [<ffffffff8167a3d0>] ? rest_init+0x80/0x80
[ 2.282071] Code: c3 64 eb b1 83 3d 15 35 70 00 00 74 05 e8 1e 32 a1 ff 48 c7 c6 40 71 d8 81 48 c7 c7 60 be 8c 81 31 c0 e8 64 02 00 00 fb 66 66 90 <66> 66 90 45 31 e4 e8 3a 49 aa ff 4d 39 ec 7c 18 41 83 f6 01 44
[ 2.282071] RIP [<ffffffff81683beb>] panic+0x1c5/0x209
[ 2.282071] RSP <ffff88003daabd70>
[ 2.282071] ---[ end trace 5c54a652a799b4be ]---

Enjoy

Jakob

[Martin Ichilevici de Oliveira]

Jakob,

Thanks for the quick reply, your comments helped me solve it. Analyzing
the grub config file in more details cleared it all, as it contained the
initrd image to use and the correct disk.

After copying the initrd to the host, I was able to boot the system with
the command below:

$ qemu-system-x86_64 -m 4G -hda image.img -kernel ./bzImage -initrd ./initramfs-3.17.0-rc5.img -append "root=/dev/mapper/centos-root console=ttyS0" -nographic

Enjoy

I sure will :)

Cheers,
Martin