Re: [libvirt-users] Suggestions on building VM disks from scratch
Then I created my blank 'disk' file and tried to run virt-rescue on it. It crashed out with an error from febootstrap.
First of all, debug this properly: (1) What is the full error message? (2) What is the complete, unedited output of 'libguestfs-test-tool'? (3) What version of libguestfs & febootstrap are using and where did you get them from? You can post the details on our mailing list libguestfs@redhat.com (no need to subscribe if you don't want to).
After finding nothing terribly useful or current on this in searching, I tried guestfish instead. After some fiddling I got it to attach my blank disk. However I cannot find a reasonable way to partition it with the part-add which seems to want me to count sectors. All I want is a 9G linux and a 2G swap.
Assuming the filesystem was in /tmp/root.tar.gz, the following code will do this: guestfish <<EOF sparse /tmp/test.img 11G run part-init /dev/sda mbr # 9GB sda1 part-add /dev/sda p 64 $(( 9*1024*1024*2 )) # remainder in sda2 part-add /dev/sda p $(( 9*1024*1024*2 + 1 )) -64 mkfs ext4 /dev/sda1 mkswap /dev/sda2 mount /dev/sda1 / tgz-in /tmp/root.tar.gz / EOF Example: $ sh test.sh $ virt-df -a test.img -h Filesystem Size Used Available Use% test.img:/dev/sda1 9.0G 276M 8.3G 4% $ ll -h test.img -rw-rw-r--. 1 rjones rjones 11G Apr 4 09:39 test.img Whether this would actually boot is another question: you may also need to add some grub commands to set up the bootloader, *or* (better and easier IMHO) set up libvirt so that it boots from an external kernel + initrd.
It also lacks access to rsync.
It's not the first time that someone has asked for rsync, and it wouldn't be too hard to add. However note that rsync really gives you no benefit when you're creating a filesystem from scratch, because there's no original to rsync against. If you are updating a filesystem image then rsync makes sense. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Apr 04, 2012 at 09:45:18AM +0100, Richard W.M. Jones wrote:
# 9GB sda1 part-add /dev/sda p 64 $(( 9*1024*1024*2 )) # remainder in sda2 part-add /dev/sda p $(( 9*1024*1024*2 + 1 )) -64
This leaves the swap partition unaligned :-( If you care about alignment (and you probably should) then this will create properly aligned partitions: part-add /dev/sda p 128 $(( 9*1024*1024*2 - 1 )) part-add /dev/sda p $(( 9*1024*1024*2 )) -128 'Course the 'virt-alignment-scan' tool can be used to detect if your partitions are aligned correctly. With the original image: $ virt-alignment-scan -a test.img /dev/sda1 32768 32K bad (alignment < 64K) /dev/sda2 9663676928 512 bad (alignment < 4K) With the correctly aligned image: $ virt-alignment-scan -a test.img /dev/sda1 65536 64K ok /dev/sda2 9663676416 1048576K ok Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Apr 04, 2012 at 09:45:18AM +0100, Richard W.M. Jones wrote:
Then I created my blank 'disk' file and tried to run virt-rescue on it. It crashed out with an error from febootstrap.
First of all, debug this properly:
=====================================================================
(1) What is the full error message?
I upgraded and installed a few more things and my error at least changed, at least so I thought: # virt-rescue -a recovery2.immortaldata.net.raw mke2fs: Attempt to write block to filesystem resulted in short write while zeroing block 1048560 at end of filesystem Could not write 6 blocks in inode table starting at 917506: Attempt to write block to filesystem resulted in short write febootstrap-supermin-helper: /sbin/mke2fs -t ext2 -Fq '/var/tmp/guestfs.cZge1g/root': failed But guestfish apparently finds my small root disk to be a tasty morsel of fish food and impolitely swallows it whole: 4.1G /var/tmp/guestfs.1PFF68 3.5G /var/tmp/guestfs.cZge1g 4.1G /var/tmp/guestfs.FYAm8O 4.1G /var/tmp/guestfs.JP2LNO 4.1G /var/tmp/guestfs.qsLaNA 4.1G /var/tmp/guestfs.wwKT58 4.1G /var/tmp/guestfs.ZueM9V This maxed out the root partition. Once I found this was happening I cleared this out and added: TMPDIR=/home/tmp This got me back to the original error message: # virt-rescue -a recovery2.immortaldata.net.raw febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup =====================================================================
(2) What is the complete, unedited output of 'libguestfs-test-tool'?
# libguestfs-test-tool ===== Test starts here ===== LIBGUESTFS_DEBUG=1 TMPDIR=(not set) libguestfs: new guestfs handle 0x23dc190 library version: 1.14.8 guestfs_get_append: (null) guestfs_get_attach_method: appliance guestfs_get_autosync: 1 guestfs_get_direct: 0 guestfs_get_memsize: 500 guestfs_get_network: 0 guestfs_get_path: /usr/lib/guestfs guestfs_get_pgroup: 0 guestfs_get_qemu: /usr/bin/kvm guestfs_get_recovery_proc: 1 guestfs_get_selinux: 0 guestfs_get_smp: 1 guestfs_get_trace: 0 guestfs_get_verbose: 1 host_cpu: x86_64 Launching appliance, timeout set to 600 seconds. libguestfs: [00000ms] febootstrap-supermin-helper --verbose -f checksum '/usr/lib/guestfs/supermin.d' x86_64 supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null) supermin helper [00000ms] inputs[0] = /usr/lib/guestfs/supermin.d checking modpath /lib/modules/3.2.0-16-generic is a directory picked vmlinuz-3.2.0-16-generic because modpath /lib/modules/3.2.0-16-generic exists checking modpath /lib/modules/3.2.0-21-generic is a directory picked vmlinuz-3.2.0-21-generic because modpath /lib/modules/3.2.0-21-generic exists checking modpath /lib/modules/2.6.32-38-server is a directory picked vmlinuz-2.6.32-38-server because modpath /lib/modules/2.6.32-38-server exists supermin helper [00000ms] finished creating kernel supermin helper [00098ms] visiting /usr/lib/guestfs/supermin.d supermin helper [00098ms] visiting /usr/lib/guestfs/supermin.d/daemon.img supermin helper [00098ms] visiting /usr/lib/guestfs/supermin.d/init.img supermin helper [00098ms] adding kernel modules supermin helper [00259ms] finished creating appliance libguestfs: [00353ms] begin building supermin appliance libguestfs: [00353ms] run febootstrap-supermin-helper libguestfs: [00353ms] febootstrap-supermin-helper --verbose -f ext2 /usr/lib/guestfs/supermin.d x86_64 /var/tmp/guestfs.xtl6Jb/kernel /var/tmp/guestfs.xtl6Jb/initrd /var/tmp/guestfs.xtl6Jb/root supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = /var/tmp/guestfs.xtl6Jb/kernel, initrd = /var/tmp/guestfs.xtl6Jb/initrd, appliance = /var/tmp/guestfs.xtl6Jb/root supermin helper [00000ms] inputs[0] = /usr/lib/guestfs/supermin.d checking modpath /lib/modules/3.2.0-16-generic is a directory picked vmlinuz-3.2.0-16-generic because modpath /lib/modules/3.2.0-16-generic exists checking modpath /lib/modules/3.2.0-21-generic is a directory picked vmlinuz-3.2.0-21-generic because modpath /lib/modules/3.2.0-21-generic exists checking modpath /lib/modules/2.6.32-38-server is a directory picked vmlinuz-2.6.32-38-server because modpath /lib/modules/2.6.32-38-server exists supermin helper [00000ms] finished creating kernel supermin helper [02595ms] finished mke2fs supermin helper [02596ms] visiting /usr/lib/guestfs/supermin.d supermin helper [02596ms] visiting /usr/lib/guestfs/supermin.d/daemon.img supermin helper [02601ms] visiting /usr/lib/guestfs/supermin.d/init.img febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup libguestfs: error: external command failed, see earlier error messages libguestfs-test-tool: failed to launch appliance libguestfs: closing guestfs handle 0x23dc190 (state 0) =====================================================================
(3) What version of libguestfs & febootstrap are using and where did you get them from?
Updated from official Ubuntu Precise Pangolin server last night. Package: libguestfs0 Architecture: amd64 Source: libguestfs Version: 1:1.14.8-1 Package: febootstrap Architecture: amd64 Version: 3.12-1
Assuming the filesystem was in /tmp/root.tar.gz, the following code will do this:
guestfish <<EOF
sparse /tmp/test.img 11G run
part-init /dev/sda mbr # 9GB sda1 part-add /dev/sda p 64 $(( 9*1024*1024*2 )) # remainder in sda2 part-add /dev/sda p $(( 9*1024*1024*2 + 1 )) -64
mkfs ext4 /dev/sda1 mkswap /dev/sda2
mount /dev/sda1 / tgz-in /tmp/root.tar.gz /
EOF
Example:
$ sh test.sh $ virt-df -a test.img -h Filesystem Size Used Available Use% test.img:/dev/sda1 9.0G 276M 8.3G 4% $ ll -h test.img -rw-rw-r--. 1 rjones rjones 11G Apr 4 09:39 test.img
Whether this would actually boot is another question: you may also need to add some grub commands to set up the bootloader, *or* (better and easier IMHO) set up libvirt so that it boots from an external kernel + initrd.
If I can get the grub installed it should, since the original are in most cases virtual servers built on vmdk disks on a licensed VMware ESX server. The other way I found to get to this particular point was: $ dd if=/dev/zero of=/recover2.us.net.raw count=20M $ losetup /dev/loop0 recover2.us.net.raw $ cfdisk /dev/loop0 create a 9G bootable linux partition and a 1G swap $ kpartx -a -v /dev/loop0 $ mkfs.ext4 /dev/mapper/loop0p1 -m 0.01 -L "RecoverTest" $ mkdir tmp $ mount /dev/mapper/loop0p1 tmp From the backup server: $ cd <to the lastest dirvish backup set> $ rsync -av tree/ root@rebuild.server.net:/srv/idcpool/tmp/ But I could not figure out a safe way to install grub that would not put the boot disk of a critical server at risk of getting its MBR clobbered. Grub is very good at doing such things and very difficult to convince to do what *you* want rather than what *it* wants. I much prefer the technique of just using a guestfish script to do it all as you showed, but I still need to load up the backup set and install grub properly.
It also lacks access to rsync.
It's not the first time that someone has asked for rsync, and it wouldn't be too hard to add. However note that rsync really gives you no benefit when you're creating a filesystem from scratch, because there's no original to rsync against. If you are updating a filesystem image then rsync makes sense.
Actually, what the situation is that for our restoration of *last resort*, we have a server that does incremental daily and for some key infrastructure 4 times daily backups via dirvish. So what we need dirvish for is so that we can connect to the backup server and copy the most recent copy of the failed device. So yes, we do have a good reason to use rsync as it gives us a very simple secure data transfer of the files from one server to the new image. The alternative is to do the rsync over to the rebuild server, being careful not to get links messed up, tar.gz the result and then pray the result will actually recreate things, links, dev files, uid.gid's, modes, etc on when unpacked. I have an inherent distrust of adding extra steps into things if I can avoid them, particularly when discussing a mission critical last resort failsafe. Oh, and yes, we do separately back up database files via the appropriate snapshotting and those would have to be overloaded seperately after the main restoration. That's one problem, the big one I am working on. The other I can talk to you about after I get this one sorted, and that is using virt-rescuse/guestfish in the virtualization of an old SuSE server on end of life hardware. It of course has lots of LVM's on the system disk just to make things interesting...
On Wed, Apr 04, 2012 at 07:53:26PM +0100, Dale Amon wrote:
On Wed, Apr 04, 2012 at 09:45:18AM +0100, Richard W.M. Jones wrote:
Then I created my blank 'disk' file and tried to run virt-rescue on it. It crashed out with an error from febootstrap.
First of all, debug this properly:
=====================================================================
(1) What is the full error message?
[...]
# virt-rescue -a recovery2.immortaldata.net.raw febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup
This is because you need to run: sudo update-guestfs-appliance
=====================================================================
(3) What version of libguestfs & febootstrap are using and where did you get them from?
Updated from official Ubuntu Precise Pangolin server last night.
Package: libguestfs0 Architecture: amd64 Source: libguestfs Version: 1:1.14.8-1
Package: febootstrap Architecture: amd64 Version: 3.12-1
Unfortunately we found out today that Ubuntu 12.04's libguestfs package is largely broken. For more information, see: https://www.redhat.com/archives/libguestfs/2012-April/msg00028.html There is no solution to the kernel bug that affects Ubuntu (only). [...] Bit busy now, but I'll have a look at the rest later. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Wed, Apr 04, 2012 at 09:09:03PM +0100, Richard W.M. Jones wrote:
Unfortunately we found out today that Ubuntu 12.04's libguestfs package is largely broken. For more information, see:
https://www.redhat.com/archives/libguestfs/2012-April/msg00028.html
There is no solution to the kernel bug that affects Ubuntu (only).
<fs> sparse /KdevArchive1/vmpool1/Mourne.sda 30G <fs> run warning: Unable to get device geometry for /var/tmp/guestfs.7NJVau/root febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup
Is this what you are talking about? -------------------------------- root@kdev:/KdevArchive1/vmpool1# guestfish Welcome to guestfish, the libguestfs filesystem interactive shell for editing virtual machine filesystems. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell libguestfs: error: external command failed, see earlier error messages
<fs>
--------------------------------
On Thu, Apr 05, 2012 at 08:10:59PM +0100, Dale Amon wrote:
On Wed, Apr 04, 2012 at 09:09:03PM +0100, Richard W.M. Jones wrote:
Unfortunately we found out today that Ubuntu 12.04's libguestfs package is largely broken. For more information, see:
https://www.redhat.com/archives/libguestfs/2012-April/msg00028.html
There is no solution to the kernel bug that affects Ubuntu (only).
Is this what you are talking about? --------------------------------
root@kdev:/KdevArchive1/vmpool1# guestfish
Welcome to guestfish, the libguestfs filesystem interactive shell for editing virtual machine filesystems.
Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell
<fs> sparse /KdevArchive1/vmpool1/Mourne.sda 30G <fs> run warning: Unable to get device geometry for /var/tmp/guestfs.7NJVau/root febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup libguestfs: error: external command failed, see earlier error messages
No, this is because febootstrap in Ubuntu 12.04 is broken, which prevents creation of the appliance. You have to build febootstrap (at least 3.14) by hand, install it, and then rebuild the appliance by hand (update-guestfs-appliance). This is explained in this email: https://www.redhat.com/archives/libguestfs/2012-April/msg00028.html We have no Ubuntu maintainer, so Ubuntu just pull in the Debian packages which they don't test and they don't work well. libguestfs works best (indeed, is commercially supported) on Fedora and RHEL. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
I've passed your info on to the Ubuntu-devel group to see if anyone picks up on it.
On Thu, Apr 05, 2012 at 10:14:06PM +0100, Dale Amon wrote:
I've passed your info on to the Ubuntu-devel group to see if anyone picks up on it.
Thanks -- we do want it to work well on Ubuntu, and hillu is doing a great job on Debian. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
[grub] You can run grub from the guest (assuming it's a Linux guest), by doing in guestfish: sh "grub-install [other parameters]" There are some provisos which you can read about here: http://libguestfs.org/guestfs.3.html#running_commands We also provide a grub-install command which runs the grub from the host on the guest. This requires that the two versions match each other, and it needs further development work to make a fully general bootloader installer that would be secure and usable. [rsync] The issue with rsync is how to make it available from libguestfs. rsync needs an 8 bit clean connection of some sort, usually an rsh or ssh connection, and we could provide that. However we are also trying to extend libguestfs to work remotely, and then it becomes complex to route that connection from the client side to the remote appliance (eg) over a libvirt connection. Anyway if you want to add rsync, just add the package name to appliance/packagelist.in and rebuild from source. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 4/4/2012 4:45 AM, Richard W.M. Jones wrote:
Then I created my blank 'disk' file and tried to run virt-rescue on it. It crashed out with an error from febootstrap.
First of all, debug this properly:
(1) What is the full error message?
(2) What is the complete, unedited output of 'libguestfs-test-tool'?
(3) What version of libguestfs& febootstrap are using and where did you get them from?
You can post the details on our mailing list libguestfs@redhat.com (no need to subscribe if you don't want to).
After finding nothing terribly useful or current on this in searching, I tried guestfish instead. After some fiddling I got it to attach my blank disk. However I cannot find a reasonable way to partition it with the part-add which seems to want me to count sectors. All I want is a 9G linux and a 2G swap.
Assuming the filesystem was in /tmp/root.tar.gz, the following code will do this:
guestfish<<EOF
sparse /tmp/test.img 11G run
part-init /dev/sda mbr # 9GB sda1 part-add /dev/sda p 64 $(( 9*1024*1024*2 )) # remainder in sda2 part-add /dev/sda p $(( 9*1024*1024*2 + 1 )) -64
mkfs ext4 /dev/sda1 mkswap /dev/sda2
mount /dev/sda1 / tgz-in /tmp/root.tar.gz /
EOF
Example:
$ sh test.sh $ virt-df -a test.img -h Filesystem Size Used Available Use% test.img:/dev/sda1 9.0G 276M 8.3G 4% $ ll -h test.img -rw-rw-r--. 1 rjones rjones 11G Apr 4 09:39 test.img
Whether this would actually boot is another question: you may also need to add some grub commands to set up the bootloader, *or* (better and easier IMHO) set up libvirt so that it boots from an external kernel + initrd.
It also lacks access to rsync.
It's not the first time that someone has asked for rsync, and it wouldn't be too hard to add. However note that rsync really gives you no benefit when you're creating a filesystem from scratch, because there's no original to rsync against. If you are updating a filesystem image then rsync makes sense.
Rich.
Also rsync does not do anything magical/differential when dealing with local copies. Even an update is still a full normal copy because there is no benefit to the differential processing (the same disk(s) have to be read and written just as much and it all has to pass through the same cpu and busses. You can force it by doing a network transfer over localhost, but "ya canna change the laws o physics" And, not only is there no differential magic benefit, there is still a major speed loss. rsync is usually a lot slower than the same tar/cpio/cp doing full copies, even over networks, even using -W and --inplace. Definitely the point of using rsync locally is not speed of the data transfer itself, but speed of the the user. The main benefit is just familiarity and convenience with the rsync command line arguments and behavior. You spend a lot of time learning how to get various common tasks done with rsync and it is really quite a swiss army knife. And you develop trust and confidence with it. You know exactly what will happen when you run a given command because you do the same thing routinely. Those features are handy enough to be worth taking the speed hit in return for not having to develop complicated, entirely different, find/cp/cpio commands that you use less often and are less sure about what they'll actually do when you press enter. You could argue the find/cpio is better for writing into scripts and apps since they are a more universal dependency as well as the performance reason, but you could also argue that it's an advantage to use a command that works equally between any to points, be they local or remote, making your script/app more useful in more unpredicted situations for free. -- bkw
On Wed, Apr 04, 2012 at 04:08:05PM -0400, Brian K. White wrote:
Also rsync does not do anything magical/differential when dealing with local copies. Even an update is still a full normal copy
But it isn't a local copy. It's from a dirvish backup server that might not even be in the same data center.
participants (3)
-
Brian K. White -
Dale Amon -
Richard W.M. Jones