[libvirt-users] How does libvirt interaction with KVM to create a VM?
All, These days I am trying to understand the interaction relationship between the libvirt and KVM kernel module, eg. kvm_intel.ko. We know that KVM kernel module expose an entry in form of device file "/dev/kvm" which can be accessed by user space application to control, for example, create a VM using KVM_CREATE_VM with help of ioctl. Now let's say the tool virsh based upon libvirt, we can create a guest domain with the command looks like: #virsh create guest.xml Obviously, the above command will create a VM. But when I try to investigate the libvirt code, I can't find any code play with the "/dev/kvm" to send KVM_CREATE_VM ioctl code to KVM kernel module. But I do found that the reference count of the kvm_intel.ko changed before the virsh create command launched and after. So my question is: how does the libvirt interaction with KVM to create a VM? Anybody can give me some tips about that, eg, the corresponding codes in libvirt? BRs. Dennis
On Thu, Jun 28, 2012 at 05:21:57PM +0800, Dennis Chen wrote:
All,
These days I am trying to understand the interaction relationship between the libvirt and KVM kernel module, eg. kvm_intel.ko.
We know that KVM kernel module expose an entry in form of device file "/dev/kvm" which can be accessed by user space application to control, for example, create a VM using KVM_CREATE_VM with help of ioctl.
Now let's say the tool virsh based upon libvirt, we can create a guest domain with the command looks like: #virsh create guest.xml Obviously, the above command will create a VM. But when I try to investigate the libvirt code, I can't find any code play with the "/dev/kvm" to send KVM_CREATE_VM ioctl code to KVM kernel module. But I do found that the reference count of the kvm_intel.ko changed before the virsh create command launched and after.
So my question is: how does the libvirt interaction with KVM to create a VM? Anybody can give me some tips about that, eg, the corresponding codes in libvirt?
The '/dev/kvm' device is the low level kernel interface for creating virtual domains. This is not actually used by libvirt at all. The QEMU binary has code that talks to /dev/kvm, so all libvirt does is to spawn a QEMU process which in turns creates the virtual machine All the libvirt code for this part is under $GIT/src/qemu/ in particular the qemu_command.c and qemu_process.c files Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Thu, Jun 28, 2012 at 05:21:57PM +0800, Dennis Chen wrote:
All,
These days I am trying to understand the interaction relationship between the libvirt and KVM kernel module, eg. kvm_intel.ko.
We know that KVM kernel module expose an entry in form of device file "/dev/kvm" which can be accessed by user space application to control, for example, create a VM using KVM_CREATE_VM with help of ioctl.
Now let's say the tool virsh based upon libvirt, we can create a guest domain with the command looks like: #virsh create guest.xml Obviously, the above command will create a VM. But when I try to investigate the libvirt code, I can't find any code play with the "/dev/kvm" to send KVM_CREATE_VM ioctl code to KVM kernel module. But I do found that the reference count of the kvm_intel.ko changed before the virsh create command launched and after.
So my question is: how does the libvirt interaction with KVM to create a VM? Anybody can give me some tips about that, eg, the corresponding codes in libvirt? The '/dev/kvm' device is the low level kernel interface for creating virtual domains. This is not actually used by libvirt at all. The QEMU binary has code that talks to /dev/kvm, so all libvirt does is to spawn a QEMU process which in turns creates the virtual machine
All the libvirt code for this part is under $GIT/src/qemu/ in particular the qemu_command.c and qemu_process.c files
Daniel Thanks Daniel, now I understand that the libvirt code will play with QEMU binary, for example, qemu-system-x86_64, the latter will talk with KVM module . But now the question is, I guess qemuProcessStart() function was used to spawn a QEMU process, so I built a virsh from the
On 06/28/2012 06:26 PM, Daniel P. Berrange wrote: source package with "-g -O0" flag, I want to gdb the virsh, but when I want to set a break point on qemuProcessStart(), I encounter issue: #gdb virsh (gdb) b main Breakpoint 1 at 0x807d480: file virsh.c, line 20270. (gdb) r Starting program: /usr/bin/virsh [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Breakpoint 1, main (argc=1, argv=0xbffff294) at virsh.c:20270 20270 { (gdb) b qemuProcessStart Function "qemuProcessStart" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 2 (qemuProcessStart) pending. (gdb)c virsh # create guest.xml Domain vdo created from vdo.xml virsh # continue instruction above doesn't hit the pending Breakpoint 2... So my question is, which kind of object file will be created from $GIT/src/qemu folder during the build process, is it a .so file and virsh will load it? BRs Dennis
On Fri, Jun 29, 2012 at 02:18:17PM +0800, Dennis Chen wrote:
On 06/28/2012 06:26 PM, Daniel P. Berrange wrote:
The '/dev/kvm' device is the low level kernel interface for creating virtual domains. This is not actually used by libvirt at all. The QEMU binary has code that talks to /dev/kvm, so all libvirt does is to spawn a QEMU process which in turns creates the virtual machine
All the libvirt code for this part is under $GIT/src/qemu/ in particular the qemu_command.c and qemu_process.c files
Daniel Thanks Daniel, now I understand that the libvirt code will play with QEMU binary, for example, qemu-system-x86_64, the latter will talk with KVM module . But now the question is, I guess qemuProcessStart() function was used to spawn a QEMU process, so I built a virsh from the source package with "-g -O0" flag, I want to gdb the virsh, but when I want to set a break point on qemuProcessStart(), I encounter issue:
#gdb virsh (gdb) b main Breakpoint 1 at 0x807d480: file virsh.c, line 20270. (gdb) r Starting program: /usr/bin/virsh [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Breakpoint 1, main (argc=1, argv=0xbffff294) at virsh.c:20270 20270 { (gdb) b qemuProcessStart Function "qemuProcessStart" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 2 (qemuProcessStart) pending.
(gdb)c
virsh # create guest.xml Domain vdo created from vdo.xml
virsh #
continue instruction above doesn't hit the pending Breakpoint 2...
So my question is, which kind of object file will be created from $GIT/src/qemu folder during the build process, is it a .so file and virsh will load it?
virsh talks to the libvirtd daemon, the libvirtd daemon run those calls from the qemu driver. The documentation is relatively sparse, but reading what is available will help: http://libvirt.org/api.html http://libvirt.org/internals.html so if you want to step though that function you must gdb the libvirt daemon, not the client application. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On 06/29/2012 02:49 PM, Daniel Veillard wrote:
On Fri, Jun 29, 2012 at 02:18:17PM +0800, Dennis Chen wrote:
On 06/28/2012 06:26 PM, Daniel P. Berrange wrote:
The '/dev/kvm' device is the low level kernel interface for creating virtual domains. This is not actually used by libvirt at all. The QEMU binary has code that talks to /dev/kvm, so all libvirt does is to spawn a QEMU process which in turns creates the virtual machine
All the libvirt code for this part is under $GIT/src/qemu/ in particular the qemu_command.c and qemu_process.c files
Daniel Thanks Daniel, now I understand that the libvirt code will play with QEMU binary, for example, qemu-system-x86_64, the latter will talk with KVM module . But now the question is, I guess qemuProcessStart() function was used to spawn a QEMU process, so I built a virsh from the source package with "-g -O0" flag, I want to gdb the virsh, but when I want to set a break point on qemuProcessStart(), I encounter issue:
#gdb virsh (gdb) b main Breakpoint 1 at 0x807d480: file virsh.c, line 20270. (gdb) r Starting program: /usr/bin/virsh [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Breakpoint 1, main (argc=1, argv=0xbffff294) at virsh.c:20270 20270 { (gdb) b qemuProcessStart Function "qemuProcessStart" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 2 (qemuProcessStart) pending.
(gdb)c
virsh # create guest.xml Domain vdo created from vdo.xml
virsh #
continue instruction above doesn't hit the pending Breakpoint 2...
So my question is, which kind of object file will be created from $GIT/src/qemu folder during the build process, is it a .so file and virsh will load it? virsh talks to the libvirtd daemon, the libvirtd daemon run those calls from the qemu driver. The documentation is relatively sparse, but reading what is available will help:
http://libvirt.org/api.html http://libvirt.org/internals.html
so if you want to step though that function you must gdb the libvirt daemon, not the client application.
Daniel
en, this Daniel is not that Daniel: ) I remember that I joined the presentation about libvirt delivered by you in Nanjing University last year. Yes, the document about libvirt is sparse, so I am often confused when I try to understand the internals of the relationship between QEMU, KVM and libvirt... Let me go through the link you mentioned to see if I can understand libvirtd comes from and the relationship with libvirt... BRs, Dennis
On Fri, Jun 29, 2012 at 03:07:54PM +0800, Dennis Chen wrote:
On 06/29/2012 02:49 PM, Daniel Veillard wrote: [...]
virsh talks to the libvirtd daemon, the libvirtd daemon run those calls from the qemu driver. The documentation is relatively sparse, but reading what is available will help:
http://libvirt.org/api.html http://libvirt.org/internals.html
so if you want to step though that function you must gdb the libvirt daemon, not the client application.
Daniel
en, this Daniel is not that Daniel: ) I remember that I joined the presentation about libvirt delivered by you in Nanjing University last year.
ah yes, there is Dan and Daniel that can be a bit confusing :-)
Yes, the document about libvirt is sparse, so I am often confused when I try to understand the internals of the relationship between QEMU, KVM and libvirt...
ah, yes the overall stack is a bit complex.
Let me go through the link you mentioned to see if I can understand libvirtd comes from and the relationship with libvirt...
Basically for some driver the application talk to the libvirt daemon to implement all the entry points of a driver (qemu/lxc for example), it uses the remote driver to connect to the daemon, and the daemon calls the real driver. Sometimes, the driver in the applcation does the work by talking directly to the hypervisor (ESX/Hyper-V). So if you want to debug/gdb the qemu code you must attach to libvirtd daemon not to the application. See slide 5 of the CLK slides ;-) http://veillard.com/Talks/CLKLinux2011.pdf Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On 06/29/2012 03:47 PM, Daniel Veillard wrote:
On Fri, Jun 29, 2012 at 03:07:54PM +0800, Dennis Chen wrote:
virsh talks to the libvirtd daemon, the libvirtd daemon run those calls from the qemu driver. The documentation is relatively sparse, but reading what is available will help:
http://libvirt.org/api.html http://libvirt.org/internals.html
so if you want to step though that function you must gdb the libvirt daemon, not the client application.
Daniel en, this Daniel is not that Daniel: ) I remember that I joined the
On 06/29/2012 02:49 PM, Daniel Veillard wrote: [...] presentation about libvirt delivered by you in Nanjing University last year. ah yes, there is Dan and Daniel that can be a bit confusing :-)
Yes, the document about libvirt is sparse, so I am often confused when I try to understand the internals of the relationship between QEMU, KVM and libvirt... ah, yes the overall stack is a bit complex.
Let me go through the link you mentioned to see if I can understand libvirtd comes from and the relationship with libvirt... Basically for some driver the application talk to the libvirt daemon to implement all the entry points of a driver (qemu/lxc for example), it uses the remote driver to connect to the daemon, and the daemon calls the real driver. Sometimes, the driver in the applcation does the work by talking directly to the hypervisor (ESX/Hyper-V). So if you want to debug/gdb the qemu code you must attach to libvirtd daemon not to the application. See slide 5 of the CLK slides ;-) http://veillard.com/Talks/CLKLinux2011.pdf
Daniel
Actually I've downloaded your ppt from http://v.csdn.hudong.com/linuxkernal2011/page2.html, slide 5 and 12 is useful for me because it's a overview picture. About daemon, in my system: root@dennis-:/home/dennis/workspace/Software# ps aux | grep libvirtd root 20308 0.0 0.1 102908 3540 ? Sl 15:25 0:00 /usr/sbin/libvirtd -d so the libvirtd daemon is there, but I have questions about this daemon: 1. where is the daemon source code, it's in the libvirt tar package? 2. Who is responsible to start this daemon and when? BRs, Dennis
On 06/29/2012 02:12 AM, Dennis Chen wrote:
so the libvirtd daemon is there, but I have questions about this daemon: 1. where is the daemon source code, it's in the libvirt tar package?
Yes, libvirt is responsible for both libvirt.so (the client code [src/remote/*], which bundles the RPC request to send to libvirtd) and for libvirtd (the daemon code [daemon/*], which receives the RPC request and then calls into the qemu driver code [src/qemu/*] to act on the request).
2. Who is responsible to start this daemon and when?
If you use Fedora or RHEL, it is the installation of the 'libvirt' package that installs libvirtd as a service, and therefore the service mechanism (systemd or initd) that fires off libvirtd when you boot. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 06/29/2012 08:50 PM, Eric Blake wrote:
On 06/29/2012 02:12 AM, Dennis Chen wrote:
so the libvirtd daemon is there, but I have questions about this daemon: 1. where is the daemon source code, it's in the libvirt tar package? Yes, libvirt is responsible for both libvirt.so (the client code [src/remote/*], which bundles the RPC request to send to libvirtd) and for libvirtd (the daemon code [daemon/*], which receives the RPC request and then calls into the qemu driver code [src/qemu/*] to act on the request).
2. Who is responsible to start this daemon and when? If you use Fedora or RHEL, it is the installation of the 'libvirt' package that installs libvirtd as a service, and therefore the service mechanism (systemd or initd) that fires off libvirtd when you boot.
Eric, Thanks for the elaboration! It's very clear for me understanding the relationship of libvirtd and qemu driver and the qemu binary, also the internal mechanism... BRs, Dennis
On 06/28/2012 05:21 PM, Dennis Chen wrote:
All,
These days I am trying to understand the interaction relationship between the libvirt and KVM kernel module, eg. kvm_intel.ko.
We know that KVM kernel module expose an entry in form of device file "/dev/kvm" which can be accessed by user space application to control, for example, create a VM using KVM_CREATE_VM with help of ioctl.
Now let's say the tool virsh based upon libvirt, we can create a guest domain with the command looks like: #virsh create guest.xml Obviously, the above command will create a VM. But when I try to investigate the libvirt code, I can't find any code play with the "/dev/kvm" to send KVM_CREATE_VM ioctl code to KVM kernel module. But I do found that the reference count of the kvm_intel.ko changed before the virsh create command launched and after.
So my question is: how does the libvirt interaction with KVM to create a VM? Anybody can give me some tips about that, eg, the corresponding codes in libvirt?
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=2f93... 114 /* device for kvm ioctls */ 115 #define KVM_DEVICE "/dev/kvm" 1022 static int kvmGetMaxVCPUs(void) { 1023 int maxvcpus = 1; 1024 1025 int r, fd; 1026 1027 fd = open(KVM_DEVICE, O_RDONLY); 1028 if (fd < 0) { 1029 virReportSystemError(errno, _("Unable to open %s"), KVM_DEVICE); 1030 return -1; 1031 } 1032 1033 r = ioctl(fd, KVM_CHECK_EXTENSION, KVM_CAP_NR_VCPUS); 1034 if (r > 0) 1035 maxvcpus = r; 1036 1037 VIR_FORCE_CLOSE(fd); 1038 return maxvcpus; 1039 } http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_process.c;h=c51... Please see qemuProcessStart() implemetation: 3288 int qemuProcessStart(virConnectPtr conn, 3289 struct qemud_driver *driver, 3290 virDomainObjPtr vm, 3291 const char *migrateFrom, 3292 int stdin_fd, 3293 const char *stdin_path, 3294 virDomainSnapshotObjPtr snapshot, 3295 enum virNetDevVPortProfileOp vmop, 3296 unsigned int flags) ......
BRs. Dennis
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 06/28/2012 07:03 PM, Alex Jia wrote:
On 06/28/2012 05:21 PM, Dennis Chen wrote:
All,
These days I am trying to understand the interaction relationship between the libvirt and KVM kernel module, eg. kvm_intel.ko.
We know that KVM kernel module expose an entry in form of device file "/dev/kvm" which can be accessed by user space application to control, for example, create a VM using KVM_CREATE_VM with help of ioctl.
Now let's say the tool virsh based upon libvirt, we can create a guest domain with the command looks like: #virsh create guest.xml Obviously, the above command will create a VM. But when I try to investigate the libvirt code, I can't find any code play with the "/dev/kvm" to send KVM_CREATE_VM ioctl code to KVM kernel module. But I do found that the reference count of the kvm_intel.ko changed before the virsh create command launched and after.
So my question is: how does the libvirt interaction with KVM to create a VM? Anybody can give me some tips about that, eg, the corresponding codes in libvirt? http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=2f93...
114 /* device for kvm ioctls */ 115 #define KVM_DEVICE "/dev/kvm"
1022 static int kvmGetMaxVCPUs(void) { 1023 int maxvcpus = 1; 1024 1025 int r, fd; 1026 1027 fd = open(KVM_DEVICE, O_RDONLY); 1028 if (fd< 0) { 1029 virReportSystemError(errno, _("Unable to open %s"), KVM_DEVICE); 1030 return -1; 1031 } 1032 1033 r = ioctl(fd, KVM_CHECK_EXTENSION, KVM_CAP_NR_VCPUS); 1034 if (r> 0) 1035 maxvcpus = r; 1036 1037 VIR_FORCE_CLOSE(fd); 1038 return maxvcpus; 1039 } Yes, I noticed the code above in libvirt, but it's not related with my question in the first message... http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_process.c;h=c51...
Please see qemuProcessStart() implemetation:
3288 int qemuProcessStart(virConnectPtr conn, 3289 struct qemud_driver *driver, 3290 virDomainObjPtr vm, 3291 const char *migrateFrom, 3292 int stdin_fd, 3293 const char *stdin_path, 3294 virDomainSnapshotObjPtr snapshot, 3295 enum virNetDevVPortProfileOp vmop, 3296 unsigned int flags) ......
BRs. Dennis
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users I guess the qemuProcessStart() is used to spawn a QEMU process, but seems that lot's of tricks in its implementation, essentially, does this function spawn the qemu process looks like, eg:
StartProcess("/usr/bin/qemu-system-x86_64", ...); BRs, Dennis
On 06/29/2012 12:25 AM, Dennis Chen wrote:
So my question is: how does the libvirt interaction with KVM to create a VM? Anybody can give me some tips about that, eg, the corresponding codes in libvirt?
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=2f93...
Yes, I noticed the code above in libvirt, but it's not related with my question in the first message...
Actually, yes it is. That's the code that libvirtd runs in reaction to your client code making an RPC code via libvirt.so.
I guess the qemuProcessStart() is used to spawn a QEMU process, but seems that lot's of tricks in its implementation, essentially, does this function spawn the qemu process looks like, eg:
StartProcess("/usr/bin/qemu-system-x86_64", ...);
Not StartProcess, so much as execve(), as wrapped in our virCommand API (src/util/command.c). If you want to see the exact command line that we constructed, look at /var/log/libvirt/qemu/$dom.log for the domain name that you are starting. It is a rather hairy command line; this is an example from one of my VMs: LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.14 -cpu qemu32 -enable-kvm -m 768 -smp 1,sockets=1,cores=1,threads=1 -name fedora-local -uuid 24ab99db-bf50-085d-d3e5-7528243ae9b1 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fedora-local.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/fedora-local.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/libvirt/images/top.img,if=none,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:65:82:9e,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -vga cirrus -device ES1370,id=sound0,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 In turn, it is that hairy command line that starts the qemu process, and qemu then takes over the actual communication with the kernel via /dev/kvm to use KVM virtualization. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (6)
-
Alex Jia -
Daniel P. Berrange -
Daniel Veillard -
Dennis Chen -
Dennis Chen -
Eric Blake