[libvirt-users] Host modifications
Hello, I test libvirt 0.9.12 on Debian. I am disappointed by changes made on my host without any notice. Examples: - editing interfaces with virsh or virt-manager modifies my /etc/network/interfaces. It's not clear at first glance that I can even cut myself from the host when editing remotely. The initial file is not even saved. - starting default network (nat) adds rules in netfilter. I have not seen how to create another network nat conf without calling clean-traffic nwfilter (it is not explicit in network XML file). Is it hardcoded ? I think it would be nice: - to be alerted before any host modification, - to be able to change the templates, for instance: - not including any nwfilter when creating a network, - script called when adding a file in a dir pool, - and so on. Did I miss something? Am I alone to disappointed by such behavior? Thanks, Benoit
On 03/25/2013 03:09 AM, Benoit Friry wrote:
Hello,
I test libvirt 0.9.12 on Debian.
I am disappointed by changes made on my host without any notice.
The whole point of libvirtd is to make changes on your host; I wouldn't go so far as to say that it was without notice, just that they are changes that you weren't aware that libvirtd was capable of exposing.
Examples: - editing interfaces with virsh or virt-manager modifies my /etc/network/interfaces. It's not clear at first glance that I can even cut myself from the host when editing remotely. The initial file is not even saved.
The initial file _is_ saved if you properly use the 'virsh iface-begin' command before making any changes, then 'virsh iface-commit' if you are happy with the changes. 'virsh iface-rollback' will revert you to a previous saved state, and since we know that an improper change can cut off connectivity, we also set things up so that a host reboot will do an implicit 'virsh iface-rollback' on any uncommitted changes.
- starting default network (nat) adds rules in netfilter. I have not seen how to create another network nat conf without calling clean-traffic nwfilter (it is not explicit in network XML file). Is it hardcoded ?
What distro are you using? The clean-traffic nwfilter is not installed by default on Fedora, so I'm wondering if you are hitting a distro-specific add-on, or something that is added by a higher layer of the virt stack than just libvirt. Libvirt's own NAT netfilter rules are required for out-of-the-box NAT to a guest, but no one says you are forced to use NAT; you can design your own bridge and take over the netfilter rules yourself if you don't want libvirt messing with iptables.
I think it would be nice: - to be alerted before any host modification,
What did you have in mind? Patches are welcome if you can come up with a proposal.
- to be able to change the templates, for instance: - not including any nwfilter when creating a network, - script called when adding a file in a dir pool, - and so on.
Did I miss something? Am I alone to disappointed by such behavior?
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 03/25/2013 16:21, Eric Blake wrote:
On 03/25/2013 03:09 AM, Benoit Friry wrote:
Hello,
I test libvirt 0.9.12 on Debian.
I am disappointed by changes made on my host without any notice.
The whole point of libvirtd is to make changes on your host; I wouldn't go so far as to say that it was without notice, just that they are changes that you weren't aware that libvirtd was capable of exposing.
I would not say libvirt is just about making changes on the host. I like the wrapping of the daemonization (instead of some "nohup kvm &") and the normalization of the commands. And when I run virt-install, I am not disappointed by the file creation.
Examples: - editing interfaces with virsh or virt-manager modifies my /etc/network/interfaces. It's not clear at first glance that I can even cut myself from the host when editing remotely. The initial file is not even saved.
The initial file _is_ saved if you properly use the 'virsh iface-begin' command before making any changes, then 'virsh iface-commit' if you are happy with the changes. 'virsh iface-rollback' will revert you to a previous saved state, and since we know that an improper change can cut off connectivity, we also set things up so that a host reboot will do an implicit 'virsh iface-rollback' on any uncommitted changes.
I did not understood the purpose of this commands. Unfortunately, they are not available in virt-manager.
- starting default network (nat) adds rules in netfilter. I have not seen how to create another network nat conf without calling clean-traffic nwfilter (it is not explicit in network XML file). Is it hardcoded ?
What distro are you using? The clean-traffic nwfilter is not installed by default on Fedora, so I'm wondering if you are hitting a distro-specific add-on, or something that is added by a higher layer of the virt stack than just libvirt. Libvirt's own NAT netfilter rules are required for out-of-the-box NAT to a guest, but no one says you are forced to use NAT; you can design your own bridge and take over the netfilter rules yourself if you don't want libvirt messing with iptables.
Debian wheezy, libvirt 0.9.12. Debian patches are listed on http://patch-tracker.debian.org/package/libvirt/0.9.12-11 I do not see anything modifying that part. I can be wrong.
I think it would be nice: - to be alerted before any host modification,
What did you have in mind? Patches are welcome if you can come up with a proposal.
For a beginning, I think it may be valuable to list such behavior in the README. http://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=README;hb=HEAD On Debian, and maybe in upstream, clean-traffic nwfilter is activated for every nat network... But without being listed in the network XML configuration.
- to be able to change the templates, for instance: - not including any nwfilter when creating a network, - script called when adding a file in a dir pool, - and so on.
Another example: what if I want to use BIND9 instead of dnsmasq? BIND9 has a dns64 capability, dnsmasq has not. dnsmasq, radvd, brctl are hardcoded. Don't you think it would be better to call a helper script, that can be tweaked by admins? Thanks & HAND, benoit
On 03/25/2013 05:12 PM, Benoit Friry wrote:
On 03/25/2013 16:21, Eric Blake wrote:
On 03/25/2013 03:09 AM, Benoit Friry wrote:
Examples: - editing interfaces with virsh or virt-manager modifies my /etc/network/interfaces. It's not clear at first glance that I can even cut myself from the host when editing remotely. The initial file is not even saved. The initial file _is_ saved if you properly use the 'virsh iface-begin' command before making any changes, then 'virsh iface-commit' if you are happy with the changes. 'virsh iface-rollback' will revert you to a previous saved state, and since we know that an improper change can cut off connectivity, we also set things up so that a host reboot will do an implicit 'virsh iface-rollback' on any uncommitted changes. I did not understood the purpose of this commands. Unfortunately, they are not available in virt-manager.
virsh iface-begin saves the current state of config for all network interfaces so that we can restore it if the new config fails virsh iface-commit marks the new state of config as "okay" and removes the backup config that had been saved. virsh iface-rollback restores the original config that was saved with virsh iface-begin (this also automatically happens if you reboot the host after calling virsh iface-begin but without calling virsh iface-commit) virt-manager is "behind" libvirt in many areas, this is just one; there has been a lack of people doing active development on virt-manager for a year or so now; hopefully that will be changing soon :-)
- starting default network (nat) adds rules in netfilter. I have not seen how to create another network nat conf without calling clean-traffic nwfilter (it is not explicit in network XML file).
nwfilter rules are applied to individual *guest interfaces* by adding a <filterref> element to the guest's <interface> element. They are completely unrelated to the netfilter rules added when a libvirt network is started. The rules added for a libvirt virtual network change depending on the setting of "<forward mode='xxx'>" in the network config 1) for <forward mode='route'> a set of rules allowing all traffic in both directions is added 2) for <forward mode='nat'> rules are added to allow outbound traffic from the guests to any destination, prevent inbound traffic from anywhere except a) the host and b) other guests attached to the same network, AND to mangle the source address of packets outbound to places beyond the host (so that they appear to be coming from the host itself). 3) if there is *no* <forward> (aka an "isolated" network), rules are added to allow traffic only between guests on the same network and the host, but block all other traffic. The rules for each of these cases is hardcoded (and also very simplistic) and can't be configured in any more finely grained manner.
Is it hardcoded ? What distro are you using? The clean-traffic nwfilter is not installed by default on Fedora, so I'm wondering if you are hitting a distro-specific add-on, or something that is added by a higher layer of the virt stack than just libvirt. Libvirt's own NAT netfilter rules are required for out-of-the-box NAT to a guest, but no one says you are forced to use NAT; you can design your own bridge and take over the netfilter rules yourself if you don't want libvirt messing with iptables. Debian wheezy, libvirt 0.9.12.
Debian patches are listed on http://patch-tracker.debian.org/package/libvirt/0.9.12-11
I do not see anything modifying that part. I can be wrong.
I think it would be nice: - to be alerted before any host modification, What did you have in mind? Patches are welcome if you can come up with a proposal. For a beginning, I think it may be valuable to list such behavior in the README.
http://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=README;hb=HEAD
On Debian, and maybe in upstream, clean-traffic nwfilter is activated for every nat network... But without being listed in the network XML configuration.
That is not true. No nwfilter rule is added for *any* "network" ever. They are added for individual guests, and only when the guest's XML specifically says to do that. My guess is that you're mistaking some of the rules added for NAT as originating from the clean-traffic filter. Can you paste the relevant parts of the output of "iptables -S" and indicate which you believe to be coming from nwfilter?
- to be able to change the templates, for instance: - not including any nwfilter when creating a network, - script called when adding a file in a dir pool, - and so on. Another example: what if I want to use BIND9 instead of dnsmasq? BIND9 has a dns64 capability, dnsmasq has not.
dnsmasq, radvd, brctl are hardcoded. Don't you think it would be better to call a helper script, that can be tweaked by admins?
No. That would be a nightmare to support; we specifically avoid doing something that free-form. If you would like a network driver that uses different utilities in place of dnsmasq and radvd, the proper way to do that is with a different network driver to be built in place of src/network/bridge_driver.c. But before you go to that trouble, are you certain that dnsmasq still doesn't have what you need? Recent versions of libvirt (since 1.0.1) have support for many new IPv6 features of dnsmasq 2.66+, including DHCP6, and I think dnsmasq has had support for IPv6 DNS resolution for quite a long time. (I couldn't figure out from a quick google search whether or not dnsmasq directly supports DNS64, or if that may be unnecessary if the dns server upstream from dnsmasq has DNS64 support.) (BTW, radvd has been replaced in newer libvirt versions by similar functionality in newer versions of dnsmasq, where available, and I'm pretty sure we've eliminated all use of brctl in favor of directly using iotctl() some time ago as well.)
Le 03/28/2013 16:59, Laine Stump wrote :
On 03/25/2013 05:12 PM, Benoit Friry wrote:
On 03/25/2013 16:21, Eric Blake wrote:
On 03/25/2013 03:09 AM, Benoit Friry wrote:
- starting default network (nat) adds rules in netfilter. I have not seen how to create another network nat conf without calling clean-traffic nwfilter (it is not explicit in network XML file).
nwfilter rules are applied to individual *guest interfaces* by adding a <filterref> element to the guest's <interface> element. They are completely unrelated to the netfilter rules added when a libvirt network is started.
Indeed, I thought the added rules where from nwfilter. I did not checked precisely. I did not suspect there were another set of rules for networks.
The rules added for a libvirt virtual network change depending on the setting of "<forward mode='xxx'>" in the network config
1) for <forward mode='route'> a set of rules allowing all traffic in both directions is added
2) for <forward mode='nat'> rules are added to allow outbound traffic from the guests to any destination, prevent inbound traffic from anywhere except a) the host and b) other guests attached to the same network, AND to mangle the source address of packets outbound to places beyond the host (so that they appear to be coming from the host itself).
3) if there is *no* <forward> (aka an "isolated" network), rules are added to allow traffic only between guests on the same network and the host, but block all other traffic.
The rules for each of these cases is hardcoded (and also very simplistic) and can't be configured in any more finely grained manner.
It's a pity. There is nothing between all hardcoded rules and all manual rules (in mode='bridge'). Manual rules are to be managed outside. It would have been nice to be able to start/stop specific rules from libvirt, linked to network start/stop commands. Thanks for the explanation.
- to be able to change the templates, for instance: - not including any nwfilter when creating a network, - script called when adding a file in a dir pool, - and so on. Another example: what if I want to use BIND9 instead of dnsmasq? BIND9 has a dns64 capability, dnsmasq has not.
dnsmasq, radvd, brctl are hardcoded. Don't you think it would be better to call a helper script, that can be tweaked by admins?
No. That would be a nightmare to support; we specifically avoid doing something that free-form. If you would like a network driver that uses different utilities in place of dnsmasq and radvd, the proper way to do that is with a different network driver to be built in place of src/network/bridge_driver.c.
Ouch. Isn't it overkill? I can manage some scripts. I will not manage patches, to be rebuild with every new release. Few organization can afford such thing! What about a overwrite conf file? Hardcoded conf by default, but one can change some element. Support only when no overwrited conf.
But before you go to that trouble, are you certain that dnsmasq still doesn't have what you need? Recent versions of libvirt (since 1.0.1) have support for many new IPv6 features of dnsmasq 2.66+, including DHCP6, and I think dnsmasq has had support for IPv6 DNS resolution for quite a long time. (I couldn't figure out from a quick google search whether or not dnsmasq directly supports DNS64, or if that may be unnecessary if the dns server upstream from dnsmasq has DNS64 support.)
AFAIK, dnsmasq has no DNS64 support. Thanks, Benoit
participants (3)
-
Benoit Friry -
Eric Blake -
Laine Stump