[libvirt PATCH 0/2] network: Drop UUID handling for default network
<blurb>cc30b49f-6084-48da-879e-55645459f2a0</blurb> Andrea Bolognani (2): network: Drop UUID handling for default network spec: Drop UUID handling for default network libvirt.spec.in | 4 ---- src/network/meson.build | 32 +++++++------------------------- 2 files changed, 7 insertions(+), 29 deletions(-) -- 2.26.2
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML; the fact that we only do this if the uuidgen command happens to be available on the build machine is further proof that we can safely skip this step. This patch is best viewed with 'git show -w'. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- src/network/meson.build | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-) diff --git a/src/network/meson.build b/src/network/meson.build index 13dd2c26b2..3ec598c3f9 100644 --- a/src/network/meson.build +++ b/src/network/meson.build @@ -84,31 +84,13 @@ if conf.has('WITH_NETWORK') runstatedir / 'libvirt' / 'network', ] - uuidgen_prog = find_program('uuidgen', required: false) - - if uuidgen_prog.found() - uuid = run_command(uuidgen_prog).stdout().strip() - - configure_file( - input: 'default.xml.in', - output: '@BASENAME@', - command: [ - 'sed', '-e', 's|</name>|</name>\\n <uuid>@0@</uuid>|'.format(uuid), - '@INPUT@', - ], - capture: true, - install: true, - install_dir: confdir / 'qemu' / 'networks', - ) - else - configure_file( - input: 'default.xml.in', - output: '@BASENAME@', - copy: true, - install: true, - install_dir: confdir / 'qemu' / 'networks', - ) - endif + configure_file( + input: 'default.xml.in', + output: '@BASENAME@', + copy: true, + install: true, + install_dir: confdir / 'qemu' / 'networks', + ) meson.add_install_script( meson_python_prog.path(), python3_prog.path(), meson_install_symlink_prog.path(), -- 2.26.2
On 11/15/20 3:43 PM, Andrea Bolognani wrote:
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML;
But that automatically generated uuid will not be stored in the original xml in /etc/libvirt, so a new and different uuid will be generated every time libvirt is restarted. I don't know if the solution to this is to modify libvirt so that it rewrites the XML for a network if it has to auto-generate any attributes[*], or to put a canned/static uuid value in the installed xml, or just to declare that we don't care if the uuid changes from one run of libvirtd to the next. But definitely these patches change behavior, so we probably need to point that out and maybe discuss it. BTW, the whole thing of generating a different default.xml for each install of libvirt is a thorn in the side anyway - I looked for the BZ describing the problem and came up with this: https://bugzilla.redhat.com/show_bug.cgi?id=1657041 which isn't the entire discussion, but is part of it. Basically while talking about this in *some venue* (maybe it was IRC, maybe a different BZ?) we decided that we shouldn't be (can't be?) generating files during the package %post_install. Our main topic of discussion was the subnet of the network, but adding/modifying the uuid is just another example of the same undesired behavior. So *that* would be an argument in favor of at least the "don't generate a different uuid at install time" part of your patches. But it doesn't (afaics) lead to a definite resolution of "what should be done instead?". (btw, I'm working / supposed to be working on eliminating the need to modify the subnet address in this same file with a new "autoaddress" function that will replace a fixed IP in a network definition with an indicator that means "find an unused subnet and use that". Doesn't solve the uuid bit though) [*] I have a vague memory that there is (or was?) at least one case where we would automatically update the persistent definition of a domain or network on disk when libvirt started (it was needed due to a change in some attribute or something), but I don't recall the details, don't feel like looking for it right now, and it anyway is not the norm.
the fact that we only do this if the uuidgen command happens to be available on the build machine is further proof that we can safely skip this step.
This patch is best viewed with 'git show -w'.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- src/network/meson.build | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-)
diff --git a/src/network/meson.build b/src/network/meson.build index 13dd2c26b2..3ec598c3f9 100644 --- a/src/network/meson.build +++ b/src/network/meson.build @@ -84,31 +84,13 @@ if conf.has('WITH_NETWORK') runstatedir / 'libvirt' / 'network', ]
- uuidgen_prog = find_program('uuidgen', required: false) - - if uuidgen_prog.found() - uuid = run_command(uuidgen_prog).stdout().strip() - - configure_file( - input: 'default.xml.in', - output: '@BASENAME@', - command: [ - 'sed', '-e', 's|</name>|</name>\\n <uuid>@0@</uuid>|'.format(uuid), - '@INPUT@', - ], - capture: true, - install: true, - install_dir: confdir / 'qemu' / 'networks', - ) - else - configure_file( - input: 'default.xml.in', - output: '@BASENAME@', - copy: true, - install: true, - install_dir: confdir / 'qemu' / 'networks', - ) - endif + configure_file( + input: 'default.xml.in', + output: '@BASENAME@', + copy: true, + install: true, + install_dir: confdir / 'qemu' / 'networks', + )
meson.add_install_script( meson_python_prog.path(), python3_prog.path(), meson_install_symlink_prog.path(),
On Sun, Nov 15, 2020 at 07:19:24PM -0500, Laine Stump wrote:
On 11/15/20 3:43 PM, Andrea Bolognani wrote:
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML;
But that automatically generated uuid will not be stored in the original xml in /etc/libvirt, so a new and different uuid will be generated every time libvirt is restarted.
I don't know if the solution to this is to modify libvirt so that it rewrites the XML for a network if it has to auto-generate any attributes[*], or to put a canned/static uuid value in the installed xml, or just to declare that we don't care if the uuid changes from one run of libvirtd to the next. But definitely these patches change behavior, so we probably need to point that out and maybe discuss it.
We definitely should not be changing UUID on every startup, so IMHO this patch should not be applied. I'm not seeing much point in creating some new solution in libvirtd to replace something which already works, unless there's actually some real problem reported which isn't mentioned in this commit message. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Sun, 2020-11-15 at 19:19 -0500, Laine Stump wrote:
On 11/15/20 3:43 PM, Andrea Bolognani wrote:
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML;
But that automatically generated uuid will not be stored in the original xml in /etc/libvirt, so a new and different uuid will be generated every time libvirt is restarted.
That doesn't seem to be the case: $ sudo cat /etc/libvirt/qemu/networks/default.xml <network> <name>default</name> <forward mode='nat'/> <bridge name='virbr0'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> $ sudo systemctl start libvirtd $ sudo systemctl stop libvirtd $ sudo cat /etc/libvirt/qemu/networks/default.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh net-edit default or other application using the libvirt API. --> <network> <name>default</name> <uuid>212a684a-4ca4-4c42-b389-aecd676868f8</uuid> <forward mode='nat'/> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:ff:27:85'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> $ sudo systemctl start libvirtd $ sudo systemctl stop libvirtd $ sudo cat /etc/libvirt/qemu/networks/default.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh net-edit default or other application using the libvirt API. --> <network> <name>default</name> <uuid>212a684a-4ca4-4c42-b389-aecd676868f8</uuid> <forward mode='nat'/> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:ff:27:85'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network>
I don't know if the solution to this is to modify libvirt so that it rewrites the XML for a network if it has to auto-generate any attributes[*], or to put a canned/static uuid value in the installed xml, or just to declare that we don't care if the uuid changes from one run of libvirtd to the next. But definitely these patches change behavior, so we probably need to point that out and maybe discuss it.
[*] I have a vague memory that there is (or was?) at least one case where we would automatically update the persistent definition of a domain or network on disk when libvirt started (it was needed due to a change in some attribute or something), but I don't recall the details, don't feel like looking for it right now, and it anyway is not the norm.
AFAICT the behavior you're talking about, and which can be seen in action in the transcript above, is caused by the following snippet in virNetworkLoadConfig(): switch ((virNetworkForwardType) def->forward.type) { case VIR_NETWORK_FORWARD_NONE: case VIR_NETWORK_FORWARD_NAT: case VIR_NETWORK_FORWARD_ROUTE: case VIR_NETWORK_FORWARD_OPEN: if (!def->mac_specified) { virNetworkSetBridgeMacAddr(def); virNetworkSaveConfig(configDir, def, xmlopt); } break; which can be tracked down to https://gitlab.com/libvirt/libvirt/-/commit/a47ae7c004e92f959b45808ca82326e8... So, it's far from a new behavior. Note that we only persist the new configuration if the bridge's MAC address has been generated, not if the network UUID has, which is something that we might want to change. But, for the use case at hand, we know that the XML template for the default network will have neither of those, so in practice they will both be generated during the first startup and the result will be persisted to disk. -- Andrea Bolognani / Red Hat / Virtualization
On 11/16/20 5:33 AM, Andrea Bolognani wrote:
On Sun, 2020-11-15 at 19:19 -0500, Laine Stump wrote:
On 11/15/20 3:43 PM, Andrea Bolognani wrote:
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML; But that automatically generated uuid will not be stored in the original xml in /etc/libvirt, so a new and different uuid will be generated every time libvirt is restarted. That doesn't seem to be the case:
$ sudo cat /etc/libvirt/qemu/networks/default.xml <network> <name>default</name> <forward mode='nat'/> <bridge name='virbr0'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> $ sudo systemctl start libvirtd $ sudo systemctl stop libvirtd $ sudo cat /etc/libvirt/qemu/networks/default.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh net-edit default or other application using the libvirt API. -->
<network> <name>default</name> <uuid>212a684a-4ca4-4c42-b389-aecd676868f8</uuid> <forward mode='nat'/> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:ff:27:85'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> $ sudo systemctl start libvirtd $ sudo systemctl stop libvirtd $ sudo cat /etc/libvirt/qemu/networks/default.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh net-edit default or other application using the libvirt API. -->
<network> <name>default</name> <uuid>212a684a-4ca4-4c42-b389-aecd676868f8</uuid> <forward mode='nat'/> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:ff:27:85'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network>
But... I did a nearly identical experiment last night before posting my reply, and came up with different results, which is why I posted... Ah! So the difference is that in your example, you have no MAC address either, and the code you point out down below is noticing that change, and causing the entire network config to be re-written to the disk. I hadn't removed the MAC address, so in my case it didn't re-write the config. (...and now I've read to the bottom of your reply and see that you've already figured that out :-P) I guess if we're okay with re-writing the config at daemon start to add a MAC address, then we should be okay with doing that to add a uuid (they're really just differently-used examples of the same thing). But if we're going to do that, the check to see if the config should be modified to check both uuid and mac address (even if in current practice both are absent from the proto-default.xml, that may not be the case in the future). Alternately, if we're *not* okay with rewriting the config at daemon start time (even though, as you've pointed out, we've been doing that for 6 years already), then we would need to *stop* doing that for MAC address. (Which would mean we would need to figure out a different way to get a fixed MAC address into the config file, and seeing that the commit you reference actually *removes* my original code that added it in at install time, we can't really go back there (also because that's moving the code in the wrong direction - Fedora SilverBlue people said in the BZ I referenced that they don't want %post_install scripts modifying the files that are installed). I think I'm okay with the former (adding a check for changed uuid onto your patches); it gives us a stable uuid (as long as the modified disk contents are saved for the next reboot) while eliminating another bit of the undesirable code that runs during %post_install. Is there anything else wrt. pre-built images that we're not taking into consideration though?
I don't know if the solution to this is to modify libvirt so that it rewrites the XML for a network if it has to auto-generate any attributes[*], or to put a canned/static uuid value in the installed xml, or just to declare that we don't care if the uuid changes from one run of libvirtd to the next. But definitely these patches change behavior, so we probably need to point that out and maybe discuss it.
[*] I have a vague memory that there is (or was?) at least one case where we would automatically update the persistent definition of a domain or network on disk when libvirt started (it was needed due to a change in some attribute or something), but I don't recall the details, don't feel like looking for it right now, and it anyway is not the norm. AFAICT the behavior you're talking about, and which can be seen in action in the transcript above, is caused by the following snippet in virNetworkLoadConfig():
switch ((virNetworkForwardType) def->forward.type) { case VIR_NETWORK_FORWARD_NONE: case VIR_NETWORK_FORWARD_NAT: case VIR_NETWORK_FORWARD_ROUTE: case VIR_NETWORK_FORWARD_OPEN: if (!def->mac_specified) { virNetworkSetBridgeMacAddr(def); virNetworkSaveConfig(configDir, def, xmlopt); } break;
which can be tracked down to
https://gitlab.com/libvirt/libvirt/-/commit/a47ae7c004e92f959b45808ca82326e8...
So, it's far from a new behavior.
Note that we only persist the new configuration if the bridge's MAC address has been generated, not if the network UUID has, which is something that we might want to change. But, for the use case at hand, we know that the XML template for the default network will have neither of those, so in practice they will both be generated during the first startup and the result will be persisted to disk.
On Mon, 2020-11-16 at 10:00 -0500, Laine Stump wrote:
On 11/16/20 5:33 AM, Andrea Bolognani wrote:
On Sun, 2020-11-15 at 19:19 -0500, Laine Stump wrote:
On 11/15/20 3:43 PM, Andrea Bolognani wrote:
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML;
But that automatically generated uuid will not be stored in the original xml in /etc/libvirt, so a new and different uuid will be generated every time libvirt is restarted.
That doesn't seem to be the case:
[... a bunch of command line nonsense ...]
But... I did a nearly identical experiment last night before posting my reply, and came up with different results, which is why I posted...
Ah! So the difference is that in your example, you have no MAC address either, and the code you point out down below is noticing that change, and causing the entire network config to be re-written to the disk. I hadn't removed the MAC address, so in my case it didn't re-write the config.
(...and now I've read to the bottom of your reply and see that you've already figured that out :-P)
"And that's why you always read the entire message before you start replying to it." -- J. Walter Weatherman
I guess if we're okay with re-writing the config at daemon start to add a MAC address, then we should be okay with doing that to add a uuid (they're really just differently-used examples of the same thing). But if we're going to do that, the check to see if the config should be modified to check both uuid and mac address (even if in current practice both are absent from the proto-default.xml, that may not be the case in the future).
Alternately, if we're *not* okay with rewriting the config at daemon start time (even though, as you've pointed out, we've been doing that for 6 years already), then we would need to *stop* doing that for MAC address. (Which would mean we would need to figure out a different way to get a fixed MAC address into the config file, and seeing that the commit you reference actually *removes* my original code that added it in at install time, we can't really go back there (also because that's moving the code in the wrong direction - Fedora SilverBlue people said in the BZ I referenced that they don't want %post_install scripts modifying the files that are installed).
I don't see how we could possibly not be okay with writing back the updated configuration: filling in whatever blanks are present in the user-provided configuration and then writing the result back to disk is quite central to how we handle domains and other objects, and I see no reason networks should behave any differently. If anything, I think we should not extend the check to UUIDs, but just drop it completely and unconditionally write the (possibly) updated file file to disk. I'll try cooking a patch for that.
I think I'm okay with the former (adding a check for changed uuid onto your patches); it gives us a stable uuid (as long as the modified disk contents are saved for the next reboot) while eliminating another bit of the undesirable code that runs during %post_install.
Is there anything else wrt. pre-built images that we're not taking into consideration though?
Honestly I was not considering the pre-built images angle at all, so there's probably *loads* that I'm not taking into account O:-) -- Andrea Bolognani / Red Hat / Virtualization
On 11/16/20 11:43 AM, Andrea Bolognani wrote:
On Mon, 2020-11-16 at 10:00 -0500, Laine Stump wrote:
On 11/16/20 5:33 AM, Andrea Bolognani wrote:
On Sun, 2020-11-15 at 19:19 -0500, Laine Stump wrote:
On 11/15/20 3:43 PM, Andrea Bolognani wrote:
We are generating a fresh UUID and storing it in the XML for the default network, but this is unnecessary because the network driver will automatically generate one if it's missing from the XML; But that automatically generated uuid will not be stored in the original xml in /etc/libvirt, so a new and different uuid will be generated every time libvirt is restarted. That doesn't seem to be the case:
[... a bunch of command line nonsense ...] But... I did a nearly identical experiment last night before posting my reply, and came up with different results, which is why I posted...
Ah! So the difference is that in your example, you have no MAC address either, and the code you point out down below is noticing that change, and causing the entire network config to be re-written to the disk. I hadn't removed the MAC address, so in my case it didn't re-write the config.
(...and now I've read to the bottom of your reply and see that you've already figured that out :-P) "And that's why you always read the entire message before you start replying to it." -- J. Walter Weatherman
I guess if we're okay with re-writing the config at daemon start to add a MAC address, then we should be okay with doing that to add a uuid (they're really just differently-used examples of the same thing). But if we're going to do that, the check to see if the config should be modified to check both uuid and mac address (even if in current practice both are absent from the proto-default.xml, that may not be the case in the future).
Alternately, if we're *not* okay with rewriting the config at daemon start time (even though, as you've pointed out, we've been doing that for 6 years already), then we would need to *stop* doing that for MAC address. (Which would mean we would need to figure out a different way to get a fixed MAC address into the config file, and seeing that the commit you reference actually *removes* my original code that added it in at install time, we can't really go back there (also because that's moving the code in the wrong direction - Fedora SilverBlue people said in the BZ I referenced that they don't want %post_install scripts modifying the files that are installed). I don't see how we could possibly not be okay with writing back the updated configuration: filling in whatever blanks are present in the user-provided configuration and then writing the result back to disk is quite central to how we handle domains and other objects, and I see no reason networks should behave any differently.
If anything, I think we should not extend the check to UUIDs, but just drop it completely and unconditionally write the (possibly) updated file file to disk. I'll try cooking a patch for that.
I understand why you would consider doing this (low/no maintenance, easier to verify correct results), but I don't think it's a generally good idea to rewrite every config file every time we restart libvirtd. Some media has slow write speed and a limited number of writes in its lifetime (sure, it's not 1982 and we're not talking about a 2716 EPROM, but still :-)) and some people may rely on /etc/libvirt to be in a read-only filesystem for some reason (I haven't tried that, but am just assuming that it would work). Also, the date stamp on a file can provide useful clues about when something got screwed up (although admittedly it is useless in the quest to actually undo the screwup) Except for three cases (that I can think of; I'm sure there are others), what is read from libvirt's config files in /etc is exactly what would be written if we were to re-write it immediately after a read-parse-format: 1) the first read of an xml file that was generated outside libvirt, and has been manually put into /etc/libvirt, e.g. xml files that are a part of a package installation/upgrade. NB: all other cases of externally generated files being placed in /etc/libvirt are expressly forbidden (well, at least "discouraged and declared unsupported") by libvirt documentation. 2) a user manually edits a file in /etc/libvirt - see (1) 3) upgrading libvirt causes something to be parsed differently than it was before. This should never happen, as there should always be enough info in the xml we write to the disk after its initial arrival via the libvirt API that any future read/write cycles are completely idempotent.
I think I'm okay with the former (adding a check for changed uuid onto your patches); it gives us a stable uuid (as long as the modified disk contents are saved for the next reboot) while eliminating another bit of the undesirable code that runs during %post_install.
Is there anything else wrt. pre-built images that we're not taking into consideration though? Honestly I was not considering the pre-built images angle at all, so there's probably *loads* that I'm not taking into account O:-)
Or maybe I'm just paranoid. That is just as likely. (Hmmph. After typing this my brain has figured out that the EPROM-writing card that should be stored away in a box somewhere inside another box on a shelf in my garage is an ISA-bus card, and so I haven't owned any computer it would plug into in at least 11 years. What will I do if I ever need to reprogram the EPROM on my 1980's homebrew computer?? :-( Should my lesson be that I should throw away less old hardware? Or more?)
On Tue, 2020-11-17 at 10:47 -0500, Laine Stump wrote:
On 11/16/20 11:43 AM, Andrea Bolognani wrote:
I don't see how we could possibly not be okay with writing back the updated configuration: filling in whatever blanks are present in the user-provided configuration and then writing the result back to disk is quite central to how we handle domains and other objects, and I see no reason networks should behave any differently.
If anything, I think we should not extend the check to UUIDs, but just drop it completely and unconditionally write the (possibly) updated file file to disk. I'll try cooking a patch for that.
I understand why you would consider doing this (low/no maintenance, easier to verify correct results), but I don't think it's a generally good idea to rewrite every config file every time we restart libvirtd. Some media has slow write speed and a limited number of writes in its lifetime (sure, it's not 1982 and we're not talking about a 2716 EPROM, but still :-)) and some people may rely on /etc/libvirt to be in a read-only filesystem for some reason (I haven't tried that, but am just assuming that it would work).
I would absolutely expect that to *not* be the case, if nothing else because I don't think anyone is actually QA-ing that scenario. Even assuming it does indeed work, you'd end up with a pretty limited version of libvirt, where features like 'virsh define' don't work. Honestly I would be very surprised if anyone actually used libvirt this way.
Except for three cases (that I can think of; I'm sure there are others), what is read from libvirt's config files in /etc is exactly what would be written if we were to re-write it immediately after a read-parse-format:
1) the first read of an xml file that was generated outside libvirt, and has been manually put into /etc/libvirt, e.g. xml files that are a part of a package installation/upgrade. NB: all other cases of externally generated files being placed in /etc/libvirt are expressly forbidden (well, at least "discouraged and declared unsupported") by libvirt documentation.
2) a user manually edits a file in /etc/libvirt - see (1)
Okay, but from libvirt's point of view there's really no way to tell apart a file that's been dropped into /etc/libvirt by the package manager as opposed to the admin, is there? So they will ultimately have to be treated the same way.
3) upgrading libvirt causes something to be parsed differently than it was before. This should never happen, as there should always be enough info in the xml we write to the disk after its initial arrival via the libvirt API that any future read/write cycles are completely idempotent.
Except for randomly-generated data such as, I don't know, MAC addresses or UUIDs :) So, I played with this a bit to understand the state of the art. $ for f in storage/default qemu/networks/default \ qemu/cirros nwfilter/allow-arp; do f="/etc/libvirt/$f.xml" echo "### $f" sudo cat "$f" done ### /etc/libvirt/storage/default.xml <pool type='dir'> <name>default</name> <target> <path>/var/lib/libvirt/images</path> </target> </pool> ### /etc/libvirt/qemu/networks/default.xml <network> <name>default</name> <forward mode='nat'/> <bridge name='virbr0'/> <mac address='52:54:00:42:5a:e7'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> ### /etc/libvirt/qemu/cirros.xml <domain type='kvm'> <name>cirros</name> <memory unit='KiB'>131072</memory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='q35'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'/> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/cirros.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <controller type='usb' model='none'/> <controller type='pci' model='pcie-root'/> <serial type='pty'> <target type='isa-serial'/> </serial> <console type='pty'> <target type='serial'/> </console> <memballoon model='none'/> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> </rng> </devices> </domain> ### /etc/libvirt/nwfilter/allow-arp.xml <filter name='allow-arp' chain='arp'> <rule action='accept' direction='inout'/> </filter> $ for f in storage/default qemu/networks/default \ qemu/cirros nwfilter/allow-arp; do f="/etc/libvirt/$f.xml" sudo md5sum "$f" done be781432b877fa98092bc45610060fff /etc/libvirt/storage/default.xml 1233c7b5bd043a8d209e18abdccbb251 /etc/libvirt/qemu/networks/default.xml 2976a665897bf8ce64a7dbf36af3adbf /etc/libvirt/qemu/cirros.xml 5fdb011f22de738feae3189f5f6b2e91 /etc/libvirt/nwfilter/allow-arp.xml $ sudo systemctl start libvirtd $ sudo systemctl stop libvirtd 2>/dev/null $ for f in storage/default qemu/networks/default \ qemu/cirros nwfilter/allow-arp; do f="/etc/libvirt/$f.xml" sudo md5sum "$f" done be781432b877fa98092bc45610060fff /etc/libvirt/storage/default.xml 1233c7b5bd043a8d209e18abdccbb251 /etc/libvirt/qemu/networks/default.xml 2976a665897bf8ce64a7dbf36af3adbf /etc/libvirt/qemu/cirros.xml 7aff93debdd3e646492d8763d03d6b62 /etc/libvirt/nwfilter/allow-arp.xml Okay, so everything is the same except for the nwfilter. Let's see what changed there: $ sudo cat /etc/libvirt/nwfilter/allow-arp.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh nwfilter-edit allow-arp or other application using the libvirt API. --> <filter name='allow-arp' chain='arp' priority='-500'> <uuid>e4487541-6b88-415c-9aaf-a7f1d3d4263a</uuid> <rule action='accept' direction='inout' priority='500'/> </filter> This behavior is implemented in virNWFilterObjListLoadConfig(): /* We generated a UUID, make it permanent by saving the config to disk */ if (!def->uuid_specified && virNWFilterSaveConfig(configDir, def) < 0) goto error; Looks reasonable enough to me. Of course, since other objects don't have similar snippets in the relevant places, they will get a different UUID every single time the daemon is restarted: $ sudo systemctl stop libvirtd 2>/dev/null $ for i in $(seq 1 3); do sleep 1; sudo systemctl start libvirtd sleep 1; sudo virsh pool-dumpxml --inactive default | grep uuid sudo systemctl stop libvirtd 2>/dev/null done <uuid>2bfe8afb-0366-443a-96cd-b706df4e5221</uuid> <uuid>6804e364-e3a9-494d-8054-93ebe85970a6</uuid> <uuid>6be460c9-8cfb-403c-a791-fdc1d505a0ee</uuid> $ sudo systemctl stop libvirtd 2>/dev/null $ for i in $(seq 1 3); do sleep 1; sudo systemctl start libvirtd sleep 1; sudo virsh net-dumpxml --inactive default | grep uuid sudo systemctl stop libvirtd 2>/dev/null done <uuid>22c61987-43a0-4499-a54d-d1247d36c881</uuid> <uuid>8360afda-4b5c-445d-97f8-abe81d877e7d</uuid> <uuid>821ddd59-c2b0-47c6-9d2f-0708e4e2a9bc</uuid> $ sudo systemctl stop libvirtd 2>/dev/null $ for i in $(seq 1 3); do sleep 1; sudo systemctl start libvirtd sleep 1; sudo virsh dumpxml --inactive cirros | grep uuid sudo systemctl stop libvirtd 2>/dev/null done <uuid>b74581ea-a215-4fe9-8365-c6cfceb2b18e</uuid> <uuid>d00d902b-ebca-4213-9411-d27f14363fdb</uuid> <uuid>74d50bb0-b54a-4a7e-82eb-410ccc12fd9b</uuid> That's not cool, right? The configuration files might not have reached /etc/libvirt the Proper Way™, but we stop so very short of dealing with them reasonably... If writing the file back to disk every single time is a concern due to performance reasons, which I can actually see being the case for large enough hosts, can we at least agree that when UUIDs and MAC addresses have been generated by libvirt then we need to store them persistently? If nothing else, I would argue we definitely need this to be the case for at least nwfilters, networks and storage pools, which are all things for which some configuration can be reasonably provided through the package manager - before the corresponding APIs are even available. -- Andrea Bolognani / Red Hat / Virtualization
On 11/17/20 12:45 PM, Andrea Bolognani wrote:
On Tue, 2020-11-17 at 10:47 -0500, Laine Stump wrote:
On 11/16/20 11:43 AM, Andrea Bolognani wrote:
I don't see how we could possibly not be okay with writing back the updated configuration: filling in whatever blanks are present in the user-provided configuration and then writing the result back to disk is quite central to how we handle domains and other objects, and I see no reason networks should behave any differently.
If anything, I think we should not extend the check to UUIDs, but just drop it completely and unconditionally write the (possibly) updated file file to disk. I'll try cooking a patch for that.
I understand why you would consider doing this (low/no maintenance, easier to verify correct results), but I don't think it's a generally good idea to rewrite every config file every time we restart libvirtd. Some media has slow write speed and a limited number of writes in its lifetime (sure, it's not 1982 and we're not talking about a 2716 EPROM, but still :-)) and some people may rely on /etc/libvirt to be in a read-only filesystem for some reason (I haven't tried that, but am just assuming that it would work).
I would absolutely expect that to *not* be the case, if nothing else because I don't think anyone is actually QA-ing that scenario.
Even assuming it does indeed work, you'd end up with a pretty limited version of libvirt, where features like 'virsh define' don't work.
Honestly I would be very surprised if anyone actually used libvirt this way.
I never presume to know the depths of... er... "uniqueness" that users will plumb. (half /s) But yeah, sure - *I* would never do that, and would never recommend anyone to do that. Hmm, although - maybe a copy-on-write overlay filesystem would be useful, and I recall (very possibly *incorrectly*) about some sort of copy-on-write overlay being used for a "live CD" (maybe even some incarnation of the Fedora Live CD?) that would keep track of all updates to the filesystem in a separate file/partition/whatever that would eventually fill up. (I may be mixing that up with some even-more-ancient memory of a filesystem for a write-once CD-R though - with my brain you just never know what you're gonna get!)
Except for three cases (that I can think of; I'm sure there are others), what is read from libvirt's config files in /etc is exactly what would be written if we were to re-write it immediately after a read-parse-format:
1) the first read of an xml file that was generated outside libvirt, and has been manually put into /etc/libvirt, e.g. xml files that are a part of a package installation/upgrade. NB: all other cases of externally generated files being placed in /etc/libvirt are expressly forbidden (well, at least "discouraged and declared unsupported") by libvirt documentation.
2) a user manually edits a file in /etc/libvirt - see (1)
Okay, but from libvirt's point of view there's really no way to tell apart a file that's been dropped into /etc/libvirt by the package manager as opposed to the admin, is there? So they will ultimately have to be treated the same way.
Yep. That's why the "see (1)". I'd already said "3", and couldn't go back on that (just pretend my up-arrow doesn't work, okay?), but realized while typing that (1) and (2) were essentially the same thing.
3) upgrading libvirt causes something to be parsed differently than it was before. This should never happen, as there should always be enough info in the xml we write to the disk after its initial arrival via the libvirt API that any future read/write cycles are completely idempotent.
Except for randomly-generated data such as, I don't know, MAC addresses or UUIDs :)
Note that I said *should*.
So, I played with this a bit to understand the state of the art.
$ for f in storage/default qemu/networks/default \ qemu/cirros nwfilter/allow-arp; do f="/etc/libvirt/$f.xml" echo "### $f" sudo cat "$f" done ### /etc/libvirt/storage/default.xml <pool type='dir'> <name>default</name> <target> <path>/var/lib/libvirt/images</path> </target> </pool> ### /etc/libvirt/qemu/networks/default.xml <network> <name>default</name> <forward mode='nat'/> <bridge name='virbr0'/> <mac address='52:54:00:42:5a:e7'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> ### /etc/libvirt/qemu/cirros.xml <domain type='kvm'> <name>cirros</name> <memory unit='KiB'>131072</memory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='q35'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'/> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/cirros.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <controller type='usb' model='none'/> <controller type='pci' model='pcie-root'/> <serial type='pty'> <target type='isa-serial'/> </serial> <console type='pty'> <target type='serial'/> </console> <memballoon model='none'/> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> </rng> </devices> </domain> ### /etc/libvirt/nwfilter/allow-arp.xml <filter name='allow-arp' chain='arp'> <rule action='accept' direction='inout'/> </filter>
$ for f in storage/default qemu/networks/default \ qemu/cirros nwfilter/allow-arp; do f="/etc/libvirt/$f.xml" sudo md5sum "$f" done be781432b877fa98092bc45610060fff /etc/libvirt/storage/default.xml 1233c7b5bd043a8d209e18abdccbb251 /etc/libvirt/qemu/networks/default.xml 2976a665897bf8ce64a7dbf36af3adbf /etc/libvirt/qemu/cirros.xml 5fdb011f22de738feae3189f5f6b2e91 /etc/libvirt/nwfilter/allow-arp.xml
$ sudo systemctl start libvirtd $ sudo systemctl stop libvirtd 2>/dev/null
$ for f in storage/default qemu/networks/default \ qemu/cirros nwfilter/allow-arp; do f="/etc/libvirt/$f.xml" sudo md5sum "$f" done be781432b877fa98092bc45610060fff /etc/libvirt/storage/default.xml 1233c7b5bd043a8d209e18abdccbb251 /etc/libvirt/qemu/networks/default.xml 2976a665897bf8ce64a7dbf36af3adbf /etc/libvirt/qemu/cirros.xml 7aff93debdd3e646492d8763d03d6b62 /etc/libvirt/nwfilter/allow-arp.xml
Okay, so everything is the same except for the nwfilter. Let's see what changed there:
$ sudo cat /etc/libvirt/nwfilter/allow-arp.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh nwfilter-edit allow-arp or other application using the libvirt API. -->
<filter name='allow-arp' chain='arp' priority='-500'> <uuid>e4487541-6b88-415c-9aaf-a7f1d3d4263a</uuid> <rule action='accept' direction='inout' priority='500'/> </filter>
This behavior is implemented in virNWFilterObjListLoadConfig():
/* We generated a UUID, make it permanent by saving the config to disk */ if (!def->uuid_specified && virNWFilterSaveConfig(configDir, def) < 0) goto error;
Looks reasonable enough to me.
Of course, since other objects don't have similar snippets in the relevant places, they will get a different UUID every single time the daemon is restarted:
$ sudo systemctl stop libvirtd 2>/dev/null $ for i in $(seq 1 3); do sleep 1; sudo systemctl start libvirtd sleep 1; sudo virsh pool-dumpxml --inactive default | grep uuid sudo systemctl stop libvirtd 2>/dev/null done <uuid>2bfe8afb-0366-443a-96cd-b706df4e5221</uuid> <uuid>6804e364-e3a9-494d-8054-93ebe85970a6</uuid> <uuid>6be460c9-8cfb-403c-a791-fdc1d505a0ee</uuid>
$ sudo systemctl stop libvirtd 2>/dev/null $ for i in $(seq 1 3); do sleep 1; sudo systemctl start libvirtd sleep 1; sudo virsh net-dumpxml --inactive default | grep uuid sudo systemctl stop libvirtd 2>/dev/null done <uuid>22c61987-43a0-4499-a54d-d1247d36c881</uuid> <uuid>8360afda-4b5c-445d-97f8-abe81d877e7d</uuid> <uuid>821ddd59-c2b0-47c6-9d2f-0708e4e2a9bc</uuid>
$ sudo systemctl stop libvirtd 2>/dev/null $ for i in $(seq 1 3); do sleep 1; sudo systemctl start libvirtd sleep 1; sudo virsh dumpxml --inactive cirros | grep uuid sudo systemctl stop libvirtd 2>/dev/null done <uuid>b74581ea-a215-4fe9-8365-c6cfceb2b18e</uuid> <uuid>d00d902b-ebca-4213-9411-d27f14363fdb</uuid> <uuid>74d50bb0-b54a-4a7e-82eb-410ccc12fd9b</uuid>
That's not cool, right?
The fact that you took the time to do such a nicely documented experiment *is* cool. The results are not, though.
The configuration files might not have reached /etc/libvirt the Proper Way™, but we stop so very short of dealing with them reasonably...
The most bothersome part of this to me is that the treatment is inconsistent. If we're going to re-write the config when parsing causes a change, then we should do that for all types of config, and all changes. And that goes double for changes to things that are supposed to remain stable (which, I guess, is "everything").
If writing the file back to disk every single time is a concern due to performance reasons, which I can actually see being the case for large enough hosts, can we at least agree that when UUIDs and MAC addresses have been generated by libvirt then we need to store them persistently?
Yep. I'm right there with you on this - I was only objecting to the idea of always re-writing no matter what.
If nothing else, I would argue we definitely need this to be the case for at least nwfilters, networks and storage pools, which are all things for which some configuration can be reasonably provided through the package manager - before the corresponding APIs are even available.
I think it should be done consistently across all config objects. I would suggest making some sort of pseudo-standardized function/table for every config object type that would maintain a list of those items that need to be checked, but that just seems like over-engineering that would lead to lots of code that not everybody would know about and/or remember, and so it wouldn't be maintained anyway...
We're no longer generating a UUID during installation, so we clearly don't need to strip it afterwards; and since the network driver is perfectly capable of generating a UUID if necessary, we don't need to do that at %post time either. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- libvirt.spec.in | 4 ---- 1 file changed, 4 deletions(-) diff --git a/libvirt.spec.in b/libvirt.spec.in index 034982809d..f4cee8940d 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -1240,8 +1240,6 @@ cp -a $RPM_BUILD_ROOT%{_sysconfdir}/libvirt/nwfilter/*.xml \ # libvirt saves these files with mode 600 chmod 600 $RPM_BUILD_ROOT%{_sysconfdir}/libvirt/nwfilter/*.xml -# Strip auto-generated UUID - we need it generated per-install -sed -i -e "/<uuid>/d" $RPM_BUILD_ROOT%{_datadir}/libvirt/networks/default.xml %if ! %{with_qemu} rm -f $RPM_BUILD_ROOT%{_datadir}/augeas/lenses/libvirtd_qemu.aug rm -f $RPM_BUILD_ROOT%{_datadir}/augeas/lenses/tests/test_libvirtd_qemu.aug @@ -1422,9 +1420,7 @@ if test $1 -eq 1 && test ! -f %{_sysconfdir}/libvirt/qemu/networks/default.xml ; ;; esac - UUID=`/usr/bin/uuidgen` sed -e "s/${orig_sub}/${sub}/g" \ - -e "s,</name>,</name>\n <uuid>$UUID</uuid>," \ < %{_datadir}/libvirt/networks/default.xml \ > %{_sysconfdir}/libvirt/qemu/networks/default.xml ln -s ../default.xml %{_sysconfdir}/libvirt/qemu/networks/autostart/default.xml -- 2.26.2
On Sun, Nov 15, 2020 at 3:43 PM Andrea Bolognani <abologna@redhat.com> wrote:
We're no longer generating a UUID during installation, so we clearly don't need to strip it afterwards; and since the network driver is perfectly capable of generating a UUID if necessary, we don't need to do that at %post time either.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- libvirt.spec.in | 4 ---- 1 file changed, 4 deletions(-)
diff --git a/libvirt.spec.in b/libvirt.spec.in index 034982809d..f4cee8940d 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -1240,8 +1240,6 @@ cp -a $RPM_BUILD_ROOT%{_sysconfdir}/libvirt/nwfilter/*.xml \ # libvirt saves these files with mode 600 chmod 600 $RPM_BUILD_ROOT%{_sysconfdir}/libvirt/nwfilter/*.xml
-# Strip auto-generated UUID - we need it generated per-install -sed -i -e "/<uuid>/d" $RPM_BUILD_ROOT%{_datadir}/libvirt/networks/default.xml %if ! %{with_qemu} rm -f $RPM_BUILD_ROOT%{_datadir}/augeas/lenses/libvirtd_qemu.aug rm -f $RPM_BUILD_ROOT%{_datadir}/augeas/lenses/tests/test_libvirtd_qemu.aug @@ -1422,9 +1420,7 @@ if test $1 -eq 1 && test ! -f %{_sysconfdir}/libvirt/qemu/networks/default.xml ; ;; esac
- UUID=`/usr/bin/uuidgen` sed -e "s/${orig_sub}/${sub}/g" \ - -e "s,</name>,</name>\n <uuid>$UUID</uuid>," \ < %{_datadir}/libvirt/networks/default.xml \ > %{_sysconfdir}/libvirt/qemu/networks/default.xml ln -s ../default.xml %{_sysconfdir}/libvirt/qemu/networks/autostart/default.xml -- 2.26.2
Looks good to me, though when did we make this change? Regardless... Reviewed-by: Neal Gompa <ngompa13@gmail.com> -- 真実はいつも一つ！/ Always, there's only one truth!
On Tue, 2020-11-17 at 13:01 -0500, Neal Gompa wrote:
On Sun, Nov 15, 2020 at 3:43 PM Andrea Bolognani <abologna@redhat.com> wrote:
- UUID=`/usr/bin/uuidgen` sed -e "s/${orig_sub}/${sub}/g" \ - -e "s,</name>,</name>\n <uuid>$UUID</uuid>," \ < %{_datadir}/libvirt/networks/default.xml \ > %{_sysconfdir}/libvirt/qemu/networks/default.xml ln -s ../default.xml %{_sysconfdir}/libvirt/qemu/networks/autostart/default.xml
Looks good to me, though when did we make this change?
In the first patch of the series ;) -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Nov 17, 2020 at 2:29 PM Andrea Bolognani <abologna@redhat.com> wrote:
On Tue, 2020-11-17 at 13:01 -0500, Neal Gompa wrote:
On Sun, Nov 15, 2020 at 3:43 PM Andrea Bolognani <abologna@redhat.com> wrote:
- UUID=`/usr/bin/uuidgen` sed -e "s/${orig_sub}/${sub}/g" \ - -e "s,</name>,</name>\n <uuid>$UUID</uuid>," \ < %{_datadir}/libvirt/networks/default.xml \ > %{_sysconfdir}/libvirt/qemu/networks/default.xml ln -s ../default.xml %{_sysconfdir}/libvirt/qemu/networks/autostart/default.xml
Looks good to me, though when did we make this change?
In the first patch of the series ;)
D'oh! This is why email patch review is hit-or-miss... :/ -- 真実はいつも一つ！/ Always, there's only one truth!
participants (4)
-
Andrea Bolognani -
Daniel P. Berrangé -
Laine Stump -
Neal Gompa