Investigating MAC Address Conflict Resolution in libvirt: Log Analysis and Code Location Inquiry
Dear Team, I am reaching out regarding an issue I encountered with libvirt and MAC address conflicts. Below is a summary of the situation: 1. Initially, the vNIC's MAC address was different from the target VM's MAC address. 2. After modifying the vNIC's MAC address to match the VM's MAC address, the network was interrupted. 3. After rebooting the VM, the vNIC's MAC address was automatically modified again. I have observed the following kernel logs during this process: Dec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 16:59:40 zstack-manager kernel: device vnic43.0 left promiscuous modeDec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: device vnic43.0 entered promiscuous modeDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered forwarding state I am looking to understand the underlying code that handles the automatic modification of the vNIC's MAC address after the conflict and how the network interruption occurs. Can you help direct me to the relevant code segment or provide any insights into this behavior? Thank you for your assistance. Best regards,
On Tue, Dec 24, 2024 at 05:26:29PM +0800, Xuda Zhang wrote:
Dear Team,
Hi, not sure if this is still relevant, but ...
I am reaching out regarding an issue I encountered with libvirt and MAC address conflicts. Below is a summary of the situation:
1. Initially, the vNIC's MAC address was different from the target VM's MAC address.
you are talking about a vNIC from the host's point of view and "target VM" as seen from the guest? I'm just trying to make sure I understand.
2. After modifying the vNIC's MAC address to match the VM's MAC address, the network was interrupted. 3. After rebooting the VM, the vNIC's MAC address was automatically modified again.
Are you using some filtering (nwfilter) on the libvirt network?
I have observed the following kernel logs during this process:
Dec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 16:59:40 zstack-manager kernel: device vnic43.0 left promiscuous modeDec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: device vnic43.0 entered promiscuous modeDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered forwarding state
I am looking to understand the underlying code that handles the automatic modification of the vNIC's MAC address after the conflict and how the network interruption occurs. Can you help direct me to the relevant code segment or provide any insights into this behavior?
What conflict are you talking precisely? Is something having the same MAC address as the VM?
Thank you for your assistance.
Best regards,
(Somehow I never received the original of this message into my libvirt folder. Possibly my email client mistakenly decided it was spam...) On 2/3/25 8:36 AM, Martin Kletzander wrote:
On Tue, Dec 24, 2024 at 05:26:29PM +0800, Xuda Zhang wrote:
Dear Team,
Hi, not sure if this is still relevant, but ...
I am reaching out regarding an issue I encountered with libvirt and MAC address conflicts. Below is a summary of the situation:
1. Initially, the vNIC's MAC address was different from the target VM's MAC address.
you are talking about a vNIC from the host's point of view and "target VM" as seen from the guest? I'm just trying to make sure I understand.
To further clarify - when you say "vNIC" do you mean the tap device that is connected to the bridge on the host side, and "target VM's MAC address" is the MAC address of the network device inside the guest (VM)? Assuming that is the case, the explanation is this: The tap device on the host *must* have a different MAC address than the device in the guest. If the two MAC addresses are the same, then when the host's network stack see a packet destined for that MAC address, it will think to itself "Hey - that's the MAC address of an interface on *this* machine*, so I don't need to forward it anywhere!" and then attempt to deliver the packet locally (i.e. send it up to IP on the host). You would of course see this as "guest networking doesn't work". The reason that you see the tap device MAC address go back to "normal" when you stop and restart the guest is because any time you stop/destroy a guest, any tap device(s) associated with that guest will be destroyed, and the next time you start the guest, *new* tap devices will be created. And whenever libvirt creates a tap device, it automatically makes a MAC address that is just "guest interface MAC address, except replace the first byte with 0xFE". This is done to guarantee that the tap device and guest interface have different MAC addresses, but they are similar to help make it more obvious which tap device is used by which guest interface (because the MACs are *almost* the same). So everything that you see is normal and, for the most part, necessary. I'm surprised that you care about the MAC address of the tap device. It really doesn't matter what it is except that it must be different from the MAC in the guest.
2. After modifying the vNIC's MAC address to match the VM's MAC address, the network was interrupted. 3. After rebooting the VM, the vNIC's MAC address was automatically modified again.
Are you using some filtering (nwfilter) on the libvirt network?
I have observed the following kernel logs during this process:
Dec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 16:59:40 zstack-manager kernel: device vnic43.0 left promiscuous modeDec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: device vnic43.0 entered promiscuous modeDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered forwarding state
I am looking to understand the underlying code that handles the automatic modification of the vNIC's MAC address after the conflict and how the network interruption occurs. Can you help direct me to the relevant code segment or provide any insights into this behavior?
See above for the explanation of why the MAC address of the tap is changed, and why networking stops if you change it to match the guest MAC. The code that sets the MAC address is in virNetDevTapCreateInBridgePort(), src/util/virnetdevtap.c:638 in current upstream sources - this is done unconditionally every time a tap device is created.
What conflict are you talking precisely? Is something having the same MAC address as the VM?
Thank you for your assistance.
Best regards,
Dear [Libvirt Developer Team], It has been quite some time since I initially sent my email, and I never expected to receive a response after so long. I sincerely appreciate your time and effort in addressing this case — thank you! Regarding your question:
To further clarify - when you say "vNIC" do you mean the tap device that is connected to the bridge on the host side, and "target VM's MAC address" is the MAC address of the network device inside the guest (VM)?
Yes, that is precisely what I meant! Now, returning to the issue at hand: the distinctiveness of MAC addresses is a fundamental principle of network communication. This issue was originally raised by a friend of mine. While I was able to explain the theoretical importance of unique MAC addresses, I couldn't provide a clear explanation of how libvirt handles the "automatic modification" of the tap device's MAC address, as I had only observed changes in the device's state but not the modification itself in the logs. After carefully reviewing your response multiple times and conducting additional tests, I now fully understand the mechanism behind the "0xFE" modification of the first byte of the MAC address, which is handled in virNetDevTapCreateInBridgePort() at line 638 (starting at line 625). However, I wanted to confirm one specific point regarding tap device behavior. According to the official documentation <https://libvirt.org/formatdomain.html#generic-ethernet-connection>:
"If no target dev is specified, libvirt will create a new standard tap device with a name of the pattern 'vnetN', where 'N' is replaced with a number. If a target dev is specified and that device doesn't exist, then a new standard tap device will be created with the exact dev name given."
In contrast, you mentioned:
"Any time you stop/destroy a guest, any tap device(s) associated with that guest will be destroyed, and the next time you start the guest, new tap devices will be created."
From this, it seems that if the tap device already exists, the MAC address modification would be a direct "modification" rather than a "recreation" of the device. Looking at the code section: --- /* We need to set the interface MAC before adding it * to the bridge, because the bridge assumes the lowest * MAC of all enslaved interfaces & we don't want it * seeing the kernel allocate random MAC for the TAP * device before we set our static MAC. */ virMacAddrSet(&tapmac, macaddr); if (!(flags & VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE)) { /* The tap device's MAC address cannot match the MAC address * used by the guest. This results in "received packet on * vnetX with own address as source address" error logs from * the kernel. Making the tap address as high as possible * discourages the bridge from using this tap's MAC as its own * (a Linux host bridge will take on the lowest numbered MAC * of all devices attached to it). */ if (tapmac.addr[0] == 0xFE) tapmac.addr[0] = 0xFA; else tapmac.addr[0] = 0xFE; } if (virNetDevSetMAC(*ifname, &tapmac) < 0) goto error; --- Could you help confirm the exact behavior in this case? Specifically: 1. If a target tap device already exists, does libvirt modify the MAC address instead of recreating the device? 2. Under what circumstances does libvirt destroy and recreate the tap device instead of modifying its attributes? Looking forward to your insights! Best regards, Xuda Zhang Laine Stump <laine@laine.org> 于2025年2月4日周二 02:34写道：
(Somehow I never received the original of this message into my libvirt folder. Possibly my email client mistakenly decided it was spam...)
On 2/3/25 8:36 AM, Martin Kletzander wrote:
On Tue, Dec 24, 2024 at 05:26:29PM +0800, Xuda Zhang wrote:
Dear Team,
Hi, not sure if this is still relevant, but ...
I am reaching out regarding an issue I encountered with libvirt and MAC address conflicts. Below is a summary of the situation:
1. Initially, the vNIC's MAC address was different from the target VM's MAC address.
you are talking about a vNIC from the host's point of view and "target VM" as seen from the guest? I'm just trying to make sure I understand.
To further clarify - when you say "vNIC" do you mean the tap device that is connected to the bridge on the host side, and "target VM's MAC address" is the MAC address of the network device inside the guest (VM)?
Assuming that is the case, the explanation is this: The tap device on the host *must* have a different MAC address than the device in the guest.
If the two MAC addresses are the same, then when the host's network stack see a packet destined for that MAC address, it will think to itself "Hey - that's the MAC address of an interface on *this* machine*, so I don't need to forward it anywhere!" and then attempt to deliver the packet locally (i.e. send it up to IP on the host). You would of course see this as "guest networking doesn't work".
The reason that you see the tap device MAC address go back to "normal" when you stop and restart the guest is because any time you stop/destroy a guest, any tap device(s) associated with that guest will be destroyed, and the next time you start the guest, *new* tap devices will be created. And whenever libvirt creates a tap device, it automatically makes a MAC address that is just "guest interface MAC address, except replace the first byte with 0xFE". This is done to guarantee that the tap device and guest interface have different MAC addresses, but they are similar to help make it more obvious which tap device is used by which guest interface (because the MACs are *almost* the same).
So everything that you see is normal and, for the most part, necessary.
I'm surprised that you care about the MAC address of the tap device. It really doesn't matter what it is except that it must be different from the MAC in the guest.
2. After modifying the vNIC's MAC address to match the VM's MAC address, the network was interrupted. 3. After rebooting the VM, the vNIC's MAC address was automatically modified again.
Are you using some filtering (nwfilter) on the libvirt network?
I have observed the following kernel logs during this process:
Dec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 16:59:40 zstack-manager kernel: device vnic43.0 left promiscuous modeDec 24 16:59:40 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered disabled stateDec 24 17:00:11 zstack-manager kernel: device vnic43.0 entered promiscuous modeDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered blocking stateDec 24 17:00:11 zstack-manager kernel: br_enp2s0: port 14(vnic43.0) entered forwarding state
I am looking to understand the underlying code that handles the
automatic
modification of the vNIC's MAC address after the conflict and how the network interruption occurs. Can you help direct me to the relevant code segment or provide any insights into this behavior?
See above for the explanation of why the MAC address of the tap is changed, and why networking stops if you change it to match the guest MAC. The code that sets the MAC address is in virNetDevTapCreateInBridgePort(), src/util/virnetdevtap.c:638 in current upstream sources - this is done unconditionally every time a tap device is created.
What conflict are you talking precisely? Is something having the same MAC address as the VM?
Thank you for your assistance.
Best regards,
On 2/4/25 11:04 PM, Xuda Zhang wrote:
Could you help confirm the exact behavior in this case? Specifically:
1. If a target tap device already exists, does libvirt modify the MAC address instead of recreating the device?
Sorry, I needed to qualify that detail a bit (and actually I probably shouldn't have even brought it up, since it doesn't apply to tap devices used to connect to a bridge device) - *only for "<interface type='ethernet' managed='no'>" * an already-existing tap device can be specified in the XML (with <target dev='blah'/>). type='ethernet' is used for tap devices that aren't connected to any bridge (all communication with the guest must then be *routed* by the host at the IP level, rather than being bridged). And a detail that I misremembered - when libvirt uses a pre-existing tap device, it assumes that the creator of the tap already set the MAC appropriately, so it doesn't modify it. But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
2. Under what circumstances does libvirt destroy and recreate the tap device instead of modifying its attributes?
Another detail that I've forgotten over the long time since I last looked at this code. libvirt doesn't explicitly delete the tap device, it just closes the device. In the case of tap devices that libvirt itself created, they are automatically deleted when they are closed. In the case of pre-existing tap devices (which again doesn't apply in your use case), 1) libvirt assumes that the creator of the pre-existing device has already set the MAC address to something appropriate, so it doesn't attempt to change it, and 2) again libvirt won't explicitly delete the tap device. If it was created as a persistent device, then closing it doesn't cause it to be auto-deleted, but if the original creator of the tap device didn't create it as persistent, and no other process has an open handle for the device, then again closing the device will auto-delete it. But again, for your use case (where the tap is connected to a bridge) the creator of the tap device is always libvirt, and so it will always be auto-deleted when libvirt closes the final handle it has open on the device.
Looking forward to your insights!
Best regards, Xuda Zhang
Hi Laine, You mentioned:
But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
However, the documentation does not *explicitly *state that already-existing tap devices *cannot *be used for interface type='bridge' or interface type='network'. I also conducted another test, which confirmed that whenever the VM is shut down, the existing tap device is deleted. This suggests that the correct understanding is *recreation *rather than modification. Can I conclude that, in a bridge network type, libvirt is expected to create and manage the tap device automatically? This would mean that the tap device is always created by libvirt, its lifecycle is tied to the VM (guest OS), and it is deleted when the VM stops. Additionally, each time the VM starts, a new tap device is created with slight variations, likely derived from the VM's MAC address. Would you agree with this assessment? Best regards, Xuda Zhang Laine Stump <laine@laine.org> 于2025年2月5日周三 14:03写道：
On 2/4/25 11:04 PM, Xuda Zhang wrote:
Could you help confirm the exact behavior in this case? Specifically:
1. If a target tap device already exists, does libvirt modify the MAC address instead of recreating the device?
Sorry, I needed to qualify that detail a bit (and actually I probably shouldn't have even brought it up, since it doesn't apply to tap devices used to connect to a bridge device) - *only for "<interface type='ethernet' managed='no'>" * an already-existing tap device can be specified in the XML (with <target dev='blah'/>). type='ethernet' is used for tap devices that aren't connected to any bridge (all communication with the guest must then be *routed* by the host at the IP level, rather than being bridged).
And a detail that I misremembered - when libvirt uses a pre-existing tap device, it assumes that the creator of the tap already set the MAC appropriately, so it doesn't modify it.
But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
2. Under what circumstances does libvirt destroy and recreate the tap device instead of modifying its attributes?
Another detail that I've forgotten over the long time since I last looked at this code. libvirt doesn't explicitly delete the tap device, it just closes the device. In the case of tap devices that libvirt itself created, they are automatically deleted when they are closed.
In the case of pre-existing tap devices (which again doesn't apply in your use case), 1) libvirt assumes that the creator of the pre-existing device has already set the MAC address to something appropriate, so it doesn't attempt to change it, and 2) again libvirt won't explicitly delete the tap device. If it was created as a persistent device, then closing it doesn't cause it to be auto-deleted, but if the original creator of the tap device didn't create it as persistent, and no other process has an open handle for the device, then again closing the device will auto-delete it.
But again, for your use case (where the tap is connected to a bridge) the creator of the tap device is always libvirt, and so it will always be auto-deleted when libvirt closes the final handle it has open on the device.
Looking forward to your insights!
Best regards, Xuda Zhang
On 2/5/25 1:39 AM, Xuda Zhang wrote:
Hi Laine,
You mentioned:
But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
However, the documentation does not *explicitly *state that already- existing tap devices *cannot *be used for interface type='bridge' or interface type='network'.
Any reasonably large piece of software has many many behaviors that aren't explicitly stated.
I also conducted another test, which confirmed that whenever the VM is shut down, the existing tap device is deleted. This suggests that the correct understanding is *recreation *rather than modification.
That's basically what I told you. It's unfortunate that I ever brought up the idea of pre-existing tap devices, since they are only allowed in one very narrow use case.
Can I conclude that, in a bridge network type, libvirt is expected to create and manage the tap device automatically? This would mean that the tap device is always created by libvirt, its lifecycle is tied to the VM (guest OS), and it is deleted when the VM stops.
Correct.
Additionally, each time the VM starts, a new tap device is created with slight variations, likely derived from the VM's MAC address.
I guess you mean "... with a MAC address that is the same as the guest interface's MAC except the first byte is changed to 0xFE." If so then yes.
Would you agree with this assessment?
Yes.
Best regards,
Xuda Zhang
Laine Stump <laine@laine.org <mailto:laine@laine.org>> 于2025年2月5日周 三 14:03写道：
On 2/4/25 11:04 PM, Xuda Zhang wrote:
> Could you help confirm the exact behavior in this case? Specifically: > > 1. If a target tap device already exists, does libvirt modify the MAC > address instead of recreating the device?
Sorry, I needed to qualify that detail a bit (and actually I probably shouldn't have even brought it up, since it doesn't apply to tap devices used to connect to a bridge device) - *only for "<interface type='ethernet' managed='no'>" * an already-existing tap device can be specified in the XML (with <target dev='blah'/>). type='ethernet' is used for tap devices that aren't connected to any bridge (all communication with the guest must then be *routed* by the host at the IP level, rather than being bridged).
And a detail that I misremembered - when libvirt uses a pre-existing tap device, it assumes that the creator of the tap already set the MAC appropriately, so it doesn't modify it.
But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
> 2. Under what circumstances does libvirt destroy and recreate the tap > device instead of modifying its attributes?
Another detail that I've forgotten over the long time since I last looked at this code. libvirt doesn't explicitly delete the tap device, it just closes the device. In the case of tap devices that libvirt itself created, they are automatically deleted when they are closed.
In the case of pre-existing tap devices (which again doesn't apply in your use case), 1) libvirt assumes that the creator of the pre-existing device has already set the MAC address to something appropriate, so it doesn't attempt to change it, and 2) again libvirt won't explicitly delete the tap device. If it was created as a persistent device, then closing it doesn't cause it to be auto-deleted, but if the original creator of the tap device didn't create it as persistent, and no other process has an open handle for the device, then again closing the device will auto-delete it.
But again, for your use case (where the tap is connected to a bridge) the creator of the tap device is always libvirt, and so it will always be auto-deleted when libvirt closes the final handle it has open on the device.
> Looking forward to your insights! > > Best regards, > Xuda Zhang
Understood completely. I appreciate your patience. Best Regards, Xuda Zhang Laine Stump <laine@redhat.com> 于2025年2月5日周三 21:41写道：
On 2/5/25 1:39 AM, Xuda Zhang wrote:
Hi Laine,
You mentioned:
But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
However, the documentation does not *explicitly *state that already- existing tap devices *cannot *be used for interface type='bridge' or interface type='network'.
Any reasonably large piece of software has many many behaviors that aren't explicitly stated.
I also conducted another test, which confirmed that whenever the VM is shut down, the existing tap device is deleted. This suggests that the correct understanding is *recreation *rather than modification.
That's basically what I told you. It's unfortunate that I ever brought up the idea of pre-existing tap devices, since they are only allowed in one very narrow use case.
Can I conclude that, in a bridge network type, libvirt is expected to create and manage the tap device automatically? This would mean that the tap device is always created by libvirt, its lifecycle is tied to the VM (guest OS), and it is deleted when the VM stops.
Correct.
Additionally, each time the VM starts, a new tap device is created with slight variations, likely derived from the VM's MAC address.
I guess you mean "... with a MAC address that is the same as the guest interface's MAC except the first byte is changed to 0xFE." If so then yes.
Would you agree with this assessment?
Yes.
Best regards,
Xuda Zhang
Laine Stump <laine@laine.org <mailto:laine@laine.org>> 于2025年2月5日周 三 14:03写道：
On 2/4/25 11:04 PM, Xuda Zhang wrote:
> Could you help confirm the exact behavior in this case? Specifically: > > 1. If a target tap device already exists, does libvirt modify the MAC > address instead of recreating the device?
Sorry, I needed to qualify that detail a bit (and actually I probably shouldn't have even brought it up, since it doesn't apply to tap devices used to connect to a bridge device) - *only for "<interface type='ethernet' managed='no'>" * an already-existing tap device can be specified in the XML (with <target dev='blah'/>). type='ethernet' is used for tap devices that aren't connected to any bridge (all communication with the guest must then be *routed* by the host at the IP level, rather than being bridged).
And a detail that I misremembered - when libvirt uses a pre-existing tap device, it assumes that the creator of the tap already set the MAC appropriately, so it doesn't modify it.
But again, already-existing tap devices can't be used for interface type='bridge' or type='network' (which also connects the tap to a bridge).
> 2. Under what circumstances does libvirt destroy and recreate the tap > device instead of modifying its attributes?
Another detail that I've forgotten over the long time since I last looked at this code. libvirt doesn't explicitly delete the tap device, it just closes the device. In the case of tap devices that libvirt itself created, they are automatically deleted when they are closed.
In the case of pre-existing tap devices (which again doesn't apply in your use case), 1) libvirt assumes that the creator of the pre-existing device has already set the MAC address to something appropriate, so it doesn't attempt to change it, and 2) again libvirt won't explicitly delete the tap device. If it was created as a persistent device, then closing it doesn't cause it to be auto-deleted, but if the original creator of the tap device didn't create it as persistent, and no other process has an open handle for the device, then again closing the device will auto-delete it.
But again, for your use case (where the tap is connected to a bridge) the creator of the tap device is always libvirt, and so it will always be auto-deleted when libvirt closes the final handle it has open on the device.
> Looking forward to your insights! > > Best regards, > Xuda Zhang
participants (4)
-
Laine Stump -
Laine Stump -
Martin Kletzander -
Xuda Zhang