Public IP on virtual machine network issue
I have been struggling with this for weeks and I was unable to find an answer on line. Perhaps someone here can help me. Oracle linux 8 running virtualization: hardware node has a public IP address on interface bridge0 and physical eno1 is a member of the bridge0 a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node. I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of. When I try tracepath or traceroute from outside to virtual I get !H on last hup same result when I try to do the same form hardware node to virtual I get !H Also, when I telnet (TCP) to a specific port on virtual where I have a daemon LISTENING OR NOT I get: No route to host. Same experiment works just fine for ssh port. Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to virtual dropping connections from all other sources This issue presented it self when I attempted to setup a galera node on virtual and ports 4567 is responding but 4568 and 4444 are not, but the daemons are running and I can clearly see lsoft showing "LISTENING" I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware of? What am I missing? -- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net
Can you post the output of iptables -L? By default, the bridge module in the kernel sends packets traversing the bridge to iptables (in the FORWARD chain I believe) for processing. So if you have configured a DENY policy on the FORWARD chain, or are otherwise filtering in the forward chain, you'll be affecting packets traversing the bridge. Check out this page for details on how to change this behavior: https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf Tom On Sun, Feb 13, 2022 at 4:08 PM Marcin Groszek <marcin@voipplus.net> wrote:
I have been struggling with this for weeks and I was unable to find an answer on line. Perhaps someone here can help me.
Oracle linux 8 running virtualization:
hardware node has a public IP address on interface bridge0 and physical eno1 is a member of the bridge0
a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node.
I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of.
When I try tracepath or traceroute from outside to virtual I get !H on last hup
same result when I try to do the same form hardware node to virtual I get !H
Also, when I telnet (TCP) to a specific port on virtual where I have a daemon LISTENING OR NOT I get: No route to host. Same experiment works just fine for ssh port.
Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to virtual dropping connections from all other sources
This issue presented it self when I attempted to setup a galera node on virtual and ports 4567 is responding but 4568 and 4444 are not, but the daemons are running and I can clearly see lsoft showing "LISTENING"
I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware of?
What am I missing?
-- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net
-- ----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon@gmail.com -----------------------------------------------------------------------------
Hi, as Tom says, check iptables forward rules. Also, you can check host sysctl ipv4/6 global and per interface rules to double check bridge forward capabilities. Finally, check your routes on guest vm, especially the default gw, sometimes you can receive the packet and the answer is sent through the wrong interface because of bad routes. Best Regards. Daniel Romero P. On Sun, Feb 13, 2022 at 7:39 PM Tom Ammon <thomasammon@gmail.com> wrote:
Can you post the output of iptables -L?
By default, the bridge module in the kernel sends packets traversing the bridge to iptables (in the FORWARD chain I believe) for processing. So if you have configured a DENY policy on the FORWARD chain, or are otherwise filtering in the forward chain, you'll be affecting packets traversing the bridge. Check out this page for details on how to change this behavior: https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf
Tom
On Sun, Feb 13, 2022 at 4:08 PM Marcin Groszek <marcin@voipplus.net> wrote:
I have been struggling with this for weeks and I was unable to find an answer on line. Perhaps someone here can help me.
Oracle linux 8 running virtualization:
hardware node has a public IP address on interface bridge0 and physical eno1 is a member of the bridge0
a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node.
I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of.
When I try tracepath or traceroute from outside to virtual I get !H on last hup
same result when I try to do the same form hardware node to virtual I get !H
Also, when I telnet (TCP) to a specific port on virtual where I have a daemon LISTENING OR NOT I get: No route to host. Same experiment works just fine for ssh port.
Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to virtual dropping connections from all other sources
This issue presented it self when I attempted to setup a galera node on virtual and ports 4567 is responding but 4568 and 4444 are not, but the daemons are running and I can clearly see lsoft showing "LISTENING"
I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware of?
What am I missing?
-- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net
--
----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon@gmail.com
-----------------------------------------------------------------------------
The issue has been resolved I had a firewald running on virtual host. Thank you for the replay. On 2/13/2022 9:17 PM, Daniel Romero wrote:
Hi,
as Tom says, check iptables forward rules. Also, you can check host sysctl ipv4/6 global and per interface rules to double check bridge forward capabilities. Finally, check your routes on guest vm, especially the default gw, sometimes you can receive the packet and the answer is sent through the wrong interface because of bad routes.
Best Regards. Daniel Romero P.
On Sun, Feb 13, 2022 at 7:39 PM Tom Ammon <thomasammon@gmail.com <mailto:thomasammon@gmail.com>> wrote:
Can you post the output of iptables -L?
By default, the bridge module in the kernel sends packets traversing the bridge to iptables (in the FORWARD chain I believe) for processing. So if you have configured a DENY policy on the FORWARD chain, or are otherwise filtering in the forward chain, you'll be affecting packets traversing the bridge. Check out this page for details on how to change this behavior: https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf
Tom
On Sun, Feb 13, 2022 at 4:08 PM Marcin Groszek <marcin@voipplus.net <mailto:marcin@voipplus.net>> wrote:
I have been struggling with this for weeks and I was unable to find an answer on line. Perhaps someone here can help me.
Oracle linux 8 running virtualization:
hardware node has a public IP address on interface bridge0 and physical eno1 is a member of the bridge0
a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node.
I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of.
When I try tracepath or traceroute from outside to virtual I get !H on last hup
same result when I try to do the same form hardware node to virtual I get !H
Also, when I telnet (TCP) to a specific port on virtual where I have a daemon LISTENING OR NOT I get: No route to host. Same experiment works just fine for ssh port.
Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to virtual dropping connections from all other sources
This issue presented it self when I attempted to setup a galera node on virtual and ports 4567 is responding but 4568 and 4444 are not, but the daemons are running and I can clearly see lsoft showing "LISTENING"
I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware of?
What am I missing?
-- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net
-- ----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon@gmail.com <mailto:thomasammon@gmail.com> -----------------------------------------------------------------------------
-- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net
On 2/13/22 5:38 PM, Tom Ammon wrote:
Can you post the output of iptables -L?
By default, the bridge module in the kernel sends packets traversing the bridge to iptables (in the FORWARD chain I believe) for processing. So if you have configured a DENY policy on the FORWARD chain, or are otherwise filtering in the forward chain, you'll be affecting packets traversing the bridge. Check out this page for details on how to change this behavior: https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf <https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf>
That information is *very* out of date; the situation has changed quite a lot since that was written in 2014. Filtering of packets traversing a bridge device are now only filtered if the br_netfilter module is loaded, which isn't done by default. It *is* autoloaded if certain types of iptables rules are added(I can't remember the details of the type of rule though - there was a bug in iptables a year or so ago where autoload of br_netfilter was triggered by libvirt attempting to *remove* a rule of whatever type it was). Anyway, unless "lsmod | grep br_netfilter" shows that you have br_netfilter loaded, this entire path is a red herring (if you do have it loaded, unload it, and try to figure out why it was loaded). (Interestingly, this is the 2nd time this particular outdated page has come up in the last week. Has something else broken somewhere that's causing people to search out this page?)
Tom
On Sun, Feb 13, 2022 at 4:08 PM Marcin Groszek <marcin@voipplus.net <mailto:marcin@voipplus.net>> wrote:
I have been struggling with this for weeks and I was unable to find an answer on line. Perhaps someone here can help me.
Oracle linux 8 running virtualization:
hardware node has a public IP address on interface bridge0 and physical eno1 is a member of the bridge0
a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node.
I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of.
When I try tracepath or traceroute from outside to virtual I get !H on last hup
same result when I try to do the same form hardware node to virtual I get !H
Also, when I telnet (TCP) to a specific port on virtual where I have a daemon LISTENING OR NOT I get: No route to host. Same experiment works just fine for ssh port.
Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to virtual dropping connections from all other sources
This issue presented it self when I attempted to setup a galera node on virtual and ports 4567 is responding but 4568 and 4444 are not, but the daemons are running and I can clearly see lsoft showing "LISTENING"
I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware of?
What am I missing?
-- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net <http://www.voipplus.net>
-- ----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon@gmail.com <mailto:thomasammon@gmail.com> -----------------------------------------------------------------------------
Laine, Though I can't remember the particulars, I have a vague memory of the sysctl settings in that article indeed solving the problem of traffic not being forwarded on the bridge when I had configured no filtering on the guest - hence my attempt to share what worked for me. Perhaps it would be good to update that page. I looked around for a link to create an account on the libvirt wiki but could find none. I'm happy to go do some more research around the items you mentioned and add a quick note to that page to keep from leading people astray in the future, if I could get an account on the wiki. Do you know how I would do that? Thanks, Tom On Mon, Feb 14, 2022 at 8:12 AM Laine Stump <laine@redhat.com> wrote:
On 2/13/22 5:38 PM, Tom Ammon wrote:
Can you post the output of iptables -L?
By default, the bridge module in the kernel sends packets traversing the bridge to iptables (in the FORWARD chain I believe) for processing. So if you have configured a DENY policy on the FORWARD chain, or are otherwise filtering in the forward chain, you'll be affecting packets traversing the bridge. Check out this page for details on how to change this behavior: https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf <https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf
That information is *very* out of date; the situation has changed quite a lot since that was written in 2014.
Filtering of packets traversing a bridge device are now only filtered if the br_netfilter module is loaded, which isn't done by default. It *is* autoloaded if certain types of iptables rules are added(I can't remember the details of the type of rule though - there was a bug in iptables a year or so ago where autoload of br_netfilter was triggered by libvirt attempting to *remove* a rule of whatever type it was).
Anyway, unless "lsmod | grep br_netfilter" shows that you have br_netfilter loaded, this entire path is a red herring (if you do have it loaded, unload it, and try to figure out why it was loaded).
(Interestingly, this is the 2nd time this particular outdated page has come up in the last week. Has something else broken somewhere that's causing people to search out this page?)
Tom
On Sun, Feb 13, 2022 at 4:08 PM Marcin Groszek <marcin@voipplus.net <mailto:marcin@voipplus.net>> wrote:
I have been struggling with this for weeks and I was unable to find
an
answer on line. Perhaps someone here can help me.
Oracle linux 8 running virtualization:
hardware node has a public IP address on interface bridge0 and
physical
eno1 is a member of the bridge0
a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node.
I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of.
When I try tracepath or traceroute from outside to virtual I get !H
on
last hup
same result when I try to do the same form hardware node to virtual I get !H
Also, when I telnet (TCP) to a specific port on virtual where I have
a
daemon LISTENING OR NOT I get: No route to host. Same experiment
works
just fine for ssh port.
Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to
virtual
dropping connections from all other sources
This issue presented it self when I attempted to setup a galera node
on
virtual and ports 4567 is responding but 4568 and 4444 are not, but
the
daemons are running and I can clearly see lsoft showing "LISTENING"
I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware
of?
What am I missing?
-- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net <http://www.voipplus.net>
--
-----------------------------------------------------------------------------
Tom Ammon M: (737) 400-9042 thomasammon@gmail.com <mailto:thomasammon@gmail.com>
-----------------------------------------------------------------------------
-- ----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon@gmail.com -----------------------------------------------------------------------------
On 2/14/22 10:18 AM, Tom Ammon wrote:
Laine,
Though I can't remember the particulars, I have a vague memory of the sysctl settings in that article indeed solving the problem of traffic not being forwarded on the bridge when I had configured no filtering on the guest - hence my attempt to share what worked for me. Perhaps it would be good to update that page.
Yeah, I had completely forgot of its existence until there were two unrelated references suddenly made to it in the last week.
I looked around for a link to create an account on the libvirt wiki but could find none. I'm happy to go do some more research around the items you mentioned and add a quick note to that page to keep from leading people astray in the future, if I could get an account on the wiki. Do you know how I would do that?
I actually tried to update the article after this second reference, and found that my password no longer works. Awhile back the decision was made to deprecate the wiki and slowly move content into "knowledgebase" articles that are included in the project git repo, and I think the wiki may have been made read-only at that time. I had planned to ask about that in IRC yesterday, but either forgot, or it was too late to catch anyone by the time I asked (I've even forgotten what happened yesterday :-/) Anyway, even in the days when the wiki was "active", automatic account creation was disabled to prevent spam articles, so creating an account required sending a message to danpb asking for an account; these days I think he'd just say "don't bother - it's going away anyway". Thanks anyway for the offer to update it though (and also for piping in with the idea in the first place - hopefully my response didn't come off as discouraging responses - even though it wasn't the source of the problem this time, next time yours might be the idea that solves the issue :-)). I'll try to take care of the wiki article in the next day or two.
Thanks, Tom
On Mon, Feb 14, 2022 at 8:12 AM Laine Stump <laine@redhat.com <mailto:laine@redhat.com>> wrote:
On 2/13/22 5:38 PM, Tom Ammon wrote: > Can you post the output of iptables -L? > > By default, the bridge module in the kernel sends packets traversing the > bridge to iptables (in the FORWARD chain I believe) for processing. So > if you have configured a DENY policy on the FORWARD chain, or are > otherwise filtering in the forward chain, you'll be affecting packets > traversing the bridge. Check out this page for details on how to change > this behavior: > https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf <https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf>
> <https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf <https://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf>>
That information is *very* out of date; the situation has changed quite a lot since that was written in 2014.
Filtering of packets traversing a bridge device are now only filtered if the br_netfilter module is loaded, which isn't done by default. It *is* autoloaded if certain types of iptables rules are added(I can't remember the details of the type of rule though - there was a bug in iptables a year or so ago where autoload of br_netfilter was triggered by libvirt attempting to *remove* a rule of whatever type it was).
Anyway, unless "lsmod | grep br_netfilter" shows that you have br_netfilter loaded, this entire path is a red herring (if you do have it loaded, unload it, and try to figure out why it was loaded).
(Interestingly, this is the 2nd time this particular outdated page has come up in the last week. Has something else broken somewhere that's causing people to search out this page?)
> > Tom > > On Sun, Feb 13, 2022 at 4:08 PM Marcin Groszek <marcin@voipplus.net <mailto:marcin@voipplus.net> > <mailto:marcin@voipplus.net <mailto:marcin@voipplus.net>>> wrote: > > I have been struggling with this for weeks and I was unable to find an > answer on line. Perhaps someone here can help me. > > Oracle linux 8 running virtualization: > > hardware node has a public IP address on interface bridge0 and physical > eno1 is a member of the bridge0 > > a virtual OS has interface bridged to lan and source is bridge0, Ip > address of virtual OS is also a public from same class as the > hardware node. > > I can route in and out of virtual, I can ping from hardware node to > virtual and vice versa, so the routing works as it should, sort of. > > When I try tracepath or traceroute from outside to virtual I get !H on > last hup > > same result when I try to do the same form hardware node to virtual > I get !H > > Also, when I telnet (TCP) to a specific port on virtual where I have a > daemon LISTENING OR NOT I get: No route to host. Same experiment works > just fine for ssh port. > > Firewalld is not running, and I just have very basic iptables rules > like > allowing external address block to ssh to hardware node and to virtual > dropping connections from all other sources > > This issue presented it self when I attempted to setup a galera node on > virtual and ports 4567 is responding but 4568 and 4444 are not, but the > daemons are running and I can clearly see lsoft showing "LISTENING" > > I capture the traffic and the tcp as well as udp are getting to the > virtual. Is there a preconfigured netfiltering that I am not aware of? > > What am I missing? > > > > > -- > Best Regards: > Marcin Groszek > Business Voip Resource. > http://www.voipplus.net <http://www.voipplus.net> <http://www.voipplus.net <http://www.voipplus.net>> > > > > -- > ----------------------------------------------------------------------------- > Tom Ammon > M: (737) 400-9042 > thomasammon@gmail.com <mailto:thomasammon@gmail.com> <mailto:thomasammon@gmail.com <mailto:thomasammon@gmail.com>> > -----------------------------------------------------------------------------
-- ----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon@gmail.com <mailto:thomasammon@gmail.com> -----------------------------------------------------------------------------
participants (4)
-
Daniel Romero -
Laine Stump -
Marcin Groszek -
Tom Ammon