10:23 p.m.
Hello!
I just spent the last four days working with nwfilters only to decide
that they are apparently unusable. I've come to the mailing list seeking
input on this subject.
First off, please forgive my offensiveness. I'm sure people worked hard
on nwfilters and it looks like a lot of effort went into providing this
functionality. This is also an extremely difficult subject to get right
in the many possible use cases, so I'm very sympathetic to how difficult
it would be to try and implement this. However, the existing system
didn't work out for me, I've found a number of other people who are
saying the same thing (it didn't work out for them), and I don't see any
hope continuing down the path of trying to make it work.
For now, I've given up on nwfilters and I created a hook script that
works with my existing iptables rules and applies network filter
policies on specific VM/guests where needed.
If you are doing extensive VM network filtering in your environment, how
did you do it?
I've listed a bunch of my gripes below. Please correct me if I've gotten
anything wrong here. I'm new to nwfilters so maybe I overlooked
something or I might just misunderstand the whole thing and could be
totally wrong.
The first and primary problem that I have with nwfilters are that the
documentation is poor. There is very little documentation which exists,
and that which does exist seems like it was spat out just to fulfill
business requirements that some documentation be produced, rather than
an effort into creating good usable documentation. I've run into large
amounts of undocumented behavior and I don't feel like reading the
source code any further to figure out what the intent of these tools were.
My second big issue, and a clue very few people actually use nwfilters
in the wild, is the low quantity of examples and how-to docs I found
while googling. Complex examples just don't seem to exist. Further, of
those complex examples I did find, people were often going down the
route of creating their own hook script programs to replace nwfilters,
indicating that this isn't just me.
Additionally...
I discovered that nwfilters do not play well with existing system
iptables/ebtables rules. There is some good examples on this regarding
Red Hat's firewalld and how libvirt's nwfilters does not play well
together if you google around a little. It seems like this was just not
considered in scope, or the assumption was that the local host would not
have any existing iptables/ebtables rules and that libvirt would have
complete control over the hypervisor host. There is no documented means
of controlling where libvirt inserts it's rules into an existing set of
rules, and libvirt creates numerous rules in both ebtables and iptables,
making the problem even more complex.
nwfilter seems to have been designed with a bias towards
user-networking. I am using bridged interfaces, and some features and
virsh commands don't apply to this mode of operation.
I've been able to produce scenarios where nwfilter would abandon rules
after changes had been made to running guests, and the only way I could
get rid of them was manual intervention (iptables/ebtables -F -X).
There is no command/control to apply an existing nwfilter to a running
guest, or to remove/clear the existing nwfilters on a running guest.
This item is a huge indication that this isn't a production-ready
feature set.
I think the worst problem I've run into, however, is that I was able to
create very simple nwfilters that either broke networking of the
hypervisor system (stopped all traffic), or failed to drop traffic which
should have been dropped. I still don't understand why nwfilter is often
creating rules in the ebtables "nat" table instead of the "filter"
table, where they belong. That one right there is a huge WTF -- packets
never get inspected because the rules are in the wrong table!
In general, I found the output iptables/ebables rules that nwfilter
generated often did not reflect the obvious intent of the rules that
went into the nwfilter xml configuration. This abstraction layer
produces unreliable and/or confusing results. I put a series of rules
into a nwfilter xml file and the iptables/ebtables rules that I get out
are insane. Nwfilter rules in = mystery meat out.
Priorities are a huge WTF that caused me a lot of grief. Are rules going
to be assembled in iptables/ebtables in the order which they are
declared in XML? (this is undocumented) If so, why do priorities exist?
(undocumented). What is the default priority? Is it zero? (undocumented).
Want to create a filter rule that will log certain packets? Apparently
there is no logging functionality at all. Can't be done. Anything beyond
the most basic packet allow/drop (even reject was an afterthought) isn't
supported by nwfilter rules.
Thanks for reading