I mildly recalls seeing a bug like this. The problem was in intel's
kernel driver. A NIC by defaul checks incoming packets whether they
match NIC's MAC. So if a TAP device was created over a NIC, it had to be
put into promisc mode (automatically done by the driver) to allow
different MACs and the check was done in kernel then. But since this
generates too much interrupts, NICs HW was extended and it can be
programmed with multiple MACs to let through. However, there was a bug
which I recall of, that intel driver was not always putting the TAP MAC
into the NIC HW correctly. Obviously, the bug was not visible if the NIC
was put into promisc mode. And this may be what you are seeing. Let me
see if I can find the bug.