I have multiple VM’s on the same KVM host. One of the VM’s is running NTP. All VM’s can reach eachother, no firewall in between. But the problem is, the VM’s cannot communicate over port 123/udp to the NTP VM.

Network: 172.24.100.0/22

KVM: 172.24.101.50

VM ntp: 172.24.102.10

VM foo: 172.24.102.20

1. On the NTP server, listen for any incoming packets from VM foo on port 123:

[ntp ~]# tcpdump -i any host 172.24.102.20 and port 123 -n

2. Execute the following on server foo. Since server ntp is listening with tcpdump, packets should be visible in tcpdump.

[foo ~]# ntpdate 172.24.102.10

This is failing:

ntpdate[30443]: no server suitable for synchronization found

No packets are coming in to the ntp server, tcpdump is just blank. Weird.

To troubleshoot further, start over and do the following:

[ntp ~]# tcpdump -i any host 172.24.102.20 and port 123 -n ## Listen for packets filtering host 172.24.102.20 and port 123

[foo ~]# tcpdump -i any host 172.24.102.10 and port 123 -n ## Listen for packets filtering host 172.24.102.10 and port 123

While both tcpdumps are running, execute the following:

[foo ~]# ntpdate 172.24.102.10

Now, on the tcpdump of VM foo, you will see outgoing packets:

19:45:26.644630 IP 172.24.102.20.ntp > 172.24.102.10.ntp: NTPv4, Client, length 48

As you can see, packets are exiting the server, but there is no response.

And the tcpdump of the ntp server is still empty, it doesn't receive the packets (so, it won't reply). But why?

Lets troubleshoot further and run ntpdate in debugging mode:

[foo ~]# ntpdate -dv 172.24.102.10

22 Aug 19:51:23 ntpdate[30465]: ntpdate 4.2.6p5@1.2349-o Wed Mar  1 09:00:52 UTC 2017 (1)

Looking for host 172.24.102.10 and service ntp

host found : some-host.com

transmit(172.24.102.10)

receive(172.24.102.10)

transmit(172.24.102.10)

receive(172.24.102.10)

server 172.24.102.10, port 123

22 Aug 19:51:29 ntpdate[30465]: step time server 172.24.102.10 offset 1.414813 sec

Wow it worked!? But it only works with the "-d" option. What is the difference between normal and debgging mode? Lets have a closer look; without the "-d" option, the src and dest ports are 123. When using the "-d" option, the src port is not 123 (it is a random high port number).

On the KVM host and on the VM's there is no firewall active, even if there was a firewall, in tcpdump the packets should have been shown.

Anyone who can help? Thanks!

Sinan