[libvirt] time: event poll may become un-triggerable after changing system clock.
event poll may become un-triggerable after changing system clock. The steps to reproduce the problem: 1 run event-test 1 define and start a domain with name vm1. 2 destroy vm1 3 change system time to 1 hour before when timer.expiresAt has been set in virEventPollUpdateTimeout (and before virEventPollDispatchTimeouts()). 4 event-test will recive no message until 1 hour later. The reasons for the problem is : 1 The value of timer.expiresAt is set by virTimeMillisNowRaw. virTimeMillisNowRaw is effectable by settimeofday(), bacause it uses CLOCK_REALTIME to get time. 2 If we change the system time to a time long before now, after that timer.expiresAt has been set. timer.expiresAt is not affected, while virEventPollDispatchTimeouts is. Suppose it's now May 12th, and we set it to 10th, then the expiresAt is 12th, and the time virEventPollDispatchTimeouts got is 10th. if (eventLoop.timeouts[i].expiresAt <= (now+20)) { // expiresAt will not be less than now until 2 days later. *Solution(not good enough)*: 1 change the clock mode in virTimeMillisNowRaw from REALTIME to MONOTONIC, which would not be affected by settimeofday(). 2 add the time got from clock_gettime(*MONOTONIC*) with the system-start-time from epoch, making it equal to the value got from REALTIME. 3 As that the timestamp of the log message should follow system time, so we keep it to REALTIME as before. However, there's still problems: 1 pthread_cond_wait() gets time with REALTIME mode. When we change system time, pthread_cond_wait() may still be affected. So, Is there any other better solution? thanks in advance. oscar
On Tue, May 12, 2015 at 04:14:37PM +0800, zhang bo wrote:
event poll may become un-triggerable after changing system clock.
The steps to reproduce the problem: 1 run event-test 1 define and start a domain with name vm1. 2 destroy vm1 3 change system time to 1 hour before when timer.expiresAt has been set in virEventPollUpdateTimeout (and before virEventPollDispatchTimeouts()). 4 event-test will recive no message until 1 hour later.
The reasons for the problem is : 1 The value of timer.expiresAt is set by virTimeMillisNowRaw. virTimeMillisNowRaw is effectable by settimeofday(), bacause it uses CLOCK_REALTIME to get time. 2 If we change the system time to a time long before now, after that timer.expiresAt has been set. timer.expiresAt is not affected, while virEventPollDispatchTimeouts is. Suppose it's now May 12th, and we set it to 10th, then the expiresAt is 12th, and the time virEventPollDispatchTimeouts got is 10th. if (eventLoop.timeouts[i].expiresAt <= (now+20)) { // expiresAt will not be less than now until 2 days later.
*Solution(not good enough)*: 1 change the clock mode in virTimeMillisNowRaw from REALTIME to MONOTONIC, which would not be affected by settimeofday(). 2 add the time got from clock_gettime(*MONOTONIC*) with the system-start-time from epoch, making it equal to the value got from REALTIME. 3 As that the timestamp of the log message should follow system time, so we keep it to REALTIME as before.
However, there's still problems: 1 pthread_cond_wait() gets time with REALTIME mode. When we change system time, pthread_cond_wait() may still be affected.
So, Is there any other better solution? thanks in advance.
Simply don't change the system time by massive deltas. Libvirt is not going to be the only app to be affected. As you mention it is going to hit the pthread_cond_wait() call which will likely affect pretty much every single non-trivial process running on the system. I'd expect other apps have much the same problem with calculating poll sleeps too. If you need to massively change the system time this should be done at single user mode, or do a reboot. Once a system is running it should be kept synced with NTPD which will only ever change system time in very small increments and so once cause thsi problem. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 2015/5/12 16:56, Daniel P. Berrange wrote:
On Tue, May 12, 2015 at 04:14:37PM +0800, zhang bo wrote:
So, Is there any other better solution? thanks in advance.
Simply don't change the system time by massive deltas. Libvirt is not going to be the only app to be affected. As you mention it is going to hit the pthread_cond_wait() call which will likely affect pretty much every single non-trivial process running on the system. I'd expect other apps have much the same problem with calculating poll sleeps too.
If you need to massively change the system time this should be done at single user mode, or do a reboot. Once a system is running it should be kept synced with NTPD which will only ever change system time in very small increments and so once cause thsi problem.
Regards, Daniel
Thank you Daniel. Would you please explain more about what "single user mode" mean here? and does "reboot" refers to "rebooting the os" or "rebooting the process libvirtd"? I think it's "the process", right?
On Wed, May 13, 2015 at 08:54:49AM +0800, zhang bo wrote:
On 2015/5/12 16:56, Daniel P. Berrange wrote:
On Tue, May 12, 2015 at 04:14:37PM +0800, zhang bo wrote:
So, Is there any other better solution? thanks in advance.
Simply don't change the system time by massive deltas. Libvirt is not going to be the only app to be affected. As you mention it is going to hit the pthread_cond_wait() call which will likely affect pretty much every single non-trivial process running on the system. I'd expect other apps have much the same problem with calculating poll sleeps too.
If you need to massively change the system time this should be done at single user mode, or do a reboot. Once a system is running it should be kept synced with NTPD which will only ever change system time in very small increments and so once cause thsi problem.
Would you please explain more about what "single user mode" mean here? and does "reboot" refers to "rebooting the os" or "rebooting the process libvirtd"? I think it's "the process", right?
Single user mode is where you boot the OS such that you only have a root shell running. No other processes are present. So you can safely mess around with core system settings like hardware clock without confusing any apps that are running. I mean rebooting the OS, not restarting the process Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (2)
-
Daniel P. Berrange -
zhang bo