addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1linklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

Re: [ljc] JVM is messed up when system clock is rolled backward

From: Bruno B.
Sent on: Wednesday, September 4, 2013 2:01 PM
HI Mikhail,

In our case we are experiencing this problem on virtualized environments of our customers, where our system is deployed. We do not have any control about those environments,. It happens  because of a sysop or NTP change or $whateverreason (these can be wide and complex environments, not hosting only our app but many other, a sort of customer cloud). If the system date change, and we deliver some items out of order, then we say the customer "hey it's not our app, its $linux/$java/$whatever": at this  they first basically laugh at us and then tell us"just fix it".

And I think they have a point.

I guarantee we do not have sleep() to synchronize things :), we are using java.concurrency primitives. The most problematic thing is the ScheduledExecutor: it just stops. The backup plan would be 1) patch the JVM or 2) rewrite in native some of those function but I am afraid of the extent of this issue. LockSupport.parkNanos(..) is basically the root of all other concurrency primitives in the JVM.

Cheers,

    Bruno


On Wed, Sep 4, 2013 at 12:53 PM, Mikhail Krivoshein <[address removed]> wrote:
Thank you for the very interesting conversation. I agree that the bug sound to be quite critical to fix, but I have a question in mind. How does no one I know suffers from this in production environments? Is it because they don't allow NTP daemon to change time backwards? Also is it because they use UTC instead of one of geographical timezones with summer time and other oddities? I understand if you deploy to a commercial cloud you have to use whatever is provided to you. My experience comes from environments where a company has a custom build of Linux and mostly high-spec hardware from top vendors. And a support contract for JVM that allows to engage Sun/Oracle engineers to solve problems.

Kind regards,
Mikhail Krivoshein


On 3 September[masked]:06, Bruno Bossola <[address removed]> wrote:
Well, the thing that worries me is that, anyway, the bug is still a P4 planned for JDK9: in my experience P4 means "we will never fix it" :)

I did not manage to write a patch myself (you basically have to change the cpp code of the hotspot VM) but I could do that, if I have time. At the moment I just reimplemente the sleep() which, to me, is useless: the whole park()/unpark() functions family have to be rewritten.

Cheers,

    Bruno



On Tue, Sep 3, 2013 at 1:54 PM, Martijn Verburg <[address removed]> wrote:
Looks like the discussion is going well over at hotspot-runtime - although it sounds like the short-term solution is to use your own patched JVM (as you've done).

Cheers,
Martijn


On 3 September[masked]:06, Bruno Bossola <[address removed]> wrote:
Thanks for your answer, I did not report back here the discussion with the engineers of the hotspot VM, I will try to summarize where we are atm.

True, the daylight saving will be sorted by TZ adjustments (thank god!) BUT the implementation of the concurrency primitive LockSupport.parkNanos(), the function that controls *every* concurrency primitive on the JVM, is flawed, and any NTP sync, or system time change, will break it (with different and funny results, yuppie!)

To be more clear about the issue, the extent of it and the concurrency library, let me introduce this very simple program:

import java.util.concurrent.locks.LockSupport;

public class Main {

    public static void main(String[] args) {

        for (int i=100; i>0; i--) {
            System.out.println(i);
            LockSupport.parkNanos(1000L*1000L*1000L);
        }

        System.out.println("Done!");
    }
}


Run it with a 64bit 1.6+ JVM on 64bit Linux, turn the clock down one hour and wait until the counter stops... magic!  I tested this on JDK6, JDK7 and latest JDK8 beta running on various Ubuntu distros. It's not just a matter of (old?) sleep() and wait() primitives, this issue it affects the whole concurrency library.

To prove that this is fixable, I reimplemented the program above above substituting  LockSupport.parkNanos()  with  a JNI call to clock_nanosleep(CLOCK_MONOTONIC...): works like a charm :(

This is due to the fact  that the CPP code is calling the pthread_cond_timedwait() using its default clock (CLOCK_REALTIME) which, unfortunately is affected by settime()/settimeofday() calls (on Linux): for that reason it cannot be used to measure nanoseconds delays, which is what the specification requires. CLOCK_REALTIME is not guaranteed to monotonically count as this is the actual "system time": each time my system syncs time using a NTP server on the net, the time might jump forward or backward. The correct call (again on Linux)  would require to use CLOCK_MONOTONIC as clock id, which are defined by POSIX specs since 2002. (or better CLOCK_MONOTONIC_RAW)

The POSIX spec is infact clear, as it states "...setting the value of the CLOCK_REALTIME clock via clock_settime() shall have no effect on threads that are blocked waiting for a relative time service based upon this clock...": it definitely states "relative".  Having a look at the hotspot code, it appears that the park() is using compute_abstime() (which uses timeofday) and then waits on an absolute period: for that reason it's influenced by the system clock change.
Very wrong
.

Consequences? If a system time change happens then all the threads parked will hang, with unpredictable/corrupted/useless results to the end user. Same applies to Future, Queue, Executor, and any other construct that it's somehow related to concurrency. This is a big issue for us and for any near time application: please think about trading and betting, where the JVM is largely used. And please do not restrain yourself to the Java language: add Scala and any other JVM-based language to the picture.

Hope this clears the confusion :), and I am still VERY scared.
Cheers,

   Bruno

p.s.
BTW I somehow managed to have the bug reopened to the public... a small step for a man, a giant leap for the JVM :)









--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Bruno Bossola ([address removed]) from LJC - London Java Community.
To learn more about Bruno Bossola, visit his/her member profile
Set my mailing list to email me As they are sent | In one daily email | Don't send me mailing list messages

Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed]





--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Martijn Verburg ([address removed]) from LJC - London Java Community.
To learn more about Martijn Verburg, visit his/her member profile





--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Bruno Bossola ([address removed]) from LJC - London Java Community.
To learn more about Bruno Bossola, visit his/her member profile
Set my mailing list to email me As they are sent | In one daily email | Don't send me mailing list messages

Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed]





--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Mikhail Krivoshein ([address removed]) from LJC - London Java Community.
To learn more about Mikhail Krivoshein, visit his/her member profile

Our Sponsors

  • Our Blog

    Read the latest news from the LJC

  • RecWorks Ltd

    Fixing Tech Recruitment using the Power of Community

  • jClarity

    Java/JVM Performance Analysis Tools & mentoring for Java related matters

  • LJC Aggrity

    Our LJC Aggrity site contains blog posts from our members

  • LJC Book Club

    Our Book club with book reviews from our members

  • Devoxx UK

    Java Community Conference in collaboration with the LJC, 8-10th June 16

  • SkillsMatter

    "Host, help organise, promote, film many of our meetings."

  • IBM

    Build Enterprise-grade apps at start-up speed.

  • New Relic

    New Relic makes sense of billions of metrics a day in real time.

  • Hazelcast

    Hazelcast is the leader in operating in-memory computing.

  • Java.Net

    We are an official Java User Group recognised by Oracle's JUG program

  • JRebel

    Free 3 month J-Rebel license.

  • O'Reilly

    40% discount on printed books and 50% on e-books.

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy