Turning off XInitThreads

I've read various threads where someone had a problem with XInitThreads.  While I see the use of it, our particular case has JOGL fill just one role in a larger system.  If we are going to call XInitThreads, we'd like to be able to do it earlier than JOGL will do it, before any X calls are made.

Would you be willing to add a way to not call XInitThreads in JOGL initialization?

There is a public static final boolean that controls the call in initSingleton() in X11Util.java.  Could we get something we could set?

thanks,
andy

On 09/11/2012 08:48 PM, Andy Skinner [via jogamp] wrote: I like to discuss the actual impact 1st.

For example, it should not make a difference if you call XInitThread()
before JOGL does it from GLProfile.initSingletion().
Read: Only the 1st XInitThread() call shall do something, subsequent calls
shall be a NOP.

You could also only call NativeWindow factory's initSingleton()
to achieve the XInitThread() call w/o initializing the shared OpenGL resources.

Would any of the above not work for you ? If so why ?
Before adding some conditional code (again) to the initialization sequence
I like to have us understand the issue 1st.

~Sven

> thanks,
> andy
>

I can't find documentation that second XInitThreads calls are NOP.  Could you point me to that?

That doesn't seem to be what we're seeing, but I'll poke at this again to make sure I'm not confusing this case with something else happening.

thanks
andy

On 09/12/2012 05:06 PM, Andy Skinner [via jogamp] wrote:
> I can't find documentation that second XInitThreads calls are NOP.  Could you
> point me to that?
>

Re-evaluating whether we still need X11 display locking.
There is a good chance this is not true, due to our impl. level locking.
Will update you here .. or better in a bug report you may enter in our
bugzilla DB. Thank you.
I will push this change to a specific branch - hope you participate in testing.

Also you are right, the X11 spec doesn't even mention the case of subsequent
XInitThreads calls - and hence YMMV.

~Sven

On 09/17/2012 09:51 AM, Sven Gothel wrote: Well, a quick review showed that it is not possible to easily
drop the multithreading requirements of XLib, b/c:
  - We reuse display connections in NEWT which
    maybe used from multiple threads. X11 locking is quite fast here,
    since usually xcb is being utilized as the backend.
    Simply removing XInitThreads()
    (even though no explicit XLockDisplay/.. is in place) will cause havoc.
    xcb_ actually detects concurrent access and aborts JOGL in this case.

  - A 'workaround' would be to instantiate one display connection per surface,
    similar as we use it currently for offscreen surfaces.

  - ...

Added experimental branch 'x11_no_xinitthreads'
for further discussions ..

<http://jogamp.org/git/?p=jogl.git;a=shortlog;h=refs/heads/x11_no_xinitthreads>


> Also you are right, the X11 spec doesn't even mention the case of subsequent
> XInitThreads calls - and hence YMMV.
>
> ~Sven
>

Sven Gothel wrote

Well, a quick review showed that it is not possible to easily
drop the multithreading requirements of XLib, b/c:
  - We reuse display connections in NEWT which
    maybe used from multiple threads. X11 locking is quite fast here,
    since usually xcb is being utilized as the backend.
    Simply removing XInitThreads()
    (even though no explicit XLockDisplay/.. is in place) will cause havoc.
    xcb_ actually detects concurrent access and aborts JOGL in this case.

Why do we reuse display connections in NEWT? Is the creation of such a connection very expensive?

On 09/17/2012 12:37 PM, gouessej [via jogamp] wrote: Good that you ask .. so allow me to elaborate on this
whole X11 Display / Locking issue .. it's a world of pain :)

Basics:

X11 is network based, using fast local transfers or other slow media.
A X11 display connection (DPY) communicates w/ a device (X server)
which may be local (fast) or remote (slow).
One site can have multiple connection names, e.g. X server etc.

There is a DPY limit, 255 connections are allowed _to_ one X11 server (?)

1 DPY gives you one communication queue to the server,
each is intrinsically thread safe if used by one thread at a time.

If sharing a DPY one would either need to use
  [1] XLockDisplay/XUnlockDisplay, which requires XInitThreads to work
  [2] Application level locking

Depending on the impl. [1] is quite fast today (XCB backend).

A bad example of [2] is AWT, i.e. using one global lock for the whole
toolkit (on X11). In general it would be a very bad idea to use
one DPY for the whole 'location'. Stuttering and other blocking
artifacts would be experienced .. as you are used w/ the
one-for-all EDT architecture as experienced w/ AWT, Cocoa, SWT, ..

Since the goal already achieved is to use one 'display' w/ multiple
rendering instances w/o blocking, we can either use:
  [a] one DPY for each thread and use case
  [b] X11 locking

The master branch uses a mix of [a] and [b] already,
where a shift towards [a] happened a while ago in the offscreen
surface and drawable implementation.

Now lets have a view back in history ..

- In the beginning .. JOGL1, followed more the EDT approach
  utilizing the ThreadUtil. Of course, at this time AWT and Swing
  were the only drawable surfaces available.
  You could disable the EDT funneling via properties, they still exist.
  However, since AWT is not implemented to be thread safe,
  but single threaded - your mileage may vary (YMMV).

- On AWT we lock surface and the so called JAWT global lock.
  Hence rendering in multiple AWT surfaces (GLCanvas)
  from multiple threads will serve you a stuttering experience.
  You will also notice a lag of input, since the global lock
  synchronizes both, the input event queue and the surfaces - all resources.  

- We created NEWT and hence AWT constraints were no more required to follow.
  We were asked which threading model NEWT follows.
  The simple answer was none at all, i.e. it should be thread safe.
  In the end - due to conveniences, the NEWT EDT semantic was born.
  NEWT's EDT intend was to only dispatch events from the native queue.
  Nevertheless we were required to also use it to handle
  the native windowing lifecycle actions like create / destroy windows.

  In the end .. as it is of today, you can use a NEWT surface
  to render into from any thread, while locking it's surface and DPY.
  Since we used [b], a shared DPY w/ XInitThreads, we are thread-safe
  and resource efficient. Still .. having multiple NEWT windows
  operating on one shared DPY may reduce some responsiveness.

  This is the balance to discuss here.

The current DPY situation regarding locking, open and close ..

- NVidia plays nice w/ open and closing of display connections

- AMD has an open/close order bug, i.e. DPY must be closed in the order
  they were opened, hence X11Util funnels all DPY open and close ops.
  If using AMD, we never close the DPY, but reuse them (not too bad).
  At JVM shutdown we close them in the proper order,
  which avoids leaks and a SEGV at JVM shutdown .. you may
  have been bothered with it.

- XCB backend introduced another hack, it's locking bug forced us
  to have 1 DPY per EDT and one per 'renderer'.
  Read: Both make up one NEWT X11 Display, so a NEWT X11 Display
  opens _two_ DPY.


++++

The new branch x11_no_xinitthreads, test it!

- One good unit test here demonstrating the difference of
  the master branch [b] and the new x11_no_xinitthreads branch [a]
  is TestInitConcurrentNEWT, which opens multiple NEWT windows
  each in it's own thread and renders them.
  With [b] you will notice that it takes a little time
  until all windows become established, where with [a]
  realization of all windows happens almost at the same time
  even though it takes longer until they all show up at once.
  The total time delta is almost zero, a few milliseconds
  pro [a]. However, resource costs for [a] explodes a little bit.
     20 NEWT Windows w/ DPY: [b] == 2 DPYs, [a] == 40 DPYs

  Of course, this is not the regular use case ..

As mentioned above, it is to discuss which solution is preferable.

PRO x11_no_xinitthreads branch:

  - more clean and less complex implementation

  - easier to maintain

  - neither requiring not bothering w/ XInitThreads,
    nor XLockDisplay/XUnlockDisplay

  - no questionable locking parts in implementation

  - may solve issues folks had w/ IDEs like Eclipse ?
    E.g. Wade and others reported different behavior
    when calling JOGL's initialization at a different time.

  - multiple NEWT windows running [each in their thread]
    on one display run more in a separate fashion.
    No interference nor blocking from other same 'display' NEWT windows    

PRO master branch (old style):

  - it's tested and used today

  - uses less DPY resources

  - much harder to hit the DPY limit of 255 connections to one server


The educated gut feeling will favor 'x11_no_xinitthreads'.

There has been quite some change in the NEWT X11 Display domain:
  NEWT: Changed Lifecycle of Display/Screen
    <http://jogamp.org/git/?p=jogl.git&a=search&h=HEAD&st=commit&s=Changed+Lifecycle+of+Display%2FScreen>

also searching for 'X11 Display' reveals above mentioned change sets:
    <http://jogamp.org/git/?p=jogl.git&a=search&h=HEAD&st=commit&s=X11+Display>

My conclusion so far - if you could test the new branch and:
  - it works well
  - even fixes maybe a bug (Eclipse, IDEs, .. Wade ?)
  - the DPY limit doesn't bother you
  - no other concerns ..

then we could move over and clean up accordingly.

Sorry for my very long writeup, but since the matter is quite complex
and important to our X11 port I believe being more verbose and detailed
is hurting less as if being too brief.

Cheers, Sven

Created bug 616:
  <https://jogamp.org/bugzilla/show_bug.cgi?id=616>

which blocks (IMHO) bug 603:
  <https://jogamp.org/bugzilla/show_bug.cgi?id=603>

Would be great if we all could cont. discussion there
and if you can the branch w/ your use cases.
Sure, the branch is more a hack than a clean & complete work,
but it should give an idea whether this approach is feasible
for your cases.

~Sven