Thread blocking issue with AWT (but not NEWT) on OSX

I found (in Processing) that issuing any call to the frame object that contains the GL canvas, for example frame.setTitle(...), hangs the application. This happens under OSX (tested on 10.6.8), using AWT as the toolkit.

I wrote a minimal self-contained test code that roughly follows the rendering flow in Processing, and exhibits the problem:

http://pastebin.com/afEPhBh1

The frame.setTitle() call is right at the beginning of the draw() method. If I replace it with:

EventQueue.invokeLater(new Runnable() {
  public void run() {
    frame.setTitle("frame " + frameCount);
  }
});

then the code runs fine. I tested it using the jogl jars from an autobuild from two days ago. I should add this wasn't an issue until something changed in jogl recently, probably in the last week or so. It doesn't affect Windows, and haven't tested on Linux yet. An equivalent test code using NEWT:

http://pastebin.com/XxDnRh3a

doesn't show the problem on OSX.

We've been hitting what may be related probelms with respect to requesting focus in AWT, below is the local patch
we've been running with and haven't seen any problems since.  Seen on Win7 64 bit fairly regularly when changing focus
between two AWT windows containing NewtCanvasAWTs


---
 .../classes/com/jogamp/newt/awt/NewtCanvasAWT.java |   36 +++++++++++++-------
 1 file changed, 23 insertions(+), 13 deletions(-)

diff --git a/src/newt/classes/com/jogamp/newt/awt/NewtCanvasAWT.java b/src/newt/classes/com/jogamp/newt/awt/NewtCanvasAWT.java
index 89a749c..b702852 100644
--- a/src/newt/classes/com/jogamp/newt/awt/NewtCanvasAWT.java
+++ b/src/newt/classes/com/jogamp/newt/awt/NewtCanvasAWT.java
@@ -163,34 +163,44 @@ public class NewtCanvasAWT extends java.awt.Canvas implements WindowClosingProto
             if(DEBUG) {
                 System.err.println("NewtCanvasAWT.FocusAction: "+Display.getThreadName()+", isOnscreen "+isOnscreen+", hasFocus "+hasFocus()+", isParent "+isParent+", isFS "+isFullscreen);
             }
-            if(isParent && !isFullscreen) {
-                // Newt-EDT -> AWT-EDT may freeze Window's native peer requestFocus.
-                if(!hasFocus()) {
-                    // Acquire the AWT focus 1st for proper AWT traversal
-                    NewtCanvasAWT.super.requestFocus();
-                }
-                if(isOnscreen) {
-                    // Remove the AWT focus in favor of the native NEWT focus
-                    KeyboardFocusManager.getCurrentKeyboardFocusManager().clearGlobalFocusOwner();
+            java.awt.EventQueue.invokeLater(new Runnable() {
+                public void run() {
+
+                    if(isParent && !isFullscreen) {
+                        // Newt-EDT -> AWT-EDT may freeze Window's native peer requestFocus.
+                        if(!hasFocus()) {
+                            // Acquire the AWT focus 1st for proper AWT traversal
+                            NewtCanvasAWT.super.requestFocus();
+                        }
+                        if(isOnscreen) {
+                            // Remove the AWT focus in favor of the native NEWT focus
+                            KeyboardFocusManager.getCurrentKeyboardFocusManager().clearGlobalFocusOwner();
+                        }
+                    }
                 }
-            }
+            });
+
             return false; // NEWT shall proceed requesting the native focus
         }
     }
     private FocusAction focusAction = new FocusAction();
-    
+
     WindowListener clearAWTMenusOnNewtFocus = new WindowAdapter() {
           @Override
           public void windowGainedFocus(WindowEvent arg0) {
               if( isParent() && !isFullscreen() ) {
-                  MenuSelectionManager.defaultManager().clearSelectedPath();
+                  java.awt.EventQueue.invokeLater(new Runnable() {
+                      public void run() {
+                          MenuSelectionManager.defaultManager().clearSelectedPath();
+                      }
+                  });
               }
           }
     };
 
     class FocusTraversalKeyListener implements KeyListener {
          boolean suppress = false;
-        
+
          public void keyPressed(KeyEvent e) {
              if( isParent() && !isFullscreen() ) {
                  handleKey(e, false);
--
1.7.10.msysgit.1

Thanks for posting this patch. However, I notice it is for NewtCanvasAWT, while the deadlock happens when I create an AWT surface with GLCanvas, so not sure yet if it applies directly to my problem. In any case, I'll take a look and see if I can figure things out.

On 10/31/2012 10:18 PM, ac [via jogamp] wrote: The last methodology is the only correct way to modify an already established AWT component.
This is mentioned in the AWT specification, where all mutable methods shall be
called from the AWT EDT (as you do in your workaround).

You also see this in all of our unit tests,
hence we don't test 'off-thread' AWT method calls, i.e. from a user thread.
Due to the latter .. we cannot track such an [unsupported by AWT] behavior.

> then the code runs fine. I tested it using the jogl jars from an autobuild
> from two days ago. I should add this wasn't an issue until something changed
> in jogl recently, probably in the last week or so. It doesn't affect Windows,
> and haven't tested on Linux yet. An equivalent test code using NEWT:

The question here is whether this is due to our changes or
an update of the Java6/7 on OSX you use.
Sure .. possible that a JOGL code change removed the tolerance,
but again - see above - it's not supported.

>
> http://pastebin.com/XxDnRh3a
>
> doesn't show the problem on OSX.

Yes, NEWT uses it's own threading system
and hence is not vulnerable to this.

The deadlocks potential of AWT is huge, since most of the AWT
methods will trigger an action on the EDT, or lock the so-called
AWT TreeLock. I would need to see the stack traces (jstack -l PID).

In the meantime .. pls run AWT actions from the designated AWT-EDT,
otherwise it's just luck that it works. And this is not just b/c of JOGL :)

If you can bisect the JOGL commits and find the culprit (something threading related, see
commit message) .. we may discuss it whether it's worth to handle this case.
IMHO - not.

~Sven

On 10/31/2012 11:56 PM, ac [via jogamp] wrote:
> Thanks for posting this patch. However, I notice it is for NewtCanvasAWT,
> while the deadlock happens when I create an AWT surface with GLCanvas, so not
> sure yet if it applies directly to my problem. In any case, I'll take a look
> and see if I can figure things out.

Yes .. and it is unrelated.
It's actually about a NewtCanvasAWT deadlock Harvey experienced
and I haven't looked into it ..

I would like to discuss this issue *now* .. if you (AC + Harvey)
could join the jogamp jabber chatroom.
   server: conference.jabber.org room: jogamp

~Sven

On 10/31/2012 10:55 PM, hharrison [via jogamp] wrote:
> We've been hitting what may be related probelms with respect to requesting
> focus in AWT, below is the local patch
> we've been running with and haven't seen any problems since.  Seen on Win7 64
> bit fairly regularly when changing focus
> between two AWT windows containing NewtCanvasAWTs
>
>

I saw your p. email from last week.
Good that it is public now.

As you have read in my reply to AC's original report,
it's desired to perform all AWT action must be dont in AWT-EDT.

Your patch seems to fix this for NewtCanvasAWT.

The only problem I have w/ it is that there is no unit test
able to reproduce the deadlock you have experienced.
(I also would have preferred a git pull req. :)

Trusting you on the issue a bit .. I will apply your patch
and run the unit tests we have to check for regressions.

Again .. I would like to see a unit test proving the issue
and that this patch fixes it.

Sorry for being 'rude' .. not intended, just damn tired
trying to get RC11 out :)

~Sven

Ok, I see, thanks for the explanation.

However, there is still something that confuses me. If I print the current thread when calling frame.setTitle() it is Thread[AWT-EventQueue-0,6,main], doesn't this mean that the call is happening from the EDT and not from the user thread?

On 11/01/2012 12:19 AM, ac [via jogamp] wrote:
> Ok, I see, thanks for the explanation.
>
> However, there is still something that confuses me. If I print the current
> thread when calling frame.setTitle() it is Thread[AWT-EventQueue-0,6,main],
> doesn't this mean that the call is happening from the EDT and not from the
> user thread?
>

Yes :(

So you found a regression due to our huge change of locking.
And I see .. you call 'later' on EDT, hence pass our locking etc.

My fault .. now we need a unit test.

Can you pls dump information of OSX, Java version etc ?

Dunno if I can make a fix into RC11 .. since it's already 'last call' here :)

~Sven

Sorry, I was waiting for comment from you before pushing further, I just got sidetracked
before pushing out a commit you could pull.  I'll see what can be done WRT a unit test.

The unit test would essentially be, create two AWT Frames with a NewtAWTCanvas in
each frame.  Alternately set each window to be the focused frame, for us it locked
up trivially.

If you hold off pushing this out, I will craft a commit w/ explanatory message for you to
Review/pull in the next few hours.

Harvey

On 11/01/2012 12:28 AM, hharrison [via jogamp] wrote: I have experienced at least one regression on Win7 w/ Java6 64bit,
see attached files.

Hence this patch is not ready yet. Yes, the NewtCanvasAWT NEWT/AWT focus handling
is complicated and .. well, not clean. Just look at the focus-handler in NEWT Window
for this task, the clear-menu thingy .. etc.

To not break other applications relying on it, we have to wait until a patch is matured more.
In general .. it does the right thing - but then again, it's tricky .. timing etc.

~Sven

>
> Harvey

Java Version            : 1.6.0_37
Java VM version       : 20.12-b01-434
Java Runtime version: 1.6.0_37-b06-434-10M3909
Architecture             : 64 bit
OSX version: 10.6.8 and 10.8.2

On 11/01/2012 12:50 AM, ac [via jogamp] wrote:
> Java Version            : 1.6.0_37
> Java VM version       : 20.12-b01-434
> Java Runtime version: 1.6.0_37-b06-434-10M3909
> Architecture             : 64 bit
> OSX version: 10.6.8 and 10.8.2
>
>

Thank you .. I am doing RC11 now.

Pls file a bugreport, attach all this info _and_ your unit tests.
[I would love to see 'em as a git pull req. ofc :]

Then .. next week or so, we have a look at it.

Thank you.

~Sven

Thanks, we'll keep looking at this issue and submit a pull request when we have some more confidence.  We sent the
patch out for comment, and now we have it :-)

On 11/01/2012 12:55 AM, hharrison [via jogamp] wrote:
> Thanks, we'll keep looking at this issue and submit a pull request when we
> have some more confidence.  We sent the
> patch out for comment, and now we have it :-)

Yup .. again, me a bit 'wasted' :)

Good job, looking fwd continuing on cleaning up the focus handling.

If possible, it would be nice to have this in a bug report.
Emails are so transient, easily gets lost and forgotten
in a folder w/ so many mails.

Ok .. thank you.

Laters, Sven

I posted the Java and OS info below.

> So you found a regression due to our huge change of locking.

In fact, I'm also experiencing another deadlock issue in Processing, but I didn't bring up at first since it involves native libraries and video, so I don't know how to report it through a small test case. But since you mention that you have done major changes in the locking mechanism in JOGL, I just want to point out to it because it might indicate other potential issues/regressions.

Very briefly, I'm pre-rolling capture devices (webcams, etc) to get the the device's supported resolutions and framerates. I do this through some native code written in Objective-C, which gets called via JNA in the first frame of my app in Processing/Java. This pre-rolling involves waiting a bit for the camera to start capture, getting a few frames, and then shutting down the camera. So I have something along these lines:

// Initial delay
NSRunLoop *mainRunLoop = [NSRunLoop mainRunLoop];
if ([mainRunLoop currentMode] == nil) {  
  [[NSRunLoop mainRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:1]];
}

// Wait until the delegate method gets a bunch of frames....
while (!gotFrame) {
    [[NSRunLoop mainRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:0.1]];
}  

In any case, this used to work w/out any problem until I updated to the autobuild jars from Oct 29th. After that, the Java program would hang when doing this pre-rolling in native code. Again, if I encapsulate the Java call that triggers the pre-rolling inside a EventQueue.invokeLater(), then the app doesn't hang. So I feel this is related to the same regression that causes the deadlock when trying to set the title of the frame. Same as before, everything gets called from the EDT, although the native calls might be happening in a separate thread, this I'm not sure about.


 

On 11/01/2012 01:13 AM, ac [via jogamp] wrote: Current locking code (since removing XInitThread) targets to be lock free,
or better block free. This is ofc impossible for AWT GLCanvas (AWT EDT issue).

Well, maybe you can find the issue - or better, produce a good
test case provoking it (like your last one).

Sure it would help if you bisect and find the offending commit / build,
i.e. the 1st which breaks sanity.

Please put this all together in a bug report, we can work better on it
this way.

~Sven

+++

From now - past (~2 Month):

jogl$ git log --oneline --follow src/jogl/classes/javax/media/opengl/awt/GLCanvas.java

51f055a GLCanvas: Avoid NPE in debug mode @ reshape (duh)
d4114a6 Refine GLDrawable.swapBuffer() fix (AWT/SWT GLCanvas and GLAutoDrawableBase) of commit 455fed40391afe10ce5ffb9146ca325af63b0a49
455fed4 Fix GLDrawable.swapBuffer() implementation in AWT/SWT GLCanvas and GLAutoDrawableBase
bb8454d Refine a3cb6bb14f410f67fccf5ccd4cd7ecc66f448389, fix deadlock (regression)
a3cb6bb Fix Bug 572 (2nd time): GLCanvas.validateGLDrawable() @ display() and reshape() ; GLCanvas.reshape() only if drawble valid ; GLCanvas.validateGLDrawable() also test isDisplayable() ; Fix size validation ;
4f8432b GLDrawableHelper.init(..): Avoid double reshape if subsequent display issues reshape
4dd44b9 Seamless Integration of an FBObject based GLFBODrawable as GLOffscreenAutoDrawable.FBO and as an OffscreenLayerSurface's drawable (OSX) - Fix Bugs 569 and 599

Ok, I will open a bug report and try to identify the commit that introduced the deadlock.

In the meantime, I'll probably need to downgrade to an earlier version of JOGL for the next beta release of Processing, since the threading issue is breaking video capture. Hopefully we will able to track this one down soon.

On 11/01/2012 02:06 AM, ac [via jogamp] wrote:
> Ok, I will open a bug report and try to identify the commit that introduced
> the deadlock.
>
> In the meantime, I'll probably need to downgrade to an earlier version of JOGL
> for the next beta release of Processing, since the threading issue is breaking
> video capture. Hopefully we will able to track this one down soon.
>

Analyzing the bug right now.

So I could easily produce the deadlock on OSX - good.

I see:

"AWT-EventQueue-0" prio=6 tid=7fe6758af000 nid=0x119282000 runnable [11927f000]
   java.lang.Thread.State: RUNNABLE
        at apple.awt.CWindow._setTitle(Native Method)
        at apple.awt.CWindow.setTitle(CWindow.java:765)
        at java.awt.Frame.setTitle(Frame.java:503)
        - locked <7f42b01b8> (a java.awt.Frame)
        at com.jogamp.opengl.test.junit.jogl.awt.TestGLCanvasAWTActionDeadlock01AWT.draw(TestGLCanvasAWTActionDeadlock01AWT.java:249)
        at com.jogamp.opengl.test.junit.jogl.awt.TestGLCanvasAWTActionDeadlock01AWT$SimpleListener.display(TestGLCanvasAWTActionDeadlock01AWT.java:347)
        at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:373)
        - locked <7f3072e58> (a java.lang.Object)
        at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:358)
        at javax.media.opengl.awt.GLCanvas$7.run(GLCanvas.java:983)
        at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:655)
        at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:594)
        at javax.media.opengl.awt.GLCanvas$8.run(GLCanvas.java:996)
        at javax.media.opengl.Threading.invoke(Threading.java:193)
        at javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:449)
        at javax.media.opengl.awt.GLCanvas.paint(GLCanvas.java:499)

_and_

"OPENGL" prio=5 tid=7fe67431a000 nid=0x11c3fa000 in Object.wait() [11c3f9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <7f31d11e8> (a java.awt.EventQueue$1AWTInvocationLock)
        at java.lang.Object.wait(Object.java:485)
        at java.awt.EventQueue.invokeAndWait(EventQueue.java:1106)
        - locked <7f31d11e8> (a java.awt.EventQueue$1AWTInvocationLock)
        at jogamp.common.awt.AWTEDTExecutor.invoke(AWTEDTExecutor.java:53)
        at jogamp.opengl.awt.AWTThreadingPlugin.invokeOnOpenGLThread(AWTThreadingPlugin.java:103)
        at jogamp.opengl.ThreadingImpl.invokeOnOpenGLThread(ThreadingImpl.java:205)
        at javax.media.opengl.Threading.invokeOnOpenGLThread(Threading.java:172)
        at javax.media.opengl.Threading.invoke(Threading.java:191)
        at javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:449)
        at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:74)
        at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:142)

I.e. both threads (AWT-EDT and 'OPENGL') perform a mutable
operation on the native peer - which is illegal so to speak.

Your 'OPENGL' thread is a custom animator based on AnimatorBase,
which does proper registration in the GLDrawable as a AnimatorCtrl.

In GLCanvas we actually try to avoid calling 'display()' if such animator
is running: GLDrawableHelper.isExternalAnimatorAnimating().

Something in this area does not work obviously, cont. finding the culprit.

~Sven

Thanks for looking into this.

Another thing to note fromt the test code I posted is that you can disable the use of the animator (set useAnimator to false), so that the frame rendering is triggered by canvas.display(). In this case the application also seems to deadlock, and the draw() method is called from the EDT, i.e.; thread "AWT-EventQueue-0,6,main".

On 11/01/2012 07:25 PM, ac [via jogamp] wrote:
> Thanks for looking into this.
>
> Another thing to note fromt the test code I posted is that you can disable the
> use of the animator (set useAnimator to false), so that the frame rendering is
> triggered by canvas.display(). In this case the application also seems to
> deadlock, and the draw() method is called from the EDT, i.e.; thread
> "AWT-EventQueue-0,6,main".

I already can confirm that this is not a regression per se,
but the deadlock on OSX is due to the newly added functionality
of using the new offscreen CALayer FBO.

Updating a similar unit test using the AWT component Applet. The latter
triggers the OSX offscreen CALayer usage.

Will update later today or tomorrow.

~Sven