Some issues using NEWT

Hello guys,

I'm currently replacing the use of AWT's GLCanvas/GLDrawable in Processing with NEWT's window and canvas. Results are quite promising so far, as I managed to get the rendering loop running within its own independent thread, fullscreen support working on Linux, etc.

However, I also encountered some issues, affecting particularly MacOSX. Since Processing's renderer is fairly large, I created a small self-contained demo application that reproduces the issues I see in Processing, and also mirrors the basic structure of the Processing renderer.

The code of the application is here: http://pastebin.com/H4pb3kik

It is organized as an Applet contained inside a Frame, and uses a custom animator to render frames on demand. The GLWindow and NewtCanvasAWT canvas are created in a setup method that allows to hook the canvas either to the root frame or the applet object. I'm testing these two alternatives because each one has different issues and I haven't managed yet to get an initialization procedure that works 100%. So the issues are the following (on MacOSX Snow Leopard and Lion, behavior on Windows and Linux is slightly different but let's focus on Mac for the time being):

1) Hooking the canvas to the frame:
frame.add(canvas, frame.getComponentCount() - 1);

* window looses decorations on OSX
* is not logically consistent: frame contains both applet and canvas as child components, but it should be frame -> applet -> canvas, shouldn't it?

2) Hooking the canvas to the applet:
applet.add(canvas, this.getComponentCount() - 1);

* target framerate (120 in the posted code) is not honored
* gives invalid drawable error one time at the beginning

I'm running all my tests with a recent autobuild package of jogl (post rc5). I'm also testing event handling, and have some questions about it as well, but will leave them for later.

So my basic question at this point is: what would be the correct setup procedure for NEWT that works within this framework (Frame->Applet)? Since Processing's Applet class is basically a heavyweight AWT component, I'm wondering if issue-free integration of NEWT is possible at all. Maybe we need to stick with AWT's GLCanvas? Please let me know your thoughts, comments, etc.

Any help will be greatly appreciated!

On 02/19/2012 12:51 AM, ac [via jogamp] wrote:
>
>
> Hello guys,
>
> I'm currently replacing the use of AWT's GLCanvas/GLDrawable in Processing
> with NEWT's window and canvas. Results are quite promising so far, as I
> managed to get the rendering loop running within its own independent thread,
> fullscreen support working on Linux, etc.

great.
Since I will head down south now (bedtime), allow me to just link you
to the Applet magic which uses NewtCanvasAWT and hooks up any GLEventListener:

<http://jogamp.org/git/?p=jogl.git;a=tree;f=src/newt/classes/com/jogamp/newt/awt/applet;hb=HEAD>

<http://jogamp.org/git/?p=jogl.git;a=blob;f=src/newt/classes/com/jogamp/newt/awt/applet/JOGLNewtApplet1Run.java;hb=HEAD>

This works well on any OSX version so far (latest master, upcoming RC6)
and ofc the other platforms.
RC5 tests of the above @
<http://jogamp.org/deployment/jogamp-current/jogl-test-applets.html>, but be
aware that not all OSX fixes are in RC5 yet.

I know your question is more complex, so I will answer and test it later on.
Maybe I can even adobt your code to one of our unit tests ..

>
> Any help will be greatly appreciated!

sorry .. later more ...

~Sven

> Since I will head down south now (bedtime), allow me to just link you
> to the Applet magic which uses NewtCanvasAWT and hooks up any GLEventListener:
> <http://jogamp.org/git/?p=jogl.git;a=tree;f=src/newt/classes/com/jogamp/newt/awt/applet;hb=HEAD> 
> <http://jogamp.org/git/?p=jogl.git;a=blob;f=src/newt/classes/com/jogamp/newt/awt/applet/JOGLNewtApplet1Run.java;hb=HEAD> 

Thanks for your reply. I see that in both cases the Newt canvas is added to the Applet object, so in this sense it it similar to what I do. However, the problem is that in Processing the applet is further contained within a frame. Perhaps this is causing the framerate issues...

> This works well on any OSX version so far (latest master, upcoming RC6)
> and ofc the other platforms.

Ok, I will run my tests again using the master.

I run my test case again using the jogl and gluegen autobuilds from today. I realized that the problem with the missing window decorations was that I had to set the bounds and location of the canvas:

canvas.setBounds(0, 0, 300, 300);
insets = frame.getInsets();
canvas.setLocation(insets.left, insets.top);  

Here is the corrected code: http://pastebin.com/DUthqZrC

So, adding the canvas to the frame works fine now (I still need to figure out the event handling, but that's a separate thing). However, I wonder why the framerate doesn't work properly when adding the frame to the applet instead (in the sample code, the framerate is set to 120, which is honored when adding the canvas to the frame, but not when adding it to the applet). Is this a expected behavior, or a possible bug in NEWT? I ask because this framerate issue appears only on OSX, but not on Linux or Windows.

Why do you still use a Frame? Is it necessary? Isn't it possible to use directly a GLWindow inside an applet?

On 02/20/2012 03:31 AM, ac [via jogamp] wrote: Awesome, thank you for saving my time looking into it.
Again: Hooking NEWT to AWT up, we have lots of unit test doing so,
and the mentioned NEWT/Applet container code.

> However, I wonder why
> the framerate doesn't work properly when adding the frame to the applet
> instead (in the sample code, the framerate is set to 120, which is honored
> when adding the canvas to the frame, but not when adding it to the applet).
> Is this a expected behavior, or a possible bug in NEWT? I ask because this
> framerate issue appears only on OSX, but not on Linux or Windows.

Right .. sorry.

This is because of using the OSX CALayer (offscreen animation layer),
which has it's own static refresh cycle.

We need to tune this code path (if possible), eg. hand 'down' the fps values.
Pls create a bug report - if you don't mind.

~Sven

Processing basic architecture is based on an AWT applet embedded into a frame. It would be difficult to change this without a major refactoring of the code... so, if it is possible to make the NEWT canvas to work within the applet, it would be great.

Ok, here is the bug report:

 https://jogamp.org/bugzilla/show_bug.cgi?id=555

Another question. I realized that I can create my GLWindow using a native window as argument in addition to the gl capabilities. Something like this:

AWTGraphicsScreen screen = (AWTGraphicsScreen)AWTGraphicsScreen.createDefault();
AWTGraphicsConfiguration config = (AWTGraphicsConfiguration)GraphicsConfigurationFactory
        .getFactory(AWTGraphicsDevice.class).chooseGraphicsConfiguration(capabilities, capabilities, null, screen);
NativeWindow natWin = NativeWindowFactory.getNativeWindow(applet, config);  
   
window = GLWindow.create(natWin, capabilities);
canvas = new NewtCanvasAWT(window);

where I pass the parent applet objet to the getNativeWindow() method. Is this recommended, not recommended, or same as creating the GLWindow passing only the capablilities?

On 02/20/2012 04:08 PM, ac [via jogamp] wrote: This cstr is avail to allow native parenting,
and actually exactly what NewtCanvasAWT does plus all the
required event handling etc.

> canvas = new NewtCanvasAWT(window);
>
> where I pass the parent applet objet to the getNativeWindow() method. Is
> this recommended, not recommended, or same as creating the GLWindow passing
> only the capablilities?

So for the 'NewtCanvasAWT' case, you just shall use it as advertised.
As described above, it handles the native AWT parenting and event translation
itself.

~Sven

> This is because of using the OSX CALayer (offscreen animation layer),
> which has it's own static refresh cycle.
>
> We need to tune this code path (if possible), eg. hand 'down' the fps values.
> Pls create a bug report - if you don't mind.

Mostly out of curiosity (since I don't think I have enough knowledge to tackle this bug myself), what is the source code file in jogl where the framerate of the OSX CALayer could be controlled? At least I'd like to take a look, maybe some idea or suggestion will come up :-)

On 02/25/2012 01:21 AM, ac [via jogamp] wrote: It's all in one file, pretty much isolated and orthogonal,
so possible to fix it w/o overall system knowledge.

Setup here:

<http://jogamp.org/git/?p=jogl.git;a=blob;f=src/jogl/native/macosx/MacOSXWindowSystemInterface-pbuffer.m;hb=HEAD#l127>

A note:

<http://jogamp.org/git/?p=jogl.git;a=blob;f=src/jogl/native/macosx/MacOSXWindowSystemInterface-pbuffer.m;hb=HEAD#l260>

SetSwapInterval (NOP with CALayer):

<http://jogamp.org/git/?p=jogl.git;a=blob;f=src/jogl/native/macosx/MacOSXWindowSystemInterface-pbuffer.m;hb=HEAD#l360>

If you know a way how to control the swap interval at least,
that would be great.

Technically the JOGL 'user' code provides a texture from
rendering into the pbuffer.

The CALayer renders the textures into
whatever the 'sink' is from it's own animation thread.

If we would have at least a vsync (swap interval),
we could sync rendering properly.

It's highly likely that I am missing something here.

~Sven

Thanks for the source links.

I managed to build jogl and played around a bit with the code in MacOSXWindowSystemInterface-pbuffer.m

As it is indicated in the comments, the [context setValues: &interval forParameter: NSOpenGLCPSwapInterval] call doesn't have any effect. I tried placing it in various other places (before making context current, in the setSwapInterval method) but didn't appear to change anything.

So at this point I don't have any specific suggestions, I will let you know if I think of anything else.

On 02/25/2012 06:24 AM, ac [via jogamp] wrote:
>
>
> Thanks for the source links.

.. a bit off-topic ..

but we (Xerxes and myself) currently try your new processing 2.0 library
on linux arm ..

and it would be lovely if you could adapt to our new JAR and native-JAR
scheme ..

Then one could just drop our original JAR files .. and thats it.

Would be great.

~Sven

Hey guys, I was away working on other projects so couldn't go back to this thread until now.

I found a way of removing the framerate capping that occurs on OSX when adding the canvas object to an AWT applet.

I realized that in the swapBuffers() method from the NSOpenGLImpl class (MacOSXCGLContext.java), there is this call:

CGL.waitUntilNSOpenGLLayerIsReady(nsOpenGLLayer, 16);

where the timeout to trigger a new draw is hardcoded to 16ms, which corresponds to a refresh rate of 60Mhz.

Even if the swap interval is set to 0 in order to disable v-sync, the refresh rate continues to be dictated by waitUntilNSOpenGLLayerIsReady() and its timeout argument. By changing the timeout to a smaller value, say 4, I can finally set my own higher framerates from the application. I eventually added some additional changes to the setSwapInterval() method also in NSOpenGLImpl, so that the correct v-sync timeout value is chosen depending on the swap interval. I will upload the patch to the bug report I opened earlier, with some further suggestions.

I will look at jogl on Android in the next few days.

Andres

Thank you so much for the tip, really :)

You are welcome. Please check the updated bug entry for the patch details:

https://jogamp.org/bugzilla/show_bug.cgi?id=555

On 03/15/2012 08:17 PM, ac [via jogamp] wrote: Well, I can't remember exactly what I was doing there
but it 'feels' like the 'wait' was 'waiting' for the
actual OSX layer animator to have the offscreen image rendered.

>
> where the timeout to trigger a new draw is hardcoded to 16ms, which
> corresponds to a refresh rate of 60Mhz.

yup ..
great will look into it soon .. when I switch my brain back into OSX mode :)

> I will look at jogl on Android in the next few days.
>
> Andres

thank you

~Sven

Going back to the Android suggestion, I looked briefly at the NewtBaseActivity class and some examples like NEWTRedSquareES2Activity.

The initialization code looks very compact thanks to the GLWindow and GLCapabilties objects. However, it also seems to me that in order to use jogl-android in Processing we'd probably need to do some fair amount of rewriting since Processing on Android is currently based on the SDK's Activity class plus our own customizations for input, etc, and uses a GLSurfaceView to handle the OpenGL surface.

Since we already got this basic stuff working and are accessing GLES2 through the native Android bindings, I'd like to know what would be the added benefits of using the JOGL bindings instead. I can imagine that having the same API on both desktop and Android is one, but we already wrote a Processing-GL abstraction layer that encapsulates the platform-specific APIs. On the negative side, Android applications would need to be bundled with additional files (jogl.all-android.apk, jogl.all-android.jar...? not exactly sure wich ones are needed).

On 03/22/2012 05:38 AM, ac [via jogamp] wrote:
>
>
> Going back to the Android suggestion, I looked briefly at the
> NewtBaseActivity class and some examples like NEWTRedSquareES2Activity.
>
> The initialization code looks very compact thanks to the GLWindow and
> GLCapabilties objects.

Assuming you refer to the latest ..

<http://jogamp.org/git/?p=gluegen.git;a=commit;h=0cfc7847c58b51c9a26b50d905b592d1fc4c8578>

yes, it's seamless using NEWT w/ Android, i.e. target to require no code change
of your Java/JOGL application.
Yes, sure. Plus soon we will enable unit testing as well etc.
So the big points for JOGL/Android IMHO is:
  - lower costs of cross-platform development and maintenance
    due to reusing the same code path
   
  - quality
    - maturing our common Android binding instead
      of every app on their own

    - benefit from our [cross platform] unit tests

> but we already wrote a
> Processing-GL abstraction layer that encapsulates the platform-specific
> APIs. On the negative side, Android applications would need to be bundled
> with additional files (jogl.all-android.apk, jogl.all-android.jar...? not
> exactly sure wich ones are needed).

As mentioned above [latest GlueGen/JOGL Android changes],

 [User:a1] -- (usr-data) --> [Launcher] -> [User:a2] + using [other packages..]
   
    [User APK]   - The user provided APK
    [JogAmp APK] - JogAmp APKs
   
    [User:a1]    - The initial user activity, which starts the [Launcher].
                   Providing data to [Launcher]: [User:a2], [User APK]
                   Resides in [User APK]
   
    [User:a2]    - The actual downstream 'real' activity, spoiled w/ full fledged ClassLoader
                   having access to all packages as requested, ie. [User APK], ..
                   Resides in [User APK]
   
    [Launcher]   - The launcher activity.
                   Gets called by [User:a1].
                   Creates a new ClassLoader, daisy chainging all requested APKs.
                   Instantiates [User:a2] w/ new ClassLoader.
                   Delegates all calls to [User:a2].
                   Resides in [JogAmp APK].

- you would only need to install the [JogAmp APK]s on the phone

- then you install your [User APK]

- then your 'dummy' [User:a1] activity kicks of your real activity [User:a2]
  while using [Launcher], which daisy chains ClassLoader and allows access to all
  pre installed JogAmp APKs ..


One thing is for sure here .. our Android binding is bleeding edge.
However, it's sort of 1st priority and we will work swiftly to fix bugs
and to enhance it, e.g. based on your requirements etc.

So I really would love to see Processing using JOGL/Android,
which would allow us to mature the binding ASAP.

So I guess the benefits are clear and the 'risks' as well.

Hope this helps a bit.

After RC6, I will try to push a 1st version to the market,
so it can be installed as usual.

Cheers, Sven

On 03/22/2012 09:46 AM, Sven Gothel wrote: To elaborate a bit more ..

<http://jogamp.org/git/?p=jogl.git;a=tree;f=src/test/com/jogamp/opengl/test/android;h=f3b2f2d71bc5b9407b8f4d4b80359d7b1c8ef1ef;hb=HEAD>

You see Demo1:
   [User:a2] NEWTGearsES2Activity.java     and [User:a1] NEWTGearsES2ActivityLauncher.java
and Demo2:    
   [User:a2] NEWTRedSquareES2Activity.java and [User:a1] NEWTRedSquareES2ActivityLauncher.java

Demo2 [User:a1] is kept very simple to demonstrate how easy you can launch the actual
activity with using JOGL:

<http://jogamp.org/git/?p=jogl.git;a=blob;f=src/test/com/jogamp/opengl/test/android/NEWTRedSquareES2ActivityLauncher.java;h=b6265b72bc221d295a16576e956b3022fb6618be;hb=HEAD>
Demo1 [User:a1]
  <http://jogamp.org/git/?p=jogl.git;a=blob;f=src/test/com/jogamp/opengl/test/android/NEWTGearsES2TransActivityLauncher.java;h=c29c0c5db6f7111cb61b541dae8620d12af6b291;hb=HEAD>

is using the helper class LauncherUtil.BaseActivityLauncher
  <http://jogamp.org/git/?p=jogl.git;a=blob;f=src/test/com/jogamp/opengl/test/android/LauncherUtil.java;h=f1f36589d0a84d8ce0e95061c5dfe15c572246b8;hb=HEAD#l70>

to generate the proper URI for passing properties easily.

Hope this helps.

~Sven