JOGL/JavaFX integration and the future

Oracle has just released the first version of JavaFX that includes proper 3D support.  This support is very rudimentary when compared to the full support offered by JOGL but it does raise a few questions.

First, I am interested in what can be done integrating JavaFX and JOGL.  JavaFX has many nice features that would be useful to integrate with JOGL such as advanced media support, hardware accelerated 2D rendering, WebKit, sophisticated binding etc.

Specifically I am thinking about what could be done by perhaps rendering a 2D scene in JavaFX and applying this scene in JOGL by using a texture on some 3D object or surface.  In particular, text rendering is problematic with OpenGL and JavaFX has good text rendering support.  Would it be possible/feasible for text and 2D shape rendering to be done in a offscreen buffer in JavaFX and then used in JOGL by using the "snapshot" feature of a JavaFX node?  I am concerned that this method would not be particularly performant but may have some benefits.

Also, the video and audio support in JavaFX is more advanced than that in JOGL e.g. there is no audio support in GLMediaPlayer yet.  Could the media player in JavaFX be used to render the video and then the current frame applied as a texture in JOGL?

The support for 3D functionality will continue to develop in JavaFX but will probably always lag behind that in JOGL.  For this reason I believe that JOGL will always have a place and, ideally, JOGL and JavaFX could be used together.  Have the JogAmp team considered this integration and the possible convergence of 3D features in these 2 products?

Thanks,

-Qu0ll

On 02/19/2013 07:33 AM, Qu0ll [via jogamp] wrote:
> Oracle has just released the first version of JavaFX that includes proper 3D
> support.  This support is very rudimentary when compared to the full support
> offered by JOGL but it does raise a few questions.

Can you pls provide a URL to the source code, assuming you refer to
the OpenJavaFX portion .. anything else might not be of interest.

>
> First, I am interested in what can be done integrating JavaFX and JOGL.
>  JavaFX has many nice features that would be useful to integrate with JOGL
> such as advanced media support, hardware accelerated 2D rendering, WebKit,
> sophisticated binding etc.

If those features are accessible w/o the need to use their
'renderer' .. sure .. why not ?

>
> Specifically I am thinking about what could be done by perhaps rendering a 2D
> scene in JavaFX and applying this scene in JOGL by using a texture on some 3D
> object or surface.  In particular, text rendering is problematic with OpenGL
> and JavaFX has good text rendering support.  Would it be possible/feasible for
> text and 2D shape rendering to be done in a offscreen buffer in JavaFX and
> then used in JOGL by using the "snapshot" feature of a JavaFX node?  I am
> concerned that this method would not be particularly performant but may have
> some benefits.

Well .. for sure it is possible to use offscreen image data
to swap back/forth .. however, this is hardly high performance.

>
> Also, the video and audio support in JavaFX is more advanced than that in JOGL
> e.g. there is no audio support in GLMediaPlayer yet.  Could the media player
> in JavaFX be used to render the video and then the current frame applied as a
> texture in JOGL?

Well, we 'only' miss the JOAL integration, which will be done
after some JOAL fixes have been made.

Further more, we like to bundle a libav pre-build
to have a higher availability on Windows/OSX.

I cannot speak about JavaFX.

>
> The support for 3D functionality will continue to develop in JavaFX but will
> probably always lag behind that in JOGL.  For this reason I believe that JOGL
> will always have a place and, ideally, JOGL and JavaFX could be used together.
>  Have the JogAmp team considered this integration and the possible convergence
> of 3D features in these 2 products?

The only product visible to me are some mercurial repos about OpenJavaFX.
Well, it's not us who hinder any migration, our sources are available
and even if a patch may take a while (.. week) .. it eventually gets merged.

So for OpenJavaFX I would be interested personally if a Maya importer is available
or indeed maybe WebKit. But for some reason I guess that
all the 'nice' modules are available in the proprietary version :)
If somebody in this forum is able to enlight us on the issue .. please go ahead.

>
> Thanks,
>
> -Qu0ll

Cheers.

~Sven

Hi

At first, there is already a request for enhancement about that:
https://jogamp.org/bugzilla/show_bug.cgi?id=607

Secondly, before speaking about supporting some kind of interoperability between JavaFX and JOGL 2.0, we have to think about the perenity of the former.

Thirdly, there is already some source code that can be ported to JOGL 2.0 in order to render some JavaFX thingy into a framebuffer object, a programmer spoke about that a few months ago on JavaGaming.org.

I really need to see which parts of JavaFX are available in OpenJFX and the limitations in terms of licensing. If we are not allowed to ship OpenJFX with any application on embedded devices, I won't invest any time in such a port. Free software is about freedom.

The previous backend based on JOGL 2.0 could be resurrected but is it worth a try? Several underlying components of JavaFX have the same role than some of the JogAmp stack, especially NEWT.

Hello,

I'm currently waiting for the remaining prism code of JavaFX 8 to be made open source (soon). IMHO the best way to go about this is to develop a JOGLGraphicsPipeline for JavaFX, then you can write a custom JavaFX node and use JOGL commands in the renderer thread. That would be cross platform (except for WinRT tablets without GL). I have experimented with this under OSX to integrate an IOSurface into a JavaFX application and it works without sacrificing performance, although a bit hacky yet.

I guess it would also be possible to use JavaFX to render into an offscreen surface and use it as a texture, but IMHO using JavaFX as your application framework (threading, events, properties, css, bindings etc are quite awesome) and using custom GL nodes is a better way.

I've chatted with Julien Gouesse a bit on JavaGaming but didn't have much time working on stuff since then. I have pushed my current highly experimental efforts to https://bitbucket.org/dejayberlin/joglfxpipeline if anyone cares to check it out. It's not working yet.



Currently I'm still stuck constructing the "Presentable" from an existing glass window:
public JOGLPresentable(PresentableState state, JOGLContext context)

The idea is you have an existing window from glass (PresentableState has a "WinView" and that has a "WinWindow" and all give the same long for getNativeHandle / getNativeView
and get a native view handle (long). You don't want jogl to handle window / input events, since glass is handling all that. All calls to the JOGLPipeline happen in the "QuantumRendererThread" which would become the GL thread and would be fine.
I assume the "state.getWindow().getNativeHandle()" the glass windows gives me is the HWND under windows. How would I go about creating a GLContext with a hardware swap chain or double buffer for this existing HWND? I guess since win/mac perform their own desktop composition a swap chain is fastest?


Do I create a child window? That seems like overhead.
com.jogamp.newt.NewtFactory.createWindow(view.getNativeView(), screen, new GLCapabilities(GLProfile.getDefault()));

Do I use the NativeWindowFactory and somehow create a GLSurface from that?
javax.media.nativewindow.NativeWindowFactory.getNativeWindow(winObj, config)

Do I use jogamp.nativewindow.WrappedSurface?
UpstreamSurfaceHookMutableSize upstream = new UpstreamSurfaceHookMutableSize(view.getWidth(), view.getHeight());
AbstractGraphicsConfiguration agc = new MutableGraphicsConfiguration(screen, null, null);
jogamp.nativewindow.WrappedSurface ws = new WrappedSurface(agc, view.getNativeView(), upstream, false);

Or should one create an offscreen surface and copy the rendered content into the window?
javax.media.opengl.GLDrawableFactory.getDesktopFactory().createOffscreenDrawable(device, caps, chooser, width, height);


The truth is I don't really know what I'm doing here since I'm a little overwhelmed with the many options =)


Thanks in advance,
Dejay

PS, about licensing, it should be fine for applications:
http://fxexperience.com/2013/02/february-open-source-update/#more-2496

Richard Bair wrote

There have been some question about licensing on iOS as Apple doesn’t allow for GPL licensed applications in their app store. OpenJFX and OpenJDK are both licensed with the same GPLv2 with Classpath Extension. My understanding (and I’m not a lawyer) is that this means that if you take OpenJFX + OpenJDK (minus any binary stubs released under a different license), then you can safely combine this with your application and release your application under your own license as a single application co-bundle. You can do the same with the GA release of JavaFX + JavaSE. The difference is that we do not yet have iOS / Android on our official release roadmap, so that for the time being, the only way to use JavaFX on iOS is via OpenJDK / OpenJFX. I can imagine this would make open source guys smile (open source goes where closed source does not!). This is also why we’ve been prioritizing open source so highly for the last little while — when OpenJFX can be built without any binary stubs, then there is nothing stopping you from taking OpenJFX with iOS port + OpenJDK and creating commercial applications that can be sold in the iOS app store.

On 02/25/2013 07:08 PM, Dejay [via jogamp] wrote:
> Hello,
>
<snip impl. questions/>

>
> PS, about licensing, it should be fine for applications:
> http://fxexperience.com/2013/02/february-open-source-update/#more-2496
>
>     Richard Bair wrote
>     My understanding (and I’m not a lawyer)

And this is the whole problem we have,
there is no clear statement about Oracle's source and binary license
towards the patent grant which they currently only grant for OpenJDK
on TCK approved builds!

Richard Blair's statement is a NULL statement ending right after his
disclaimer,since nobody really cares what he 'understands' personally.

Elaboration of this is available here:
  https://jogamp.org/bugzilla/show_bug.cgi?id=682

As long Oracle is not capable of publishing a legally binding statement
we all are able to understand properly, there is a lot of FUD going around.

The issue Oracle vs Google might be able to solve it,
however decisions in such cases lie very often in the very detail
and may not give small companies 'peace of mind'.

IMHO it would be very helpful if Oracle finally could turn around
and give a patent grant to all products based on OpenJDK
including mobile/embedded.
AFAIK, the latter is excluded in their binary license terms.

Many questions to be answered.

~Sven

On 02/25/2013 07:26 PM, Sven Gothel wrote: Not to forget: _And_ products build from the OpenJDK source code,
not having passed the TCK!

> AFAIK, the latter is excluded in their binary license terms.
>
> Many questions to be answered.
>
> ~Sven

On 02/25/2013 07:08 PM, Dejay [via jogamp] wrote: Here you would already sacrifice concurrency of UI and GL rendering,
i.e. the UI thread will block your GL renderer - e.g. by input events.
Well, this is a JavaFX design decision quite similar w/ AWT and one reason
we have NEWT in place. But well, everyone their own toy.

See http://jogamp.org/jogl/doc/NEWT-Overview.html

> I assume the "state.getWindow().getNativeHandle()" the glass windows gives me
> is the HWND under windows. How would I go about creating a GLContext with a
> hardware swap chain or double buffer for this existing HWND? I guess since
> win/mac perform their own desktop composition a swap chain is fastest?
>
>
> Do I create a child window? That seems like overhead.
> com.jogamp.newt.NewtFactory.createWindow(view.getNativeView(), screen, new
> GLCapabilities(GLProfile.getDefault()));

Using native parenting w/ NEWT would give you best performance
and it would follow our NewtCanvasAWT and NewtCanvasSWT pattern
of integrating such toolkits w/ NEWT.

Have a look into NewtCanvasSWT implementation,
which should be simple enough to allow a JFX native parenting solution.

This is one solution and I like it the most,
since it allows dropping a platform agnostic NEWT [GL]Window into
another toolkit. NEWT code and the GLEventListener don't need to change
and can be re-parented in and out of your alien toolkit.

>
> Do I use the NativeWindowFactory and somehow create a GLSurface from that?
> javax.media.nativewindow.NativeWindowFactory.getNativeWindow(winObj, config)

This currently supports only external AWT objects via JAWTWindow,
but we could extend it.

>
> Do I use jogamp.nativewindow.WrappedSurface?
> UpstreamSurfaceHookMutableSize upstream = new
> UpstreamSurfaceHookMutableSize(view.getWidth(), view.getHeight());
> AbstractGraphicsConfiguration agc = new MutableGraphicsConfiguration(screen,
> null, null);
> jogamp.nativewindow.WrappedSurface ws = new WrappedSurface(agc,
> view.getNativeView(), upstream, false);

Pls use GLDrawableFactory.createProxySurface(..) which already
decides whether to use WrappedSurface / GDISurface.
Have a look at our SWT GLCanvas code to see how to use it properly.

This is the solution to give the alien toolkit users (AWT, SWT, ..)
a way to stay within their world w/o touching NEWT etc.

Ofc you have to deal w/ negative side-effects like lack of UI concurrency
and toolkit dependencies (-> see NEWT approach above).

>
> Or should one create an offscreen surface and copy the rendered content into
> the window?
> javax.media.opengl.GLDrawableFactory.getDesktopFactory().createOffscreenDrawable(device,
> caps, chooser, width, height);

That would be another valid but slow way to do it.
Look in GLJPanel how we use it :)

>
>
> The truth is I don't really know what I'm doing here since I'm a little
> overwhelmed with the many options =)

Yup, I agree, we have a lot of options to migrate toolkits.
Nevertheless this flexibility allows you to have a choice.

~Sven

A developer using our competitor uses an FBO to render JavaFX widgets, it is viable but not optimal. Sven's suggestion is better but I would prefer us to provide a solution that wouldn't rely on any UI tool kit if possible. I'm already busy with tons of things but I know something has to be done for JavaFX.

Awesome, thank you very much for your help. I can glClear something :D

Btw Javafx is already multi threaded, e.g. render thread, animation thread, and application thread (all events get dispatched there). So doing stuff in response to events would not block rendering.

I've used the createProxySurface / createGLDrawable similar to swt.GLCanvas. The example greatly helped. Now I have to figure out how to render stuff :D

BTW if one only uses GLES2 for rendering, does the same code work on PC / android? (within capabilities)

And do any gl commands get dispatched into another thread if I only use makeCurrent and glCommands in a single rendering thread?

Cheers,
Dejay

gouessej wrote

A developer using our competitor uses an FBO to render JavaFX widgets, it is viable but not optimal. Sven's suggestion is better but I would prefer us to provide a solution that wouldn't rely on any UI tool kit if possible. I'm already busy with tons of things but I know something has to be done for JavaFX.

Is this the LWJGL solution from Spasi? One problem with LWJGL is that it doesn't support multiple screens / windows AFAIK. Jogl is much better for that. I personally even prefer the static gl methods of LWJGL, but for openGLES a GL object might be a great advantage.

I'm not sure what you mean by not relying on an UI toolkit? Do you mean you do not want to use JavaFX as an application framework, only to render UI textures? Just curious.

Dejay wrote

gouessej wrote

A developer using our competitor uses an FBO to render JavaFX widgets, it is viable but not optimal. Sven's suggestion is better but I would prefer us to provide a solution that wouldn't rely on any UI tool kit if possible. I'm already busy with tons of things but I know something has to be done for JavaFX.

Is this the LWJGL solution from Spasi? One problem with LWJGL is that it doesn't support multiple screens / windows AFAIK. Jogl is much better for that. I personally even prefer the static gl methods of LWJGL, but for openGLES a GL object might be a great advantage.

Yes this is Spasi's solution.

Dejay wrote

I'm not sure what you mean by not relying on an UI toolkit? Do you mean you do not want to use JavaFX as an application framework, only to render UI textures? Just curious.

I thought we could resurrect the old JOGL 1 backend and use only OpenGL calls without needing to use NEWT but you've already done a nice job.

On 02/25/2013 09:49 PM, Dejay [via jogamp] wrote:
> Awesome, thank you very much for your help. I can glClear something :D
>
> Btw Javafx is already multi threaded, e.g. render thread, animation thread,
> and application thread (all events get dispatched there). So doing stuff in
> response to events would not block rendering.
>
> I've used the createProxySurface / createGLDrawable similar to swt.GLCanvas.
> The example greatly helped. Now I have to figure out how to render stuff :D
Sweet.

>
> BTW if one only uses GLES2 for rendering, does the same code work on PC /
> android? (within capabilities)
Use GL2ES2, check our OpenGL UML diagram on the jogamp homepage
and see the many unit tests (*GearsES2* etc).

Using a common profile like GL2ES2 allows you to run your stuff on desktop
and mobile w/o any change, if toolkit is provided.
Due to the latter - I also strongly recommend to walk the NEWT path .. well :)
[Since we support NEWT on desktop and mobile.]

>
> And do any gl commands get dispatched into another thread if I only use
> makeCurrent and glCommands in a single rendering thread?
Nope.

The special thread dispatch is done in the AWT toolkit's GLCanvas
just to serve constraints of those .. it's not a constrain of JOGL.

~Sven

On 02/25/2013 11:38 PM, gouessej [via jogamp] wrote:

>     Dejay wrote
>     I'm not sure what you mean by not relying on an UI toolkit?
>
> I thought we could resurrect the old JOGL 1 backend and use only OpenGL calls
> without needing to use NEWT but you've already done a nice job.

You don't need to have NEWT or anything else, able to have simply a NativeSurface 'on the fly'.
However, this is not suitable for long term - better to have a class denominating
at least the alien toolkit's UI component. Regardless, it's just that NativeWindow
wrapper instance - quite simple.

~Sven