Packaging JOGL projects to be cross-platform?

Hey all,

I'm looking to find a way to package up Processing projects (which use JOGL and gluegen) as a single executable jar. Processing only gives you the natives for the current system you're on (in my case, gluegen-rt-natives-windows-amd64.jar and jogl-all-natives-windows-amd64.jar), but I want to be able to send my projects to people on mac or linux machines as well. To that end, I downloaded the natives as described on this page, and I copied all of the native jars into another directory for easy access.

To create a single jar, I've been trying to use JarSplice: for those not familiar, JarSplice simply takes multiple jars, extracts the files from each, and adds those files to a new jar to create a "fat jar" that contains everything in one file. I've tried simply adding the jars exported by Processing as well as the jars I copied into the directory I just mentioned, hoping that would cause the natives for every system to be included.

However, when I run JarSplice, I get this error: duplicate entry: libgluegen-rt.so. A little more investigation shows why this happens: all of the native JOGL and gluegen jars for a particular system contain files with the same name. For example, the native jars for linux-amd64 contain files with the same name as linux-armv6, etc.

So I know why this is happening, but is there a better approach to creating a cross-platform application that uses JOGL?

Please take a look at: http://jogamp.org/wiki/index.php/JogAmp_JAR_File_Handling
you need to place the natives inside a folder named natives with a subfolder for the os.and.arch:

If you want to use fat-jar you need to use the following jar layout.
 /com/lala1/Lala1.class
 /com/lala2/Lala2.class
 /natives/<os.and.arch>/libLala1.so
 /natives/<os.and.arch>/libLala2.so

We have some test scripts inside gluegen that we use to test this fat-jar functionality
gluegen/test/TestMultiAndFatJar
http://jogamp.org/git/?p=gluegen.git;a=tree;f=test/TestMultiAndFatJar;hb=HEAD


It is also possible to bundle all JogAmp jars inside the main jar if your classloader supports jar-in-jar resources. jar-in-jar is known to work using the Eclipse
"export as runnable jar"  -> "         + Library handling:
           Package required libraries into generated JAR"
deployment option.

cheers
Xerxes

Hi

I advise you to read my comment:
https://jogamp.org/bugzilla/show_bug.cgi?id=845#c4

The biggest problem is that the default archiver may open your JAR instead of letting Java run it.

There are several solutions to package JOGL projects:
- applets (they won't work when NPAPI stops being supported in the web browsers, they're dying, JDT is already blacklisted and you have the same constraint than Java Webstart for the certificate)
- Java Webstart / Icedtea Webstart (fast, simple but now you have to buy a "trusted" certificate, Oracle often modifies its security requirements which drives it very difficult to run anything without getting tons of warnings and you need to do something under Mac to allow applications from Internet)
- self-contained applications with native installers (you need a different native installer per OS, you can bundle the JRE with your application)
- other cross-platform installers: IzPack, GetDown, ...

Good luck.

Edit.: I already talked about Gatekeeper under Mac here:
http://forum.jogamp.org/JNLP-apps-crash-on-OS-X-10-8-Mountain-Lion-tp4025732p4026647.html
http://forum.jogamp.org/ZipException-classpath-problem-with-jocl-trying-to-run-HelloJOCL-tp4030658p4030666.html

Gatekeeper with Java is explained here:
https://blogs.oracle.com/talkingjavadeployment/entry/java_applications_and_gatekeeper

Thanks for the replies.

Xerxes- So I can take the natives (not the native jars, but the files inside them) and then put them in the correct folder? For example, I extract the files in gluegen-rt-natives-linux-amd64 into /natives/linux-amd64/ and so on? That'll be the next thing I try.

gouessej- You raise some valid points. I'm actually trying to get away from applets and webstarts, since Java update 51 breaks them by default. I'm trying to come up with a process that novices can follow (I run a website geared towards helping novice programmers deploy Java and Processing games). I'm also looking into JWrapper, but I just really like how JarSplice handles natives and everything. It's just unfortunate that JOGL's natives are all named the same thing, otherwise I could just package them all up and let JarSplice handle it!

You can use an Ant script to make your fat JAR, it would be very easy to reuse, I indicated which tasks to use to do that in my comment in the bug report. Red Line RPM can be used to create RPM for GNU Linux even under Windows. I don't know whether JarSplice still supports .app files for Mac. In my humble opinion, JogAmp can't support all tools that may have contradictory requirements.

I agree, you guys can't support every use of JOGL. I'm actually more annoyed at Processing for hiding a lot of stuff from the users, and from what I'm learning about JOGL, you guys have done some really clever stuff. Anyway, I think I have it figured out using Xerxes's approach of having a natives/os.and.arch/ directory inside the fat jar that holds the native libs for each system.

That seems to work on my system, but I'm not sure how it works on other systems. For example, for the mac natives, should I have a natives/macosx-universal/ directory inside my fat jar, following the naming conventions I've seen so far, or is it looking for natives/ppc/ as described at this link?

I agree, you guys can't support every use of JOGL. I'm actually more annoyed at Processing for hiding a lot of stuff from the users, and from what I'm learning about JOGL, you guys have done some really clever stuff. Anyway, I think I have it figured out using Xerxes's approach of having a natives/os.and.arch/ directory inside the fat jar that holds the native libs for each system.

That seems to work on my system, but I'm not sure how it works on other systems. For example, for the mac natives, should I have a natives/macosx-universal/ directory inside my fat jar, following the naming conventions I've seen so far, or is it looking for natives/ppc/ as described at this link?

macosx-universal is correct, the name is JogAmp gluegen specific and composed by the Platform.getOSAndArch function.

http://jogamp.org/git/?p=gluegen.git;a=blob;f=src/java/jogamp/common/os/PlatformPropsImpl.java;hb=HEAD#l373

Use the following names, the rule is that the folder name should match the * name of the unpacked natives-*.jar
This is the list of os.and.arch natives names that JogAmp currently provide builds for:
natives/android-armv6/
natives/linux-amd64/
natives/linux-armv6/
natives/linux-armv6hf/
natives/linux-i586/
natives/macosx-universal/
natives/solaris-amd64/
natives/solaris-i586/
natives/windows-amd64/
natives/windows-i586/


This is the full list that JogAmp gluegen know about:
natives/freebsd-i586
natives/freebsd-amd64
natives/hpux-hppa
natives/linux-amd64
natives/linux-ia64
natives/linux-i586
natives/linux-armv6
natives/linux-armv6hf
natives/android-armv6
natives/macosx-universal
natives/solaris-sparc
natives/solaris-sparcv9
natives/solaris-amd64
natives/solaris-i586
natives/windows-amd64
natives/windows-i586

Cheers
Xerxes

On 01/22/2014 01:43 PM, Xerxes Rånby [via jogamp] wrote: And Fat-Jar here ..
  <https://jogamp.org/wiki/index.php/JogAmp_JAR_File_Handling#Fat-Jar>

General:
  <https://jogamp.org/wiki/index.php/JogAmp_JAR_File_Handling>
  <https://jogamp.org/jogl/doc/deployment/JOGL-DEPLOYMENT.html>

Note: Deployment docs are updated ..

Thanks Sven, the documentation is very good.

Thanks guys, this is exactly what I was looking for. Keep up the awesome work, the more I learn about this project the more impressed I am!

Kevin, would you be interested in using Ant or any build tool to make your fat JARs? Do you think Ant is a bit too much for the beginners?

Compared to using JarSplice (and the jar I just put together that contains the JOGL natives and other Processing-centric libraries), I would say that ant is much more complicated, especially for beginners who are accustomed to just pressing a play button and seeing their code work.

Processing is great because it lets novices and non-programmers see their code in action very easily, but how much they hide makes it much harder to deal with deployment. Most users don't even know they're using JOGL (I had to look it up) or native libraries, which is made doubly worse by the fact that Processing only exports the natives for the current system (so compiling on Windows means Mac users can't use your program). I think I finally have it all figured out and I've come up with a workaround (packaging all of the JOGL natives into a single jar and telling the kids to add that to JarSplice), thanks to you guys.

The nice thing about JarSplice is that it automatically extracts the natives from the fat jar to a temp directory before running the actual program, then deletes them when the program exits. I gather that JOGL does something very similar, but JarSplice is a nice one-size-fits-all solution for people who are using natives other than the JOGL ones. It makes it easier for me to tell everyone to use JarSplice than it would be to write specific instructions for each library kids might be using.

That being said, I am flirting with the idea of creating my own GUI that takes as input a set of Jars, external files, natives, etc. and exports a runnable jar and packaged executables, maybe by using JWrapper, and maybe using ant internally. That's a bit down the road though, as right now I'm just trying to come up with general workarounds now that Java update 51 has made applets and webstarts less of an option.

I'm still figuring all of this out as I go, so I'm very open to suggestions!

Hi

You really need a GUI for kids, whatever you use behind the hood. I have already planned to add a similar feature into JFPSM. I suggest you to create your own GUI and to use Ant internally. I can write several Ant macros (under GPL v2 like for TUER and JFPSM) to make fat JARs and you can reuse them in your own project. You can use JWrapper if you want, it's free of charge but neither free nor open source, it's a good solution in the meantime. I can write the first macro very soon, probably before the end of the week.

I have started planning for packaging my latest hobby game project. I was thinking about the fat jar approach compared to what I usually do and pack a ZIP-file with a native launcher alongside readme, license, lib-directory and so on. Sometimes using http://launch4j.sourceforge.net/ to create a native launcher (mainly for windows).

I can see that having the user unpack a zip is an extra step on their part and that can be a problem, but are there any other advantages to the fat jar approach?

I already said that a JAR can be opened as a ZIP by the default archiver. In my humble opinion, the "best" option consists in forcing Oracle to accept free of charge CACert certificates so that we can go on using Java Webstart despite the security concerns (or use JiGong Web Plugin?). I'm just open minded enough to help someone else even though I think the fat jar approach isn't the best.

Several JogAmp contributors could buy a trusted certificate and use it together, it would be less expensive.

Yeah, a GUI would definitely be optimal, I'm just trying to come up with a nice one-size-fits-all solution that handles Processing/JOGL, Java/other natives, LWJGL, libGDX, external files, etc etc etc. If I could build my own fat jars, that would be a step in the right direction, especially if I could automatically extract the external files and natives to a temp directory.

I've seen the problem with jar files being opened by the zip handler, but IMHO that's an easy enough fix (right click, open with...). I like the idea of jars because they seem more "light weight" than any other option, except of course webstarts and jnlp! But then again, fat jars can't handle non-JOGL natives or external files, so that puts me back to the JWrapper option (which I'm currently struggling with, JOGL doesn't seem to like being in a JWrapper application).

I briefly considered purchasing a certificate and letting the users of my site use that for free, but the problem with that is all it takes is one person to "accidentally" do something bad, and then the certificate for *everybody* is revoked.

Sorry to bother you guys again, but I want to make sure I understand: do the JOGL natives have to be in the same jar as the main class? Or can they be in another jar that's on the classpath?

I have one jar that simply contains the main class, and then a fat jar that contains all of the Processing classpath (including JOGL), as well as all of the natives as discussed above.

If I create a fat jar out of the main jar and the classpath jar, everything works fine.

However, if I try to simply use the classpath jar as a classpath member (using java -classpath=Main.jar;Classpath.jar Main), I get an UnsatisfiedLinkError.

I was hoping that JOGL would automatically extract itself from any jar on the classpath (especially since the jar contains both the JOGL libraries and the JOGL natives), but that doesn't seem to be the case. Am I missing something?

On 01/24/2014 01:53 AM, KevinWorkman [via jogamp] wrote:
> Sorry to bother you guys again, but I want to make sure I understand: do the
> JOGL natives have to be in the same jar as the main class?
No.
> Or can they be in
> another jar that's on the classpath?
Yes. (All our applet tests work that way)

>
> I have one jar that simply contains the main class, and then a fat jar that
> contains all of the Processing classpath (including JOGL), as well as all of
> the natives as discussed above.
Whatever pleases you.

Reasoning for us to not use a fat-jar,
is that we like to save some little bandwidth .. thats all.

You may have noticed the *-all* naming in the jar files and the
'atomic' subfolder, they are a relict of a very fine grained deployment.
Later we found the 'reasonable' subset for the jar file (we hope),
i.e. nativewindow, jogl and newt - for desktop and mobile.

>
> If I create a fat jar out of the main jar and the classpath jar, everything
> works fine.
Good.

>
> However, if I try to simply use the classpath jar as a classpath member (using
> java -classpath=Main.jar;Classpath.jar Main), I get an UnsatisfiedLinkError.
>
> I was hoping that JOGL would automatically extract itself from any jar on the
> classpath (especially since the jar contains both the JOGL libraries and the
> JOGL natives), but that doesn't seem to be the case. Am I missing something?

I just tested this case .. it [still] works, see:
  <http://jogamp.org/git/?p=gluegen.git;a=commit;h=2868816971bb80eb226e6edffc527d909ced755b>

~Sven

Thank you, that's actually really helpful. I think I'm doing pretty much the same thing as you, just on Windows. I've created a Test.jar that contains my main class, a ProcessingClasspath.jar that contains all the Processing and JOGL classes, and then ProcessingNatives.jar that contains all of the natives. However, when I run the command, I get an UnsatisfiedLinkError. Here are the contents of each of my jars (I left out ProcessingClasspath.jar because it's huge). Strangely, if I create a fat jar out of all of these jars, it works.

Does anything obvious stick out below?



C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeTest>
C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeTest>jar -tf Test.jar
META-INF/MANIFEST.MF
Test.class

C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeTest>jar -tf ProcessingNatives.j
ar
META-INF/
META-INF/MANIFEST.MF
natives/
natives/linux-amd64/
natives/linux-amd64/libgluegen-rt.so
natives/linux-amd64/libjogl_desktop.so
natives/linux-amd64/libjogl_mobile.so
natives/linux-amd64/libnativewindow_awt.so
natives/linux-amd64/libnativewindow_x11.so
natives/linux-amd64/libnewt.so
natives/linux-armv6hf/
natives/linux-armv6hf/libgluegen-rt.so
natives/linux-armv6hf/libjogl_desktop.so
natives/linux-armv6hf/libjogl_mobile.so
natives/linux-armv6hf/libnativewindow_awt.so
natives/linux-armv6hf/libnativewindow_x11.so
natives/linux-armv6hf/libnewt.so
natives/linux-i586/
natives/linux-i586/libgluegen-rt.so
natives/linux-i586/libjogl_desktop.so
natives/linux-i586/libjogl_mobile.so
natives/linux-i586/libnativewindow_awt.so
natives/linux-i586/libnativewindow_x11.so
natives/linux-i586/libnewt.so
natives/macosx-universal/
natives/macosx-universal/libgluegen-rt.jnilib
natives/macosx-universal/libjogl_desktop.jnilib
natives/macosx-universal/libjogl_mobile.jnilib
natives/macosx-universal/libnativewindow_awt.jnilib
natives/macosx-universal/libnativewindow_macosx.jnilib
natives/macosx-universal/libnewt.jnilib
natives/windows-amd64/
natives/windows-amd64/gluegen-rt.dll
natives/windows-amd64/jogl_desktop.dll
natives/windows-amd64/jogl_mobile.dll
natives/windows-amd64/nativewindow_awt.dll
natives/windows-amd64/nativewindow_win32.dll
natives/windows-amd64/newt.dll
natives/windows-i586/
natives/windows-i586/gluegen-rt.dll
natives/windows-i586/jogl_desktop.dll
natives/windows-i586/jogl_mobile.dll
natives/windows-i586/nativewindow_awt.dll
natives/windows-i586/nativewindow_win32.dll
natives/windows-i586/newt.dll

C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeTest>java -cp Test.jar;Processin
gNatives.jar;ProcessingClasspath.jar Test
Catched FileNotFoundException: C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeT
est\ProcessingClasspath-natives-windows-amd64.jar (The system cannot find the fi
le specified), while addNativeJarLibsImpl(classFromJavaJar class com.jogamp.comm
on.os.Platform, classJarURI jar:file:/C:/Users/Kevin/Desktop/JWrapper/Processing
NativeTest/ProcessingClasspath.jar!/com/jogamp/common/os/Platform.class, nativeJ
arBaseName ProcessingClasspath-natives-windows-amd64.jar): [ file:/C:/Users/Kevi
n/Desktop/JWrapper/ProcessingNativeTest/ProcessingClasspath.jar -> file:/C:/User
s/Kevin/Desktop/JWrapper/ProcessingNativeTest/ ] + ProcessingClasspath-natives-w
indows-amd64.jar -> slim: jar:file:/C:/Users/Kevin/Desktop/JWrapper/ProcessingNa
tiveTest/ProcessingClasspath-natives-windows-amd64.jar!/
Exception in thread "Animation Thread" java.lang.UnsatisfiedLinkError: Can't loa
d library: C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeTest\gluegen-rt.dll
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
        at java.lang.Runtime.load0(Runtime.java:795)
        at java.lang.System.load(System.java:1062)
        at com.jogamp.common.jvm.JNILibLoaderBase.loadLibraryInternal(JNILibLoad
erBase.java:551)
        at com.jogamp.common.jvm.JNILibLoaderBase.access$000(JNILibLoaderBase.ja
va:64)
        at com.jogamp.common.jvm.JNILibLoaderBase$DefaultAction.loadLibrary(JNIL
ibLoaderBase.java:96)
        at com.jogamp.common.jvm.JNILibLoaderBase.loadLibrary(JNILibLoaderBase.j
ava:414)
        at com.jogamp.common.os.DynamicLibraryBundle$GlueJNILibLoader.loadLibrar
y(DynamicLibraryBundle.java:388)
        at com.jogamp.common.os.Platform$1.run(Platform.java:203)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.jogamp.common.os.Platform.<clinit>(Platform.java:173)
        at javax.media.opengl.GLProfile.<clinit>(GLProfile.java:82)
        at processing.opengl.PJOGL.initSurface(PJOGL.java:238)
        at processing.opengl.PGraphicsOpenGL.initPrimary(PGraphicsOpenGL.java:59
88)
        at processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:16
00)
        at processing.core.PApplet.run(PApplet.java:2177)
        at java.lang.Thread.run(Thread.java:744)

C:\Users\Kevin\Desktop\JWrapper\ProcessingNativeTest>

On 01/24/2014 03:55 AM, KevinWorkman [via jogamp] wrote:
> Thank you, that's actually really helpful. I think I'm doing pretty much the
> same thing as you, just on Windows. I've created a Test.jar that contains my
> main class, a ProcessingClasspath.jar that contains all the Processing and
> JOGL classes, and then ProcessingNatives.jar that contains all of the natives.
> However, when I run the command, I get an UnsatisfiedLinkError. Here are the
> contents of each of my jars (I left out ProcessingClasspath.jar because it's
> huge). Strangely, if I create a fat jar out of all of these jars, it works.
>
> Does anything obvious stick out below?

Sorry .. I got it ..

That is a config we do _not_ support [yet],
i.e. you have 3 jar archives:

  - ProcessingClasspath.jar [gluegen, jogl, processing classes]
  - ProcessingNatives.jar   [gluegen, jogl, processing native libs]
  - Test.jar                [user stuff]

Our 'fat' approach assumes the native libraries
in the same jar file of the classes for gluegen and jogl, i.e.:

  - jogl-fat.jar            [gluegen, jogl, classes and native-libs]
  - Test.jar                [user stuff]

or in your case (if possible):

  - ProcessingFat.jar       [gluegen, jogl, processing classes and native-libs]
  - Test.jar                [user stuff]

If you cannot do the ProcessingFat.jar,
stuff gluegen's and jogl's classes into your ProcessingNatives.jar.

See:

<http://jogamp.org/git/?p=gluegen.git;a=blob;f=test/TestMultiAndFatJar/make-fat_and_multi.sh;h=de031be231c51153f44f3b9535a0e145e5f3cef9;hb=2868816971bb80eb226e6edffc527d909ced755b>

Problem is that we need to find the native libs
in relation to the gluegen-rt or jogl-all jar URI
and also validate their signatures if a security manager is enabled.

We would need to introduce a 3rd way of finding your 'ProcessingNatives.jar',
which is not [yet] implemented, i.e. via '-natives-fat' postfix analogue
to '-natives-linux-amd64' etc.

It is possible that we include this, if this path is really necessary.
Maybe you can contribute this yourself ?
At least we should add junit test for all these configurations ..

~Sven