jogamp › jogl

ZipException when loading native jars

Classic

List

Threaded

35 messages Options

ryanmohr

ZipException when loading native jars

Here's the error that I'm getting:

ZipException: error in opening zip file, while TempJarCache.bootstrapNativeLib() of jar:file:/test.jar!/lib/gluegen-rt-natives-macosx-universal.jar!

Any ideas what would be causing this? The jar unzips fine when running under eclipse but the above error happens when trying to run as onejar.

Looks like it could be a permissions issue but you'd think eclipse wouldn't work then either.

gouessej

Re: ZipException when loading native jars

Administrator

Hi

JogAmp is not yet compatible with the "one single fat JAR approach" and it is clearly not the advised way of deploying applications based on JOGL 2.0. We have to fix a bug preventing the loading and the extraction of native libraries from JARs when these JARs are inside another JAR to make it work with onejar. I know lots of Mac developers love bundling their applications as Mac bundles with a single JAR but you should rather find a way of using several JARs to create a bundle or use any other working installer (Java Web Start, GetDown, IzPack, ...).

Julien Gouesse | Personal blog | Website

ryanmohr

Re: ZipException when loading native jars

The purpose of using onejar is to allow us to have a single file that can then be packaged as a native app / installer for any platform (wrapped into an app bundle and disk image for macs, wrapped into an executable and installer using launch4j and inno setup for windows, and linux users just get the jar compressed into a .tar.gz). Works great and simplifies our deployment process. Only thing that doesn't work is jogl.

I can easily skip the native jars and write my own native library loader that detects the platforms and loads the right libs. Just didn't want to go to the effort if you guys had already written something that would work for us.

On Tue, Dec 4, 2012 at 3:34 AM, gouessej [via jogamp] <[hidden email]> wrote:

Hi

JogAmp is not yet compatible with the "one single fat JAR approach" and it is clearly not the advised way of deploying applications based on JOGL 2.0. We have to fix a bug preventing the loading and the extraction of native libraries from JARs when they are inside a JAR to make it work with onejar. I know lots of Mac developers love bundling their applications as Mac bundles with a single JAR but you should rather find a way of using several JARs to create a bundle or use any other working installer (Java Web Start, GetDown, IzPack, ...).
Julien Gouesse
http://tuer.sourceforge.net
http://gouessej.wordpress.com

If you reply to this email, your message will be added to the discussion below:
http://forum.jogamp.org/ZipException-when-loading-native-jars-tp4027387p4027392.html

To unsubscribe from ZipException when loading native jars, click here.
NAML

Sven Gothel

Re: ZipException when loading native jars

Administrator

On 12/04/2012 06:16 PM, ryanmohr [via jogamp] wrote:
> The purpose of using onejar is to allow us to have a single file that can then
> be packaged as a native app / installer for any platform (wrapped into an app
> bundle and disk image for macs, wrapped into an executable and installer using
> launch4j and inno setup for windows, and linux users just get the jar
> compressed into a .tar.gz). Works great and simplifies our deployment
> process. Only thing that doesn't work is jogl.

Gentlemen, I recently summarized whats possible in this regard
in this forum - pls read:
<http://forum.jogamp.org/appbundler-jogl-jnilib-dylib-one-jar-confusion-td4027169.html#a4027245>

It would be great if you could pick that up and evaluate for your needs.

We might be able to accommodate you if proper collaboration is possible,
i.e. discussing the deployment specifications.

Please note that none of the mentioned external deployment models
are known to 'us' officially. So if this discussion should earn
some meaning - pls detail the specifications, compare it w/ what we
have w/ JogAmp and whats required. [-> Requirement management].

If the above is possible, we sure maybe able to arrange a solution.
It's not like we are stubborn :) Example: We did the same for maven2
packaging support.

Thank you.

~Sven

signature.asc (909 bytes) Download Attachment

gouessej

Re: ZipException when loading native jars

Administrator

In reply to this post by ryanmohr

ryanmohr wrote

The purpose of using onejar is to allow us to have a single file that can
then be packaged as a native app / installer for any platform (wrapped into
an app bundle and disk image for macs, wrapped into an executable and
installer using launch4j and inno setup for windows, and linux users just
get the jar compressed into a .tar.gz). Works great and simplifies our
deployment process. Only thing that doesn't work is jogl.

You are exposed to bugs of Launch4J and inno setup whereas Java Web Start is reliable enough to do something similar, it's even better for updates, the end user doesn't have to download the fat JAR each time, Java Web Start can update only the modified JAR(s). We cannot support all installers.

ryanmohr wrote

I can easily skip the native jars and write my own native library loader
that detects the platforms and loads the right libs. Just didn't want to
go to the effort if you guys had already written something that would work
for us.

Why not helping us to fix this bug? It would benefit to the whole community and you would not have to maintain a separate loader alone.

Edit.: Sven's Jar-In-Jar approach should work in your case.

Julien Gouesse | Personal blog | Website

ryanmohr

Re: ZipException when loading native jars

whereas Java Web Start is reliable enough to do something similar, it's even better for updates

In theory yes. In practice, not even close. We've used web start to deploy our application for the past ten years and we're finally giving up on it. We've repeatedly been bitten by signing issues, offline issues, missing desktop icons, broken file associations... all the kinds of things our customers expect to just work. It's great for little hobby projects but just doesn't make the cut in production.

Our entire application is less than 10mb. Not an issue to have to redownload, especially since it happens in the background automatically.

Why not helping us to fix this bug? It would benefit to the whole community and you would not have to maintain a separate loader alone.

I took a look at the source and it was too messy for me to want to tinker with. Plus there is the issue of what happens if the jars can't be unzipped to the tmp directory for whatever reason (you wouldn't believe the shape some of our customers computers are in). I'll be open sourcing my custom loader solution as its own project that can be used for any java projects that need to load or execute platform specific code and libraries.

ryanmohr

Re: ZipException when loading native jars

In reply to this post by gouessej

Just about finished with my loader but ran into a little hiccup. When jogamp.gluegen.UseTempJarCache is false the Platform class expects the library to exist in java.library.path. Since the path can't be modified at runtime this severely limits our loading options. I was able to work around it but the loader design is not as clean as it could be because of this restriction.

Would it be alright to add a new property ("jogamp.gluegen.LoadLibraries", defaults to true) that when false causes the Platform class to assume the libraries have already been loaded? That way in special cases the libraries can be loaded based on the discretion of the programmer and the Platform class can just relax.

On Tue, Dec 4, 2012 at 9:59 AM, Ryan Mohr <[hidden email]> wrote:

whereas Java Web Start is reliable enough to do something similar, it's even better for updates

In theory yes. In practice, not even close. We've used web start to deploy our application for the past ten years and we're finally giving up on it. We've repeatedly been bitten by signing issues, offline issues, missing desktop icons, broken file associations... all the kinds of things our customers expect to just work. It's great for little hobby projects but just doesn't make the cut in production.

Our entire application is less than 10mb. Not an issue to have to redownload, especially since it happens in the background automatically.

Why not helping us to fix this bug? It would benefit to the whole community and you would not have to maintain a separate loader alone.

I took a look at the source and it was too messy for me to want to tinker with. Plus there is the issue of what happens if the jars can't be unzipped to the tmp directory for whatever reason (you wouldn't believe the shape some of our customers computers are in). I'll be open sourcing my custom loader solution as its own project that can be used for any java projects that need to load or execute platform specific code and libraries.

gouessej

Re: ZipException when loading native jars

Administrator

ryanmohr wrote

> whereas Java Web Start is reliable enough to do something similar, it's
> even better for updates
>

In theory yes. In practice, not even close. We've used web start to
deploy our application for the past ten years and we're finally giving up
on it. We've repeatedly been bitten by signing issues, offline issues,
missing desktop icons, broken file associations... all the kinds of things
our customers expect to just work. It's great for little hobby projects
but just doesn't make the cut in production.

Our entire application is less than 10mb. Not an issue to have to
redownload, especially since it happens in the background automatically.

If your nice exe. file doesn't want to work, the end user will have to open the JAR by double-clicking onto it and most of the time it will open it with the default archiver instead of running your program. Lots of people use WinRAR under Windows and Ark under GNU Linux... There are workarounds to succeed in using Java Web Start in good conditions (I rewrote the mechanism that creates the desktop shortcuts). JavaFX native packaging is an interesting option too.

ryanmohr wrote

Since the path can't be modified at
runtime

It's wrong:
http://fahdshariff.blogspot.fr/2011/08/changing-java-library-path-at-runtime.html

Julien Gouesse | Personal blog | Website

ryanmohr

Re: ZipException when loading native jars

There are workarounds to succeed in using Java Web Start in good conditions.

Trust me. We wouldn't be going through all the extra work to package our app natively if we got good reliable service out of web start. As I said, we've been using it for ten years.

What happens if you want to install the software on a machine that's not connected to the internet? These are serious issues we have to deal with for some of our major customers (the computers are attached to CNC machines that cost close to $100K and they don't want to run the risk of catching a virus). With a native installer it's just a matter of throwing the files on a USB key, running a virus scanner, and off you go.

ryanmohr wrote

Since the path can't be modified at runtime

It's wrong:

I know it's wrong. That's why I said "since the path can't be modified at runtime" and why I asked for your opinion: "Would it be alright to add a new property ("jogamp.gluegen.LoadLibraries", defaults to true) that when false causes the Platform class to assume the libraries have already been loaded? That way in special cases the libraries can be loaded based on the discretion of the programmer and the Platform class can just relax."

Regardless of whether the idea sits well with you, I'm making the modifications in my own fork. It's just a matter of whether I'm going to be forced to maintain the fork indefinitely for a two line change or am going to be able to have the changes accepted back into the master.

Sven Gothel

Re: ZipException when loading native jars

Administrator

On 12/05/2012 01:10 AM, ryanmohr [via jogamp] wrote:
>
> Regardless of whether the idea sits well with you, I'm making the
> modifications in my own fork. It's just a matter of whether I'm going to be
> forced to maintain the fork indefinitely for a two line change or am going to
> be able to have the changes accepted back into the master.

Dear Ryan,

I am the maintainer of GlueGen and JOGL, and in this role
I have replied the post:

<http://forum.jogamp.org/ZipException-when-loading-native-jars-tp4027387p4027396.html>

Please read carefully .. and we may be able to fix the issue
in a constructive manner.

Further more it would be great, if you can provide some test cases.

~Sven

signature.asc (909 bytes) Download Attachment

ryanmohr

Re: ZipException when loading native jars

Sorry Sven, I totally missed your original post. I took a look at the related posts and issues. It would help if you could describe the jar-in-jar loader better.

For example, in Test-1 of https://jogamp.org/bugzilla/show_bug.cgi?id=522#c7 I'm surprised that the jogl jars are found under lala01_lib. What is the full list of paths that are checked in this case?

From my understanding, the auto extraction process simply copies the required libraries out of the necessary jars into a temporary directory and loads them from there. If that's the case, why does it matter where the native jars are as long as they're included in the jar's class path? The application has no problem finding the jogl-related classes themselves, just the libraries.

Here's the structure of our application if it helps:

application.jar -- the application itself
lib/
resources.jar -- all the resources the application relies on
foo.jar -- other dependencies
bar.jar
jogl/
jogl-all.jar
gluegen-rt.jar
... and so on for all the required jars

Onejar apparently doesn't support subdirectories in the lib dir so we flatten the lib structure before packaging it.

gouessej

Re: ZipException when loading native jars

Administrator

In reply to this post by ryanmohr

ryanmohr wrote

> There are workarounds to succeed in using Java Web Start in good
> conditions.
>

Trust me. We wouldn't be going through all the extra work to package our
app natively if we got good reliable service out of web start. As I said,
we've been using it for ten years.

What happens if you want to install the software on a machine that's not
connected to the internet? These are serious issues we have to deal with
for some of our major customers (the computers are attached to CNC machines
that cost close to $100K and they don't want to run the risk of catching a
virus). With a native installer it's just a matter of throwing the files
on a USB key, running a virus scanner, and off you go.

Sony uses Java Web Start for PoxNora. I have used Java Web Start both for my hobby projects and at work since almost seven years. It is possible to run it locally but you have to detect proxies because localhost can be used only without proxy. Java Web Start must be configured correctly when a firewall is used if it doesn't automatically detect its parameters. If Java Web Start isn't reliable enough for you, why not trying JavaFX native packaging (and future versions will include built-in support for autoupdate)?

I hope Sven's suggestion will work fine in your case.

Julien Gouesse | Personal blog | Website

Sven Gothel

Re: ZipException when loading native jars

Administrator

In reply to this post by ryanmohr

On 12/05/2012 07:06 AM, ryanmohr [via jogamp] wrote:
> It would help if you could describe the jar-in-jar loader
> better.
>
> For example, in Test-1 of
> https://jogamp.org/bugzilla/show_bug.cgi?id=522#c7 I'm surprised that the jogl
> jars are found under lala01_lib. What is the full list of paths that are
> checked in this case?

Test-1 is not Jar-In-Jar, but simple Side-by-Side,
hence it's simply the default case.

./lala01.jar
./lala01_lib/jogl.all.jar
./lala01_lib/jogl-all-natives-linux-amd64.jar
./lala01_lib/... etc ..

Test-2 is Jar-In-Jar:

./lala02.jar:
Manifest-Version: 1.0
Rsrc-Class-Path: ./ gluegen-rt-natives-linux-amd64.jar gluegen-rt.jar
jogl-all-natives-linux-amd64.jar jogl.all.jar
Class-Path: .
Rsrc-Main-Class: Test01
Main-Class: org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader

meaning that the Jar contains the Jars listed in the manifest
in it's root folder.

Jar-In-Jar simply works by passing the big jar to the commandline.
The used classloader then is able to simply use resources from within
the jar, exposed as a normal URI.
Hence our GlueGen native lib loading mechanism simply uses the
inner-jar URI and appends the native jar files to it.
No actual extraction happens until then.
After having the native JAR URI (pointing to the native jar file
inside the jar), we extract it and proceed as usual.

>
> From my understanding, the auto extraction process simply copies the required
> libraries out of the necessary jars into a temporary directory and loads them
> from there. If that's the case, why does it matter where the native jars are
> as long as they're included in the jar's class path?

They shall not be in the jar's class path, but simply parallel
to the non-native jar file with the same basename!

> The application has no
> problem finding the jogl-related classes themselves, just the libraries.
>
> Here's the structure of our application if it helps:
>
> application.jar -- the application itself
> lib/
> resources.jar -- all the resources the application relies on
> foo.jar -- other dependencies
> bar.jar
> jogl/

I would rename jogl/ -> jogamp/

> jogl-all.jar
jogl-all-natives-linux-amd64.jar
...
> gluegen-rt.jar
gluegen-rt-natives-linux-amd64.jar
...
> ... and so on for all the required jars
>
> Onejar apparently doesn't support subdirectories in the lib dir so we flatten
> the lib structure before packaging it.

as long the extracted stuff by Onejar will result in something like:
/wherever/
jogl-all.jar
jogl-all-natives-linux-amd64.jar
...
gluegen-rt.jar
gluegen-rt-natives-linux-amd64.jar
...

it should work.

The basename, e.g. gluegen-rt does not matter,
but the -native-<OS.and.ARCH> part.

Can you provide a test case?
- a project which builds a onejar binary? (-> all open source)
- a onejar binary ? (-> if onejar is not open source)

Is Onejar freely available ? Is it open source ?

+++

The original exception shows me that the jar file could not be found,
or loaded. Maybe you can also provide the complete exception w/ stack trace,
maybe even a log file w/ all jogamp debug flags enabled (-> Wiki/FAQ/Bug..).

In such case the traditional way (native lib path) is being used.

~Sven

signature.asc (909 bytes) Download Attachment

ryanmohr

Re: ZipException when loading native jars

In reply to this post by gouessej

Thanks for the suggestion Julien. JavaFX sounds promising and I've never heard of it before. I'll have to look into it some more.

On Wed, Dec 5, 2012 at 2:05 AM, gouessej [via jogamp] <[hidden email]> wrote:

ryanmohr wrote

> There are workarounds to succeed in using Java Web Start in good
> conditions.
>

Trust me. We wouldn't be going through all the extra work to package our
app natively if we got good reliable service out of web start. As I said,
we've been using it for ten years.

What happens if you want to install the software on a machine that's not
connected to the internet? These are serious issues we have to deal with
for some of our major customers (the computers are attached to CNC machines
that cost close to $100K and they don't want to run the risk of catching a
virus). With a native installer it's just a matter of throwing the files
on a USB key, running a virus scanner, and off you go.

Sony uses Java Web Start for PoxNora. I have used Java Web Start both for my hobby projects and at work since almost seven years. It is possible to run it locally but you have to detect proxies because localhost can be used only without proxy. Java Web Start must be configured correctly when a firewall is used if it doesn't automatically detect its parameters. If Java Web Start isn't reliable enough for you, why not trying JavaFX native packaging (and future versions will include built-in support for autoupdate)?

I hope Sven's suggestion will work fine in your case.
Julien Gouesse
http://tuer.sourceforge.net
http://gouessej.wordpress.com

If you reply to this email, your message will be added to the discussion below:

http://forum.jogamp.org/ZipException-when-loading-native-jars-tp4027387p4027409.html

To unsubscribe from ZipException when loading native jars, click here.
NAML

ryanmohr

Re: ZipException when loading native jars

In reply to this post by Sven Gothel

Sven Gothel wrote

Test-1 is not Jar-In-Jar, but simple Side-by-Side,
hence it's simply the default case.

./lala01.jar
./lala01_lib/jogl.all.jar

What strikes me as odd is that it's using the name of the jar file to determine the lib path. Normally I'd expect to see something more like this:

./lala01.jar
./lib/jogl.all.jar

Sven Gothel wrote

as long the extracted stuff by Onejar will result in something like:
/wherever/
jogl-all.jar
jogl-all-natives-linux-amd64.jar
...
gluegen-rt.jar
gluegen-rt-natives-linux-amd64.jar
...

it should work.

Only problem I see with onejar is that the main jar never sits beside the dependencies. Onejar is very specific about the folder structure. Has to look like this:

main/
application.jar
lib/
[all other jars]

Sven Gothel wrote

Can you provide a test case?
- a project which builds a onejar binary? (-> all open source)
- a onejar binary ? (-> if onejar is not open source)

If I'm not able to get things working based on your suggestions I'll create a project on github that demonstrates the issue.

Sven Gothel wrote

Is Onejar freely available ? Is it open source ?

Yes, free and open source on source forge (http://one-jar.sourceforge.net/)

ryanmohr

Re: ZipException when loading native jars

In reply to this post by ryanmohr

Well JavaFX looked like the holy grail until I came across this part:

Bundle per target platform
Bundle formats are platform specific. Currently a native bundle can only be produced for the same system you are building on. That is, if you want to deliver native app bundles on Windows, Linux and Mac you will have to build your project on all three platforms.

That simply won't work for us and I'm not sure how that design was ever considered acceptable.

My current release process is as simple as running "rake release master vX.Y.Z" which checks out the master branch of the source, tags it with the given version number, compiles it, obfuscates it, packages it into native bundles / installers, uploads the releases to our server, adds the new release to our database, and notifies users of a new update.

Sven Gothel

Re: ZipException when loading native jars

Administrator

In reply to this post by ryanmohr

On 12/05/2012 07:31 PM, ryanmohr [via jogamp] wrote:

> Sven Gothel wrote
> Test-1 is not Jar-In-Jar, but simple Side-by-Side,
> hence it's simply the default case.
>
> ./lala01.jar
> ./lala01_lib/jogl.all.jar
>
> What strikes me as odd is that it's using the name of the jar file to
> determine the lib path. Normally I'd expect to see something more like this:
>
> ./lala01.jar
> ./lib/jogl.all.jar
>
>
> Sven Gothel wrote
> as long the extracted stuff by Onejar will result in something like:
> /wherever/
> jogl-all.jar
> jogl-all-natives-linux-amd64.jar
> ...
> gluegen-rt.jar
> gluegen-rt-natives-linux-amd64.jar
> ...
>
> it should work.
>
> Only problem I see with onejar is that the main jar never sits beside the
> dependencies. Onejar is very specific about the folder structure. Has to
> look like this:
>
> main/
> application.jar
> lib/
> [all other jars]

as long you have
lib/
> jogl-all.jar
> jogl-all-natives-linux-amd64.jar
> ...
> gluegen-rt.jar
> gluegen-rt-natives-linux-amd64.jar

it should work.

The base URI of gluegen-rt.jar is being used,
i.e. 'file:///lalala/lib/'
and the module's base name 'gluegen-rt'
+ '-native-'<os.and.arch>.jar
appended.

>
>
> Sven Gothel wrote
> Can you provide a test case?
> - a project which builds a onejar binary? (-> all open source)
> - a onejar binary ? (-> if onejar is not open source)
>
> If I'm not able to get things working based on your suggestions I'll create a
> project on github that demonstrates the issue.

That would be great, thank you.

>
>
> Sven Gothel wrote
> Is Onejar freely available ? Is it open source ?
>
> Yes, free and open source on source forge (http://one-jar.sourceforge.net/)

Ok, good to know, thx.

~Sven

signature.asc (909 bytes) Download Attachment

Sven Gothel

Re: ZipException when loading native jars

Administrator

In reply to this post by gouessej

On 12/05/2012 01:05 PM, gouessej [via jogamp] wrote:
>
> Sony uses Java Web Start for PoxNora. I have used Java Web Start both for my
> hobby projects and at work since almost seven years. It is possible to run it
> locally but you have to detect proxies because localhost can be used only
> without proxy. Java Web Start must be configured correctly when a firewall is
> used if it doesn't automatically detect its parameters.
AFAIK the offline mode of Webstart has been fixed a while ago,
but I am no more an expert in this regard and have not tested it
w/ all the different configurations (firewall, proxy etc).

> If Java Web Start
> isn't reliable enough for you, why not trying JavaFX native packaging
> <https://blogs.oracle.com/talkingjavadeployment/entry/native_packaging_for_javafx> (and
> future versions will include built-in support for autoupdate)?

To allow a wide spread availability, relying on a proprietary
distribution model might be difficult.
Disclaimer: I don't know whether this javafx pack200 magic deployment model
is open source'ed yet and available in e.g. OpenJDK in free software form.

>
> I hope Sven's suggestion will work fine in your case.

Since Rayan will provide a test project and gave me information about
the onejar source code, I guess I will be able to fix it.

Even though this might not be our default use case,
different deployment scenarios can only better our code.

~Sven

> Julien Gouesse
> http://tuer.sourceforge.net
> http://gouessej.wordpress.com

signature.asc (909 bytes) Download Attachment

Sven Gothel

Re: ZipException when loading native jars

Administrator

In reply to this post by ryanmohr

On 12/05/2012 08:27 PM, ryanmohr [via jogamp] wrote:

> Well JavaFX looked like the holy grail until I came across this part:
>
> *Bundle per target platform*
> Bundle formats are platform specific. Currently *a native bundle can only
> be produced for the same system you are building on*. That is, if you
> want to deliver native app bundles on Windows, Linux and Mac you will have
> to build your project on all three platforms.
>
> That simply won't work for us and I'm not sure how that design was ever
> considered acceptable.
>
> My current release process is as simple as running "rake release master
> vX.Y.Z" which checks out the master branch of the source, tags it with the
> given version number, compiles it, obfuscates it, packages it into native
> bundles / installers, uploads the releases to our server, adds the new release
> to our database, and notifies users of a new update.

.. plus my other reply regarding the proprietary status of that method.
I would not dare to include a blob in my project, depending on some weirdo
deployment and maybe even market scheme :)

~Sven

signature.asc (909 bytes) Download Attachment

ryanmohr

Re: ZipException when loading native jars

Sven Gothel wrote

.. plus my other reply regarding the proprietary status of that method.
I would not dare to include a blob in my project, depending on some weirdo
deployment and maybe even market scheme :)

Exactly! The only reason we're willing to go to the work of writing a custom deployment process is to remove or internalize as many variables as possible.

We need to be able to guarantee that the application we send to our customers just works (to the point that we'll soon be giving customers the option to have the JRE embedded, removing one more variable that's given our customers problems in the past).

Sven -- I'm giving your suggestions a go right now. Will let you know if I make any progress. Appreciate the guidance!