gluInvertMatrixf problems(started with gluUnproject)

Well we have been struggling with gluUnproject returning false for totally valid passed arguments for a few days now, we also noticed that by taking the code from http://www.opengl.org/wiki/GluProject_and_gluUnProject_code and converting it to java it works flawlessly, so we dug deep and saw that for some matrices your gluInvertMatrixf function returns false stating that the given matrix is not invertible but it in fact is. I am attaching example matrices for you to check out, hopefully this is of some help to you :)

For example your invert function fails on the following matrix:
[-0.8184068703526464, -0.7313346429703671, 0.8265999715430894, 0.8265727758407593,
 0.0, 2.266778784171848, 0.3441178170910746, 0.3441064953804016,
 1.5188486310796492, -0.3940677440777307, 0.44539996103581814, 0.44538530707359314,
 -29.837189131809282, 7.880600278695056, 31788.731450702064, 31789.685546875]

and this matrix has an inverse of approximately

[-0.2749384784384102, -8.508771838660298E-9, 0.5102473310792097, 0.0,
 -0.12547721590338706, 0.3889178409085109, -0.06761135604083059, 0.0,
 13141.636777184758, 5470.961521026936, 7071.334994947186, -0.49999177469301814,
 -13141.24258735293, -5470.797418788947, -7071.122269004663, 0.5000082253043491]

Have a nice day :)

Good catch. Do you use floats or doubles?

Thanks for reporting this! This code has been in JOGL unchanged since 2004 (the original JOGL 1 from way back), so if this bug is real, it'll be a great catch

I've added a bug report for this at https://jogamp.org/bugzilla/show_bug.cgi?id=483. If you add yourself to the CC list, you'll see the progress reports. I assigned the bug to myself, since the fix should be straightforward (i.e. no digging into assembly or native windowing code is required, and unit testing is easy).

I'll let you know if I have trouble reproducing this, so please add yourself to the CC list or set your preferences to see followups here.

fails with both floats and doubles.