Posted by lukej on May 02, 2012; 12:44pm
URL: https://forum.jogamp.org/Problems-with-gluUnProject-tp3954975.html

I recently upgraded from JOGL 1.1 to JOGL 2.0 and I noticed some strange things were happening when I translated screen coordinates to world coordinates.  I have tracked it down to what appears to be a bug in the JOGL 2.0 implementation of GLU.gluUnProject.  

To make sure it wasn't something in my scene I took the source code from here and converted it to Java code.  When I replaced the GLU.gluUnProject call with that one, things work as expected.

Next I created some unit tests that test the resluts of the glhUnProjectf function vs. GLU.gluUnProject.  I found that in cases where the modelview matrix is the identity matrix, the answers match.  However, if the modelview matrix is not the identity matrix, the answers do not match (I did not rigorously test if this was always true, but it seemed to hold true).  It is worth noting that the GLU.gluUnProject call DOES return true, but the answer is incorrect.

Here are 2 such examples (the first uses the identity for a modelview matrix, the second does not):

Modelview Matrix: [1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0]
Projection Matrix: [2.3464675, 0.0, 0.0, 0.0, 0.0, 2.4142134, 0.0, 0.0, 0.0, 0.0, -1.0002, -1.0, 0.0, 0.0, -20.002, 0.0]
Viewport: [0, 0, 1000, 1000]
Screen Coordinates: [250.0, 250.0, 0.5]
JOGL GLU Result: [-4.261299, -4.141722, -19.998001]
Official OGL Code Result: [-4.261299, -4.141722, -19.998001]

Modelview Matrix: [1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, -200.0, 1.0]
Projection Matrix: [2.3464675, 0.0, 0.0, 0.0, 0.0, 2.4142134, 0.0, 0.0, 0.0, 0.0, -1.0002, -1.0, 0.0, 0.0, -20.002, 0.0]
Viewport: [0, 0, 1000, 1000]
Screen Coordinates: [250.0, 250.0, 0.5]
JOGL GLU Result: [0.02141787, 0.020816857, 0.10051267] *** WRONG
Official OGL Code Result: [-4.2612987, -4.1417212, 180.002]