Posted by lukej on May 02, 2012; 2:42pm
URL: https://forum.jogamp.org/Problems-with-gluUnProject-tp3954975p3955187.html

I've done a little more research on the issue and looked at the JOGL 2.0 source code a little bit.  If I'm following this correctly, it looks like:

GLU.gluUnProject calls ProjectFloat.gluUnProject
ProjectFloat.gluUnProject calls FloatUtil.multMatrixf passing in the parameters (modelMatrix, projMatrix, matrixBuf)

According to the javadoc for multMatrixf, this should put a*b in d.  However, I believe it actually puts the matrix b*a in d.

I found this because I made a similar mistake looking at the OGL example code that I linked to in my original post.  That code has a function MultiplyMatrices4by4OpenGL_FLOAT that takes result, matrix1, matrix2.  However, what is actually put into result is matrix2 * matrix 1.  That code works fine because it passes in projection as matrix1 and modelview as matrix2.  (note that the desired result is modelview * projection).

I'm not sure what other code (if any) you have calling multMatrixf, but I think you should consider changing the order of the parameters in ProjectFloat.gluUnProject, or fixing FloatUtil.multMatrixf so that it puts a*b in d, as advertised.  If multMatrixf is left as is, the javadoc should be corrected to reflect that d is the result of b*a.

EDIT: Some of the a*b vs. b*a issue may have to do with my own confusion of OpenGL's matrix ordering.  Regardless, the multiplication for the gluUnProject seems to be applied in the incorrect order.

Hope this helps,
Luke