Mouse Button Modifiers Lost In Newt mouseDragged Events

Fedora 12
JRE/JDK 7u7
GeForce 9800 GT + NVIDIA driver 304.37
JOGAMP/JOGL custom build from 10/09/2012 repos (RC11 in progress)

This probably should have gone straight to Bugzilla, but first I thought I'd see what the forum had to say:

I noticed today that newt events lose information about the mouse button state in at least some circumstances with JDK 7u7.  For example, if I left-click-and-drag with the mouse, when I dump out event.getModifiers() and event.getButton() they are both zero when the left mouse button is held.

The center and right buttons result in a non-zero getModifiers() value, but getButton() returns zero in those cases too in mouseDragged events.

The JDK7 AWT MouseEvent class appears to only set its "button" field for MOUSE_PRESSED, MOUSE_RELEASED, and MOUSE_CLICKED event ids.  The newt stuff bases the newt mouse event getButton() return value on the AWT event's getButton() result, so this might explain why getButton() returns zero in mouseDragged, and probably in mouseMoved events too though I haven't tried this yet.

The behavior of the getModifiers() return value is slightly trickier.  The button state is also in the modifiers for all three buttons.  In the AWT goo the middle and right mouse button masks are set to Event.ALT_MASK and Event.META_MASK respectively.  The AWTNewtEventFactory.awtModifiers2Newt method pays attention to ALT_MASK and EVENT_MASK, but not to BUTTON1_MASK.  The end result is that the middle and right mouse buttons make it through in the modifiers, but the left mouse button is lost.

Not sure what the "correct" behavior would be here.  It seems wrong that the AWT is not setting the button state for mouse event ids other than pressed/released/clicked unless there is some special meaning to the getButton() return value.  Could newt work around this by basing its button state on the AWT modifiers instead of the result of getButton()?

On 10/09/2012 11:35 PM, rhatcher [via jogamp] wrote: Your post is a bit confusing, since it is not clear 'from the top'
which module you are referring here. I guess mentioning JDK7 is the culprit
here or the fact that I just woke up :)

NEWT [X11, Win, OSX, ..], SWT, AWT, ... who knows ?

> For example, if I
> left-click-and-drag with the mouse, when I dump out event.getModifiers() and
> event.getButton() they are both zero when the left mouse button is held.
>
I assume NEWT ?
If so .. have you digged into the source code and checked?
Understand this as a kudos to you, i.e. familiar w/ our stuff so maybe
beginning to read the source and becoming a patch contributor via git
would be something desired (for me). Then .. it's still early in the morning
and hopefully you are enjoying my entertainment :)

> The center and right buttons result in a non-zero getModifiers() value, but
> getButton() returns zero in those cases too in mouseDragged events.
Realizing that our NEWT UI test framework is truly crap, i.e.
self written AWT robot methods performing too simple and not so reliable
actions and their validation.
Expanding this to test mouse actions might be a challenge.

>
> The JDK7 AWT MouseEvent class appears to only set its "button" field for
> MOUSE_PRESSED, MOUSE_RELEASED, and MOUSE_CLICKED event ids.  The newt stuff
> bases the newt mouse event getButton() return value on the AWT event's
> getButton() result, so this might explain why getButton() returns zero in
> mouseDragged, and probably in mouseMoved events too though I haven't tried
> this yet.
We are talking NewtCanvasAWT here ? I finally get a clue .. :)

>
> The behavior of the getModifiers() return value is slightly trickier.  The
> button state is also in the modifiers for all three buttons.  In the AWT goo
> the middle and right mouse button masks are set to Event.ALT_MASK and
> Event.META_MASK respectively.  The AWTNewtEventFactory.awtModifiers2Newt
> method pays attention to ALT_MASK and EVENT_MASK, but not to BUTTON1_MASK.
>  The end result is that the middle and right mouse buttons make it through in
> the modifiers, but the left mouse button is lost.
Nice, seems like you found a bug.
Would you be so kind and maybe fix it ?
Looks like you already validated the source code (KUDOS),
so a fix might be easy for you now.

>
> Not sure what the "correct" behavior would be here.  
I hate AWT :)

> It seems wrong that the
> AWT is not setting the button state for mouse event ids other than
> pressed/released/clicked unless there is some special meaning to the
> getButton() return value.  Could newt work around this by basing its button
> state on the AWT modifiers instead of the result of getButton()?

Policy is, as you feel as well, to provide all information in a reasonable
way w/o jumping through hoops.

Since you are already on this matter, pls file a bug report
assign it to yourself and fix it. I will help & assiste ofc.

I am very happy that we have such a huge feedback already,
which helps stabilizing and fixing bugs as well as enhancing stuff.

If we could come to the point that we have more source code contributors,
people who are also willing to patch and fix bugs and to provide unit tests
so their work is being seen via our glorious Jenkins receiving some Chuck'les,
 .. well .. that would be even more awesome :)

So .. I hope you all read this with lots of humor, it's not meant to be mean,
on the contrary - KUDOS.
Just me begging to move on to the next level, kicking my behind to
review your commits. Maybe somebody will even review my boring elaborations
and criticizes it, constructively of course :)

Wish you a great Wednesday and good morning!

~Sven

Heh.  Your response made me laugh.  Have some coffee!

The bottom line summary is that the NEWT events aren't telling the whole story about the mouse button state during mouseDragged and probably mouseMoved too, at least with JDK 7u7.  The reason appears to be because a) AWT's getButton() method doesn't provide button info for those events (who knows why), and b) NEWT's translation of modifier flags only picks up two of the buttons by happenstance due to their association with the META and ALT masks.

It is odd that this hasn't been seen before, so perhaps it works differently in older versions of the AWT.

We do our own builds routinely and will of course be happy to submit a change for consideration.  The biggest issue we have is that we aren't set up to test these changes across a wide variety of platforms; really just Linux, and a relatively old version at that.  So... you will need to consider any such changes as "suggestions" much like we did the X11 screen issue a few weeks ago and adjust as you see fit.

A couple questions/clarifications:

1) So for now the process for submission of patches for less familiar persons such as our group will be to attach modified code in Bugzilla and avoid direct commits.  Obviously it is important to maintain some order in the code base and avoid a free-for-all, so somebody on your end still needs to approve such changes.

2) In your ant build infrastructure, is there a magic property somewhere that will turn on debug info (locals, line numbers, etc) across the board for gluegen, jogl, and joal?  Seems like the last time I was through some of this code in the debugger some of that info was missing.

On 10/10/2012 02:02 PM, rhatcher [via jogamp] wrote:
> Heh.  Your response made me laugh.  Have some coffee!
good and the coffe is hot :)

>
> The bottom line summary is that the NEWT events aren't telling the whole story
> about the mouse button state during mouseDragged and probably mouseMoved too,
> at least with JDK 7u7.  The reason appears to be because a) AWT's getButton()
> method doesn't provide button info for those events (who knows why), and b)
> NEWT's translation of modifier flags only picks up two of the buttons by
> happenstance due to their association with the META and ALT masks.

Again: We seem to talk about AWT -> NEWT event translation ?
Ok, your writup targets AWT -> NEWT event translation
as implemented in AWTNewtEventFactory and used e.g. in
  AWTMouseAdapter, AWTKeyAdapter, AWTWindowAdapter

These adapters are being used e.g. for:
  - Using platform agnostic NEWT events even w/ AWT UI elements

  - NewtCanvasAWT:
      - AWTWindowAdapter: All types for AWT state notifications
      - AWTMouseAdapter, AWTKeyAdapter: NEWT offscreen only (OSX, ..)

So you claim that the resulting NEWT events
from AWT mouse-dragging and AWT mouse-moving are incomplete
in regards to the modifier bits.

This is _not_ regards native NEWT events.

.. now we can go on :)

>
> It is odd that this hasn't been seen before, so perhaps it works differently
> in older versions of the AWT.

Meaning you didn't test against JDK6 ?

TBH I haven't tested modifier bits a lot,
especially not the AWT translated ones
- hence there is no unit test for it yet.

>
> We do our own builds routinely and will of course be happy to submit a change
> for consideration.  The biggest issue we have is that we aren't set up to test
> these changes across a wide variety of platforms; really just Linux, and a
> relatively old version at that.  So... you will need to consider any such
> changes as "suggestions" much like we did the X11 screen issue a few weeks ago
> and adjust as you see fit.

Sounds great.
Well, the platform problem will be easily solved,
if you create unit tests w/ your changes (there is no other way anyways to
commit).
Then our next Jenkins build will show you the results, voila.
Ofc .. given that the same tests have passed at least on one platform :)

>
> A couple questions/clarifications:
>
> 1) So for now the process for submission of patches for less familiar persons
> such as our group will be to attach modified code in Bugzilla and avoid direct
> commits.  
No way - sorry.

I really don't wanne keep track of emails and attachments,
that would be too expensive - thats why we do have git/bugzilla etc.
Have you noted the nice SCM Ref field in our bugzilla DB ?
All fixed bugs can be tracked back to commits.

And I really can use the time better like answering emails here :)

> Obviously it is important to maintain some order in the code base
> and avoid a free-for-all, so somebody on your end still needs to approve such
> changes.

Pls do it the proper way and use git, then you can send via:
  - git pull request a'la github
  - git email patch
  - git readable repository of your w/ all the sha1 you want me to pull
    or cherry pick
  - .. whatever (don't know of any other way)

Again - you should provide unit tests, especially for new features.
Even though in the very past we didn't do it this way,
since around later 2009 we do unit test a lot.
Only this way a feature and sane behavior can be guaranteed.

>
> 2) In your ant build infrastructure, is there a magic property somewhere that
> will turn on debug info (locals, line numbers, etc) across the board for
> gluegen, jogl, and joal?  Seems like the last time I was through some of this
> code in the debugger some of that info was missing.

You mean to have the compiler/linker leave/produce the debug info ?
Yes there is (from jogl/scripts/make.jogl.all.linux-x86_64.sh):
#    -Dc.compiler.debug=true \
#    -Djavacdebug="true" \
#    -Djavacdebuglevel="source,lines,vars" \

~Sven

> Again: We seem to talk about AWT -> NEWT event translation?

Sorry I'm not being clear about what's going on; yes, AWT->NEWT event translation is the issue, specifically related to mouse button state.  We have used JOGL 1.1.1a for a long time, but I am new to NEWT and still don't know everything it does yet so my discussion may not be (and obviously is not) complete.


> Meaning you didn't test against JDK6?

Only 7u7 so far, only because that's what we use from day to day.

What is your policy on backward compatibility of JOGL changes?  For example, do changes need to work with 1.4 forward, or 5 forward, or... ?  I have JDKs installed for 1.5.0-22 and 6u33.  Nothing earlier than that currently.


> No way - sorry.

Heh.  However you want to do it is fine of course.  I suppose your git process described below still allows you to decide what you want to keep, so you are protected from noob contributors such as myself - which is all I am concerned about as far as how we submit stuff.


> Pls do it the proper way and use git, then you can send via:
>   - git pull request a'la github
>   - git email patch
>   - git readable repository of your w/ all the sha1 you want me to pull
>     or cherry pick
>   - .. whatever (don't know of any other way)

Sure - though... all I know how to do with git currently is clone your repos, so I probably will mess this up :-).

At this point I have a modified version of AWTNewtEventFactory.java that illustrates at least one way to work around this button/modifier AWT->NEWT translation issue.  It is the only file that changed, so maybe the "git email patch" option is the best way in this case (?).  I will go try to absorb some git and figure out what to do.

The exact form of the unit test is less clear.  In this case the goal is to see if information in the AWT event makes it through to the NEWT event.  So... I guess the most obvious thing to do is construct AWT mouse events representing known button states of interest, run them through AWTNewtEventFactory, and see if the bits set in the resulting NEWT event are correct.


I'd like your input on a few questions before I propose a fix:

1) Generally, do we want NEWT event methods named similarly/identical to AWT event methods to have the same meaning/semantics?  The answer here influences the next question:

2) If a mouse button is held down do we expect someNewtMouseEvent.getButton() to return anything?  The equivalent AWT event method says this will be the button that changed, so I guess it's OK that the NEWT mouse dragged event returns zero from getButton() if the answer to 1) was yes.

3) Do we expect someNewtMouseEvent.getButtonsDown() to return an array containing all currently pressed buttons - and by association someNewtMouseEvent.isButtonDown(someButton) to return true if the indicated button is down - for all mouse events (i.e. including mouse dragged and mouse moved) (?).

On 10/10/2012 07:38 PM, rhatcher [via jogamp] wrote:
>> Again: We seem to talk about AWT -> NEWT event translation?
>
> Sorry I'm not being clear about what's going on; yes, AWT->NEWT event
> translation is the issue, specifically related to mouse button state.  We have
> used JOGL 1.1.1a for a long time, but I am new to NEWT and still don't know
> everything it does yet so my discussion may not be (and obviously is not)
> complete.

Ok, so one or all of the listed uses cases of my previous post.

>
>
>> Meaning you didn't test against JDK6?
>
> Only 7u7 so far, only because that's what we use from day to day.
>
> What is your policy on backward compatibility of JOGL changes?  For example,
> do changes need to work with 1.4 forward, or 5 forward, or... ?  I have JDKs
> installed for 1.5.0-22 and 6u33.  Nothing earlier than that currently.

Currently our minimum requirement is 1.6 'core', allowing some flavors
and custom configurations (headless, android, etc).

>
>
>> No way - sorry.
>
> Heh.  However you want to do it is fine of course.  I suppose your git process
> described below still allows you to decide what you want to keep, so you are
> protected from noob contributors such as myself - which is all I am concerned
> about as far as how we submit stuff.

Ofc, using DSCM is always a sane & safe process w/ implicit review
before you merge - no central repo for all folks to check-in per se.
Of course .. after your commit #1000 .. I may just press 'merge' :)
to validate - I am able to track back where your change comes from.
Take your last patch for example, only b/c it was simple enough
and only in one place - it was easy with you given me the point in time etc.
But you know .. usually it's not just that easy .. and there are more
complicated patches.
Many words short sentiment: It will save all our time and nerves
to use out SCM process.
THANK YOU

> At this point I have a modified version of AWTNewtEventFactory.java that
> illustrates at least one way to work around this button/modifier AWT->NEWT
> translation issue.  It is the only file that changed, so maybe the "git email
> patch" option is the best way in this case (?).  I will go try to absorb some
> git and figure out what to do.
As you wish. Any git way will preserve it's source, i.e. from which sha1 you have branched of. Of course - it must be one of the sha1 I have.
Scenario which would not work, after you have changed file A
in your git repo, you send me another single patch based on your sha1.
This one I cannot grok, since I don't have the whole history.

>
> The exact form of the unit test is less clear.  In this case the goal is to
> see if information in the AWT event makes it through to the NEWT event.  So...
> I guess the most obvious thing to do is construct AWT mouse events
> representing known button states of interest, run them through
> AWTNewtEventFactory, and see if the bits set in the resulting NEWT event are
> correct.
AWT Robot .. we already even have one mouse drag use case
TestSwingAWTRobotUsageBeforeJOGLInitBug411.
And a few w/ key types / mouse click ones. Just see call-tree of our AWTRobotUtil class.

>
>
> I'd like your input on a few questions before I propose a fix:
>
> 1) Generally, do we want NEWT event methods named similarly/identical to AWT
> event methods to have the same meaning/semantics?  
We don't need a copy of AWT behavior, especially not if it's not so sane :)

We like to produce the most identical behavior for all platforms,
best example it the KeyEvent class, pls read it's class header doc
and the note for getKeyChar().
This was the result to alias the behavior of all major platforms.

So .. lets try to do the most intuitive and platform neutral.

One thing for sure - we need to document the NEWT event classes.
Thx to your questions and review and to lates attempt to better the
documentation, I cannot agree more that we need more work here!

> The answer here influences
> the next question:

[We speak about NEWT's MouseEvent.EVENT_MOUSE_DRAGGED event, I assume.]
[Please don't be afraid of redundancies in your descriptions ..]

>
> 2) If a mouse button is held down do we expect someNewtMouseEvent.getButton()
> to return anything?  
> The equivalent AWT event method says this will be the
> button that /changed/, so I guess it's OK that the NEWT mouse dragged event
> returns zero from getButton() if the answer to 1) was yes.
Well, my intuition would say 'yes the pressed button information
should be delivered with the MouseEvent.EVENT_MOUSE_DRAGGED event.

WindowImpl.java line 2061:
                e = new MouseEvent(MouseEvent.EVENT_MOUSE_DRAGGED, this, when,
                                   modifiers, x, y, 1, mouseButtonPressed, 0);

This shows how we actually synthesize the _native_ mouse-drag-event.
Only this way we are able to harmonize the dragging in NEWT for
all platforms.

Maybe we can do the same for the AWT->NEWT event factory, i.e. properly track
the mouse states. One thing comes to mind now is the fact that we follow 2 path
for key/mouse events.
  1) Native delivery of details via WindowImpl.doMouseEvent()
  2) TK -> NEWT factory, which creates event itself and queues them

(1) does simple state tracking and synthesizes missing event etc
(2) does this not properly and is a diff. code path

Maybe we can change our factories (2) in a way to use method (1),
so it follows same path. What do you think ?
Then we can at least assure (better) that native events
and translated events behave similar.

>
> 3) Do we expect someNewtMouseEvent.getButtonsDown() to return an array
> containing all currently pressed buttons - and by association
> someNewtMouseEvent.isButtonDown(someButton) to return true if the indicated
> button is down - for /all/ mouse events (i.e. including mouse dragged and
> mouse moved) (?).

We already have solved the multi-touch / multi-pointer problem
with current API, see the [encrypted] doc for
getPointerCount(), getPointerId(int index), getX(int index) .. etc (all w/ index as argument).
Look at unit test: TextureSequenceCubeES2 line 107
Granted .. right now multi-pointer mode is currently only implemented
for Android, see AndroidNewtEventFactory.createMouseEvents(..),
which is very similar to the AWT->NEWT factory.

IMHO we should implement multi-pointer mode for other platforms ASAP,
it's a bug for sure.

I hope my above elaborations give you more insight to the design principles
and ideas (which you might have already found).

So .. I am all up to work w/ you and the whole _TEAM_ here to make
event handling more uniform.

~Sven

> Well, my intuition would say 'yes the pressed button information
> should be delivered with the MouseEvent.EVENT_MOUSE_DRAGGED event.

Yes - one has to believe that it's common to want to know what the current button state is for any event, including dragging.  At least I want to know it :-).


> WindowImpl.java line 2061:
>                e = new MouseEvent(MouseEvent.EVENT_MOUSE_DRAGGED, this, when,
>                                  modifiers, x, y, 1, mouseButtonPressed, 0);
>
> This shows how we actually synthesize the _native_ mouse-drag-event.
> Only this way we are able to harmonize the dragging in NEWT for
> all platforms.

WindowImpl does something slightly different from the AWT, which gets into the issue of preserving (or not) the semantics of similarly named methods.

As we have mentioned before though, we shouldn't do wrong things just because the AWT is doing wrong things, but in this case I'm not sure which one is wrong.

As we mentioned earlier, the AWT event's getButton() method says it returns the button that changed.  The current comprehensive button state is also available in the event's modifiers and extended modifiers, so all information about button state is still available.  My observation so far is that a lot of the time AWT's getButton() method returns 0, which is consistent with the associated javadocs.  I guess the getButton() method was intended to be a convenience to keep the user from having to track historical button state themselves and compare to current button state to figure out what changed.  For example, if I set up some listeners to watch some of this stuff and dump out modifier bits I'll see something like this when a click-and-drag is initiated:

...
AWT mouse moved, mods = 0, mods ex = 0, button = 0
AWT mouse pressed, mods = 10000, mods ex = 10000000000, button = 1
AWT mouse dragged, mods = 10000, mods ex = 10000000000, button = 0
...

The getButton() method returns a 1 on the press, but then for the drag it's returning 0.  I guess for mouse moved  the getButton() result would always be 0 anyway because if a button is pressed then it's a mouse drag, not a mouse move.

The pattern above is similar if buttons 2 and 3 are used instead, the only difference being which button is returned by getButton() for the mousePressed event that initiates the drag.

If a second button is pressed while the button used to initiate the drag operation is still down, then a mousePressed event will show up in the middle of the mouseDragged events and getButton() will return the additional button that was pressed:

...
AWT mouse dragged, mods = 10000, mods ex = 10000000000, button = 0
AWT mouse pressed, mods = 1000, mods ex = 110000000000, button = 2
AWT mouse dragged, mods = 11000, mods ex = 110000000000, button = 0
...

If the second button pressed is then released then a released event shows up, but the drag events continue:

...
AWT mouse dragged, mods = 11000, mods ex = 110000000000, button = 0
AWT mouse released, mods = 1000, mods ex = 10000000000, button = 2
AWT mouse dragged, mods = 10000, mods ex = 10000000000, button = 0
...

What WindowImpl is doing - at least for mouse dragged events - is always returning the button that was most recently pressed from getButton() during dragging.

So there is still a question here of the semantics of getButton() for NEWT - is it intended to indicate the button that changed, or indicate current button state?  The modifiers already tell the comprehensive story of button state - or at least they will after we fix the bug that started this discussion - and in NEWT's case we also have getButtonsDown() and isButtonDown(someButton) to provide this information.  So... maybe NEWT's getButton() should follow the semantics of the AWT (keep in mind that when I say this I am not promoting AWT's behavior as being the correct behavior).  Or... deprecate getButton() and eventually eliminate it altogether, and make the user figure out what buttons changed.


I noticed another oddity while I was looking at this: in the case above where a drag is initiated then a second button is pressed and released while the button that initiated the drag remains held down, the AWT will continue to send mouse dragged events following the release of the second button, but WindowImpl will lose the drag state and start sending mouse moved events after the second button is released even if the button that initiated the drag remains held.

This probably is another bug.  However, it highlights the question again of what NEWT's semantics should be.  Should this remain a series of drag events after the second button is released?  I think so if the button that initiated the drag remains held.


> Maybe we can do the same for the AWT->NEWT event factory, i.e. properly track
> the mouse states. One thing comes to mind now is the fact that we follow 2 path
> for key/mouse events.
>  1) Native delivery of details via WindowImpl.doMouseEvent()
>  2) TK -> NEWT factory, which creates event itself and queues them
>
> (1) does simple state tracking and synthesizes missing event etc
> (2) does this not properly and is a diff. code path
>
> Maybe we can change our factories (2) in a way to use method (1),
> so it follows same path. What do you think ?
> Then we can at least assure (better) that native events
> and translated events behave similar.

Your option 1 above sounds like a much better approach: back up to more primitive concepts that all the underlying toolkits can agree on, and reconstruct your own semantics from there.  It will insulate NEWT from the oddities of the various toolkits.  Though... I don't have broad enough platform exposure past Linux (Mac, Android, etc) at the moment to know where difficulties might lie, especially in the multi-touch world.


FYI I got set up on github this morning and forked the repos under sgothel/jogl etc.  I've sync'd those down to my local box and was going to make the bug fix for mouse button state there and push it back up to github.  I guess then I have to create a pull request.  Please bear with me on this; I'm a total git noob and am "drinking from a fire hose" at the moment, so I'm not quite sure what I'm doing yet particularly in the "best practices" area.  Any recommendations about the best way to handle this current bug fix would be welcome.