Posted by bananafish on Jan 25, 2022; 9:23pm
URL: https://forum.jogamp.org/Infinite-recursion-with-in-GLAutoDrawableBase-reshape-tp4041605.html

If GLDrawableHelper.invoke() is called from a GLEventListener.reshape(), reshape() goes into an unbreakable recursion loop. Code of concern is at GLAutoDrawableHelper:435:

protected final Runnable defaultDisplayAction = new Runnable() {
        @Override
        public final void run() {
            // Lock: Locked Surface/Window b

y display _and_ MakeCurrent/Release
            if (sendReshape) {
                helper.reshape(GLAutoDrawableBase.this, 0, 0, getSurfaceWidth(), getSurfaceHeight());
                sendReshape = false;
            }
            helper.display(GLAutoDrawableBase.this);
            fpsCounter.tickFPS();
        } };

If the helper.reshape() invocation results in GLDrawableHelper.invoke() being called,

sendReshape=false;

... never gets executed and the operation recurs forever.

Proposed fix is to set sendReshape to false immediately before any outside code can be executed by reshape(...):

protected final Runnable defaultDisplayAction = new Runnable() {
        @Override
        public final void run() {
            // Lock: Locked Surface/Window by display _and_ MakeCurrent/Release
            if (sendReshape) {
                sendReshape = false;
                helper.reshape(GLAutoDrawableBase.this, 0, 0, getSurfaceWidth(), getSurfaceHeight());
            }
            helper.display(GLAutoDrawableBase.this);
            fpsCounter.tickFPS();
        } };

This is part of an effort to migrate some mature code from GLCanvas to NewtWindow+NewtCanvasAWT in order to be able to remove decorations from the window mid-runtime without destroying the canvas context. It's been a little rough.