Multiple OpenCL contexts

Hi,

I have a problem regarding the following pull request to jME3 (https://github.com/jMonkeyEngine/jmonkeyengine/pull/494).
I want to create multiple OpenCL contexts at the same time.
I have experienced, however, that when a second context is created, the previous one is automatically released. This invalidates all associated programs, buffers and so on, leading to an INVALID_CONTEXT, INVALID_MEM_OBJECT, ... being thrown.

What causes this problem? Is it intended (I do not hope so)? How can I work around it?

Wade, do you have an idea?

I can't see anything in the code of CLContext that kills previous contexts when you create a new one (see https://github.com/WadeWalker/jocl/blob/master/src/com/jogamp/opencl/CLContext.java). If you call CLContext.create(), it just calls the CLContext constructor, which just initializes itself as far as I can see.

shaman, do you have a simple example of trying to create two contexts and failing? If you create a minimal test for only this problem, I can check it out.

Here is a simple example. I copy'n'paste all important parts from jME3 together so that it works (more or less).
Now the second context fails in creating the OpenCL context.


import com.jogamp.opencl.CLBuffer;
import com.jogamp.opencl.CLDevice;
import com.jogamp.opencl.CLMemory;
import com.jogamp.opencl.CLPlatform;
import com.jogamp.opencl.gl.CLGLContext;
import com.jogamp.opengl.*;
import com.jogamp.opengl.awt.GLCanvas;
import com.jogamp.opengl.util.Animator;
import com.jogamp.opengl.util.AnimatorBase;
import com.jogamp.opengl.util.FPSAnimator;
import java.awt.*;
import java.awt.event.WindowAdapter;
import java.awt.event.WindowEvent;
import java.lang.reflect.InvocationTargetException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.SwingUtilities;


/**
 *
 * @author Sebastian Weiss
 */
public class MultiContextTest implements GLEventListener {

    protected GraphicsDevice device;
    protected GLCanvas canvas;
    protected AnimatorBase animator;
    protected Frame frame;
    protected CLDevice clDevice;
    protected CLGLContext clContext;
    protected int index;
   
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        //query platform
        final CLPlatform platform = CLPlatform.listCLPlatforms()[0];
        System.out.println("Platform Name: "+platform.getName());
        final CLDevice[] devices = platform.listCLDevices();
        for (int i=0; i<devices.length; ++i) {
            final CLDevice device = devices[i];
            final int ii = i;
            new Thread() {
                public void run() {
                    new MultiContextTest(device, ii);
                }
            }.start();
            try {
                Thread.sleep(200);
            } catch (InterruptedException ex) {
                Logger.getLogger(MultiContextTest.class.getName()).log(Level.SEVERE, null, ex);
            }
            //break;
        }
    }

    public MultiContextTest(CLDevice device, int index) {
        this.index = index;
        launch(device);
    }
   
    private void launch(CLDevice clDevice) {
        this.clDevice = clDevice;
        System.out.println("Device Name: "+clDevice.getName());
        if (!clDevice.isAvailable()) {
            System.out.println("Device not available");
            return;
        }
        if (!clDevice.isGLMemorySharingSupported()) {
            System.out.println("OpenGL sharing not supported");
            return;
        }
       
        //create OpenGL context
        create(true);
       
    }
   
    private void createCLContext(GLContext context) {
        //create OpenCL context
        try {
            clContext = CLGLContext.create(context, new CLDevice[]{clDevice});
            System.out.println("CL context created");
        } catch (Exception ex) {
            System.err.println("Unable to create OpenCL context");
            ex.printStackTrace();
            return;
        }
    }
   
    private void update() {
        if (clContext == null) {
            return;
        }
       
        //test OpenCL
        try {
        CLBuffer buffer = clContext.createBuffer(1024, CLMemory.Mem.READ_WRITE);
        buffer.release();
        } catch (Exception ex) {
            System.err.println(index+": OpenCL exception: "+ex.getLocalizedMessage());
            CLGLContext c = clContext;
            clContext = null;
            c.release();
            return;
        }
       
        System.out.println(index+": update");
    }
   
    private void initGLCanvas() {
        device = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();
       
        GLCapabilities caps = new GLCapabilities(GLProfile.getMaxProgrammable(true));
       
        caps.setHardwareAccelerated(true);
        caps.setDoubleBuffered(true);
        caps.setStencilBits(8);
        caps.setDepthBits(24);

        canvas = new GLCanvas(caps) {
            @Override
            public void addNotify() {
                super.addNotify();
                onCanvasAdded();
            }

            @Override
            public void removeNotify() {
                onCanvasRemoved();
                super.removeNotify();
            }
        };
        canvas.invoke(false, new GLRunnable() {
            public boolean run(GLAutoDrawable glad) {
                canvas.getGL().setSwapInterval(1);
                return true;
            }
        });
        canvas.setFocusable(true);
        canvas.requestFocus();
        canvas.setSize(800, 600);
        canvas.setIgnoreRepaint(true);
        canvas.addGLEventListener(this);
    }
   
    protected void createGLFrame(){
        frame = new Frame("Test: "+clDevice.getName());
        frame.setResizable(false);
        frame.add(canvas);
       
        applySettings();
       
        // Make the window visible to realize the OpenGL surface.
        frame.setVisible(true);
       
        canvas.setVisible(true);
    }

    protected void applySettings(){
        final boolean isDisplayModeModified;
        final GraphicsDevice gd = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();
        // Get the current display mode
        final DisplayMode previousDisplayMode = gd.getDisplayMode();
        // center the window on the screen.
        isDisplayModeModified = false;
        frame.pack();

        int x, y;
        x = (Toolkit.getDefaultToolkit().getScreenSize().width - 800) / 2;
        y = (Toolkit.getDefaultToolkit().getScreenSize().height - 600) / 2;
        frame.setLocation(x, y);
       
        frame.addWindowListener(new WindowAdapter() {
            @Override
            public void windowClosing(WindowEvent evt) {
                // If required, restore the previous display mode
                if (isDisplayModeModified) {
                    gd.setDisplayMode(previousDisplayMode);
                }
                // If required, get back to the windowed mode
                if (gd.getFullScreenWindow() == frame) {
                    gd.setFullScreenWindow(null);
                }
//                windowCloseRequest.set(true);
            }
            @Override
            public void windowActivated(WindowEvent evt) {
//                active.set(true);
            }

            @Override
            public void windowDeactivated(WindowEvent evt) {
//                active.set(false);
            }
        });
    }
   
    protected void startGLCanvas() {
        animator = new FPSAnimator(canvas, 30);

        animator.start();
    }
   
    private void initInEDT(){
        initGLCanvas();

        createGLFrame();

        startGLCanvas();
    }


    public void create(boolean waitFor){
        if (SwingUtilities.isEventDispatchThread()) {
            initInEDT();
        } else {
            try {
                if (waitFor) {
                    try {
                        SwingUtilities.invokeAndWait(new Runnable() {
                            public void run() {
                                initInEDT();
                            }
                        });
                    } catch (InterruptedException ex) {
                        System.err.println("Interrupted");
                    }
                } else {
                    SwingUtilities.invokeLater(new Runnable() {
                        public void run() {
                            initInEDT();
                        }
                    });
                }
            } catch (InvocationTargetException ex) {
                throw new AssertionError(); // can never happen
            }
        }
    }
   
    @Override
    public void init(GLAutoDrawable drawable) {
        canvas.requestFocus();
        createCLContext(drawable.getContext());
    }

    @Override
    public void dispose(GLAutoDrawable drawable) {

    }

    @Override
    public void display(GLAutoDrawable drawable) {
        update();
    }

    @Override
    public void reshape(GLAutoDrawable drawable, int x, int y, int width, int height) {
       
    }
   
    protected void onCanvasRemoved(){
    }

    protected void onCanvasAdded(){

    }
}

This example runs fine for me :) But it only creates one CLContext, is this what you meant for it to do? It creates one CLContext, then uses it to create/release a CLBuffer over and over.

Then you only have one OpenCL device (I have two, my CPU and the GPU, both from Intel).
OpenCL allows multiple contexts for the same device, so just change the loop at the very top to not loop through the devices but instead use the first device 2-3 times.
The error should occur then.

I modified your test to create four windows, each creating one CLContext on my only HW device (an nvidia GTX 660). It worked without any problems -- four windows popped up, and I saw messages to the console indicating that the update methods were being called, with no exceptions or error messages.

Maybe this is some restriction of Intel OpenCL devices, that they can only handle one context per process? You also might try running this test on a different type of device, and make sure your driver is updated to the latest version. Another possibility is the GLCLContext -- maybe vanilla CLContexts don't have this problem on Intel?

For reference, I ran this with nvidia driver version 353.90 on Windows 10 64-bit, with Java 1.8.0_91.

This is getting weirder and weirder.
The following is the output when I run it only with 4 contexts on the cpu

Platform Name: Intel(R) OpenCL
Device Name:        Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz
Device Name:        Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz
Device Name:        Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz
Device Name:        Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz
CL context created
0: update
Unable to create OpenCL context
com.jogamp.opencl.CLException$CLInvalidOperationException: can not create CL context [error: CL_INVALID_OPERATION]
        at com.jogamp.opencl.CLException.checkForError(CLException.java:73)
        at com.jogamp.opencl.CLContext.createContext(CLContext.java:227)
        at com.jogamp.opencl.gl.CLGLContext.create(CLGLContext.java:129)
        at MultiContextTest.createCLContext(MultiContextTest.java:116)
        at MultiContextTest.init(MultiContextTest.java:279)
        at jogamp.opengl.GLDrawableHelper.init(GLDrawableHelper.java:644)
        at jogamp.opengl.GLDrawableHelper.init(GLDrawableHelper.java:667)
        at com.jogamp.opengl.awt.GLCanvas$10.run(GLCanvas.java:1407)
        at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1291)
        at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147)
        at com.jogamp.opengl.awt.GLCanvas$12.run(GLCanvas.java:1438)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:241)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
        at java.awt.EventQueue.access$200(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:694)
        at java.awt.EventQueue$3.run(EventQueue.java:692)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
CL context created
2: update
Unable to create OpenCL context
com.jogamp.opencl.CLException$CLInvalidOperationException: can not create CL context [error: CL_INVALID_OPERATION]
        at com.jogamp.opencl.CLException.checkForError(CLException.java:73)
        at com.jogamp.opencl.CLContext.createContext(CLContext.java:227)
        at com.jogamp.opencl.gl.CLGLContext.create(CLGLContext.java:129)
        at MultiContextTest.createCLContext(MultiContextTest.java:116)
        at MultiContextTest.init(MultiContextTest.java:279)
        at jogamp.opengl.GLDrawableHelper.init(GLDrawableHelper.java:644)
        at jogamp.opengl.GLDrawableHelper.init(GLDrawableHelper.java:667)
        at com.jogamp.opengl.awt.GLCanvas$10.run(GLCanvas.java:1407)
        at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1291)
        at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147)
        at com.jogamp.opengl.awt.GLCanvas$12.run(GLCanvas.java:1438)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:241)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
        at java.awt.EventQueue.access$200(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:694)
        at java.awt.EventQueue$3.run(EventQueue.java:692)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
0: update
2: update
0: update
2: update
0: update
2: update
0: update
2: update
0: update
2: update
0: update
2: update
0: update


I definitively have to investigate this more. Might it be a synchronization error?

The man page at https://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/clCreateContext.html says it's only supposed to return CL_INVALID_OPERATION if CL_INVALID_D3D10_DEVICE_KHR is set, but we're not setting it :)

One thing you might try is creating vanilla CLContexts instead of CLGLContexts. There may be some limit on those. You might also see whether you can create 4 processes each with one context -- this would indicate that there's some sort of per-process limit in the driver.

I'll try but even if different processes or only CLContext would work, it would not help.
I need it for multiple jME3 applications in the same JVM. So the sharing with OpenGL is the key part.

Ok but we have to find the origin of the limitation as a first step.

Hi,
a long time has passed. I've been trying the stuff out on different platforms: Intel embedded, NVIDIA, AMD.
I get different results every time: from a total crash to partially (e.g. only the first two execute) or everything works fine.
I'm out of ideas what to do, how to resolve the issue. I also searched the NVIDIA and AMD forums but it seems, no one has tried to create multiple contexts before. There is simply no need for it, you can utilize all possible devices just by choosing all devices in the context creation and use multiple command queues.

Hi shaman,

It seems like you've discovered that behavior varies across vendor when you're creating multiple CL contexts, so apparently we can't count on it working correctly :) Given this, is there some way you can accomplish your goal without relying on multiple contexts? As you say, if you're simply trying to use all the devices, you can do that from one context. Is there some specific reason you need more than one context, or was this just something you were trying because it seemed like it should work?

Hi Wade,
If I want to use all devices, I combine them in one single context. This is better also from a point of memory management.
As I said at the very beginning, I've written an OpenCL wrapper for the jMonkeyEngine. In jME, it is possible to theoretically create multiple applications (i.e. multiple OpenGL windows and contexts) that run in parallel. For me, this only works with JOGL, LWJGL throws errors :) .
Since the OpenCL context is attached to the OpenGL context (for OpenGL-OpenCL interop), it would be nice to also have an OpenCL context for every application. This leads exactly to the question I've asked.
For simplicity, I would now just say: If someone really needs multiple OpenCL contexts for some reason, he/she should just try it out if it works on his/her platform.
That's not a satisfying result, I know, but I'm out of ideas and would stop working on this issue now.

Since you've tried this with nvidia, AMD, and Intel, could you tell us which ones work correctly? nvidia worked for me, and you mentioned before that Intel didn't work for you, but what about AMD?

Here are the results:

Laptop (Win 10):
Platform: Intel OpenCL, Device: i7 --> every second application crashes
Platform: Intel OpenCL, Device: Intel HD Graphics 4000 (embedded) --> total crash with  CL_INVALID_GL_SHAREGROUP_REFERENCE_KHR even for a single application
Platform: NVIDIA CUDA, Device: GeForce GT 735M --> tested up to 10 parallel applications without any issue

Workstation (Win 8.1):
Platform: AMD, Device: AMD FirePro W9100 --> works without any problems
Platform: AMD, Device: Intel Xeon CPU --> works without any problems
Platform: Intel OpenCL, Device: Intel Xeon CPU --> every second application crashes
Platform: Experimental OpenCL 2.0 CPU Only Platform, Device: Intel Xeon CPU --> every second application crashes

So, only Intel crashes. If you see it that way, it's not too bad.
(Just sad that on my laptop I usually use only the CPU to save battery ;) )

Here is the whole test suite to reproduce the results:
http://www.filedropper.com/multicontexttest

A bit off topic but your findings reminds me of this

http://richg42.blogspot.se/2014/05/the-truth-on-opengl-driver-quality.html

Yeah, unfortunately with results like that (works on Nvidia and AMD, fails on Intel), it seems pretty sure that it's an Intel OpenCL driver bug :( I assume you're running the most recent Intel driver version -- if so, you're pretty much out of luck. As you said, you might just have to warn the user that it won't work on Intel, or only enable OpenCL if you're on Nvidia or AMD.