CL_DEVICE_NOT_AVAILABLE when creating CLContext with 64-bit Java on GTX 970

Heyo everyone!

I'm having trouble running JOCL with 64-bit versions of Java on an nVidia GPU.
When I run the program with a 32-bit version (I tested jre1.6.0_45-32bit) everything works flawlessly, but when I switch to a 64-bit version (tested some versions of Java 7 and 8) I get an exception.

Basically, the program fails at the very first line where JOCL is involved: clContext = CLContext.create(CLDevice.Type.GPU);
1st line of the stack trace: Exception in thread "main" com.jogamp.opencl.CLException$CLDeviceNotAvailableException: can not create CL context [error: CL_DEVICE_NOT_AVAILABLE]

When I run the program on the CPU instead of the GPU, everything works perfectly with both 32-bit and 64-bit Java.

Relevant system specs:
- Intel i7-2600k
- nVidia GTX 970
- Windows 10 64-bit
- Using JOCL 2.3.2

At first I thought this was a driver problem, so I updated both the general nVidia driver to v364.72 (which includes its OpenCL drivers) and the Intel OpenCL driver to v15.1. Unfortunately, this didn't fix the problem.

Then I got my brother who has a very similar computer (Win10-64bit, GTX 970, Java 8 64-bit) to run the program and he ran into the exact same problem, so it's not just my computer either.

I even went so far as to re-code everything with both the JOCL from jocl.org and with JavaCL.
The results were similar, but not identical: Both worked with 32-bit Java and failed with 64-bit Java, but the error message was different. Both frameworks returned CL_OUT_OF_RESOURCES instead of CL_DEVICE_NOT_AVAILABLE.
I assume this is because these frameworks both call clCreateContext directly, while JOCL calls clCreateContextFromType.

At this point, I don't know what is causing this issue and what I should troubleshoot next.
I'm considering nuking all my nVidia drivers because there might be old left-over junk and then reinstalling the drivers. I'm not sure this would help, though, as brother's computer, which has been set up just recently, encounters the same problem.

Any idea what might be causing this issue or what I should be trying next?

When you run with 64-bit Java, are you also using the 64-bit version of the JOCL native libraries? Not sure if this could be the problem though... normally mismatched library architectures causes a library loading error.

I've tested JOCL with 64-bit Java on Windows 10 64-bit with an nvidia graphics card, but mine is not as new as yours :)But it can't be completely broken at least.

You could try running the JOCL "info" demo that dumps information about your drivers and devices. The code is at https://github.com/JogAmp/jocl-demos, it's pretty easy to clone and build. This might give us more information about what's going on.

it's pretty easy to clone and build

If you already have a development environment set up for C / C++, sure, but for someone who's only ever worked with Java... not so much ^^
But today I learned MinGW 32-bit and 64-bit are two completely separate projects and the 32-bit one completely fails on 64-bit systems :P

You should consider providing some binary builds for people who want to quickly run the demos. I mean, isn't the point of a demo being able to quickly test something without having to commit fully?

As for the native libraries:
I had just been using maven to include JOCL and to figure out the dependencies. It automatically imported some native libraries which I assumed to be correct - but this assumption was probably completely wrong.

With the self-built demo binaries (and thus also JOCL binaries), I've been able to run both the info demo and the Mandelbrot display. Both work as expected, which implies the native libraries maven imported weren't the right ones.

So I'll try to copy those over / change the maven dependency / whatever and let you know if that fixed my problem. I might do that tomorrow though, it's already getting close to 1am here ^^

By the way, the build scripts and the demos are both EPIC. I'm can't even imagine how much time must have gone into those. Thank you and the whole team for this awesome project!

And it turns out, I celebrated just a bit too early. Neither the info demo nor the fractal demo directly create a CLContext. The info demo doesn't create one at all and the fractal demo uses the CLGLContext.

One of the demos that actually creates a CLContext is the bandwidth demo, and of course that one doesn't work :(
The stack trace is pretty much what we'd expect:

     [java] Exception in thread "main" com.jogamp.opencl.CLException$CLOutOfResourcesException: can not create CL context [error: CL_OUT_OF_RESOURCES]
     [java]     at com.jogamp.opencl.CLException.checkForError(CLException.java:67)
     [java]     at com.jogamp.opencl.CLContext.createContext(CLContext.java:225)
     [java]     at com.jogamp.opencl.CLContext.create(CLContext.java:190)
     [java]     at com.jogamp.opencl.demos.bandwidth.BandwidthBenchmark.main(BandwidthBenchmark.java:99)
     [java] Java Result: 1

I also forgot to include the output from the info demo, here it is:

CL_PLATFORM_NAME	NVIDIA CUDA	Intel(R) OpenCL
CL_PLATFORM_EXTENSIONS	[cl_khr_global_int32_base_atomics, cl_khr_fp64, cl_nv_compiler_options, cl_khr_byte_addressable_store, cl_nv_copy_opts, cl_khr_global_int32_extended_atomics, cl_khr_icd, cl_nv_pragma_unroll, cl_nv_d3d10_sharing, cl_nv_device_attribute_query, cl_khr_local_int32_extended_atomics, cl_nv_d3d11_sharing, cl_khr_gl_sharing, cl_khr_d3d10_sharing, cl_khr_local_int32_base_atomics]	[cl_khr_global_int32_extended_atomics, cl_khr_icd, cl_khr_global_int32_base_atomics, cl_khr_fp64, cl_khr_local_int32_extended_atomics, cl_khr_spir, cl_khr_3d_image_writes, cl_khr_depth_images, cl_intel_exec_by_local_thread, cl_khr_byte_addressable_store, cl_khr_local_int32_base_atomics]
CL_PLATFORM_PROFILE	FULL_PROFILE	FULL_PROFILE
CL_PLATFORM_VERSION	OpenCL 1.2 CUDA 8.0.0	OpenCL 1.2
CL_PLATFORM_VENDOR	NVIDIA Corporation	Intel(R) Corporation
CL_PLATFORM_ICD_SUFFIX_KHR	NV	INTEL
CL_DEVICE_NAME	GeForce GTX 970	Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
CL_DEVICE_TYPE	GPU	CPU
CL_DEVICE_EXTENSIONS	[cl_khr_global_int32_base_atomics, cl_khr_fp64, cl_nv_compiler_options, cl_khr_byte_addressable_store, cl_nv_copy_opts, cl_khr_global_int32_extended_atomics, cl_khr_icd, cl_nv_pragma_unroll, cl_nv_d3d10_sharing, cl_nv_device_attribute_query, cl_khr_local_int32_extended_atomics, cl_nv_d3d11_sharing, cl_khr_gl_sharing, cl_khr_d3d10_sharing, cl_khr_local_int32_base_atomics]	[cl_khr_global_int32_extended_atomics, cl_khr_icd, cl_khr_global_int32_base_atomics, cl_khr_fp64, cl_khr_local_int32_extended_atomics, cl_khr_spir, cl_khr_3d_image_writes, cl_khr_depth_images, cl_intel_exec_by_local_thread, cl_khr_byte_addressable_store, cl_khr_local_int32_base_atomics]
CL_DEVICE_PROFILE	FULL_PROFILE	FULL_PROFILE
CL_DEVICE_VERSION	OpenCL 1.2 CUDA	OpenCL 1.2 (Build 57)
CL_DEVICE_AVAILABLE	true	true
CL_DEVICE_VENDOR	NVIDIA Corporation	Intel(R) Corporation
CL_DEVICE_MAX_CLOCK_FREQUENCY	1329	3400
CL_DEVICE_MAX_COMPUTE_UNITS	13	4
CL_DEVICE_ENDIAN_LITTLE	true	true
CL_DEVICE_MAX_WORK_ITEM_SIZES	[1024, 1024, 64]	[8192, 8192, 8192]
CL_DEVICE_MAX_CONSTANT_BUFFER_SIZE	65536	131072
CL_DEVICE_GLOBAL_MEM_CACHE_TYPE	READ_WRITE	READ_WRITE
CL_DEVICE_ERROR_CORRECTION_SUPPORT	false	false
cl_khr_gl_sharing | cl_APPLE_gl_sharing	true	false
CL_DEVICE_NATIVE_VECTOR_WIDTH_DOUBLE	1	4
CL_DEVICE_MAX_PARAMETER_SIZE	4352	3840
CL_DEVICE_GLOBAL_MEM_CACHE_SIZE	212992	262144
CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS	3	3
cl_khr_fp64	true	true
CL_DEVICE_PREFERRED_VECTOR_WIDTH_LONG	1	1
CL_DEVICE_NATIVE_VECTOR_WIDTH_FLOAT	1	8
CL_DEVICE_NATIVE_VECTOR_WIDTH_HALF	0	0
CL_DEVICE_PREFERRED_VECTOR_WIDTH_SHORT	1	1
CL_DEVICE_MAX_WORK_GROUP_SIZE	1024	8192
CL_DEVICE_NATIVE_VECTOR_WIDTH_INT	1	4
CL_DEVICE_MEM_BASE_ADDR_ALIGN	4096	1024
CL_DEVICE_MAX_MEM_ALLOC_SIZE	1073741824	4286350336
CL_DEVICE_GLOBAL_MEM_CACHELINE_SIZE	128	64
CL_DEVICE_MAX_CONSTANT_ARGS	9	480
CL_DEVICE_IMAGE_SUPPORT	true	true
CL_DEVICE_PREFERRED_VECTOR_WIDTH_FLOAT	1	1
CL_DEVICE_IMAGE2D_MAX_HEIGHT	16384	16384
CL_DEVICE_IMAGE3D_MAX_HEIGHT	4096	2048
CL_DEVICE_IMAGE3D_MAX_DEPTH	4096	2048
CL_DEVICE_EXECUTION_CAPABILITIES	[EXEC_KERNEL]	[EXEC_KERNEL, EXEC_NATIVE_KERNEL]
CL_DEVICE_SINGLE_FP_CONFIG	[DENORM, INF_NAN, ROUND_TO_NEAREST, ROUND_TO_INF, ROUND_TO_ZERO, FMA]	[DENORM, INF_NAN, ROUND_TO_NEAREST]
CL_DEVICE_PREFERRED_VECTOR_WIDTH_INT	1	1
CL_DEVICE_MAX_WRITE_IMAGE_ARGS	16	480
CL_DEVICE_COMPILER_AVAILABLE	true	true
cl_khr_fp16	false	false
CL_DEVICE_PROFILING_TIMER_RESOLUTION	1000	300
CL_DEVICE_NATIVE_VECTOR_WIDTH_CHAR	1	16
CL_DEVICE_IMAGE3D_MAX_WIDTH	4096	2048
CL_DEVICE_QUEUE_PROPERTIES	[OUT_OF_ORDER_MODE, PROFILING_MODE]	[OUT_OF_ORDER_MODE, PROFILING_MODE]
CL_DEVICE_PREFERRED_VECTOR_WIDTH_CHAR	1	1
CL_DEVICE_IMAGE2D_MAX_WIDTH	16384	16384
CL_DEVICE_NATIVE_VECTOR_WIDTH_SHORT	1	8
CL_DEVICE_PREFERRED_VECTOR_WIDTH_DOUBLE	1	1
CL_DEVICE_DOUBLE_FP_CONFIG	[DENORM, INF_NAN, ROUND_TO_NEAREST, ROUND_TO_INF, ROUND_TO_ZERO, FMA]	[DENORM, INF_NAN, ROUND_TO_NEAREST, ROUND_TO_INF, ROUND_TO_ZERO, FMA]
CL_DEVICE_NATIVE_VECTOR_WIDTH_LONG	1	2
CL_DEVICE_MAX_READ_IMAGE_ARGS	256	480
CL_DEVICE_GLOBAL_MEM_SIZE	4294967296	17145401344
CL_DEVICE_HOST_UNIFIED_MEMORY	false	true
CL_DEVICE_VENDOR_ID	4318	32902
CL_DEVICE_MAX_SAMPLERS	32	480
CL_DEVICE_ADDRESS_BITS	64	64
CL_DEVICE_HALF_FP_CONFIG	[]	[]
CL_DEVICE_LOCAL_MEM_TYPE	LOCAL	GLOBAL
CL_DEVICE_OPENCL_C_VERSION	OpenCL C 1.2	OpenCL C 1.2
CL_DRIVER_VERSION	364.72	5.0.0.57
CL_DEVICE_LOCAL_MEM_SIZE	49152	32768
cl_khr_icd	true	true

SuspiciousDroid wrote

it's pretty easy to clone and build

If you already have a development environment set up for C / C++, sure, but for someone who's only ever worked with Java... not so much ^^

Yeah good point, I had forgotten that little detail :)

SuspiciousDroid wrote

it's pretty easy to clone and build

You should consider providing some binary builds for people who want to quickly run the demos. I mean, isn't the point of a demo being able to quickly test something without having to commit fully?

We used to be able to do that, but it's broken for the moment while our certificate gets updated. You should be able to run them from https://jogamp.org/jocl-demos/www/, but browsers give an error right now until the signing certificate gets fixed.

SuspiciousDroid wrote

The stack trace is pretty much what we'd expect:
     [java] Exception in thread "main" com.jogamp.opencl.CLException$CLOutOfResourcesException: can not create CL context [error: CL_OUT_OF_RESOURCES]
     [java]     at com.jogamp.opencl.CLException.checkForError(CLException.java:67)
     [java]     at com.jogamp.opencl.CLContext.createContext(CLContext.java:225)
     [java]     at com.jogamp.opencl.CLContext.create(CLContext.java:190)
     [java]     at com.jogamp.opencl.demos.bandwidth.BandwidthBenchmark.main(BandwidthBenchmark.java:99)
     [java] Java Result: 1

I've seen stuff on the web saying that maybe this sort of thing is a driver conflict or corruption, e.g. http://forum.jogamp.org/How-to-run-jocl-tests-td4029688.html. In some cases, it seemed to help to completely uninstall all nvidia and OpenCL stuff, then just put the nvidia drivers back on.

Also, it might be worthwhile to try some of nvidia's C/C++ OpenCL samples, like the ones at https://developer.nvidia.com/opencl. If those fail in the same way, they we definitely know there's some driver weirdness going on. I've done this a few times with strange OpenGL bugs, to rule out anything related to Java or Jogamp.

Thanks for pointing me to the nvidia samples. I tried them out, and most of them failed.
Most of them don't even print an error message but just crash when they're trying to load an OpenCL kernel.

I then proceeded to completely uninstall all intel and nvidia drivers and then only reinstalled the nvidia ones.
I also checked if the OpenCL.dll in C:\Windows\System32 and C:\Program Files\NVIDIA Corporation\OpenCL are the same,
and now (unlike before the reinstall) they actually are.

Unfortunately, this still didn't fix the issue, and everything was still crashing like it did before :(

I then downloaded GPU Caps Viewer, which proceeded to correctly detect the GTX 970 as an OpenCL device.
Even better, its OpenCL demos even worked! This completely surprised me, as nvidias own samples didn't work,
but the ones of a utility program do?

After that, I went on to run more of the jocl-demos. Most of them still didn't work, but the Julia3d demo
strangely started working. After checking out the source code, I only found one difference between that
demo and the other ones; it first runs this line:

GLProfile.initSingleton();

So I swiftly imported JOGL into my project and added that line, and my with this hacky workaround, my program
actually started working!

This really leaves me with only one reaction: wat.

It happens that I also have Win10-64bit, GTX 970, Java 8 64-bit. Which of the nVidia demos didn't work for you? I can try and see if I have the same crash.

Almost all of them don't work ^^

But you can try oclVectorAdd. I get "oclVectorAdd.exe has stopped working" when the console output reaches "oclLoadProgSource (VectorAdd.cl)..."
which is about on line 16.

I did not have any problems with the demos I tested (neither the oclVectorAdd or any others). Below is the ouput from oclSimpleTexture3D.exe
Maybe we have different drivers?


  CL_DEVICE_NAME:                       GeForce GTX 970
  CL_DEVICE_VENDOR:                     NVIDIA Corporation
  CL_DRIVER_VERSION:                    364.72
  CL_DEVICE_VERSION:                    OpenCL 1.2 CUDA
  CL_DEVICE_OPENCL_C_VERSION:           OpenCL C 1.2
  CL_DEVICE_TYPE:                       CL_DEVICE_TYPE_GPU
  CL_DEVICE_MAX_COMPUTE_UNITS:          13
  CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS:   3
  CL_DEVICE_MAX_WORK_ITEM_SIZES:        1024 / 1024 / 64
  CL_DEVICE_MAX_WORK_GROUP_SIZE:        1024
  CL_DEVICE_MAX_CLOCK_FREQUENCY:        1329 MHz
  CL_DEVICE_ADDRESS_BITS:               64
  CL_DEVICE_MAX_MEM_ALLOC_SIZE:         1024 MByte
  CL_DEVICE_GLOBAL_MEM_SIZE:            4096 MByte
  CL_DEVICE_ERROR_CORRECTION_SUPPORT:   no
  CL_DEVICE_LOCAL_MEM_TYPE:             local
  CL_DEVICE_LOCAL_MEM_SIZE:             48 KByte
  CL_DEVICE_MAX_CONSTANT_BUFFER_SIZE:   64 KByte
  CL_DEVICE_QUEUE_PROPERTIES:           CL_QUEUE_OUT_OF_ORDER_EXEC_MODE_ENABLE
  CL_DEVICE_QUEUE_PROPERTIES:           CL_QUEUE_PROFILING_ENABLE
  CL_DEVICE_IMAGE_SUPPORT:              1
  CL_DEVICE_MAX_READ_IMAGE_ARGS:        256
  CL_DEVICE_MAX_WRITE_IMAGE_ARGS:       16
  CL_DEVICE_SINGLE_FP_CONFIG:           denorms INF-quietNaNs round-to-nearest round-to-zero round-to-inf fma

  CL_DEVICE_IMAGE <dim>                 2D_MAX_WIDTH     16384
                                        2D_MAX_HEIGHT    16384
                                        3D_MAX_WIDTH     4096
                                        3D_MAX_HEIGHT    4096
                                        3D_MAX_DEPTH     4096

  CL_DEVICE_EXTENSIONS:                 cl_khr_global_int32_base_atomics
                                        cl_khr_global_int32_extended_atomics
                                        cl_khr_local_int32_base_atomics
                                        cl_khr_local_int32_extended_atomics
                                        cl_khr_fp64
                                        cl_khr_byte_addressable_store
                                        cl_khr_icd
                                        cl_khr_gl_sharing
                                        cl_nv_compiler_options
                                        cl_nv_device_attribute_query
                                        cl_nv_pragma_unroll
                                        cl_nv_d3d10_sharing
                                        cl_khr_d3d10_sharing
                                        cl_nv_d3d11_sharing


  CL_DEVICE_COMPUTE_CAPABILITY_NV:      5.2
  NUMBER OF MULTIPROCESSORS:            13
MapSMtoCores SM 5.2 is undefined (please update to the latest SDK)!
  NUMBER OF CUDA CORES:                 4294967283
  CL_DEVICE_REGISTERS_PER_BLOCK_NV:     65536
  CL_DEVICE_WARP_SIZE_NV:               32
  CL_DEVICE_GPU_OVERLAP_NV:             CL_TRUE
  CL_DEVICE_KERNEL_EXEC_TIMEOUT_NV:     CL_TRUE
  CL_DEVICE_INTEGRATED_MEMORY_NV:       CL_FALSE
  CL_DEVICE_PREFERRED_VECTOR_WIDTH_<t>  CHAR 1, SHORT 1, INT 1, LONG 1, FLOAT 1, DOUBLE 1

We seem to both be using the official nvidia drivers v364.72, so it can't be that.

I must be either missing some other device drivers or have something else installed that messes with stuff.

I'm more leaning towards "something is messing with stuff" because first running some OpenGL initialization was able to make OpenCL work. If I were missing drivers, loading OpenGL stuff really shouldn't matter.

SuspiciousDroid wrote

I must be either missing some other device drivers or have something else installed that messes with stuff.

I'm more leaning towards "something is messing with stuff" because first running some OpenGL initialization was able to make OpenCL work. If I were missing drivers, loading OpenGL stuff really shouldn't matter.

The GLProfile.initSingleton() call that you inserted just changes exactly when the nvidia library gets loaded. Putting that call in a static block makes the init happen at class load time, instead of the first time you use the API.

One thing to check would be to make sure you don't have any outdated nvidia or OpenGL libraries lying around your hard drive anywhere. The loadLibrary() call searches a bunch of different paths to find stuff, so it might be finding an old one in the wrong place unless you force it. That might be why the non-nvidia OpenCL demos work -- they might have hard-coded their library path, or they may be searching manually in a different order.

Another thing to try is Process Explorer (https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx). It will show you the path to every DLL loaded by a process, so you can compare where your working vs. non-working programs are getting their DLLs, or see what else might be getting loaded.

Putting that call in a static block makes the init happen at class load time, instead of the first time you use the API.

I didn't even make it static, I just put it one line above CLContext.create ^^

I also checked with process explorer, but the location of the OpenCL DLL doesn't change between the two versions, both load it from system32.

The only real difference that the non-working version is missing some DLLs, all of which I think are related me including JOGL in the working one.

Here's the list of missing DLLs, just so I don't skip over any details:
- d3d9.dll
- dbghelp.dll
- dciman32.dll
- ddraw.dll
- dwmapi.dll
- glu32.dll
- jawt.dll
- jogl_desktop.dll
- jogl_mobile.dll
- msctf.dll
- nativewindow_awt.dll
- nativewindow_win32.dll
- ntmarta.dll
- nvcompiler.dll <-- maybe it's missing this one? nvidia compiler seems important
- nvd3dumx.dll
- nvoglv64.dll
- nvSCPAPI64.dll
- nvspcap64.dll
- opengl32.dll
- sspicli.dll
- winhttp.dll
- winsta.dll
- wtsapi32.dll


On a side note: Why does JOGL load DLLs used for HTTP and Windows Remote Desktop sessions?

SuspiciousDroid wrote

The only real difference that the non-working version is missing some DLLs, all of which I think are related me including JOGL in the working one.

Well, the missing DLLs include pretty much all the Jogamp DLLs, so I can see why it wouldn't work :) Question is, why are they missing? The initSingleton() call is supposed to be optional -- it gets called when you try to create a context, if you haven't already called it. So for some reason it must be failing when you create your context, but working if you call it earlier.

If you look inside the code for those initSingleton() calls, you can see that in some places it's swallowing exceptions silently. So perhaps that's why we're not seeing an error message. So one possibility is, you could try running against a debug version of Jogamp, and trace through the execution of initSingleton() to see where the error is happening :) I hesitate to suggest this, but you seem pretty hardcore, since you've already compiled the demos and installed the C toolchain :) Or if you don't want to wrestle with the build, I can supply you with a debug build and source zips, if you'd like to try this. Otherwise, you could just use the workaround you've found, though that is a bit unsatisfying.

SuspiciousDroid wrote

On a side note: Why does JOGL load DLLs used for HTTP and Windows Remote Desktop sessions?

That, I don't know. This must be some indirect dependency of the DLLs. You could probably take a look with Windows Dependency Walker and see who's pulling those in: http://www.dependencywalker.com/

For the hacky, now working version I added
  a) the JOGL dependency (which at least according to http://jogamp.org/wiki/index.php/JOCL_FAQ isn't required for OpenCL stuff) and
  b) added that call to GLProfile.initSingleton() - so it's stuff that should be unrelated to OpenCL.
      The version that didn't work did not initialize any OpenGL stuff.

So it's not unexpected that this version loads more libraries for all the OpenGL stuff, including some jogamp dlls related to OpenGL.

The non-working version that does not contain the JOGL dependency still loads OpenCL.dll, jocl.dll, gluegen-rt.dll, nvapi64.dll, nvopencl.dll and some others, so it's not like is obviously missing dependencies.

For the DLLs, I was interested in what path they're loaded from -- if they come from a different path in the working vs. non-working version, that would be a smoking gun. Dependencies between DLLs should be automatically loaded by Windows, but if you've got a bad version of some DLL out there on your system, loading jocl.dll could cause it to load an incompatible version of some other DLL that might cause the kind of weird behavior you're seeing.

Also, another thing you can try is setting some Jogamp debug flags on the command line, and comparing the dumps of your working vs. non-working versions. Try setting

-Dnewt.debugl -Dnativewindow.debug -Djogl.debug -Djocl.debug

on your command line, and this should produce logs with tons of info in them. Then post the logs here, and I'll take a look (if you can't see any obvious difference). You might start with just

-Djocl.debug

 to see if there's something obvious, then add more if not.

SuspiciousDroid, do you really need OpenCL? Because otherwise you may use OpenGL compute shaders

I've been trying to solve this problem myself.  I've tried lots of things, and am finally making progress by following these steps:
http://streamcomputing.eu/wp-content/uploads/kalins-pdf/singles/install-opencl-on-debianubuntu-orderly.pdf

Specifically:
1) do:
  echo "/usr/local/cuda/lib64" > /etc/ld.so.conf.d/opencl-vendor-nvidia.conf
  echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/opencl-vendor-nvidia.conf

then run:
  ldconfig

2) then do this (your actual directories and ".so" suffixes may vary)
  sudo ln -s /usr/local/cuda/lib64/libOpenCL.so /usr/lib64/libOpenCL.so.1.0
  sudo ln -s /usr/local/cuda/lib/libOpenCL.so /usr/lib/libOpenCL.so.1.0

and also:
  sudo ln -s /usr/lib64/libOpenCL.so.1.1 /usr/lib64/libOpenCL.so.1
  sudo ln -s /usr/lib64/libOpenCL.so.1 /usr/lib64/libOpenCL.so
  sudo ln -s /usr/lib64/libOpenCL.so.1.1 /usr/lib64/libOpenCL.so.1.0

3) AND MOST IMPORTANTLY:
check "/etc/OpenCL/vendors/nvidia.icd" to make sure they reference the OpenCL library!
My Ubuntu setup only had a reference to libopencl.so, even though my computer didn't have such a file installed.  All my shared OpenCL libraries were named "libOpenCL.so", so I added this to the nvidia.icd file:
"libOpenCL.so"
"libcuda.so"

Now I get error -1001 (No valid ICDs found) instead of -5 (CL_OUT_OF_RESOURCES).  So I'm not done yet, but I'm making progress!  I'm thinking I need to install some ICD software that matches my driver, hope that's it.

Good luck!  Hope this helps someone.

Sorry, in step (3), I meant to say that in my computer, "/etc/OpenCL/vendors/nvidia.icd" only had a reference to "libnvidia-opencl.so" and not "libOpenCL.so".  Adding "libOpenCL.so" (and making sure my computer knows where to find it) seems to help move things along.

Ok, I've solved the problem.  I was writing a program that forked a process after having initialized the NVIDIA device in the parent process.  Apparently Nvidia OpenCL doesn't like this.

I've modified my code to delay initializing the GPU until after the fork, this allows me to run multiple concurrent OpenCL forked processes for one program.  I can now create the context without getting any errors. :)