Maximum size of CLBuffer<FloatBuffer>

Hi all,

here is my problem. When I run:

context.createFloatBuffer(bufLength, Mem.WRITE_ONLY);

I need to make sure that "bufLength" is in the range of [0, 2^29-1] otherwise the buffer will throw an error like this:

java.lang.IllegalArgumentException: Negative capacity: -2147483648
        at java.nio.Buffer.<init>(Buffer.java:191)
        at java.nio.ByteBuffer.<init>(ByteBuffer.java:276)
        at java.nio.ByteBuffer.<init>(ByteBuffer.java:284)
        at java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:89)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:119)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
        at com.jogamp.common.nio.Buffers.newDirectByteBuffer(Buffers.java:73)
        at com.jogamp.common.nio.Buffers.newDirectFloatBuffer(Buffers.java:115)
        at com.jogamp.opencl.CLContext.createFloatBuffer(CLContext.java:316)

Apparently the float buffer will instantiate a byte buffer, i.e. the length for the byte buffer is now multiplied by 4.
I guess this results in an overflow of the integer array indexing because 2^29*4 = 2^31, which is already considered to be a negative number!
This effectively limits the CLMemory usage to < 2GB for a single float buffer. If float could be indexed directly, the range would be extended to < 8GB. We are working in the field of medical image processing where we are dealing with large volumes of up to 1024^3 = 2^30 float values just in case one might wonder why we would need such large linear data arrays.

Is there a known solution to this problem? Or can you give me a hint on how to implement a workaround for such large arrays?

Thanks
Martin

I'm not sure how we'd get around this, since java.nio.ByteBuffer.allocateDirect() takes an int size, and in Java int is always 32-bit signed two's complement. Even array indices in Java are 32-bit ints, so a one-dimensional array is no bigger. The native OpenCL clCreateBuffer() function can create bigger buffers if CL_DEVICE_ADDRESS_BITS from clGetDeviceInfo() is 64, but you can't create a CL buffer from a Java direct buffer that way, since JOCL uses the direct buffer's size to call clCreateBuffer() with :)

One option might be to subdivide your buffer into chunks of 1GB or so, and process the chunks one at a time. Even on a huge graphics card, you'd only have 12 chunks to deal with. You'd have to do something like this if your array is bigger than your physical memory anyway, which it sounds like you're getting close to.

On 11/18/2014 03:23 AM, Wade Walker [via jogamp] wrote: GlueGen's MappedByteBufferInputStream employs using NIO memory chunks,
here a file's memory mapped region:
  <http://forum.jogamp.org/How-to-delete-direct-Buffers-tp4033631p4033659.html>

IMHO you can use that .. or we may add a specialization.

~Sven