Quantcast

Maximum size of CLBuffer<FloatBuffer>

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Maximum size of CLBuffer<FloatBuffer>

Martin
Hi all,

here is my problem. When I run:

context.createFloatBuffer(bufLength, Mem.WRITE_ONLY);

I need to make sure that "bufLength" is in the range of [0, 2^29-1] otherwise the buffer will throw an error like this:

java.lang.IllegalArgumentException: Negative capacity: -2147483648
        at java.nio.Buffer.<init>(Buffer.java:191)
        at java.nio.ByteBuffer.<init>(ByteBuffer.java:276)
        at java.nio.ByteBuffer.<init>(ByteBuffer.java:284)
        at java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:89)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:119)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
        at com.jogamp.common.nio.Buffers.newDirectByteBuffer(Buffers.java:73)
        at com.jogamp.common.nio.Buffers.newDirectFloatBuffer(Buffers.java:115)
        at com.jogamp.opencl.CLContext.createFloatBuffer(CLContext.java:316)

Apparently the float buffer will instantiate a byte buffer, i.e. the length for the byte buffer is now multiplied by 4.
I guess this results in an overflow of the integer array indexing because 2^29*4 = 2^31, which is already considered to be a negative number!
This effectively limits the CLMemory usage to < 2GB for a single float buffer. If float could be indexed directly, the range would be extended to < 8GB. We are working in the field of medical image processing where we are dealing with large volumes of up to 1024^3 = 2^30 float values just in case one might wonder why we would need such large linear data arrays.

Is there a known solution to this problem? Or can you give me a hint on how to implement a workaround for such large arrays?

Thanks
Martin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Maximum size of CLBuffer<FloatBuffer>

Wade Walker
Administrator
I'm not sure how we'd get around this, since java.nio.ByteBuffer.allocateDirect() takes an int size, and in Java int is always 32-bit signed two's complement. Even array indices in Java are 32-bit ints, so a one-dimensional array is no bigger. The native OpenCL clCreateBuffer() function can create bigger buffers if CL_DEVICE_ADDRESS_BITS from clGetDeviceInfo() is 64, but you can't create a CL buffer from a Java direct buffer that way, since JOCL uses the direct buffer's size to call clCreateBuffer() with :)

One option might be to subdivide your buffer into chunks of 1GB or so, and process the chunks one at a time. Even on a huge graphics card, you'd only have 12 chunks to deal with. You'd have to do something like this if your array is bigger than your physical memory anyway, which it sounds like you're getting close to.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Maximum size of CLBuffer<FloatBuffer>

Sven Gothel
Administrator
On 11/18/2014 03:23 AM, Wade Walker [via jogamp] wrote:

> I'm not sure how we'd get around this, since
> java.nio.ByteBuffer.allocateDirect() takes an int size, and in Java int is
> always 32-bit signed two's complement. Even array indices in Java are 32-bit
> ints, so a one-dimensional array is no bigger. The native OpenCL
> clCreateBuffer() function can create bigger buffers if CL_DEVICE_ADDRESS_BITS
> from clGetDeviceInfo() is 64, but you can't create a CL buffer from a Java
> direct buffer that way, since JOCL uses the direct buffer's size to call
> clCreateBuffer() with :)
>
> One option might be to subdivide your buffer into chunks of 1GB or so, and
> process the chunks one at a time. Even on a huge graphics card, you'd only
> have 12 chunks to deal with. You'd have to do something like this if your
> array is bigger than your physical memory anyway, which it sounds like you're
> getting close to.
GlueGen's MappedByteBufferInputStream employs using NIO memory chunks,
here a file's memory mapped region:
  <http://forum.jogamp.org/How-to-delete-direct-Buffers-tp4033631p4033659.html>

IMHO you can use that .. or we may add a specialization.

~Sven



signature.asc (828 bytes) Download Attachment
Loading...