Speeding up writing to JPG file

Hello,

I have been working on an application that needs to render directly to the disk. The output image is pretty big (2550x2362 to be precise).

Currently, I'm using a GLOffscreenAutoDrawable and an AWTGLReadBufferUtil. I render the drawable, then use the buffer utility to write the data to a BufferedImage, and then output said image to a jpg file (the format is important) using standard Java functions.

Correct me if I'm wrong, but isn't it an overkill writing to the BufferedImage first and THEN to the jpg? I know we can't just dump pixels to the jpg file (as it needs to do the whole compression thing), but I assume the Java functions do that when writing the BufferedImage, so the writing to BufferedImage itself doesn't use compression. If I can skip that step and write directly to the file with compression, it might improve the overall speed of the whole process.

The methods that look like they'd be better (i.e. reading into a GLReadBufferUtil and then outputting to a jpg) are actually much worse in performance, because they seem to be allocating new data each time, whereas the AWT utility tries to reuse and update its data.

Unfortunately, the process of writing to the BufferedImage is the bottleneck of the whole operation, as it adds 40-50 milliseconds for each image output (and it stacks up when exporting multiple images). The rendering itself doesn't take long for most inputs. The AWTGLReadBufferUtil utility is reusing the same BufferedImage (I checked), so it's not that.

My guess is that it's not possible to apply the compression unless the data is copied elsewhere for processing, but I want to make sure that this is the case. That being said... is there ANY way I can speed this up? Any help is appreciated.

Thank you.

Hi

Can you indicate which code path is slower please? Post some code.

Hello,

The code that I have now is
drawable.display();
image = bufferUtil.readPixelsToBufferedImage(drawable.getGL(), false);
ImageIO.write(image, "jpg", outputFile);

...where bufferUtil is AWTGLReadBufferUtil.

An alternative code would have been using a GLReadBufferUtil:
drawable.display();
bufferUtil.readPixels(drawable.getGL(), false);
bufferUtil.write(outputFile);

The second approach is slower, because it internally creates a BufferedImage for every layer that I render, whereas the first method reuses the same BufferedImage.

I'm just wondering if there's anything I can do to speed this up (without reducing image size), or an alternative method I might have missed that is better for just writing.

Actually, I wonder whether you really need AWT. You can use GLReadBufferUtil with TextureIO.write() if you want to get rid of AWT and Swing.

I have the exact same issue. I am looking into wether libjpeg-turbo is faster for reading and writing textures

http://www.libjpeg-turbo.org/Documentation/OfficialBinaries

gouessej: The second code example in my previous post uses GLReadBufferUtil.write(File), which simply calls TextureIO.write(File). That internally creates a new BufferedImage every time, and has a noticeably worse performance than using AWTGLReadBufferUtil instead.

Ryan: Thank you, I'll have a look into that later. We're trying to stay away from C++/Java interop, so if there's no Java solution this will get a lower priority than other tasks.

Cmar wrote

gouessej: The second code example in my previous post uses GLReadBufferUtil.write(File), which simply calls TextureIO.write(File). That internally creates a new BufferedImage every time, and has a noticeably worse performance than using AWTGLReadBufferUtil instead.

TextureIO shouldn't create a BufferedImage, maybe you make a confusion with AWTTextureIO or maybe TextureIO uses the default SPI based on AWT instead of our own encoders and decoders.

IIOTextureWriter creates a BufferedImage. Actually, JOGL has its own decoder for JPEG but no encoder yet.

You have two options:
- use IIOTextureWriter as a source of inspiration to implement a better writer based on AWT
- write your own texture writer for JPEG

Then, drop IIOTextureWriter and add your writer into TextureIO.

I made a mistake in claiming TextureIO.write(...) is noticeably slower, when I measured the time I was accidentally comparing jpg with png files... so, my bad >.< It turns out TextureIO+jpg has about the same performance with when using AWT+jpg. But yes, the thing was that IIOTextureWriter uses a BufferedImage too, so whether I was using AWT or not didn't make any difference.

HOWEVER, I managed to gain a massive improvement by simply removing the call to ImageUtil.flipImageVertically(image). I now simply flip the image vertically through the rendering process itself.

Thank you for your help!

I advise you to make a small request for enhancement. This behaviour should be controlled by a flag, it would allow you to disable this flip with a single line of code.

I'll make a request for enhancement anyway. Let me know whether you need some help.

I can confirm that TurboJpeg is a massive improvement over the current implementation for both reading in jpegs as textures and writing jpegs.  There are a few issues such as trying to reduce the byte array copies as the official JNI wrapper for TurboJpeg does not support NIO.

I'm planning to open source what Im working on, if anyone is interested in helping out, i will go through the process early to get more eyes on it.


The example code for writing a jpeg is pretty simple and dropped compression time from ~30ms to 1ms on my sample
    public InputStream encode(ByteBuffer buffer, Dimension dimension) throws IOException {

        final byte[] unneccessaryCopy;
        if (buffer.hasArray()) {
            unneccessaryCopy = buffer.array();
        } else {
            unneccessaryCopy = new byte[buffer.remaining()];
            buffer.get(unneccessaryCopy);
        }
        try (TJCompressor compressor = new TJCompressor(unneccessaryCopy, 0, 0, dimension.width, 0, dimension.height, TJ.PF_RGBX)) {
           
            compressor.setJPEGQuality(90);
            compressor.setSubsamp(TJ.SAMP_420);
           
            return new ByteArrayInputStream(compressor.compress(0),0,compressor.getCompressedSize());
        } catch (Exception ex) {
            throw new IOException("Error in turbojpeg comressor: " + ex.getMessage(), ex);
        }
    }



Ryan Barker
Software Architect
Eharmony.com

I still think that we could write a decent decoder in Java but feel free to share your findings.

Thanks, I'll definitely try TurboJPEG today, I didn't realize it already has java bindings (I was trying to avoid doing that myself). I'll also keep an eye on this thread and see how this thing works out.

Currently the GL -> BufferedImage transfer can be made faster by assuming the target image is TYPE_3BYTE_BGR and using GL_BGR as the GLPixelBuffer/TextureData format, to skip the swizzling post-process. Obviously an encoder to skip the whole BufferedImage step would be better, but this is still an improvement:

import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.nio.ByteBuffer;

import com.jogamp.common.nio.Buffers;
import com.jogamp.opengl.GL;
import com.jogamp.opengl.GL2ES3;
import com.jogamp.opengl.GLException;
import com.jogamp.opengl.util.GLBuffers;
import com.jogamp.opengl.util.GLPixelBuffer;
import com.jogamp.opengl.util.GLPixelBuffer.GLPixelAttributes;
import com.jogamp.opengl.util.GLPixelBuffer.GLPixelBufferProvider;
import com.jogamp.opengl.util.GLPixelStorageModes;
import com.jogamp.opengl.util.texture.TextureData;

public class BufferUtil {
    private GLPixelBuffer readPixelBuffer;
    private TextureData texData;
    private final GLPixelStorageModes psm;
    private final int tmp[] = new int[1];
    private final GLPixelBufferProvider pixelBufferProvider;
    private final GLPixelAttributes pixelAttribs;
	
    public BufferUtil() {
        psm = new GLPixelStorageModes();
        pixelBufferProvider = GLPixelBuffer.defaultProviderNoRowStride;
        pixelAttribs = new GLPixelAttributes(GL.GL_BGR, GL.GL_UNSIGNED_BYTE);
    }
    
    public void read(GL gl, BufferedImage destination) {
        if (destination.getType() != BufferedImage.TYPE_3BYTE_BGR) {
            throw new IllegalArgumentException("Destination image must have a type of TYPE_3BYTE_BGR");
        }
        
        readPixels(gl, destination.getWidth(), destination.getHeight());

        final byte[] imageData = ((DataBufferByte) destination.getRaster().getDataBuffer()).getData();
        ByteBuffer buf = (ByteBuffer) texData.getBuffer();
        if (buf == null) {
            buf = (ByteBuffer) texData.getMipmapData()[0];
        }
        buf.rewind();
        buf.get(imageData);
        buf.rewind();
    }
    
    public BufferedImage createCompatibleImage(int width, int height) {
        return new BufferedImage(width, height, BufferedImage.TYPE_3BYTE_BGR);
    }
    
    private boolean readPixels(final GL gl, final int width, final int height) {
        final int glerr0 = gl.glGetError();
        if (GL.GL_NO_ERROR != glerr0) {
            System.err.println("Info: GLReadBufferUtil.readPixels: pre-exisiting GL error 0x" + Integer.toHexString(glerr0));
        }

        final int readPixelSize = GLBuffers.sizeof(gl, tmp, pixelAttribs.pfmt.comp.bytesPerPixel(), width, height, 1, true);

        if (readPixelBuffer == null || readPixelBuffer.requiresNewBuffer(gl, width, height, readPixelSize)) {
            readPixelBuffer = pixelBufferProvider.allocate(gl, null, pixelAttribs, true, width, height, 1, readPixelSize);
            Buffers.rangeCheckBytes(readPixelBuffer.buffer, readPixelSize);
            try {
                texData = new TextureData(gl.getGLProfile(), pixelAttribs.format, width, height, 0, pixelAttribs, false, false, false, readPixelBuffer.buffer, null);
            }
            catch (final Exception e) {
                texData = null;
                readPixelBuffer = null;
                throw new RuntimeException("can not fetch offscreen texture", e);
            }
        }

        boolean res = readPixelBuffer != null && readPixelBuffer.isValid();
        if (res) {
            psm.setPackAlignment(gl, 1);
            if (gl.isGL2ES3()) {
                final GL2ES3 gl2es3 = gl.getGL2ES3();
                psm.setPackRowLength(gl2es3, width);
                gl2es3.glReadBuffer(gl2es3.getDefaultReadBuffer());
            }
            readPixelBuffer.clear();
            try {
                gl.glReadPixels(0, 0, width, height, pixelAttribs.format, pixelAttribs.type, readPixelBuffer.buffer);
            }
            catch (final GLException gle) {
                res = false;
                gle.printStackTrace();
            }
            readPixelBuffer.position(readPixelSize);
            readPixelBuffer.flip();
            final int glerr1 = gl.glGetError();
            if (GL.GL_NO_ERROR != glerr1) {
                System.err.println("GLReadBufferUtil.readPixels: readPixels error 0x" + Integer.toHexString(glerr1) + " " + width + "x" + height + ", " + pixelAttribs + ", " + readPixelBuffer + ", sz " + readPixelSize);
                res = false;
            }
            
            psm.restore(gl);
        }
        return res;
    }
}

Still working on speeding up the code a bit. I used Bridj to create NIO bindings. Its looking pretty good but pulling bytes out is still a bit slower than I would like. Will try try the code above. Since you all seem interested, I will try and get permission to opensource the work I'm doing. There are a few different pieces and its still a bit of a mess structurally though.

Feel free to email me at $firstInitial$lastname@eharmony.com if you are willing to help me speed things up a bit more.

Ryan Barker
Software Architect
Eharmony.com

This code will only be of benefit to the existing jogl solution for large images, so it'll probably still be much slower than using JpegTurbo. It's only a temporary remedy.

I sent you an email.

A decoder based on TurboJPEG can be useful, there is already a binding based on JNA:
https://github.com/geosolutions-it/imageio-ext/wiki/TurboJPEG-Bindings-1.1

Wouldn't a pure Java decoder supporting at least a subset of the JPEG spec still be useful if it was more efficient than using AWT and Swing?

Look at the bug report 1321, a brand new pure Java JPEG encoder is already planned.

I sent an email to legal regarding open sourcing my work

3 projects

Turbojpeg patch
- Added maven pom.xml
- refactored code to maven structure
- Added bridj implementation for NIO support

Jogl-turbojpeg patch/examplecode
- Added TurboJpegEncoder, TurboJpegDecoder, TurboJpegNioEncoder, TurboJpegNioDecoder and a couple interfaces.
- Code is definitely messy as I'm rapidly iterating
- Prefer to give you guys the code and have you add it as an optional dependency/submodule but we can host it on eharmony github instead if you would rather not

image processing
- idea is to build something like GIMP where you can chain together various shaders. Currently implementing lancoz3 single pass image resizing but plan to add some other things like blur, 3d rounded edges, etc.


Will give performance numbers tomorrow on the various encoders/decoders.

I would prefer you to host it on eharmony github but it doesn't meant that we won't help you if you need some guidance.