Red-colored videos using libav

Hello, I'm trying to have a simple example on playing a video.
As a library for FFMPEGMediaPlayer I've taken libav-11.3-win64.

GLMediaPlayer player;

public void init(GLAutoDrawable drawable) {
    GL2 gl = drawable.getGL().getGL2();

    gl.glClearDepth(1f);
    gl.glEnable(GL.GL_DEPTH_TEST);
    gl.glDepthFunc(GL.GL_LEQUAL);
    player = GLMediaPlayerFactory.create(GLMediaPlayer.class.getClassLoader(),
                 "jogamp.opengl.util.av.impl.FFMPEGMediaPlayer");
    try {
        player.initStream(Uri.cast("BigBuckBunny_320x180.mp4"),
                GLMediaPlayer.STREAM_ID_AUTO, GLMediaPlayer.STREAM_ID_AUTO,
                GLMediaPlayer.TEXTURE_COUNT_DEFAULT);
    } catch (java.net.URISyntaxException e) {
        e.printStackTrace();
        return;
    }
    while (player.getState() != GLMediaPlayer.State.Initialized) {}
    try {
        player.initGL(gl);
    } catch (GLMediaPlayer.StreamException e) {
        e.printStackTrace();
    }
    while (player.getState() != GLMediaPlayer.State.Paused) {
    }
    player.play();
    while (player.getState() != GLMediaPlayer.State.Playing) {
    }
}

public void reshape(GLAutoDrawable drawable, int x, int y, int width, int height) {
    GL2 gl = drawable.getGL().getGL2();
    if (height == 0) height = 1;
    this.width = width;
    this.height = height;
    gl.glViewport(0, 0, width, height);

    gl.glMatrixMode(GL2.GL_PROJECTION);
    gl.glLoadIdentity();
    gl.glOrtho(0, Widget.BASE_WIDTH, 0, Widget.BASE_HEIGHT, 0, 1);
 
    gl.glMatrixMode(GL2.GL_MODELVIEW);
    gl.glLoadIdentity();
}

public void display(GLAutoDrawable drawable) {
    GL2 gl = drawable.getGL().getGL2();
    gl.glClear(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT);
    gl.glLoadIdentity();
    gl.glClearColor(1, 1, 1, 1);

    Texture texture = player.getNextTexture(gl).getTexture();
    texture.enable(gl);
    texture.bind(gl);
    TextureCoords coords = texture.getImageTexCoords();
    gl.glTexEnvi(GL2.GL_TEXTURE_ENV, GL2.GL_TEXTURE_ENV_MODE, GL2.GL_REPLACE);
    gl.glBegin(GL2.GL_QUADS);
    gl.glTexCoord2f(coords.left(), coords.bottom());
    gl.glVertex2i(0, 0);
    gl.glTexCoord2f(coords.left(), coords.top());
    gl.glVertex2i(0, 180);
    gl.glTexCoord2f(coords.right(), coords.top());
    gl.glVertex2i(320, 180);
    gl.glTexCoord2f(coords.right(), coords.bottom());
    gl.glVertex2i(320, 0);
    gl.glEnd();
    texture.disable(gl);
}

Unfortunately, the code above doesn't work as expected. It somehow loses blue and green channels, because avplay.exe plays video properly. Do you know where the problem could be?

It seems, the code to show a texture should work, because I used it draw pictures.

Thanks in advance,
Vitaly Aksenov

Hi

Please provide a full example, not just a sample of code. Imagine that the culprit is elsewhere... Why not trying with FFMPEG just to confirm whether the problem comes from LibAV?

Hello.
It seems to be almost full example. Except that I throw off the animator part. (Do you think it is better to put here the full version? I could share the code on the github, but it is mess right now due to excessive commenting. :-))

At first, I was trying to work with FFMpeg, unfortunately, the jogl couldn't load libraries from it. Even using two different version (I do not remember each, right now) for each x86 and x64 and putting the path into PATH. (I've read the post about not loading the libraries, and it suggested to put into PATH.) Then, I've tried LibAV of two different versions 9 and 11. (Indeed, jogl could load libraries for me only from PATH) And with both versions I have only-red stream.

AV

Yes, post the full version and compare it to this simple example that should work:
https://github.com/sgothel/jogl/blob/master/src/test/com/jogamp/opengl/test/junit/jogl/demos/es2/av/MovieSimple.java

What do you mean by "the jogl couldn't load libraries from it"? If you use the Java library path correctly, it should find the libraries.

Thank you. I'll post sligthly later.

About the libraries, I have the problem similar to: http://forum.jogamp.org/Win7-x64-Trouble-with-ffmpeg-shared-libraries-td4028589.html.

AV

Ok, you're right to use the PATH until we provide a smarter solution. In the meantime, the only solution consists in identifying which native libraries are necessary to load LibAV or FFMPEG successfully (possibly by using DependencyWalker under Windows) and in loading all those libraries in the correct order. You might have to use System.load() (not System.loadLibrary()) to do so.

Maybe MonsieurMax can suggest a less painful solution.

I've uploaded my example to https://github.com/Aksenov239/icpc-live-v2/blob/master/src/main/java/testing/PlayerGL.java. The example is pretty straightforward. (No, I still didn't try to check with SimpleMovie, because it is not enough to easily copy 3 files out of demo-repo. It still needs some files with shaders. So, I need more time to dig into it)

Yes, it still plays in red-color.

Unfortunately, as you could see in the comments at the static section the load of FFMpeg libraries in the order happens (I omitted the full path).

The error there:
Fail: Could not resolve symbol <avcodec_alloc_frame>: not optional, no alternatives.
Fail: Could not resolve symbol <avcodec_get_frame_defaults>: not optional, no alternatives.
Fail: Could not resolve symbol <av_destruct_packet>: not optional, no alternatives.
Fail: Could not resolve symbol <av_pix_fmt_descriptors>: not optional, no alternatives.
LIB_AV Not Matching
LIB_AV Not Matching Compile-Time / Runtime Major-Version

It seems that it is not enough to download ffmpeg-3.2-win64-shared and copy .dll from bin folder. Or probably I'm doing something wrong.

AV

Thank you for posting your code but please run SimpleMovie because it's known to work correctly.

You need at least LibAV codec 55.28.1 to benefit of avcodec_alloc_frame.

I've copied the MovieSimple with all the dependencies and shaders to my project. I works without any problems with my libav library.
Unfortunately, the code is much more compicated (than could be expected). I do not understand everything out of there. And it doesn't really answers what is wrong with my *simple* code...

AV

Hi

At first, you should enable the debug logs to see which libraries are loaded when running MovieSimple:
https://github.com/sgothel/jogl/blob/master/src/jogl/classes/jogamp/opengl/util/av/impl/FFMPEGMediaPlayer.java#L216

Secondly, this feature heavily relies on shaders, mixing it with fixed pipeline terribly old method calls (immediate rendering for example) might cause some troubles:
https://github.com/sgothel/jogl/blob/master/src/jogl/classes/jogamp/opengl/util/av/impl/FFMPEGMediaPlayer.java#L706

You can try to simplify MovieSimple step by step to find what is wrong in your example.

This is wrong:
https://github.com/Aksenov239/icpc-live-v2/blob/master/src/main/java/testing/PlayerGL.java#L77

Use GL_DEPTH_BUFFER_BIT instead of GL_DEPTH_TEST.

You don't ensure that you use the correct texture unit whereas it's done in MovieSimple:
https://github.com/sgothel/jogl/blob/master/src/test/com/jogamp/opengl/test/junit/jogl/demos/es2/av/MovieSimple.java#L796

Thank you very much, Julien. It I'll dig into it more. I wish the problem is in with the bit flag. (Honestly, I'm not sure what they really mean and I copied them out of other examples without thinking due to limited time) Otherwise, it would not be very pleasant to simplify the code for MovieSimple. :-)

By myself, I thought that the main problem is in the proper RGBA format setting. So, it reads in one format and then *simple* texture painter draws in different format. (Like confusing integer with float and etc)

AV

Hi

If you fix the flag and the active unit handling, it won't be enough. If you don't want to use shaders, you'll have to implement the treatment currently done in the fragment shader elsewhere, it's not very difficult for BGR24:
https://github.com/sgothel/jogl/blob/master/src/jogl/classes/jogamp/opengl/util/av/impl/FFMPEGMediaPlayer.java#L802

You have to change the texture and to send the changes with glTexSubImage. This way, you won't need any shader :)

Oh, I see. The pixel format is really YUV. (It is definitely not RGB) :-)
Yeah, I've tried yesterday the flags and indeed they didn't help. I will try your suggestion with Subimage.

Thank you, Julien.

AV

"singleTexComp" should be 'r' in your case as you see only red. You should create your own texture and write it by converting the data from the next texture returned by the player into your newly created texture. If you succeed and if you share your findings, we'll be able to provide a very simple demo working without shaders. I fear that it will be a bit slow, you'll have to minimize the updates, i.e don't call glTexSubImage on an area containing only unchanged pixels, maybe rather use glTexImage2D on the whole (one call of glTexImage2D per display instead of tons of glTexSubImage2D calls).

Julien,
I'm not sure how could I change the Texture. Is there way to get the data out of it?

AV

You can use com.jogamp.opengl.util.texture.Texture.updateImage(GL, TextureData) to update the whole texture. You have to use the buffer of the texture data to store the pixels.

You can use GL2GL3.getTexImage() to get the data out from the texture.

Julien, thank you!
I've written some small image processing part.

Unfortunately, two problems appeared:
1) The texture seems to be shrink in compare to original video. (Some right-bottom part is missing)
2) The artifacts appeared. I mean, there could be the areas covered with blue (or something other) color that seems to be irrelevant to the area. (Like on a broken TV) Probably, there is a problem with corner case (0 and 255) byte values.

The code is still available by the same link.
Probably, you could assume were the mistakes are?..

AV

A small update. I've solved the mentioned problems:
1) Somehow the coordinates of  the texture were wrong. And to draw it I used only 0 and 1.
2) I've bounded the values of colors with 0 and 255.

So, by the link there exists a small working example.

AV