playing audio from an AVI file...

Hi,
I am trying to play the audio track of an avi file using joal.

I am hoping what I am doing wrong will appear simple to someone and I will very much appreciate some help thanks. I will explain where I am so far...

the avi is encoded LINEAR,8 bits, unsigned

I create a joal buffer using...

configure(ByteBuffer data, int format, int freq)

data is ByteBuffer with the chunk buffer written directly.
format=MONO8
freq=11024;

AudioSystem3D.generateSource(MyJoalBuffer);

I am guessing that the above call is expecting the buffer to be the contents of a wav file and that is why I am hearing nonsense. However, least I am hearing something :)

I am guessing there is another way to have the AudioSystem3D to accept a raw buffer?
Not sure if I need to use a codec as the format is fairly basic.

please help thanks :)

On 03/14/2014 01:48 AM, TheLittleP [via jogamp] wrote:
> Hi,
> I am trying to play the audio track of an avi file using joal.
>
..
>
> the avi is encoded LINEAR,8 bits, unsigned
>
...
> ... why I am hearing nonsense. However, least I am
> hearing something :)

Sure the AVI file does not contain a lossy encoded audio stream,
like MP123 .. etc ?

If so .. you might need to use GLMediaPlayer of JOGL,
which offers a binding to libav/ffmpeg.

Otherwise .. pls ignore this thought.

hiya,

thanks for that. I will investigate tomorrow.

im not sure about MP123. I have only been able to determine it is LINEAR mono 8 bits. I cannot seem to find any details on what "WAVEFORMATEX wfx" represents in the audio header. the handler FCC is zero.

I have also been unable to determine what buffer formats the AudioSystem3D accepts.

I will definitely see if I can use GLMediaPlayer.

many thanks

hiya,

how can I tell how the audio buffer is encoded?? I have spent hours digging around trying to unravel this.

The details of the audio on the AVI is LINEAR, 8 bit mono - I was hoping the Source could play the raw chunk buffer data but it cant. there isn't any information on how the audio buffer is encoded, or I don't know how to find out.

It seems WAVLoader and WAVData simply pass the raw buffer of the WAV InputStream to the Source and it plays.

I will keep digging through the source code of GLMediaPlayer to try and find out how it detects the encode and what it does to decode but I thought I might ask the forum in case anyone can explain and save me hours :)

many thanks
P

2014-03-14 15:23, TheLittleP [via jogamp] skrev:
> hiya,
>
> how can I tell how the audio buffer is encoded?? I have spent hours digging around trying to unravel this.
>
> The details of the audio on the AVI is LINEAR, 8 bit mono - I was hoping the Source could play the raw chunk buffer data but it cant. there isn't any information on how the audio buffer is encoded, or I don't know how to find out.
>
> It seems WAVLoader and WAVData simply pass the raw buffer of the WAV InputStream to the Source and it plays.

WAVLoader looks for the WAVEfmt header where channels, sample-rate, bytes per second and sample size is stored. WAVLoader currently only support uncompressed WAVE files.
http://jogamp.org/git/?p=joal.git;a=blob;f=src/java/com/jogamp/openal/util/WAVLoader.java;hb=HEAD#l131

Is quite easy to pass raw data from a file to the sound-card if the audio format is known.
WAVLoader  is used by the ALut.alutLoadWAVFile to load simple wav files for the JOAL-demos lessions.
http://jogamp.org/joal-demos/www/devmaster/lesson3.html
http://jogamp.org/joal-demos/www/

>
> I will keep digging through the source code of GLMediaPlayer to try and find out how it detects the encode and what it does to decode but I thought I might ask the forum in case anyone can explain and save me hours :)
>
> many thanks
> P

GLMediaPlayer is using the FFMPEGMediaPlayer backend on desktop JogAmp systems.

By using libffmpeg/libav the audo format auto-detection is handled by libffmpeg/libav.
We can then tell how the audio buffer is encoded by simply asking the AVCodecContext that belong to the ffmpeg/libav audio decoder.

I recommend you to take a look at the ffmpeg tutorials at dranger.com
http://dranger.com/ffmpeg/tutorial03.html - Tutorial 03: Playing Sound
http://dranger.com/ffmpeg/data.html#AVCodecContext

The jogamp FFMPEGMediaPlayer backend code work quite similar to this tutorial,
you can find the audio buffer detection code inside src/jogl/native/libav/ffmpeg_impl_template.c here:
http://jogamp.org/git/?p=jogl.git;a=blob;f=src/jogl/native/libav/ffmpeg_impl_template.c;hb=HEAD#l872

We then check the AudioSink (usually JOAL ALAudioSink) if the audio format is supported by the sound-card using FFMPEGMediaPlayer.isAudioFormatSupported:
http://jogamp.org/git/?p=jogl.git;a=blob;f=src/jogl/classes/jogamp/opengl/util/av/impl/FFMPEGMediaPlayer.java;hb=HEAD#l451

For situations when the decoded audio-buffer format do not match any of the sound cards audio format then we can then ask libav/libffmpeg to re-sample all audio to match the sound cards audio format.
This fallback happens inside ffmpeg_impl_template.c using SWRESAMPLE here:
http://jogamp.org/git/?p=jogl.git;a=blob;f=src/jogl/native/libav/ffmpeg_impl_template.c;hb=HEAD#l971

Cheers
Xerxes

I have found your post really interesting thanks.

I have been going through all the links in trying to understand what I need to do. but failing miserably lol.

The audio track header says the chunk data is LINEAR  pcm unsigned with a bit rate of 11024. The data chunk size is 11024 so does that mean it is uncompressed? ie I do not need a codec such as ulaw, alaw, mpa etc.

The Source will play a raw chunk from a wav file ok and I understand this is uncompressed linear.

I really don't understand. JMStudio plays the audio ok and it uses the JavaSoundRenderer. The JavaSoundRenderer only uses a codec if the raw is in alaw or ulaw, if it is linear it just passes the raw chunk.

please help thanks

in trying to understand the difference between a raw chunk in a wav file and a raw chunk in my AVI file I did consider that as the AVI audio reports it is LINEAR PCM I tried running the chunk through both a alaw and a ulaw codec but without any success.

On 03/14/2014 10:52 PM, TheLittleP [via jogamp] wrote:
> in trying to understand the difference between a raw chunk in a wav file and a
> raw chunk in my AVI file I did consider that as the AVI audio reports it is
> LINEAR PCM I tried running the chunk through both a alaw and a ulaw codec but
> without any success.
>

Hmm .. we do have a WAV loader in JOAL,
and if it can parse this file ..

If not, and you think it should, i.e. some uncompressed non-lossy native PCM
data - then we may need to fix the Wav-Loader, i.e. add support if possible.

Otherwise .. i.e. if it is a codec .. -> ffmpeg/libav -> GLMediaPlayer

Sorry for the confusion ..

~Sven

hiya,

I have the audio playing using JavaSound the format is...

AudioFormat audioFormat=new AudioFormat(Encoding.PCM_UNSIGNED,11024.0f,8,1,1,11024.0f,false);

using AudioSystem3D I tried the following...

AudioSystem3D buffer format only offers 4 states so I did a little digging in the structures and found that ALHelpers helps put formats together so I came up with this...

 private static int getFormat(int _channels,int _bits,boolean _signed,boolean _floatingPoint)
  {
  AL al=ALFactory.getAL();
  ALExt alExt=ALFactory.getALExt();
  int sampleType=ALHelpers.getALSampleType(_bits,_signed,!_floatingPoint);
  int channelLayout=ALHelpers.getDefaultALChannelLayout(_channels);
  int format=ALHelpers.getALFormat(channelLayout,sampleType,true,al,alExt);
  return format;
  }

Buffer.configure(bb,getFormat(1,8,false,false),11024);

but it didn't work. I guess the format in AL works different that what I have deducted from the JavaDocs.

maybe someone with more experience could see what is wrong.

thanks

I have just realised that my little routine returns AL_FORMAT_MONO8.

The wav file that AudioSystem3D plays ok is 16 bit signed.

So I tried converting my Mono8 unsigned to a signed 16bit.

I tried little endian and big endian but still I cannot get it to play.

heres my routine...

 public static byte[] covertUnsigned8ToSigned16(byte[] _unsigned)
  {
  byte[] result=new byte[_unsigned.length*2];
  int pos=0;
  for(int i=0;i<_unsigned.length;i++)
   {
   int val=(_unsigned[i] & 0xff);
   byte lo=(byte)(val & 0xff);
   byte hi=(byte)((val >> 8) & 0xff);

   result[pos]=lo;
   pos++;
   result[pos]=hi;
   pos++;
   }
  return result;
  }

still didn't work.

I really would appreciate some help is how to get PCM_UNSIGNED MONO 8 bit to play in AudioSystem3D thanks.

please upload a sample wav file with this 8bit encoding.

Here is the dropbox link to my test AVI which I have been attempting to play the audio on joal.

The wav file aa.wav I have successfully played using AudioSystem3D

https://www.dropbox.com/s/yexz08u6l9hwdh0/PETE.avi
https://www.dropbox.com/s/bz9g8vj3gj9aay0/aa.wav

Playing the audio track successfully using java sound says the format is..

AudioFormat audioFormat=new AudioFormat(Encoding.PCM_UNSIGNED,11024.0f,8,1,1,11024.0f,false);

The wav file aa.wav I have successfully played using AudioSystem3D

it reports its format is...
ff=WAVE (.wav) file, byte length: 846748, data format: PCM_SIGNED 44100.0 Hz, 16 bit, stereo, 4 bytes/frame, little-endian, frame length: 211676

I have tried using javax.sound.AudioSystem to convert the AVI track to the same format as aa.wav, but mono instead of stereo,  but it still wont play.

I will appreciate your insight thanks :)

The file PETE.avi contains both audio and video
I will quickly demonstrate playback using the GLMediaPlayer examples of this file.

#Audio only playback using GLMediaPlayer
2.1.5/jogamp-all-platforms/jar$ java -cp joal.jar:jogl-test.jar:gluegen-rt.jar:jogl-all.jar com.jogamp.opengl.test.junit.jogl.demos.es2.av.CrossFadePlayer PETE.avi

GLMediaPlayer[Playing, vSCR 2147483647, frames[p 0, d 0, t 0 (45.399 s), z 0 / 166], speed 1.0, 2330928 bps, hasSW true, Texture[count 0, free 0, dec 0, tagt 0xde1, ifmt 0xffffffff, fmt 0xffffffff, type 0xffffffff], Video[id -2, <unknown>, 0x0, glOrient false, 0.0 fps, 0.0 fdur, 0 bps], Audio[id 1, <Audio: pcm_u8, 11024 Hz, 1 channels, u8, 88 kb/s>, 88192 bps, 500490 frames], uri file:/home/familjen/Hämtningar/PETE.avi]
ALAudioSink[init true, playRequested true, device OpenAL Soft, ctx 0x310ecc43, alSource 1, chosen AudioDataFormat[sampleRate 11024, sampleSize 8, channelCount 1, signed false, fixedP true, packed, little-endian], al[chan Mono, type u8, fmt 0x1100, soft true], playSpeed 0.0, buffers[total 16, avail 14, queued[2, apts 0, 2004 ms, 22048 bytes], queue[g 16, l 32]


#Audio & Video playback using GLMediaPlayer
2.1.5/jogamp-all-platforms/jar$ java -cp joal.jar:jogl-test.jar:gluegen-rt.jar:jogl-all.jar com.jogamp.opengl.test.junit.jogl.demos.es2.av.MovieCube -file PETE.avi

reports:
Video[id 0, <Video: mjpeg, yuvj422p, 320x240>, 320x240, glOrient false, 15.00015 fps, 66.666 fdur, 0 bps], Audio[id 1, <Audio: pcm_u8, 11024 Hz, 1 channels, u8, 88 kb/s>, 88192 bps, 500490 frames]



Getting the GLMediaPlayer to connect with the AudioSystem3D will require some API changes inside the JOGL and JOAL project.
At a minimum, AudioSystem3D and GLMediaPlayers ALAudioSink has to use the same ALCcontext in order for the ALAudoSink "Source" to be positioned inside the AudioSystem3D.
This will require an enhancement bug-report.

thank you very much for taking the time and effort in investigating my problem.

I am sure many people will benefit with the addition of being able to use GLMediaPlayer with JOAL.

However I have written my own player which processes the raw data buffers.

I am a little miffed as to why joal doesn't support com.jopamp.opengl.util.av.AudioSink.AudioFormat or something similar.

I notice jogamp.opengl.openal.av.ALAudioSink generates sources and buffers the same way AudioSystem3D does so tomorrow I will dig around the source code to see how the AudioFormat is implemented to which I will write a method, maybe create my own version of AudioSystem3D that supports AudioFormat.

If my AVI audio plays using ALAudioSink then I believe this will be my solution. If successful I will be happy to share my alterations to AudioSystem3D that enables it to support AudioFormat.

regards

On 03/17/2014 04:49 AM, TheLittleP [via jogamp] wrote:
> thank you very much for taking the time and effort in investigating my problem.
>
> I am sure many people will benefit with the addition of being able to use
> GLMediaPlayer with JOAL.
>
> However I have written my own player which processes the raw data buffers.
>
> I am a little miffed as to why joal doesn't support
> com.jopamp.opengl.util.av.AudioSink.AudioFormat or something similar.

We are talking about stream _deplexing_ _and_ audio _decoding_ here!

First your stream seems to be a multiplexed one, i.e. combines many stream (A,
V, ..).

Second a stream may be encoded .. or not.

Q: What is OpenAL/JOAL ?
It provides you access to the soundcard, i.e. allow you to stream
bits to the channels. It also gives you some 3d spatial abilities.

Q: What is OpenAL not ?
A decoder and demultiplexer for a/v streams.

>
> I notice jogamp.opengl.openal.av.ALAudioSink generates sources and buffers the
> same way AudioSystem3D does so tomorrow I will dig around the source code to
> see how the AudioFormat is implemented to which I will write a method, maybe
> create my own version of AudioSystem3D that supports AudioFormat.

ALAudioSink .. simply receives the _demultiplexed_ and _decoded_
bitstream from libav/ffmpeg.
Then it funnels them to OpenAL/JOAL ..

>
> If my AVI audio plays using ALAudioSink then I believe this will be my
> solution. If successful I will be happy to share my alterations to
> AudioSystem3D that enables it to support AudioFormat.

A demultiplexer is a very complex thing and not part of OpenAL/JOAL.

We added GLMediaPlayer (libav/ffmpeg .. or otherwise)
as a nice feature to 'show off'.

It is not exactly part of OpenGL/JOGL .. however .. it must be available
somewhere / somehow. And I was too lazy to setup another project :)


Q: What could you do ?

As you just said - use the ALAudioSink and depend on JOGL.
Look at the player codes provided to you .. it's not that complicated.

Hope all this helps ..

~Sven

hiya sven,
thanks for showing interest in my problem and the information you have provided.

I have written my own avi demultiplexer and decoder. the video plays fine but its just that I cant seem to get JOAL to play the audio buffers. the audio plays fine in java sound but I would really like to take advantage of joal for the 3D effects.

I wanted to have a play with ALAudioSink but as I cant instantiate AudioFormat, as shown below. I find it rather odd considering FFMPEGMediaPlayer uses it...

.\test\Test.java:15: error: cannot find symbol
import com.jogamp.opengl.util.av.AudioSink.AudioFormat;
                                          ^
  symbol:   class AudioFormat
  location: interface AudioSink
1 error

I did try converting the buffer to the same format as the working wav but it didn't work for a reason I am not aware of. heres my code...

 public static byte[] test(byte[] _unsigned)
  {
  try{
   AudioFormat audioFormatIn=new AudioFormat(Encoding.PCM_UNSIGNED,11024.0f,8,1,1,11025.0f,false);
   ByteArrayInputStream bais=new ByteArrayInputStream(_unsigned);
   AudioInputStream aisin=new AudioInputStream(bais,audioFormatIn,_unsigned.length);

   AudioFormat audioFormatOut=new AudioFormat(Encoding.PCM_SIGNED,44100.0f,16,1,2,44100.0f,false);
   AudioInputStream aisout=AudioSystem.getAudioInputStream(audioFormatOut,aisin);
   byte[] out=new byte[88200];
   aisout.read(out);

DB.print("in FORMAT="+audioFormatIn);
DB.print("out FORMAT="+audioFormatOut);

   return out;
   }catch(Throwable t){
   DB.handle(t);
   return null;
   }
  }

as you can see the buffer is raw decoded so I don't understand why it is not working.
why cant I reference com.jogamp.opengl.util.av.AudioSink.AudioFormat ??
regards

2014-03-17 12:47, TheLittleP [via jogamp] skrev:
> hiya sven,
> thanks for showing interest in my problem and the information you have provided.
>
> I have written my own avi demultiplexer and decoder. the video plays fine but its just that I cant seem to get JOAL to play the audio buffers. the audio plays fine in java sound but I would really like to take advantage of joal for the 3D effects.

Please publish your avi demultiplexer and decode code inside a git repository under a foss license and I can take a look at it.

>
> I wanted to have a play with ALAudioSink but as I cant instantiate AudioFormat, as shown below. I find it rather odd considering FFMPEGMediaPlayer uses it...

com.jogamp.opengl.util.av.AudioSink.AudioFormat
is not the same thing as
javax.sound.sampled.AudioFormat

Jogamp do not use any of the javax.sound.* classes because these classes are not available on mobile systems such as Android.
If you find jogamp code that still uses javax.sound.* then it is a bug: https://jogamp.org/bugzilla/show_bug.cgi?id=684

>
> .\test\Test.java:15: error: cannot find symbol
> import com.jogamp.opengl.util.av.AudioSink.AudioFormat;
>                                           ^
>   symbol:   class AudioFormat
>   location: interface AudioSink
> 1 error

com.jogamp.opengl.util.av.AudioSink.AudioFormat is a local class of com.jogamp.opengl.util.av.AudioSink
thats why you cant find the symbol.

>
> I did try converting the buffer to the same format as the working wav but it didn't work for a reason I am not aware of. heres my code...
>
>  public static byte[] test(byte[] _unsigned)
>   {
>   try{
>    AudioFormat audioFormatIn=new AudioFormat(Encoding.PCM_UNSIGNED,11024.0f,8,1,1,11025.0f,false);
>    ByteArrayInputStream bais=new ByteArrayInputStream(_unsigned);
>    AudioInputStream aisin=new AudioInputStream(bais,audioFormatIn,_unsigned.length);
>
>    AudioFormat audioFormatOut=new AudioFormat(Encoding.PCM_SIGNED,44100.0f,16,1,2,44100.0f,false);
>    AudioInputStream aisout=AudioSystem.getAudioInputStream(audioFormatOut,aisin);
>    byte[] out=new byte[88200];
>    aisout.read(out);
>
> DB.print("in FORMAT="+audioFormatIn);
> DB.print("out FORMAT="+audioFormatOut);
>
>    return out;
>    }catch(Throwable t){
>    DB.handle(t);
>    return null;
>    }
>   }
>
> as you can see the buffer is raw decoded so I don't understand why it is not working.
> why cant I reference com.jogamp.opengl.util.av.AudioSink.AudioFormat ??
> regards

looks like your code is written for javax.sound.sampled.AudioFormat
try change the import.
if you run into issues with javax.sound.sampled.AudioFormat then you need to ask on
http://openjdk.java.net/projects/audio-engine/
who maintains this API.

Cheers
Xerxes

hiya xerxes,

I wonder what could be gained from examining my avi extracter code when the problem I am trying to solve is why an audio buffer doesn't work in joal when it works fine in java sound. I would say the audio buffer extraction is good.

the problem I have with joal is that it doesn't provide an "AudioFormat" equivalent so it is expecting the audio buffer to be different that what I am providing.

I used javax.sound.sampled.AudioFormat to convert the audio buffers to the format that works from the wav file. I don't understand why that didn't work. my code for that is in a previous post.

I will copy the jogamp source code to my test package so I can play with the AudioFormat class.

regards

I don't know how to use git hub. Everything I code is done using a text editor. Its just the way I am, bit old school I guess.

I have put the test project into a zip...

https://www.dropbox.com/s/4jvpfjomau1p0i1/test%20avi.zip

I compile the code using javac.exe with the classpath set to the locations of the support jars.

and I run using javaw.exe

its all very rough and ready. the purpose was to get it working and then create a nice package from the result. I realise the frame handling isn't ideal so with that in mind I would like anyone examining to focus on why joal cannot play the audio buffer.

many thanks