Browser decode frame by frame example

[initialneil]

Thanks for the great code!

I'm using Broadway to decode mp4 for frames used as THREEJS texture. Here's my example for reference.

1. to enable Broadway, add <script> to index.html, so that Broadway, Player, etc. become available in all other js files.

   <!-- index.html -->
   <script type="text/javascript" src="Broadway/mp4.js"></script>
   <script type="text/javascript" src="Broadway/Decoder.js"></script>
   <script type="text/javascript" src="Broadway/YUVCanvas.js"></script>
   <script type="text/javascript" src="Broadway/Player.js"></script>
   <script type="text/javascript" src="Broadway/stream.js"></script>

   Because I put those js files in the folder Broadway, I needed to change the code in Player.js line 57

   this._config.workerFile = this._config.workerFile || "Decoder.js";
   // to 
   this._config.workerFile = this._config.workerFile || "Broadway/Decoder.js";

2. After reading the code in mp4.js, I decided to use the Mp4Player class to decode mp4 file in the background.

   var url = 'test.mp4';
   var player = new MP4Player(new Stream(url), true, false);
   var textures = [];
   
   player.avc.onPictureDecoded = function (buffer, width, height, infos) {
       var size = width * height;
       var data = new Uint8Array(3 * size);
       for (var y = 0; y < height; y++) {
           for (var x = 0; x < width; x++) {
               data[((y * width) + x) * 3 + 0] = buffer[((y * width) + x) * 4 + 0];
               data[((y * width) + x) * 3 + 1] = buffer[((y * width) + x) * 4 + 1];
               data[((y * width) + x) * 3 + 2] = buffer[((y * width) + x) * 4 + 2];
           }
       }
   
       let texture = new THREE.DataTexture(data, width, height, THREE.RGBFormat);
       texture.flipY = true;
       textures.push(texture);	
   };
   
   player.play();

• These are some tricky parts.
  • The third parameter of Mp4Player, which is webgl, must be false. When the webgl is true, you will get YUV420p data instead of RGBA.
  • The resolution is critical. THREEJS requires resolution dividable by 2 and no larger than 1024. Larger texture will cause iOS browser to crash. You heard right, neither Windows nor Android will crash, only iOS's browser will crash. You can do resize when copy data from buffer.
  • The decoded buffer cannot be given to THREE.DataTexture directly, I had to make a copy. Don't know why.
  • The decoded buffer needs a Y-axis flip before set as texture. It's because OpenGL texture memory start from bottom-left, instead of top-left.

Hope the example helps.

[huangfe]

@initialneil add support for main profile？？？

[initialneil]

@huangfe No. I had to convert my mp4 to baseline.