Question 252515 in Flutter Question posted in
The official Flutter documentation can be found here.

Converting VideoFrame to JPEG in Agora Flutter

I’m working on a project where I need to apply a TFLite model for each frame of the local user’s video during a video call using the Agora Flutter SDK.

I came across the VideoFrameObserver in the documentation, which seems to be a promising tool for this task. Here’s the code snippet I found:

onCaptureVideoFrame: (VideoSourceType sourceType, VideoFrame videoFrame) async {
  Uint8List yData = videoFrame.yBuffer!;
  Uint8List uData = videoFrame.uBuffer!;
  Uint8List vData = videoFrame.vBuffer!;
  int height = videoFrame.height!;
  int width = videoFrame.width!;

  // how to create JPEG image
  // ???

},

I’ve been trying to figure out how to convert the videoFrame to a JPEG image, but I haven’t been successful so far. I’ve also checked this issue , but the solution provided there didn’t work for me.

My question is : How to create JPEG image from yBuffer, uBuffer, vBuffer ?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Solution provided by brendan-duncan the owner of image package and it's working fine !

    onCaptureVideoFrame: (VideoSourceType sourceType, VideoFrame videoFrame) async {
  final directory = await getTemporaryDirectory();
  final filePath = '${directory.path}/frame.jpeg';
  int height = videoFrame.height!;
  int width = videoFrame.width!;
  final img = Image(width: width, height: height);
  final yData = videoFrame.yBuffer!;
  final uData = videoFrame.uBuffer!;
  final vData = videoFrame.vBuffer!;
  final yStride = videoFrame.yStride!;
  final uStride = videoFrame.uStride!;
  final vStride = videoFrame.vStride!;
          
  for (int y = 0; y < height; y++) {
    var y2 = y ~/ 2;
    for (int x = 0; x < width; x++) {
        var x2 = x ~/ 2;
        var yIndex = y * yStride + x;
        var uIndex = y2 * uStride + x2;
        var vIndex = y2 * vStride + x2;

        int yValue = yData[yIndex].toInt();
        int uValue = uData[uIndex].toInt();
        int vValue = vData[vIndex].toInt();

        // YUV to RGB conversion
        int r = (yValue + 1.13983 * (vValue - 128)).toInt();
        int g = (yValue - 0.39465 * (uValue - 128) - 0.58060 * (vValue - 128)).toInt();
        int b = (yValue + 2.03211 * (uValue - 128)).toInt();

        // Clamp the values to the range [0, 255]
        r = r.clamp(0, 255);
        g = g.clamp(0, 255);
        b = b.clamp(0, 255);
        
        img.setPixelRgb(x, y, r, g, b);
    }
  }
          
  File(filePath).writeAsBytesSync(img_lib.encodeJpg(img));
  await OpenFile.open(filePath);
}

  2. 0

    You can use the image package to handle the conversion here’s how to do this:

    1. Add the image package to your pubspec.yaml file:
    dependencies:
  image: ^3.0.1
    1. Inside your onCaptureVideoFrame callback, convert the yBuffer, uBuffer, and vBuffer into a JPEG image:
    onCaptureVideoFrame: (VideoSourceType sourceType, VideoFrame videoFrame) async {
  Uint8List yData = videoFrame.yBuffer!;
  Uint8List uData = videoFrame.uBuffer!;
  Uint8List vData = videoFrame.vBuffer!;
  int height = videoFrame.height!;
  int width = videoFrame.width!;
 
  Image yuvImage = Image(width, height);
  int uvIndex = 0;
  for (int y = 0; y < height; y++) {
    for (int x = 0; x < width; x++) {
      int yValue = yData[y * width + x];
      int uValue = uData[uvIndex];
      int vValue = vData[uvIndex];
      uvIndex++;

      // Convert YUV to RGB
      int r = (yValue + 1.13983 * (vValue - 128)).round().clamp(0, 255);
      int g = (yValue - 0.39465 * (uValue - 128) - 0.5806 * (vValue - 128)).round().clamp(0, 255);
      int b = (yValue + 2.03211 * (uValue - 128)).round().clamp(0, 255);

      yuvImage.setPixel(x, y, getColor(r, g, b));
    }
  }

  Uint8List jpegData = Uint8List.fromList(encodeJpg(yuvImage));
},
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search