Question 234147 in Flutter Question posted in
The official Flutter documentation can be found here.

Transform a CameraImage/Uint8List to a rgb array in flutter

I’m currently working on connecting my Python backend with my Flutter frontend. My backend hosts a pose estimation model that requires a numpy array as input, representing an image. As a result, I’m aiming to convert the CameraImage data I obtain in Dart into an RGB/BGR numpy array directly.

with mp_pose.Pose(min_detection_confidence=0.5, min_tracking_confidence=0.5) as pose:

    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    image.flags.writeable = False
    results = pose.process(image)
    #...

The challenge lies in the fact that the CameraImage data format can vary depending on the device’s operating system.

I also though of transforming it to a png and sending the png file to my backend but I think that this approach is highly inefficient.

I was able to transform it into Uint8List and I tried to follow the steps here

controller!.startImageStream((CameraImage img) {
        if (!isDetecting) {
          isDetecting = true;
          Uint8List imageByte = Uint8List.fromList(img.planes.expand((plane) => plane.bytes).toList());
          final decoder = Imagi.JpegDecoder();
          final decodedImg = decoder.decodeImage(imageByte);
          // ...
          isDetecting = false;
        }
      });

(Convert Image object to rgb pixel array and back in Flutter)

and explicatly tried, but decodeImage is not available anymore, probably depreciated.

I would be really happy if somebody could help me here out :D.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    So here I twisted the code a little bit. It's important to know that Android CameraImage is in the format ImageFormatGroup.yuv420 and for IOS ImageFormatGroup.bgra8888.So that is only for YUV420.

    List<List<List<int>>>? convertYUV420toRGBArray(CameraImage image) {
  try {
      final int width = image.width;
      final int height = image.height;
      final int uvRowStride = image.planes[1].bytesPerRow;
      final int? uvPixelStride = image.planes[1].bytesPerPixel;

      List<List<List<int>>> rgbList = [];

      List<List<int>> dfR = [];
      List<List<int>> dfG = [];
      List<List<int>> dfB = [];

      for (int y = 0; y < height; y++) {
        List<int> rowR = [];
        List<int> rowG = [];
        List<int> rowB = [];
        for (int x = 0; x < width; x++) {
          final int uvIndex = (uvPixelStride! * (x / 2).floor()) + (uvRowStride * (y / 2).floor());
          final int index = y * width + x;

          final int yp = image.planes[0].bytes[index];
          final int up = image.planes[1].bytes[uvIndex];
          final int vp = image.planes[2].bytes[uvIndex];

          int r = (yp + (vp * 1436 / 1024 - 179)).round().clamp(0, 255);
          int g = (yp - (up * 46549 / 131072) + 44 - (vp * 93604 / 131072) + 91).round().clamp(0, 255);
          int b = (yp + (up * 1814 / 1024 - 227)).round().clamp(0, 255);

          rowR.add(r);
          rowG.add(g);
          rowB.add(b);
        }
        dfR.add(rowR);
        dfG.add(rowG);
        dfB.add(rowB);
      }
      rgbList.add(dfR);
      rgbList.add(dfG);
      rgbList.add(dfB);
      return rgbList;
    } catch (e) {
      print("Error>>> $e");
      return null;
    }
}

  2. 0

    With image: ^4.0.17:

    You usually import it with a prefix:

    import 'package:image/image.dart' as image;

    Then you can:

    • Get a particular decoder instance and use it:
    final decoder = image.JpegDecoder();
final decodedImage = decoder.decode(bytes) as image.Image;
    • Use the shortcut function to directly decode a known image format:
    final decodedImage = image.decodeJpg(bytes) as image.Image;
    • Use the generic decodeImage if you don’t know the source image format.
      This one is the most expensive as it tries all the available decoders until the right one is found.
    final decodedImage = image.decodeImage(bytes) as image.Image;

    Once you have a decoded image instance you can obtain the bytes with a custom channel order:

    final data = decodedImage.getBytes(order: image.ChannelOrder.rgb);

    If going through all this is really required I would also try to use the CameraImage.format. Maybe the images are already in the right format and you don’t have to decode them. At least it will help you decide what decoder to use.

    Personally, I would send the image as a JPEG to the server and decode it there as transferring it in a raw format will take a lot more bandwidth.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search