Question 120849 in Firebase Question posted in
An extensive official documentation can be found here.

Firebase – Internal Error when executing TextRecognition using CameraX in realtime

I am attempting to us ML Kit to do some realtime text detection from CameraX on an Android app.

My ImageAnalyser is as follows:

package com.renegade.CoWarrior;


import android.annotation.SuppressLint;
import android.media.Image;
import android.util.Log;

import androidx.annotation.NonNull;
import androidx.camera.core.ImageAnalysis;
import androidx.camera.core.ImageProxy;

import com.google.android.gms.tasks.OnCompleteListener;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.android.gms.tasks.Task;
import com.google.mlkit.vision.common.InputImage;
import com.google.mlkit.vision.text.Text;
import com.google.mlkit.vision.text.TextRecognition;
import com.google.mlkit.vision.text.TextRecognizer;
import com.google.mlkit.vision.text.latin.TextRecognizerOptions;

public class CoWarriorAnalyzer implements ImageAnalysis.Analyzer {
    TextRecognizer recognizer;
    public CoWarriorAnalyzer(){
        // When using Latin script library
        recognizer = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS);
    }

    @Override
    public void analyze(@NonNull ImageProxy image) {

        try (@SuppressLint({"UnsafeExperimentalUsageError", "UnsafeOptInUsageError"}) Image mediaImage = image.getImage()) {

            if (mediaImage != null) {
                InputImage theimage =
                        InputImage.fromMediaImage(mediaImage, image.getImageInfo().getRotationDegrees());
                // Pass image to an ML Kit Vision API
                // ...
                Task<Text> result =
                        recognizer.process(theimage)
                                .addOnSuccessListener(new OnSuccessListener<Text>() {
                                    @Override
                                    public void onSuccess(Text visionText) {
                                        // Task completed successfully
                                        // ...
                                        String text = visionText.getText();


                                        Log.d("cortana","Success:"+text);
                                    }
                                })
                                .addOnFailureListener(
                                        new OnFailureListener() {
                                            @Override
                                            public void onFailure(@NonNull Exception e) {
                                                // Task failed with an exception
                                                // ...
                                                Log.d("cortana","Failure:"+e.getMessage());
                                            }
                                        })
                                .addOnCompleteListener(
                                       new OnCompleteListener(){
                                           @Override
                                           public void onComplete(@NonNull Task task) {
                                               image.close();
                                             
                                           }


                                        });



            }
        }
        image.close();
    }
}

I configure this analyzer by this code:

 ImageAnalysis imageAnalysis =
            new ImageAnalysis.Builder()
                    // enable the following line if RGBA output is needed.
                    //.setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
                    .setTargetResolution(new Size(1280, 720))
                    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                    .build();

    imageAnalysis.setAnalyzer(new CortanaExecutor(), new CoWarriorAnalyzer());

I didn’t know what to put for the Executor so I made this :

public class CortanaExecutor implements Executor {
        public void execute(Runnable r) {
            r.run();
        }
}

The error I am being flooded with is :

2022-07-15 21:22:06.148 18115-18115/com.renegade.halostatstracker D/cortana: Failure:Internal error has occurred when executing ML Kit tasks

The log slightly before the first of these errors shows:

2022-07-15 21:22:02.685 18115-18180/com.renegade.halostatstracker D/CaptureSession: Opening capture session.
2022-07-15 21:22:02.691 18115-18180/com.renegade.halostatstracker D/DeferrableSurface: New surface in use[total_surfaces=2, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@4b5b1b5}
2022-07-15 21:22:02.691 18115-18180/com.renegade.halostatstracker D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@4b5b1b5
2022-07-15 21:22:02.691 18115-18180/com.renegade.halostatstracker D/DeferrableSurface: New surface in use[total_surfaces=2, used_surfaces=2](androidx.camera.core.impl.ImmediateSurface@b6b1ef9}
2022-07-15 21:22:02.691 18115-18180/com.renegade.halostatstracker D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.impl.ImmediateSurface@b6b1ef9
2022-07-15 21:22:02.952 18115-18180/com.renegade.halostatstracker D/CaptureSession: Attempting to send capture request onConfigured
2022-07-15 21:22:02.952 18115-18180/com.renegade.halostatstracker D/CaptureSession: Issuing request for session.
2022-07-15 21:22:02.955 18115-18180/com.renegade.halostatstracker D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED
2022-07-15 21:22:02.955 18115-18180/com.renegade.halostatstracker D/CaptureSession: CameraCaptureSession.onReady() OPENED
2022-07-15 21:22:02.957 18115-18130/com.renegade.halostatstracker I/CameraManagerGlobal: Camera 0 facing CAMERA_FACING_BACK state now CAMERA_STATE_ACTIVE for client com.renegade.halostatstracker API Level 2
2022-07-15 21:22:03.098 18115-18127/com.renegade.halostatstracker W/System: A resource failed to call close. 
2022-07-15 21:22:03.123 18115-18130/com.renegade.halostatstracker I/BufferQueueProducer: [SurfaceView - com.renegade.halostatstracker/com.renegade.halostatstracker.MainActivity@508a84@0#1(BLAST Consumer)1](id:46c300000003,api:4,p:1616,c:18115) queueBuffer: queued for the first time.
2022-07-15 21:22:03.170 18115-18181/com.renegade.halostatstracker D/StreamStateObserver: Update Preview stream state to STREAMING
2022-07-15 21:22:03.192 18115-18130/com.renegade.halostatstracker I/BufferQueueProducer: [ImageReader-2992x2992f23m4-18115-0](id:46c300000001,api:4,p:1616,c:18115) queueBuffer: queued for the first time.
2022-07-15 21:22:03.219 18115-18192/com.renegade.halostatstracker D/TransportRuntime.SQLiteEventStore: Storing event with priority=VERY_LOW, name=FIREBASE_ML_SDK for destination cct
2022-07-15 21:22:03.223 18115-18192/com.renegade.halostatstracker D/TransportRuntime.JobInfoScheduler: Upload for context TransportContext(cct, VERY_LOW, MSRodHRwczovL2ZpcmViYXNlbG9nZ2luZy5nb29nbGVhcGlzLmNvbS92MGNjL2xvZy9iYXRjaD9mb3JtYXQ9anNvbl9wcm90bzNc) is already scheduled. Returning...
2022-07-15 21:22:03.272 18115-18383/com.renegade.halostatstracker I/Manager: DeviceManager::DeviceManager
2022-07-15 21:22:03.272 18115-18383/com.renegade.halostatstracker I/Manager: findAvailableDevices
2022-07-15 21:22:03.285 18115-18383/com.renegade.halostatstracker I/Manager: Found interface qti-default
2022-07-15 21:22:03.285 18115-18383/com.renegade.halostatstracker I/Manager: Found interface qti-dsp
2022-07-15 21:22:03.285 18115-18383/com.renegade.halostatstracker I/Manager: Found interface qti-gpu
2022-07-15 21:22:03.288 18115-18387/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.288467   18387 text_detector_thread_pool_context.cc:38] Compute manager max in flight region detector overwrite: 1
2022-07-15 21:22:03.288 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.288589   18385 text_classifier.cc:29] Creating classifier TfliteTextClassifier
2022-07-15 21:22:03.288 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.288874   18385 common_util.h:38] Resizing Thread Pool: ocr_segm to 3
2022-07-15 21:22:03.289 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.289362   18385 tflite_lstm_client_base.cc:367] Resizing interpreter pool to 4
2022-07-15 21:22:03.289 18115-18387/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.289428   18387 tflite_detector_client_base.cc:379] Resizing interpreter pool to 4
2022-07-15 21:22:03.289 18115-18385/com.renegade.halostatstracker I/tflite: Initialized TensorFlow Lite runtime.
2022-07-15 21:22:03.290 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.290584   18385 multi_pass_line_recognition_mutator.cc:284] Preloading recognizers.
2022-07-15 21:22:03.290 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.290857   18385 multi_pass_line_recognition_mutator.cc:291] Preloading a recognizer for "Latn"
2022-07-15 21:22:03.291 18115-18387/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.291418   18387 tflite_model_pooled_runner.cc:530] Loading /data/user/0/com.renegade.halostatstracker/files/mlkit-google-ocr-models/gocr/layout/line_splitting_custom_ops/model.tflite
2022-07-15 21:22:03.291 18115-18387/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.291636   18387 tflite_model_pooled_runner.cc:641] Resizing interpreter pool to 1
2022-07-15 21:22:03.291 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.291838   18385 multi_pass_line_recognition_mutator.cc:291] Preloading a recognizer for ""
2022-07-15 21:22:03.292 18115-18387/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.292126   18387 tflite_model_pooled_runner.cc:530] Loading /data/user/0/com.renegade.halostatstracker/files/mlkit-google-ocr-models/gocr/layout/line_clustering_custom_ops/model.tflite
2022-07-15 21:22:03.292 18115-18391/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.292178   18391 tflite_model_pooled_runner.cc:530] Loading /data/user/0/com.renegade.halostatstracker/files/mlkit-google-ocr-models/gocr/gocr_models/line_recognition_legacy_mobile/Latn_ctc/optical/conv_model.fb
2022-07-15 21:22:03.292 18115-18392/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.292244   18392 multi_pass_line_recognition_mutator.cc:304] Finished preloading a recognizer for ""
2022-07-15 21:22:03.292 18115-18387/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.292296   18387 tflite_model_pooled_runner.cc:641] Resizing interpreter pool to 1
2022-07-15 21:22:03.292 18115-18391/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.292356   18391 tflite_model_pooled_runner.cc:536] Loading /data/user/0/com.renegade.halostatstracker/files/mlkit-google-ocr-models/gocr/gocr_models/line_recognition_legacy_mobile/Latn_ctc/optical/lstm_model.fb
2022-07-15 21:22:03.292 18115-18391/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.292598   18391 tflite_model_pooled_runner.cc:641] Resizing interpreter pool to 4
2022-07-15 21:22:03.296 18115-18391/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.296457   18391 mobile_langid_v2.cc:58] MobileLangID V2 initialized.
2022-07-15 21:22:03.296 18115-18391/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.296535   18391 multi_pass_line_recognition_mutator.cc:304] Finished preloading a recognizer for "Latn"
2022-07-15 21:22:03.296 18115-18385/com.renegade.halostatstracker I/native: I0000 00:00:1657916523.296587   18385 multi_pass_line_recognition_mutator.cc:308] Finished preloading recognizers.
2022-07-15 21:22:03.298 18115-18115/com.renegade.halostatstracker D/cortana: Failure:Internal error has occurred when executing ML Kit tasks

This is the first time I have used CameraX, ML Kit or anything like this – so I could very well have something configured badly.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    It looks like this message is actually not catastrophic - and with more experimenting, it actually worked, the code as we see it above does actually perform realtime text recognition. If anyone can explain a bit more about this internal error, that would be great.


  2. 0

    Hope you solved your problem in the meantime!
    Nevertheless I’d like to share my experience in the ML Kit text-recognition module use.

    1. Need not to create a specific executor like new CortanaExecutor().
      You can use the main executor: ContextCompat.getMainExecutor(this).

    2. The use of the task:

    Task<Text> result = recognizer.process(theimage)

    As coded is problematic, it does not work as such.
    A Task is a Thread and in this case use, that Task is launched and executed at the frame rate of the camera (FPS)!
    To get the result of a task (Thread) you must "wait" for the task to be completed.
    So what I suggest to do, is to add a waiting loop after the code lines of the task:

    Task<Text> result = recognizer.process(theimage);
    
while (!result.isComplete()) {
  anytextView.setText("");
}      
if (result.isComplete() & result.isSuccessful())
  processResult(result.getResult());
else
  handleError();

    You’ll notice that the listeners are not coded.
    I can assure you that it works!

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search