Question 247126 in Azure Question posted in
The official documentation to get you started can be found here.

Azure SDK v2 – unable to download job output using mlclient

I was using functionality from last few months, but recently started facing some issue with the below code. I’m running the code using JupyterLab and unable to download the predictions result. However, The batch endpoint is getting invoked and able to stream the job. Additionally, I can see the prediction output saved in Azure Blob Storage.

I’m also not getting any error message.

# invoke the endpoint for batch scoring job
print("Invoking the Batch Endpoint...")
job = ml_client.batch_endpoints.invoke(
    endpoint_name=batch_endpoint_name,
    input=score_dataset_input,                    
    deployment_name=batch_deployment_name,     
    params_override=[
        {"mini_batch_size": str(mini_batch_size)}, 
        {"compute.instance_count": str(compute_instance_count)},
        {"output_file_name": f"{prediction_output_file_name}.csv"}
     ]           
)
print("Batch Endpoint invoked with the provided payload....")

print("Streaming the Job...")
job_name = job.name
batch_job_stream = ml_client.jobs.stream(name=job_name)

# download the job logs and output
ml_client.jobs.download(batch_job.name, 
                        download_path= f"{batch_prediction_dir}/csv/", 
                        output_name="predictions")
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Whenever, we invoke batch-endpopint, it will create a new job and save the prediction output at path f"container_name/{job-run-id}/score" in workspace default storage.

    Below is the code, I'm currently using to fetch the prediction output.Thanks @ooxvyd, for suggesting alternate approach:)

    job_name = job.name
batch_job = ml_client.jobs.get(name=job_name)
print(f"Job Name: {job_name}")

# stream the job logs
ml_client.jobs.stream(name=job_name)
print("Prediction complete")

# Get final batch processing Job <run id>
for job in ml_client.jobs.list(parent_job_name=job_name):
    final_job_run_id = job.name
print(f"Final Job Run Id: {final_job_run_id}")

blob_storage = ws.get_default_datastore()
prediction_output_path = f"azureml/{final_job_run_id}/score/predictions.csv"

blob_storage.download(target_path="./", prefix=prediction_output_path, overwrite=True, show_progress=False)
pred_data = pd.read_csv(prediction_output_path, delimiter= " ", names=["hash", "y_preds"])```

  2. 0

    It appears that you are attempting to run a batch scoring task in Azure Machine Learning and are having difficulties downloading the prediction results. Given that the batch endpoint is being executed successfully and you can see the prediction output saved in Azure Blob Storage, the problem is most likely in the code used to retrieve the prediction results. Because the code given for obtaining the prediction results lacks context, I’ll present a basic way for downloading files from Azure Blob Storage using Python and the Azure SDK. Assuming ml_client is an Azure Machine Learning SDK instance, you may download files from Azure Blob Storage as follows:

    from azureml.core import Workspace

# Define your Azure Machine Learning workspace
workspace = Workspace.get(name="<your_workspace_name>", subscription_id="<your_subscription_id>", resource_group="<your_resource_group>")

# Define the blob store associated with your workspace
blob_store = workspace.get_default_datastore()

# Define the container and blob names
container_name = "<container_name>"
blob_name = "<blob_name>"

# Define the local path where you want to save the downloaded file
local_path = "<local_path>"

# Download the file from Azure Blob Storage
blob_store.download(src=container_name, target_path=local_path, target_name=blob_name)

    Replace your particular values for your_workspace_name>, your_subscription_id>, your_resource_group>, container_name>, blob_name>, and local_path>. Check that your JupyterLab environment has the Azure SDK for Python (azureml-sdk) and Azure Blob Storage (azure-storage-blob) installed. You may use pip to install them:

    pip install azureml-sdk azure-storage-blob

    This code will download the provided blob from Azure Blob Storage to the supplied local directory.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search