skip to Main Content

Is there a way to search for file at a specific location instead of searching through an entire container? I have this query running on Azure CLI but it searches for specific patterns in the entire container and issue I am facing is after a while, query just freezes over. I have tried several ways but nothing works. Here is the query that runs fine for an hour and then gives me a nextMarker that will not continue the search. (Did I maximize my quote of search on Azure?) Please take a look and let me know what you think.

container_name=pure3
numResults=1000
marker=---some value----
az storage blob list 
       --account-name $account_name 
       --account-key $key 
       --num-results $numResults 
       --show-next-marker 
       --container-name $container_name 
       --prefix "pure3/fastqs/170707_"  
       --marker $marker 
       --query "[[].name, [].nextMarker]" --output yamlc

and I have this Shell script that continue the search one after another.

#/usr/bin/

./Azure_Search_newQ.sh > out

tail -1 out
line=$(tail -1 out);
echo "Length is:${#line}"
expr=${line:4:${#line}}
echo $expr

head -n -1 out >>Blob_Files
while [ ! -z "$expr" ]
do
        sed "s/FLAG/${expr}/g" Azure_listBlob_wMarker_FLAG.sh > Azure_Search_newQ.sh
        chmod +x Azure_Search_newQ.sh
        ./Azure_Search_newQ.sh > out
        head -n -1 out >>Blob_Files
        line=$(tail -1 out)
        echo "Length is:${#line}"
        expr=${line:4:${#line}}
        echo $expr
done

I tried running the code after 24 hrs and use the nextMarker that was stuck in the last run, same result.
I tried starting my run from 2 previous nextMarker and it ran fine and got stuck on the same nextMarker.

So far, I have searched through 1076000 files.

Here is the snapshot of azure cloud storage.

2

Answers


  1. To search for blobs in a particular folder inside a container, please use —-prefix parameter and specify the folder name there.

    From the documentation:

    enter image description here

    Login or Signup to reply.
  2. Azure CLI query into specific folder

    I do agree with @alex, I have reproduced in my environment and got expected results as below and I followed Microsoft-document:

    az storage blob list --account-name "name of storage account" --container-name "name of the container" --account-key "RqOU/00Q+p6CRj2kvF/FVb3bP3X+AStV19sEg==" --prefix Nameoffolder/
    

    Output:

    enter image description here

    Also if you want all file names in a folder you can use:

     $x=az storage blob list --account-name "rithwikstorage" --container-name "rithwik" --account-key "RqOU1A6rGSaRWEu42IOVb3bP3X+AStV19sEg==" --prefix EmoFolder/ | ConvertFrom-Json
     $x.name             
    

    enter image description here

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search