Is there a way to search for file at a specific location instead of searching through an entire container? I have this query running on Azure CLI but it searches for specific patterns in the entire container and issue I am facing is after a while, query just freezes over. I have tried several ways but nothing works. Here is the query that runs fine for an hour and then gives me a nextMarker that will not continue the search. (Did I maximize my quote of search on Azure?) Please take a look and let me know what you think.
container_name=pure3
numResults=1000
marker=---some value----
az storage blob list
--account-name $account_name
--account-key $key
--num-results $numResults
--show-next-marker
--container-name $container_name
--prefix "pure3/fastqs/170707_"
--marker $marker
--query "[[].name, [].nextMarker]" --output yamlc
and I have this Shell script that continue the search one after another.
#/usr/bin/
./Azure_Search_newQ.sh > out
tail -1 out
line=$(tail -1 out);
echo "Length is:${#line}"
expr=${line:4:${#line}}
echo $expr
head -n -1 out >>Blob_Files
while [ ! -z "$expr" ]
do
sed "s/FLAG/${expr}/g" Azure_listBlob_wMarker_FLAG.sh > Azure_Search_newQ.sh
chmod +x Azure_Search_newQ.sh
./Azure_Search_newQ.sh > out
head -n -1 out >>Blob_Files
line=$(tail -1 out)
echo "Length is:${#line}"
expr=${line:4:${#line}}
echo $expr
done
I tried running the code after 24 hrs and use the nextMarker that was stuck in the last run, same result.
I tried starting my run from 2 previous nextMarker and it ran fine and got stuck on the same nextMarker.
So far, I have searched through 1076000 files.
Here is the snapshot of azure cloud storage.
2
Answers
To search for blobs in a particular folder inside a container, please use
—-prefix
parameter and specify the folder name there.From the documentation:
I do agree with @alex, I have reproduced in my environment and got expected results as below and I followed Microsoft-document:
Output:
Also if you want all file names in a folder you can use: