i have a bucket with 100,000 images(pngjpg) and i want to read them all in one s3 request.
this is my code:
# geting connection to s3
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
# geting list of all objectsummery in the relevant dir - this is a request to the api
crops_path_list = bucket.objects.filter(Prefix=prefix)
# iterating on the list and geting objects
for index, crop in enumerate(crops_path_list):
# getting the bytes from s3 - this a request to the api
crop = crop.get()
# working with the bytes to make them image
image_content = crop['Body'].read()
bytes_images = BytesIO(image_content)
image = Image.open(bytes_images)
image = image.convert("RGB")
image = np.asarray(image)
image = np.ascontiguousarray(image.transpose(2, 0, 1))
image = torch.from_numpy(np.array(image)).unsqueeze(0).to(dtype=torch.float32, device=device)
# adding to list of all images
images.append(image)
it takes agessss because every .get()
is a request to the api. could not find any use to get the whole crops_path_list
list in one request. other then running threadssubprocessziping, any other ideas how to use less IO and api requests?
thanks 🙂
2
Answers
PySpark works great for me, solved
This is not possible. Downloading each file is a request.