Question 174969 in Azure Question posted in
The official documentation to get you started can be found here.

Can we rewrite this?

I can copy a folder or a small file with Azcopy (copy) command but if i apply the command to a large file the transfer fails with no clear logs.

Command:

azcopy copy 'test' 'https://mystorage.blob.core.windows.net/...pKmRpGCVO8B' --recursive --preserve-smb-info=true

Test is a folder containing a large file. (approx 10 GB)

The logs simply stops when the process get killed.
Output is as follows:

INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job a50e4368-5fd0-534b-657d-70216b0e2 has started
Log file is located at: /root/.azcopy/a50e4368-5fd0-534b-657d-70216b0e2.log

Killed

Logs only shows:

cat /root/.azcopy/a50e4368-5fd0-534b-657d-70216b0e2.log
2023/05/23 17:51:19 AzcopyVersion  10.18.1
2023/05/23 17:51:19 OS-Environment  linux
2023/05/23 17:51:19 OS-Architecture  amd64
2023/05/23 17:51:19 Log times are in UTC. Local time is 23 May 2023 17:51:19
2023/05/23 17:51:20 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2023-05-23T17:51:14Z or --include-after=2023-05-23T17:51:14Z
2023/05/23 17:51:20 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2023/05/23 17:51:20 Job-Command copy test https://[...]rw&sr=c&st=2023-05-23t15%3A56%3A08z&sv=2022-11-02 --recursive --preserve-smb-info=true
2023/05/23 17:51:20 Number of CPUs: 1
2023/05/23 17:51:20 Max file buffer RAM 0.500 GB
2023/05/23 17:51:20 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2023/05/23 17:51:20 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2023/05/23 17:51:20 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2023/05/23 17:51:20 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2023/05/23 17:51:20 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2023/05/23 17:51:20 Max open files when downloading: 1048152 (auto-computed)
2023/05/23 17:51:20 Final job part has been created
2023/05/23 17:51:20 JobID=a50e4368-5fd0-534b-657d-70216b4960e2, credential type: Anonymous
2023/05/23 17:51:20 Final job part has been scheduled
2023/05/23 17:51:20 INFO: [P#0-T#0] Starting transfer: Source "/home/user/test/myfile.gz" Destination "https://mystorage.blob.core.windows.net/[...]02". Specified chunk size 8388608

The process works with smaller files but fails with large files.
I couldn’t find a file limit on the Azcopy setting or a related limitation from the logs.

Tags:

2

Answers


  1. 0

    Smaller files but fails with large files. I couldn’t find a file limit on the Azcopy setting or a related limitation from the logs.

    According to the log you gave, it appears that smaller files are successfully sent while bigger files are unsuccessfully transferred.

    The reason may be your network bandwidth is low also check the AZCOPY_CONCURRENCY_VALUE and AZCOPY_BUFFER_GB in your system.

    You can follow this MS-Document to increase the count of concurrency value and memory use.

    For Linux, you can use this

    export AZCOPY_CONCURRENCY_VALUE=64 
export AZCOPY_BUFFER_GB=4

    Once you are all set try with below azcopy command with --block-size-mb=100

    Command:

    azcopy copy "C:UsersDownloadsabcd" "https://venkat123.blob.core.windows.net/test?sp=racwdli&st=2023-05-24T05:47:05Z&se=2023-05-24T13:47:05Z&spr=https&sv=2022-11-02&sr=c&sig=xxxxxxxx" --recursive=true --block-size-mb=100

    Output:

    INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 28b71922-9c5b-b34e-4aa4-9c000a0dfce1 has started
Log file is located at: C:Usersxxxxx.azcopy28b71922-9c5b-b34e-4aa4-9c000a0dfce1.log

INFO: azcopy.exe: A newer version 10.18.1 is available to download

100.0 %, 1 Done, 0 Failed, 0 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 2.3541


Job 28b71922-9c5b-b34e-4aa4-9c000a0dfce1 summary
Elapsed Time (Minutes): 10.7021
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Total Number of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 10073742336
Final Job Status: Completed

    enter image description here

    Portal:

    enter image description here

    Reference:
    Transfer data to or from Azure Files by using AzCopy v10 | Microsoft Learn

    Login or Signup to reply.
  2. 0

    You can solve by adjusting the following values/parameters:

    — block-size-mb (in line parameter): to set the chuck size and

    — AZCOPY_CONCURRENCY_VALUE (session variable): to set the parallelization degree needed ( based on the free ram available and the block-size-mb)

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search