I have two Azure storage accounts – staging-account.blob.core.windows.net and production-public-account.blob.core.windows.net.
I would like to copy a blob mycontainer/mydir/myblob.bin from staging-account to production-public-account.
I am running this on a host that is serving as an Azure DevOps build agent – it has access to both the production-public-account (via public internet) and the staging-account (via Private Endpoint / VNet).
Is it possible to copy from the private account to the public account?
I’ve used azcopy with SAS tokens and it is failing with CannotVerifyCopySource. The same error does not occur when I go from staging-account to a different private-production-account over a private VNet, and if I temporarily enable public access on the source account it works fine as well. We would really like to ensure that our staging account is not accessible to the public internet for this.
Function Sync-Directory {
param (
[string] $Container,
[string] $Directory,
[string] $TargetAccount,
[string] $Exclude,
[string] $Include,
[string] $Command
)
$SrcAccount = $Env:SRC_ACCOUNT_NAME
if ($Include) {
Write-Host ("Copying {0}/{1}**/{2} from {3} to {4}" -f $Container,$Directory,$Include,$SrcAccount,$TargetAccount)
} elseif ($Exclude) {
Write-Host ("Copying {0}/{1} (excluding **/{2}) from {3} to {4}" -f $Container,$Directory,$Exclude,$SrcAccount,$TargetAccount)
} else {
Write-Host ("Copying {0}/{1} from {2} to {3}" -f $Container,$Directory,$SrcAccount,$TargetAccount)
}
$Expiry = $(Get-Date).AddMinutes(90)
Set-AzContext -Subscription $Env:SRC_ACCOUNT_SUBSCRIPTION | Out-Null
$SrcContext = New-AzStorageContext -StorageAccountName $SrcAccount -UseConnectedAccount
$SrcToken = Remove-LeadingQuestionMark((New-AzStorageContainerSASToken -Context $SrcContext -Name $Container -
ExpiryTime $Expiry -Permission "rl"))
Set-AzContext -Subscription $Env:OUR_BLOB_STORAGE_SUBSCRIPTION_ID | Out-Null
$DstContext = New-AzStorageContext -StorageAccountName $TargetAccount -UseConnectedAccount
$DstToken = Remove-LeadingQuestionMark((New-AzStorageContainerSASToken -Context $DstContext -Name $Container -
ExpiryTime $Expiry -Permission "rcwdl"))
$SrcHost = "{0}.blob.core.windows.net" -f $SrcAccount
$TargetHost = "{0}.blob.core.windows.net" -f $TargetAccount
$RawSrcURL = "https://{0}/{1}/{2}" -f $SrcHost,$Container,$Directory
$RawDstURL = "https://{0}/{1}/{2}" -f $TargetHost,$Container,$Directory
$SrcURL = "{0}?{1}" -f $RawSrcURL,$SrcToken
$DstURL = "{0}?{1}" -f $RawDstURL,$DstToken
if ($Command -eq $null) {
$Command = "sync"
}
# azcopy doesn't have great regex support when directories to be synced are at the root level in the container
if ($Exclude) {
$Results = & sh -c "/tmp/azcopy '$Command' --recursive=true --exclude-pattern '$Exclude;$GlobalExcludePath' --delete-destination=true --s2s-preserve-access-tier=false '$SrcURL' '$DstURL'"
} elseif ($Include) {
$Results = & sh -c "/tmp/azcopy '$Command' --recursive=true --exclude-pattern '$GlobalExcludePath' --include-pattern '$Include' --s2s-preserve-access-tier=false --delete-destination=true '$SrcURL' '$DstURL'"
} else {
$Results = & sh -c "/tmp/azcopy '$Command' --recursive=true --exclude-pattern '$GlobalExcludePath' --delete-destination=true --s2s-preserve-access-tier=false '$SrcURL' '$DstURL'"
...
2023/11/15 16:19:46 AzcopyVersion 10.21.2
2023/11/15 16:19:46 OS-Environment linux
2023/11/15 16:19:46 OS-Architecture amd64
2023/11/15 16:19:46 Log times are in UTC. Local time is 15 Nov 2023 16:19:46
2023/11/15 16:19:46 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2023-11-15T16:19:41Z or --include-after=2023-11-15T16:19:41Z
2023/11/15 16:19:47 WARN: Failed to create destination container mycontainer. The transfer will continue if the container exists
2023/11/15 16:19:47 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2023/11/15 16:19:47 Job-Command cp --s2s-preserve-access-tier=false https://[private-staging-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15t17%3A49%3A43z&sig=-REDACTED-&ske=2023-11-15t17%3A49%3A43z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15t16%3A19%3A46z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=b&sv=2023-01-03 https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15t17%3A49%3A43z&sig=-REDACTED-&ske=2023-11-15t17%3A49%3A43z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15t16%3A19%3A45z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03
2023/11/15 16:19:47 Number of CPUs: 3
2023/11/15 16:19:47 Max file buffer RAM 1.500 GB
2023/11/15 16:19:47 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2023/11/15 16:19:47 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2023/11/15 16:19:47 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2023/11/15 16:19:47 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2023/11/15 16:19:47 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2023/11/15 16:19:47 Max open files when downloading: 1048152 (auto-computed)
2023/11/15 16:19:47 Final job part has been created
2023/11/15 16:19:47 JobID=993eaa9f-5646-1f49-6fe4-c720f7f75ac1, credential type: Anonymous
2023/11/15 16:19:47 Final job part has been scheduled
2023/11/15 16:19:47 INFO: [P#0-T#0] Starting transfer: Source "https://[private-staging-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A46Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=b&sv=2023-01-03" Destination "https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A45Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03". Specified chunk size 8388608
2023/11/15 16:19:47 ==> REQUEST/RESPONSE (Try=1/324.295011ms, OpTime=415.886833ms) -- RESPONSE STATUS CODE ERROR
PUT https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?blockid=MDAwMDCfqj6ZRlZJH2%2FkxyD391rBMDAwMDAwMDAwMDAwMDI0&comp=block&se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A45Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03
Accept: application/xml
Content-Length: 0
User-Agent: AzCopy/10.21.2 azsdk-go-azblob/v1.1.0 (go1.19.12; linux)
X-Ms-Client-Request-Id: 44431710-0710-4b9b-5800-0544edea6fe0
x-ms-copy-source:
x-ms-source-range:
x-ms-version:
--------------------------------------------------------------------------------
RESPONSE Status: 403 This request is not authorized to perform this operation.
Content-Length: 248
Content-Type: application/xml
Date: Wed, 15 Nov 2023 16:19:47 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Client-Request-Id: 44431710-0710-4b9b-5800-0544edea6fe0
X-Ms-Error-Code: CannotVerifyCopySource
X-Ms-Request-Id: 8d5e2152-201e-0004-43df-173200000000
X-Ms-Version: 2020-10-02
Response Details: <Code>CannotVerifyCopySource</Code><Message>This request is not authorized to perform this operation. </Message>
2023/11/15 16:19:47 ERR: [P#0-T#0] COPYFAILED: https://[private-staging-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A46Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=b&sv=2023-01-03 : 403 : 403 This request is not authorized to perform this operation.. When Staging block from URL. X-Ms-Request-Id: 8d5e2152-201e-0004-43df-173200000000
Dst: https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A45Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03
2023/11/15 16:19:47 JobID=993eaa9f-5646-1f49-6fe4-c720f7f75ac1 canceled
2023/11/15 16:19:47 all parts of entire Job 993eaa9f-5646-1f49-6fe4-c720f7f75ac1 successfully completed, cancelled or paused
2023/11/15 16:19:47 all parts of entire Job 993eaa9f-5646-1f49-6fe4-c720f7f75ac1 successfully cancelled
2023/11/15 16:19:47 is part of Job which 1 total number of parts done
2023/11/15 16:19:49 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 0, E: 1, T: 1, GRs: 32
2023/11/15 16:19:49 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 1072.8358
2023/11/15 16:19:49
Diagnostic stats:
IOPS: 13
End-to-end ms per request: 418
Network Errors: 0.00%
Server Busy: 0.00%
Job 993eaa9f-5646-1f49-6fe4-c720f7f75ac1 summary
Elapsed Time (Minutes): 0.0334
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Number of Symlink Transfers: 0
Total Number of Transfers: 1
Number of File Transfers Completed: 0
Number of Folder Transfers Completed: 0
Number of File Transfers Failed: 0
Number of Folder Transfers Failed: 0
Number of File Transfers Skipped: 0
Number of Folder Transfers Skipped: 0
TotalBytesTransferred: 0
Final Job Status: Cancelled
2
Answers
After much poking around at this, I finally figured out that
azcopyrequires a private endpoint for both the private and public accounts (even though the public account will not restrict traffic).My understanding (based on reading this somewhat-confusing recent blog from Microsoft) is that different routing / header magic is used on the Azure backend depending on whether the initial request to an account is done publicly or via a VNet address.
Alternatively, to copy Azure Storage Blob from a Private Storage Account to a Public Storage account you can make use of below steps:
Created storage account with private endpoint. when access to the public endpoint is disabled, the storage account can still be accessed through its private endpoints using selected virtual networks and IP addresses.
In the private storage account, enable
Enabled from selected virtual networks and IP addressesand in the firewall, checkAdd your client IP address, add it, and save like below:Select
Allow Azure services on the trusted services list to access this storage accountto allow trusted first party Microsoft services.Now, when I ran the
azcopyscript it transferred successfully like below:Example:
Output: