skip to Main Content

I am trying to perform the aws S3 multipart api calls via my command line using perl scripting below is my relevant code snippet:

    my $part_no = 1;
    my %final_upload_file;
    my @multi_part_hash;
    
    foreach my $temp_file (@split_files) {
       my $tota_no_parts= @split_files;
       my $upload_part = "aws s3api upload-part --bucket <s3_bucket_name> --key ".$filename." --part-number ".$part_no." --body ".$temp_file." --upload-id ".$UploadId;
       $temp_json=`$upload_part`;
       
       my %upload_response = %{decode_json $temp_json};
    
       my $eTag = $upload_response{'ETag'};
       my %temp_part_hash;
       $temp_part_hash{'PartNumber'} = int($part_no);
       $temp_part_hash{'ETag'} = $eTag;
       push @multi_part_hash, %temp_part_hash;
    
        if( $part_no le $tota_no_parts)
        {
         $part_no++;
        }
    }
my %final_hash;
$final_hash{'Parts'} = @multi_part_hash;
print Dumper(%final_hash);

So till this point data type of the values in the hash is correct only below is the o/p:

$VAR1 = {
‘Parts’ => [
{
‘PartNumber’ => 1,
‘ETag’ => ‘"46f438ba2afd8d6dc03958b867a349b1"’
},
{
‘ETag’ => ‘"7b1e03e9ed7bb49d907e5a2d25b89159"’,
‘PartNumber’ => 2
},
{
‘ETag’ => ‘"30c23faad792c9b7c4b05356305cb6b0"’,
‘PartNumber’ => 3
},
{
‘ETag’ => ‘"979174e5353386a75d8e36d223d22ea2"’,
‘PartNumber’ => 4
},
{
‘PartNumber’ => 5,
‘ETag’ => ‘"ee93774d0a2ec9c27fae6a4fb3496804"’
},
{
‘PartNumber’ => 6,
‘ETag’ => ‘"9a2c5a1abe6869f370b07d76c270e569"’
}
] };

Below is the further operation i am trying to in order to do the final complete-multipart-upload API call:

my $json_text = encode_json %final_hash;
print("$json_text: $json_textn");
my $json_filename = 'test.json';
 open(FH, '>', $json_filename) or die $!;
 print FH $json_text;
 close(FH);

But if we see below o/p carefully seems like now all the values of part numbers are getting converted to string PSB:

$json_text:
{"Parts":[{"PartNumber":"1","ETag":""46f438ba2afd8d6dc03958b867a349b1""},{"ETag":""7b1e03e9ed7bb49d907e5a2d25b89159"","PartNumber":"2"},{"ETag":""30c23faad792c9b7c4b05356305cb6b0"","PartNumber":"3"},{"ETag":""979174e5353386a75d8e36d223d22ea2"","PartNumber":"4"},{"PartNumber":"5","ETag":""ee93774d0a2ec9c27fae6a4fb3496804""},{"PartNumber":"6","ETag":""9a2c5a1abe6869f370b07d76c270e569""}]}

So what could be the probable fix here that i can apply in order fix other-wise complete-multipart-upload api call is falling as it is expecting integer for partNumber instead of a String.

Error message:

Parameter validation failed: Invalid type for parameter
MultipartUpload.Parts[0].PartNumber, value: 1, type: <class ‘str’>,
valid types: <class ‘int’> Invalid type for parameter
MultipartUpload.Parts[1].PartNumber, value: 2, type: <class ‘str’>,
valid types: <class ‘int’> Invalid type for parameter
MultipartUpload.Parts[2].PartNumber, value: 3, type: <class ‘str’>,
valid types: <class ‘int’> Invalid type for parameter
MultipartUpload.Parts[3].PartNumber, value: 4, type: <class ‘str’>,
valid types: <class ‘int’> Invalid type for parameter
MultipartUpload.Parts[4].PartNumber, value: 5, type: <class ‘str’>,
valid types: <class ‘int’> Invalid type for parameter
MultipartUpload.Parts[5].PartNumber, value: 6, type: <class ‘str’>,
valid types: <class ‘int’>


update


My use modules:

use strict;
use FileHandle;
use Getopt::Long;
use POSIX;
use Data::Dumper;
use feature q(say);
use JSON;

2

Answers


  1. That’s why Cpanel::JSON::XS::Type exists.

    #! /usr/bin/perl
    use warnings;
    use strict;
    
    use Cpanel::JSON::XS qw{ encode_json };
    use Cpanel::JSON::XS::Type;
    
    my $struct = {Parts => [{PartNumber => "1",
                             ETag => "46f438ba2afd8d6dc03958b867a349b1"},
                            {ETag => "7b1e03e9ed7bb49d907e5a2d25b89159",
                             PartNumber => "2"},
                            {ETag => "30c23faad792c9b7c4b05356305cb6b0",
                             PartNumber => "3"},
                            {ETag => "979174e5353386a75d8e36d223d22ea2",
                             PartNumber => "4"},
                            {PartNumber => "5",
                             ETag => "ee93774d0a2ec9c27fae6a4fb3496804"},
                            {PartNumber => "6",
                             ETag => "9a2c5a1abe6869f370b07d76c270e569"}
                 ]};
    print encode_json($struct,
                      {Parts => json_type_arrayof({
                          PartNumber => JSON_TYPE_INT,
                          ETag       => JSON_TYPE_STRING})});
    

    Output:

    {"Parts":[{"ETag":"46f438ba2afd8d6dc03958b867a349b1","PartNumber":1},{"ETag":"7b1e03e9ed7bb49d907e5a2d25b89159","PartNumber":2},{"ETag":"30c23faad792c9b7c4b05356305cb6b0","PartNumber":3},{"PartNumber":4,"ETag":"979174e5353386a75d8e36d223d22ea2"},{"PartNumber":5,"ETag":"ee93774d0a2ec9c27fae6a4fb3496804"},{"PartNumber":6,"ETag":"9a2c5a1abe6869f370b07d76c270e569"}]}
    

    BTW: If you know something is a number, don’t use string comparison on it.

    Login or Signup to reply.
  2. If you treat a number as a string, its internal representation will change. Try the following:

    perl -MDevel::Peek -E'$a=99; say Dump $a; $b = "number: $a"; say Dump $a; say $b'
    

    You will get

    SV = IV(0x5592264fbed0) at 0x5592264fbee0
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 99
    
    SV = PVIV(0x5592264fe270) at 0x5592264fbee0
      REFCNT = 1
      FLAGS = (IOK,pIOK,pPOK)
      IV = 99
      PV = 0x5592264e7970 "99"
      CUR = 2
      LEN = 16
    
    number: 99
    

    If you use sprintf instead of the interpolation, it will remain the same:

    $ perl -MDevel::Peek -E'$a=99; say Dump $a; $b = sprintf "number: %d", $a; say Dump $a; say $b'
    SV = IV(0x56006d729e70) at 0x56006d729e80
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 99
    
    SV = IV(0x56006d729e70) at 0x56006d729e80
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 99
    
    number: 99
    

    This means: Don’t use $part_no in string comparison (as @choroba said), and build variable $upload_part with sprintf.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search