Question posted in
Some documentation to get you started can be found here.

Apache – How to pass JSON using WWW::Mechanize in perl

We are using WWW::Mechanize to send json data to our api.

My code is below.

#!/usr/local/bin/perl

use WWW::Mechanize;

sub get_csrf_token {
    my $sso = shift;
    my $referer = shift;
    my @cookie_items;
    my $rsp = $sso->get($referer);
    my $set_cookie = $rsp->header('set-cookie');
    if ($set_cookie =~ /csrftoken=(S+);/) {
        push(@cookie_items, $1);
    }
    if ($set_cookie =~ /sessionid=(S+);/) {
        push(@cookie_items, $1);
    }
    else {
        die 'Failed to get CSRF Token';
    }
    return @cookie_items;
}

my $sso = WWW::Mechanize->new(
        keep_alive  => 0,
        timeout     => 3000,
        stack_depth => 1,
        debug       => 1,
        verbose     => 1,
        key         => "/home/user/.athenz/key",
        cert        => "/home/user/.athenz/cert",
    );

my $referer = "https://myapi.domain.com/athenz/api/healthcheck/";
my @cookies = get_csrf_token($sso, $referer);
$sso->add_header('Referer' => $referer);
$sso->add_header('X-CSRFToken' => $cookies[0]);
$sso->add_header('Cookie' => "sessionid=$cookies[1]; csrftoken=$cookies[0]");

my %content = (
          'value' => '127.0.0.1',
          'key_name' => 'NTP_SERVERS'
);

my $response = $sso->put("https://myapi.domain.com/athenz/api/NTP_SERVERS/", %content);

print "^^^^^^^^^^^^^^^^^^^^^^^^^n";
print $response->code();
print "n^^^^^^^^^^^^^^^^^^^^^^^^^n";

its give me error Error PUTing https://myapi.domain.com/athenz/api/NTP_SERVERS/: Bad Request at /tmp/test.pm line 53

When I check server logs, its seems, the payload which I am sending, is deliver to server as HASH instead of json.

Below is apache logs.

[Mon Oct 07 18:32:27.987189 2024] [ssl:info] [pid 19745:tid 19745] [client 127.0.0.1:58372] AH01964: Connection to child 29 established (server myapi.domain.com:443)
[Mon Oct 07 18:32:27.987600 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [init-blocking] 0 readbytes
[Mon Oct 07 18:32:27.987964 2024] [ssl:debug] [pid 19745:tid 19745] ssl_engine_kernel.c(2393): [client 127.0.0.1:58372] AH02043: SSL virtual host for servername myapi.domain.com found
[Mon Oct 07 18:32:27.992817 2024] [ssl:debug] [pid 19745:tid 19745] ssl_engine_kernel.c(2256): [client 127.0.0.1:58372] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
[Mon Oct 07 18:32:27.992936 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.993929 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 53 bytes
[Mon Oct 07 18:32:27.993964 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): PUT /athenz/api/NTP_SERVERS/ HTTP/1.1

[Mon Oct 07 18:32:27.994033 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994047 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 24 bytes
[Mon Oct 07 18:32:27.994056 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): TE: deflate,gzip;q=0.3

[Mon Oct 07 18:32:27.994065 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994089 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 23 bytes
[Mon Oct 07 18:32:27.994093 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): Connection: TE, close

[Mon Oct 07 18:32:27.994100 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994104 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 23 bytes
[Mon Oct 07 18:32:27.994123 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): Accept-Encoding: gzip

[Mon Oct 07 18:32:27.994127 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994132 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 45 bytes
[Mon Oct 07 18:32:27.994135 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): Host: myapi.domain.com

[Mon Oct 07 18:32:27.994139 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994143 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 84 bytes
[Mon Oct 07 18:32:27.994147 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): Referer: https://myapi.domain.com/athenz/api/healthcheck/

[Mon Oct 07 18:32:27.994151 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994155 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 32 bytes
[Mon Oct 07 18:32:27.994158 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): User-Agent: WWW-Mechanize/1.72

[Mon Oct 07 18:32:27.994185 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994190 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 128 bytes
[Mon Oct 07 18:32:27.994193 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): Cookie: sessionid=XXXXXXX; csrftoken=YYYYYYY

[Mon Oct 07 18:32:27.994198 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994201 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 19 bytes
[Mon Oct 07 18:32:27.994205 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): HASH(0x1d5b760):

[Mon Oct 07 18:32:27.994209 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(140): [client 127.0.0.1:58372] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:32:27.994213 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(63): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): 79 bytes
[Mon Oct 07 18:32:27.994217 2024] [dumpio:trace7] [pid 19745:tid 19745] mod_dumpio.c(103): [client 127.0.0.1:58372] mod_dumpio:  dumpio_in (data-TRANSIENT): X-CSRFToken: YYYYYYY

[Mon Oct 07 18:32:27.994223 2024] [core:debug] [pid 19745:tid 19745] protocol.c(1290): [client 127.0.0.1:58372 {YRA:127.0.0.1:58372, YPA:127.0.0.1:58372}] AH02426: Request header field name is malformed: HASH(0x1d5b760):, referer: https://myapi.domain.com/athenz/api/healthcheck/
[Mon Oct 07 18:32:27.994328 2024] [core:debug] [pid 19745:tid 19745] protocol.c(1482): [client 127.0.0.1:58372 {YRA:127.0.0.1:58372, YPA:127.0.0.1:58372}] AH00567: request failed: error reading the headers, referer: https://myapi.domain.com/athenz/api/healthcheck/

If I use same API with curl, it works fine and apache logs gives no error.

curl --cert ~/.athenz/cert --key ~/.athenz/key -H "Content-type: application/json" -H "X-CSRFToken: $TOKEN" --referer "https://myapi.domain.com/athenz/api/healthcheck/" --cookie "csrftoken=$TOKEN;sessionid=$SESSIONID;" "https://myapi.domain.com/athenz/api/NTP_SERVERS/" -X PUT -d '{"key_name": "NTP_SERVERS", "value": "98.139.133.27"}' -i
HTTP/1.1 200 OK
Date: Mon, 07 Oct 2024 18:26:50 GMT
P3P: policyref="https://policies.domain.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
Allow: GET, PUT, DELETE, HEAD, OPTIONS
X-Frame-Options: DENY
Vary: Cookie
Content-Length: 0
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Set-Cookie: csrftoken=YYYYYYYYYYY; Domain=.domain.com; expires=Mon, 06 Oct 2025 18:26:53 GMT; Max-Age=31449600; Path=/; SameSite=Lax
Set-Cookie: sessionid=XXXXXXXXXXXX; expires=Mon, 21 Oct 2024 18:26:53 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax
Cache-Control: private
Connection: close

The apache logs, which print the json I passed in the curl.

[Mon Oct 07 18:26:50.576736 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935742 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 53 bytes
[Mon Oct 07 18:26:50.935767 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): PUT /athenz/api/NTP_SERVERS/ HTTP/1.1

[Mon Oct 07 18:26:50.935821 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935840 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 31 bytes
[Mon Oct 07 18:26:50.935844 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): Host: myapi.domain.com

[Mon Oct 07 18:26:50.935848 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935852 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 24 bytes
[Mon Oct 07 18:26:50.935857 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): User-Agent: curl/8.7.1

[Mon Oct 07 18:26:50.935863 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935867 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 13 bytes
[Mon Oct 07 18:26:50.935871 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): Accept: */*

[Mon Oct 07 18:26:50.935875 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935880 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 43 bytes
[Mon Oct 07 18:26:50.935883 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): Referer: https://myapi.domain.com/api/healthcheck/

[Mon Oct 07 18:26:50.935887 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935890 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 128 bytes
[Mon Oct 07 18:26:50.935894 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): Cookie: csrftoken=YYYYYYYY;sessionid=XXXXXXX;

[Mon Oct 07 18:26:50.935908 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935913 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 32 bytes
[Mon Oct 07 18:26:50.935917 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): Content-type: application/json

[Mon Oct 07 18:26:50.935922 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935926 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 79 bytes
[Mon Oct 07 18:26:50.935930 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): X-CSRFToken: YYYYYYYYY

[Mon Oct 07 18:26:50.935934 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935937 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 20 bytes
[Mon Oct 07 18:26:50.935942 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): Content-Length: 91

[Mon Oct 07 18:26:50.935946 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Mon Oct 07 18:26:50.935949 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 2 bytes
[Mon Oct 07 18:26:50.935953 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT):

[Mon Oct 07 18:26:50.937521 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(140): [client 127.0.0.1:57073] mod_dumpio: dumpio_in [readbytes-blocking] 91 readbytes
[Mon Oct 07 18:26:50.937532 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(63): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): 91 bytes
[Mon Oct 07 18:26:50.937537 2024] [dumpio:trace7] [pid 19725:tid 19725] mod_dumpio.c(103): [client 127.0.0.1:57073] mod_dumpio:  dumpio_in (data-TRANSIENT): {"key_name": "NTP_SERVERS", "value": "98.139.133.27"}
[Mon Oct 07 18:26:50.937558 2024] [ssl:info] [pid 19725:tid 19725] [client 127.0.0.1:57073 {YRA:127.0.0.1:57073, YPA:127.0.0.1:57073}] AH02221: Requesting connection re-negotiation, referer: https://myapi.domain.com/api/healthcheck/

How can I make sure, perl script pass json data to the apache server? I tried setting Content-Type: application/json, that didnt work.

Tags:

2

Answers


  1. 0

    Use something like Cpanel::JSON::XS to encode the hash into a json.

    The documentation of WWW::Mechanize shows the correct syntax for put:

    my $json = 'Cpanel::JSON::XS'->new->utf8;
my $response = $sso->put('https://myapi.domain.com/athenz/api/NTP_SERVERS/',
                         content => $json->encode(%content));
    Login or Signup to reply.
  2. 0

    WWW::Mechanize’s put is basically a call to HTTP::Request::Common’s PUT (except that a couple of special pseudo-headers of the form :name are supported).

    So the usage is

    ->put( $url
->put( $url, Header => Value,. .. )
->put( $url, $form_ref, Header => Value, ... )
->put( $url, Header => Value, ..., Content => $form_ref )
->put( $url, Header => Value, ..., Content => $content )

    Your usage matches the second, which should have set the Content-Type header to application/x-www-form-urlencoded and provided value=127.0.0.1&key_name=NTP_SERVERS as the document.

    I don’t know how you got the stringification of the a hash, but the solution is to provide the correct argument to ->put.

    use Cpanel::JSON::XS qw( encode_json );

my $response = $sso->put( $url,
   Content_Type => "application/json",       # Optional.
   Content => encode_json( %content ),
);
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search