Question posted in
The official documentation can be found here.

Redis – Problem with dict/json payload in Flask tests – Object of type bytes is not JSON serializable

Using Flask RestX, I have an endpoint that sets the entire value of the request payload to a key in the Redis database. Using Swagger and Postman I can confirm that the endpoint works.

But when I try to test it I get an error.

tray_info_api.py

def create_tray_info_api(api: Api, db: AsyncGetSetDatabase):
    tray_info_endpoint = api.namespace(
        'trayinfo', description='APIs to Send tray information to the Front End'
    )

    wild = fields.Wildcard(fields.String)

    well_meta = api.model('Well Metadata', {
        'label': fields.String,
        'type': fields.String,
        'value': wild
    })

    well = api.model('Well', {
        'metadata': fields.List(fields.Nested(well_meta)),
        'status': fields.String(enum=('ready', 'sampled'), required=True)
        # and some other fields with string/integer/datetime types
    })

    tray = api.model('Well Tray', {
        'rows_count': fields.Integer(required=True),
        'columns_count': fields.Integer(required=True),
        'wells': fields.List(fields.Nested(well), required=True)
    })

    @tray_info_endpoint.route('/')
    class TrayInfoEndpoint(Resource):

        @tray_info_endpoint.expect(tray, validate=False)
        def put(self):
            run(db.set('tray_info', request.data))
            return run(db.get('tray_info'))

test_info_endpoint.py

def test_put_info(app):
    info = {
        "rows_count": 0,
        "columns_count": 0,
        "date_loaded": "2020-08-19T14:11:29.320Z",
        "date_processed": "2020-08-19T14:11:29.320Z",
        # ... and all the other fields; this data is copy/pasted from a working Postman request
    }
    res = app.test_client().put('/trayinfo/', json=info)   
    data = json.loads(res.data)
    assert data == info

The debugger stops in the test on the line starting with res = app.test_client, saying "TypeError: Object of type bytes is not JSON serializable".

If I put a breakpoint inside the api’s put method and go into the console, I see that request.data is the data I sent, with the b' prefix on the entire JSON:

>>> request.data
b'{"columns_count": 0, "date_loaded": "2020-08-19T14:11:29.320Z", "date_processed": "2020-08-19T14:11:29.320Z", "rows_count": 0}'

I know this indicates that it’s bytes, which seems to point toward the answer, except if that’s the problem, then shouldn’t the app not actually work in Postman? I can step through the entire put method without errors, it seems like the error is coming frmo the line in the test itself, which is really weird.

I’ve also tried res = app.test_client().put(url, data=info), which results in request.data as b'' (an empty payload), so that’s not right.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Thanks to Joran Beasley for pointing me in the right direction. I finally figured out how to apply his advice on json decoding, but it wasn't something that was anywhere in my original post.

    You can see that the create_tray_info_api method takes a database (dependency injection). In the real app, this is created with a Redis database, but in tests I use a fixture to inject a simple mock database, whose set method was:

    async def set(self, key, value):
    self.db[key] = value

    The request.data, as I noted, was a bytes object containing the json info, which my put method writes to the database. I guess Redis takes care of encoding/decoding itself in some way (or maybe the browser/swagger/postman does that) which is why it works in the real app.

    The mock database, however, just writes and reads it as is. So I needed to decode the bytes when saving to the database. Which only works, of course, for bytes, so I have to make sure that integers, strings, etc, pass through normally:

    async def set(self, key, value):
    self.db[key] = value.decode('utf8') if isinstance(value, bytes) else value

    Then in the test, data returns as a string, so we need to convert it back to json in order to assert its equality:

    assert info == json.loads(data)

  2. 0

    decode your bytes to unicode before hand

    import json

data1 = {
  "field":b"bytes"
}
json.dumps(data1) # ERROR!!!

data2 = {
  "field":b"bytes".decode("utf8")
}
json.dumps(data2) # ALL Good now its a string

    If you dump your json ahead of time you can include a "default_encoder" to handle objects that cant be encoded normally

    def encoder(obj):
    if isinstance(obj,bytes):
       return obj.decode("utf8")
    return obj

json.dumps(data1,default=encoder)
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search