Redis Design Pattern: Supporting data pagination, sorting, filtering and searching

I have an application where the backend data store is Redis. This application (interface) provides users with a table that must support searching, pagination, sorting and filtering.

My Redis design includes the use of sorted sets and standard key:value pairs. As an example, consider a used car dealership. Lets say I have a set, toyota who’s members are a list of all available cars for sale, related to that brand. Each member is a combination of the car model, and some unique identifier that relates to a physical car.

toyota
  - corolla:100
  - corolla:200
  - corolla:300
  - sienna:100
  - sienna:200

Each member of that set, has a separate key, for example toyota:corolla:100, where the value is an object containing various information about that particular car:

{
  id: 100,
  brand: "Toyota",
  model: "Corolla",
  color: "red",
  cost: 15000
}

Understanding this basic data relationship, I find myself in a scenario where I want to offer the ability to sort data within this front-end table, by some property contained within the object of each key. Let’s say for example, the color of the car. Of course, in order to do that, I need to compare all of the objects.

My predicament, is how to implement that while taking pagination into consideration. In reality, my sets are not cars, and they can easily contain thousands of members. But the data relationship is the same concept. I do not want to have to fetch all of the keys in order to determine this sort, as it defeats the purpose of the pagination.

I’ll clarify that I am not artificially paginating the results in my API layer. I am achieving pagination by directly limiting the redis results through utilizing zrangebylex (to provide some basic ordering), along with a limit offset.

$results = [];
$cars = $redis->zRangeByLex("toyota", '-', '+', 0, 1);

foreach( $cars as $car ) {
    $results[] = json_decode($redis->get($car), true);
}

// example $cars return:
// [ "corolla:100", "corolla:200" ]

// example $results return:
// [
//   { id: 100, brand: "Toyota", model: "Corolla", color: "red", cost: 15000 },
//   { id: 200, brand: "Toyota", model: "Corolla", color: "blue", cost: 14000 },
// ]

I want to avoid artificially paginating results, as fetching thousands of records on each API call, then iterating through them, takes longer than is acceptable.

I’ll also note, when it comes to searching, I am searching against the sets by utilizing zscan — which isn’t ideal, as it means I am limited by the value of the members within each set.

$search = "corolla"; # user search term
$cars = []; # result container

$it = NULL; # iterator
$redis->setOption(Redis::OPT_SCAN, Redis::SCAN_RETRY);

while($matches = $redis->zScan('toyota', $it, "*{$search}*")) {
    foreach($matches as $key => $score) {
        $cars[] = $key;
    }
}

// example $cars return:
// [ "corolla:100", "corolla:200", "corolla:300" ]

While I could redesign this application within a SQL environment and achieve all of these features with relative ease, I’m more interested in making this work using Redis. What would be a more appropriate Redis design/pattern look like, that would support all of the features I want to implement within this front-end table (sorting, pagination, searching, filtering)?