Twitter API Question posted in
The official Twitter API documentation can be found here.

Facing error while rollover based on alias and template in elasticsearch 5.6.0 – Twitter API

I am facing a strange problem while rollover of a particular alias.
Error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "index name [twitter] does not match pattern '^.*-\d+$'"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "index name [twitter] does not match pattern '^.*-\d+$'"
  },
  "status": 400
}

I have a template as below:

{
     "template": "twitter-*",
     "settings": {
       "number_of_shards": 3
     },
     "mappings" : {
           "user" : {
             "properties" : {
                "name" : { "type" : "text" },
                    "id" : { "type" : "text" }
              }
          }
      }
 }

And alias list

 GET /_aliases

{
  "twitter": {
    "aliases": {}
  },
  ".kibana": {
    "aliases": {}
  },
  "twitter-2019.06.09-1": {
    "aliases": {
      "twitter-alias": {}
    }
  },
  "twitter123": {
    "aliases": {}
  }
}

I have created twitter-2019.06.09-1 from twitter index by takng snapshot and restore process.

Whenever I try following rollover manually(I was setting it up by curator), I face above error.

#Running manual rollover
POST /twitter-alias/_rollover/
{
  "conditions": {
    "max_age":   "7d",
    "max_docs":  1
  }
}

twitter-2019.06.09-1 has following 3 records:

GET twitter-2019.06.09-1/user/_search?size=0
{
  "took": 44,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": 0,
    "hits": [ ]
  }
}

I have tried deleting the twitter index because it is of no use but same error happening by rollover api.

Please help.

Update:

After reading below post:
https://discuss.elastic.co/t/rollover-failing/153676/5

I updated the template and index alias list as below:

#Creating template for new index creation
PUT _template/template_1
{
  "template": "twitter-*",
  "settings": {
    "number_of_shards": 3
  },
"mappings" : {
        "user" : {
            "properties" : {
                "name" : { "type" : "text" },
                "id" : { "type" : "text" }
            }
        }
    },
    "aliases":
    {
    "search-all":{}
    }
}

And Index:

GET twitter-2019.06.09-1
{
  "twitter-2019.06.09-1": {
    "aliases": {
      "search-all": {},
      "twitter-alias": {}
    },
    "mappings": {
      "user": {
        "properties": {
          "id": {
            "type": "text"
          },
          "name": {
            "type": "text"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1559904447882",
        "number_of_shards": "1",
        "number_of_replicas": "1",
        "uuid": "DnUZcHiyQIi2ab8XVKKVuA",
        "version": {
          "created": "5060099"
        },
        "provided_name": "twitter"
      }
    }
  }
}

Issue still exit.

I noticed

“provided_name”: “twitter”

while checking index info.

Is it related to the above error posted? It should not be twitter. Is it a bug?

My renaming index stretegy was:

#Taking snapshot
PUT /_snapshot/eaa-backup/twitter_snapshot
{
  "indices": "twitter",
  "ignore_unavailable": true,
  "include_global_state": false
}

And Restore:

#Now restore with new name
POST /_snapshot/eaa-backup/twitter_snapshot/_restore
{
 "indices": "twitter",
 "ignore_unavailable": "true",
 "include_global_state": false,
 "rename_pattern": "twitter",
 "rename_replacement": "twitter-2019.06.09-1"
 }

Please help to find out the cause of above error. Also, I have posted same issue in elastic discussions but it seems it will taking days to find help there. So , posting here. Thanks in advance.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Apart from @untergeek's reply. Below was the cause of the problem:

    Actually, I tried today morning and found out the same @untergeek mentioned after doing everything fresh. It seems somehow while renaming index by taking snapshot and restore (ref: here), that 

    "provided_name": "twitter"

    property (description in the question asked) was wrongly set and that is why I was facing problem. It should be set to 

    twitter-2019.06.09-1

    Problem is not with the index name but the above property.

    My scenario includes renaming the index and then rolling over.


  2. 0

    I already answered this here, but include the answer I provided there for reference.

    Rollover indices must end in a dash and a number, typically (you have to go through hoops with the rollover API calls to get around this). Curator was checking for the index associated with your rollover alias (which is twitter in the pasted block), which does not end with a dash and a number.

    I’m not quite sure what you’re doing wrong. Here’s a working example:

    PUT twitter-2019.06.09-1
{
  "aliases": {
    "twitter-alias": {
      "is_write_index": true
    }
  }
}

PUT twitter-alias/_doc/1
{
  "doc1": "dummy_doc1"
}
PUT twitter-alias/_doc/2
{
  "doc2": "dummy_doc2"
}

    RESULT:

    {
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "old_index" : "twitter-2019.06.09-1",
  "new_index" : "twitter-2019.06.09-000002",
  "rolled_over" : true,
  "dry_run" : false,
  "conditions" : {
    "[max_docs: 1]" : true
  }
}

    Additionally:

    GET twitter-alias

    Yields:

    {
  "twitter-2019.06.09-000002" : {
    "aliases" : {
      "twitter-alias" : {
        "is_write_index" : true
      }
    },
    "mappings" : {...},
    "settings" : {
      "index" : {
        "creation_date" : "1560216263049",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "uuid" : "kMd6jt_dSWKSCW4ertSuLg",
        "version" : {
          "created" : "7010099"
        },
        "provided_name" : "twitter-2019.06.09-000002"
      }
    }
  },
  "twitter-2019.06.09-1" : {
    "aliases" : {
      "twitter-alias" : {
        "is_write_index" : false
      }
    },
    "mappings" : {...},
    "settings" : {
      "index" : {
        "creation_date" : "1560216199250",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "uuid" : "zjIKove5SXqAXSEfseYpQg",
        "version" : {
          "created" : "7010099"
        },
        "provided_name" : "twitter-2019.06.09-1"
      }
    }
  }
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search