Memcached Question posted in
A decent usage manual can be found here.

Memcached – GAE shutdown or restart all the active instances of a service/app

In my app (Google App Engine Standard Python 2.7) I have some flags in global variables that are initialized (read values from memcache/Datastore) when the instance start (at the first request). That variables values doesn’t change often, only once a month or in case of emergencies (i.e. when google app engine Taskqueue or Memcache service are not working well, that happened not more than twice a year as reported in GC Status but affected seriously my app and my customers: https://status.cloud.google.com/incident/appengine/15024 https://status.cloud.google.com/incident/appengine/17003).

I don’t want to store these flags in memcache nor Datastore for efficiency and costs.

I’m looking for a way to send a message to all instances (see my previous post GAE send requests to all active instances ):

As stated in https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed

Note: Targeting an instance is not supported in services that are configured for auto scaling or basic scaling. The instance ID must be an integer in the range from 0, up to the total number of instances running. Regardless of your scaling type or instance class, it is not possible to send a request to a specific instance without targeting a service or version within that instance.

but another solution could be:

1) Send a shutdown message/command to all instances of my app or a service

2) Send a restart message/command to all instances of my app or service

I use only automatic scaling, so I’cant send a request targeted to a specific instance (I can get the list of active instances using GAE admin API).

it’s there any way to do this programmatically in Python GAE? Manually in the GCP console it’s easy when having a few instances, but for 50+ instances it’s a pain…

Tags:

2

Answers


  1. 0

    One possible solution (actually more of a workaround), inspired by your comment on the related post, is to obtain a restart of all instances by re-deployment of the same version of the app code.

    Automated deployments are also possible using the Google App Engine Admin API, see Deploying Your Apps with the Admin API:

    To deploy a version of your app with the Admin API:

    1. Upload your app’s resources to Google Cloud Storage.
    2. Create a configuration file that defines your deployment.
    3. Create and send the HTTP request for deploying your app.

    It should be noted that (re)deploying an app version which handles 100% of the traffic can cause errors and traffic loss due to:

    • overwriting the app files actually being in use (see note in Deploying an app)
    • not giving GAE enough time to spin up sufficient instances fast enough to handle high income traffic rates (more details here)

    Using different app versions for the deployments and gradually migrating traffic to the newly deployed apps can completely eliminate such loss. This might not be relevant in your particular case, since the old app version is already impaired.

    Automating traffic migration is also possible, see Migrating and Splitting Traffic with the Admin API.

    Login or Signup to reply.
  2. 0

    It’s possible to use the Google Cloud API to stop all the instances. They would then be automatically scaled back up to the required level. My first attempt at this would be a process where:

    1. The config item was changed
    2. The current list of instances was enumerated from the API
    3. The instances were shutdown over a time period that allows new instances to be spun up and replace them, and how time sensitive the config change is. Perhaps close on instance per 60s.

    In terms of using the API you can use the gcloud tool (https://cloud.google.com/sdk/gcloud/reference/app/instances/):

    gcloud app instances list

    Then delete the instances with:
    gcloud app instances delete instanceid --service=s1 --version=v1

    There is also a REST API (https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1/apps.services.versions.instances/list):

    GET https://appengine.googleapis.com/v1/{parent=apps/*/services/*/versions/*}/instances
DELETE https://appengine.googleapis.com/v1/{name=apps/*/services/*/versions/*/instances/*}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search