Question posted in
The official documentation can be found here.

Redis – kubernetes – How best to isolate resources based on environment, labels and selectors or namespacing?

I’ve encountered a rather rookie issue with k8s. I’m particularly new to k8s, and setup staging and production services / deployments for a Django – celery – redis application within a cluster. However. In my excitement that I actually managed to get something working, I didn’t check to think if it was 100% correct.

Essentially, I’ve noticed that the pre-production Django application doesn’t care which celery deployment it references when dispatching a periodic task. It might go to staging, it might try the pre-production deployment. THIS IS BAD.

So, I’ve been looking at labels and selectors, as well as namespaces.

However, I should probably slow down – my first question, how would I use something native to k8s to run different environments of deployments, such that they are all isolated from each other. So the pre-production Django application can only talk to the pre-production celery-worker or pre-production celery-beat deployments…

*My answer I feel is to use labels and selectors? But … is it best to use namespaces?

Any pro-guidance around this subject would be amazing.

Tags:

2

Answers


  1. 0

    You could go with a namespace per environment and then make sure that when calling another service you always use the dns shorthand of a a single service name.

    This will, however, not limit a specific service to call another in another namespace/environment. If you are the sole developer, or trust all other this might be fine.

    An other alternative it so run different clusters but run them identically. When your number of deployments grows you’ll likely end up with this anyway.
    This way you can have one namespace per team, or per domain, and it will look identical in all environments.

    Login or Signup to reply.
  2. 0

    Aside from creating a new cluster per environment, you can separate deployments by namespace or just different stacks in one namespace. The last case (the one you use now) is the easiest to shoot in the leg since you have to change a lot of thing to make it isolated. At very least you need a different set of resource names (both in manifests and configuration) and labels to match.

    Out of the three methods I think namespace separation is the easiest; it works on DNS-based service discovery. Suppose you have a copy of redis and your application in two different namespaces (dev and prod for example). Both instances of the app are configured to use redis at redis:6379. When they call DNS to resolve the hostname, CoreDNS (the internal DNS service) would respond with different answers depending on which namespace the request came from. And so your app in dev namespace will get an IP-address of redis in dev namespace, and the app from prod namespace will contact redis from prod namespace. This method does not apply any restriction, if you wish you can specifically make it so that both apps use the same copy of redis. For that, instead of redis:6379 you have to use a full service DNS name, like this:

    redis.<namespace>.svc.cluster.local:6379

    Regardless of whatever method you choose to go with, I strongly recommend you to get familiar with Kustomize, Helm, or both. These tools are to help you avoid duplicating resource manifests and thus spend less time spawning and managing instances. I will give you a minimal example for Kustomize because it is built in kubectl. Consider the following directory structure:

    .
├── bases
│   └── my-app
│       ├── deployment.yml # your normal deployment manifest
│       └── kustomization.yml
└── instances
    ├── prod
    │   ├── kustomization.yml
    │   └── namespace.yml # a manifest that creates 'prod' namespace
    └── test
        ├── kustomization.yml
        └── namespace.yml # a manifest that creates 'test' namespace

    bases is where you keep a non-specific skeleton of your application. This isn’t meant to be deployed, like a class it has to be instantiated. instances is where you describe various instances of your application. Instances are meant to be deployed.

    bases/my-app/kustomization.yml:

    # which manifests to pick up
resources:
  - deployment.yml

    instances/prod/kustomization.yml:

    # refer what we deploy
bases:
  - ../../bases/my-app
resources:
  - namespace.yml
# and this overrides namespace attribute for all manifests referenced above
namespace: prod

    instances/test/kustomization.yml:

    # the same as above, only the namespace is different
bases:
  - ../../bases/my-app
resources:
  - namespace.yml
namespace: test

    Now if you go into instances directory and use kubectl apply -k prod you will deploy deployment.yml to prod namespace. Similarly kubectl apply -k test will deploy it to the test namespace.

    And this is how you can create several identical copies of your application stack in different namespaces. It should be fairly isolated unless some shared resources from other namespaces involved. In other words, if you deploy each component (such as the database) per namespace and those components are not configured to access components from other namespaces – it will work as expected.

    I encourage you to read more on Kustomize and Helm, since namespace overriding is just a basic thing these can do. You can manage labels, configuration, names, stack components and more.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search