Question posted in
Some documentation to get you started can be found here.

Django + Celery + Apache mod_wsgi + Postgres + RabbitMQ Application for multiple clients

I have a Django Application that uses Celery, RabbitMQ, with Apache mod_wsgi. Currently all on one server. Each client has their own URL mount, eg:

www.example.com/Client001

www.example.com/Client002

www.example.com/Client003

Each client has their own database and project directory with local_setting.py for their Django settings.

I’m using supervisord to manage Celery Worker + Celery Beat for each client.

As I get more clients so maintaining gets more time consuming.

I’ve started playing with Docker to try and simplify deployments, and probably scale across multiple hosts.

Whilst it’s quite easy setting up Docker Compose to run a group of services for one client, I’m trying to figure out the best approach for multiple clients that is easy to manage, e.g. quickly setup a new client mounted under the main URL.

I’m thinking that the Postgres database instance should be share to hold each clients database, much as it is now. And to have a shared NGIX instance to handle the HTTP side. For each client use a Kubernetes Pod consisting:

  • Gunicorn to handle Django
  • Celery Beat
  • Celery Worker
  • Light weight HTTP server for static files.

So the question is, is this a good way or is there a better way of approaching and dealing with this?

I’m also wondering if I should go down the route of building an image for each client as that might be easier to manage?

Any advice welcome.

Tags:

2

Answers


  1. 0

    There are three ways on handling clients:

    1. Isolated Approach: Separate Databases. Each tenant has it’s own database.
    2. Semi Isolated Approach: Shared Database, Separate Schemas. One database for all tenants, but one schema per tenant.
    3. Shared Approach: Shared Database, Shared Schema. All tenants share the same database and schema. There is a main tenant-table, where all other tables have a foreign key pointing to.

    django-tenents package uses 2nd method and have a sub-domains for client such as client1.example.com, client2.example.com etc.

    I have used django-tenants also 3rd with adding a Company model foreign key for every Model that I created.

    django tenant helps but has different schema in postgres; has less overhead integration.

    Adding Company must be implement in every model and should be handled with middleware or mixins if you’re using class based views.

    Login or Signup to reply.
  2. 0

    My suggestion would be keeping one codebase and one server running(or multiple server of the same Django application without any customization based on clients) for this. Main reason is to maintenance easier. You do not want to make changes multiple times to provide a feature to multiple clients.

    As you already have a Django application, I think it is best to utilize that code to accommodate the approach given above with minimal change of code. Meaning, you need some way to handle multiple clients connecting to multiple DB. I would suggesting is to use a middleware and database router. Like this:(codes based on this snippet).

    import threading

request_cfg = threading.local()

class RouterMiddleware (object):
    def process_view( self, request, view_func, args, kwargs ):
        if 'client' in kwargs:
            request_cfg.client = kwargs['client']
            request.client = client 
            # Here, we are adding client info with request object.
            # It will make the implementation bit easier because
            # then you can access client information anywhere in the view/template.

    def process_response( self, request, response ):
        if hasattr( request_cfg, 'client' ):
            del request_cfg.client
        return response

class DatabaseRouter (object):
    def _default_db( self ):
        from django.conf import settings
        if hasattr( request_cfg, 'client' ) and request_cfg.client in settings.DATABASES:
            return request_cfg.client            
        else:
            return None

    def db_for_read( self, model, **hints ):
        return self._default_db()

    def db_for_write( self, model, **hints ):
        return self._default_db()

    Then add those to settings.py:

    DATABASES = {
    'default': {
        'NAME': 'user',
        'ENGINE': 'django.db.backends.postgresql',
        'USER': 'postgres_user',
        'PASSWORD': 's3krit'
    },
    'client1': {
        'NAME': 'client1',
        'ENGINE': 'django.db.backends.postgresql',
        'USER': 'postgres_user',
        'PASSWORD': 's3krit'
    },
    'client2': {
        'NAME': 'client2',
        'ENGINE': 'django.db.backends.postgresql',
        'USER': '_user',
        'PASSWORD': 'priv4te'
    }
}

DATABASE_ROUTERS = [
    'path.to.DatabaseRouter', 
]

MIDDLEWARE = [
    # middlewares
    'path.to.RouterMiddleware'
]

    Finally update the urls.py:

    urlpatterns = [
    path('<str:client>/admin/', admin.site.urls),
    path('<str:client>/', include('client_app.urls')),
    # and so on
]

    Advantage of this approach is that you do not have to configure anything for new clients, all you need to do is add a new database in settings and run the migrations as per described in the documentation. No need to configure reverse proxy server or anything else.

    Now, when it comes to handling tasks in celery, you can provide which database you will be using to run the the queries(reference to docs). Here is an example:

    @app.task
def some_task():
    logger.info("-"*25)
    for db_name in settings.DATABASES.keys():
        Model.objects.using(db_name).filter(some_condition=True)
    logger.info("-"*25)
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search