Redis – Django Tenants – I spent a lot of time on it

User amnesia with Django_tenants.

Dear @user25066324,

Have you found a solution yet?

The problem is that simply a lot of code lacks in Django_tenants. First of all, Django_tenants preconizes the use of subdomains for its tenants. There is indeed also a possibility to go for the use of subfolders, but that is only a beta version and generates many errors.

Django_tenants works beautifully with me, but not after I have spent about two months of research and work to make that happen.

There are three problems to solve:

1. the proper working with subfolders,
2. authentication with subfolders,
3. Django sessions with subfolders.

I first had the subfolders working, but still kept having “user-amnesia” with Django_tenants, i.e. the user could log in and was accepted, but at the next page the user had become Anonymous again. For pages that required authorization, Django_tenants kept therefore running around.

There is a huge mass to cover here, so let’s kick off right away.

Let’s first start with the

proper management of the subfolders.

New middleware needs to be written. The beta version of the subfolder middleware of the Django_tenants does not suffice. Normally, you’re supposed to just switch the middleware in settings.py to the the subfolder middleware. This doesn’t work and I wrote my own middleware: subfolderhans.py, which I include here and you’re free to use. I have put this in the general section of my Django project, and refer to it in my settings.py like this:

from subfolderhans import TenantSubfolderMiddlewareHans

MIDDLEWARE = [
        # 'django_tenants.middleware.main.TenantMainMiddleware',
        'subfolderhans.TenantSubfolderMiddlewareHans',

Then, there is the urlresolver.py, also added to this answer, and that you have to write over the original one in the Django_tenant source files:
COPY urlresolvers.py /usr/local/lib/python3.10/dist-packages/django_tenants/urlresolvers.py
(on Linux), on Windows it is the c:users<your name>AppDataRoaming...
Pay attention, AppData is a hidden directory, you have to put in your view settings “Show hidden files” to True.

Then, there is the context_processors.py, also included here and that you have to put in your general section of your Django_tenants project. You can use this if you want to have in your static map a directory per app, and then refer in your template files to images etc by using the context_processors.py. Personnaly, I don’t have so many graphics, so I put them all together and just make sure the names of the images are unique.

Second , the context_processors is also used in the settings.py:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'csb_tenants', 'templates', 'csb_tenants')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'context_processors.get_program_settings', # for templates
                'context_processors.admin_header_processor', # for admin titles in registration section: password change/reset, ...
                
            ],
        },
    },
]

Csb_tenants in my case is my tenant_app.

For the sake of completeness, here a copy of my shared and tenant app settings in settings.py:

TENANT_SUBFOLDER_PREFIX = "huurder"

# Application definition

SHARED_APPS = [
    'django_tenants',
    'csb_tenants',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

TENANT_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'csb_tenant',
]

INSTALLED_APPS = list(SHARED_APPS) + [app for app in TENANT_APPS if app not in SHARED_APPS]

I specifically didn’t use “tenant” as the tenant prefix, for obvious reasons.

Furthermore, I wrote a small procedure in a global.py file to get the tenant primary key from the url, which is more or less the same as the get_subfolder(request) function in the context_processors.py:

def fetch_customer_stripeid_from_url(request):
    request_string = request.get_full_path()
    start_marker = "cus_"
    end_marker = "/"
    # Find the start and end of the substring to extract
    start = request_string.find(start_marker)
    end = request_string.find(end_marker, start + len(start_marker))
    # If end_marker is not found, include the rest of the string
    if end == -1:
        end = None
    customer_stripeid = request_string[start:end] 
    return customer_stripeid

So, this for so much as the subfolder treatment.

Now the next step:

Authentication in Django_tenants with subfolders

Authentication goes wrong in django_tenants with subfolders. You need an extra layer of authentication in settings.py:

AUTHENTICATION_BACKENDS = [
    'csb_tenant.views.TenantAuthenticationBackend',
    'django.contrib.auth.backends.ModelBackend',
]

Both of the authentications will be executed, first the above one, then the latter one, and if one of those give a positive authentication, the authentications goes through. My TenantAuthenticationBackend will do the authentication for tenants, and if that doesn’t work, the classic Modelbackend can try for the public schema, and if that also doesn’t work, then the user clearly is not authorized.

This is the code for the TenantAuthenticationBackend, that I have put in the views of my model app, but that’s personal.

from csb_tenants.csglobals import fetch_customer_stripeid_from_url
from django import forms
from django.contrib.auth import get_user_model
from django.contrib.auth.backends import ModelBackend
from django.contrib.auth.models import AnonymousUser
from django_tenants.utils import schema_context
from django.contrib.auth.forms import AuthenticationForm
from django.db import connection

class CustomAuthenticationForm(AuthenticationForm):
    def clean(self):
        username = self.cleaned_data.get('username')
        password = self.cleaned_data.get('password')
        return super().clean()


class TenantAuthenticationBackend(ModelBackend):
    def authenticate(self, request, username=None, password=None, **kwargs):
        UserModel = get_user_model()
        try:
            user = UserModel.objects.get(username=username)
            if user:
                if user.check_password(password):
                    return user
            else:
                tenant = csbtenants.objects.get(customer_stripeid=username)
                if tenant.password == password:
                    return AnonymousUser()
        except UserModel.DoesNotExist:
            cust_id = fetch_customer_stripeid_from_url(request)
            tenant = csbtenants.objects.get(customer_stripeid=cust_id)
            with schema_context(tenant.schema_name):
                try:
                    users = UserModel.objects.all()
                    for user in users:
                        print(user.username)

                    user = UserModel.objects.get(username=username)
                    if user:
                        if user.check_password(password):
                            return user
                except Exception as e:
                    print(f'TenantAuthenticationBackend: with schema_context(tenant.schema_name): An unexpected error occurred: {e}')
                    pass    
            pass
        except csbtenants.DoesNotExist:
            print(f'TenantAuthenticationBackend: csbtenants.DoesNotExist') 
            pass

        except Exception as e:
            print(f'TenantAuthenticationBackend: An unexpected error occurred: {e}')

        return None

As you can see, you have to inherit from the classic Modelbackend, and then override the authenticate function. (I should say “method”, I know, but I’m from the old generation of IT engineers, when object orientation theorema was not commonly used but just started).

This authentication backend is key. Now we can use it a bit everywhere, most importantly with logging in, but also for the finding of the next page after the logging in:

class TenantLoginView(auth_views.LoginView):
    customer_id = ''
    form_class = CustomAuthenticationForm

    def get_context_data(self, **kwargs):
        print_user_data(self.request)
        context = super().get_context_data(**kwargs)
        self.customer_id = fetch_customer_stripeid_from_url(self.request) 
        context['cust_id'] = self.customer_id 
        try:
            tenant = csbtenants.objects.get(customer_stripeid=self.customer_id)
            context['tenant_name'] = tenant.companyname
        except ObjectDoesNotExist:
            context['tenant_name'] = ''
        return context

    def form_valid(self, form):
        self.success_url = '/' + TENANT_SUBFOLDER_PREFIX + '/' + self.customer_id + '/tenantconfig/'
        
        tenant = csbtenants.objects.get(customer_stripeid=self.customer_id)
        with schema_context(tenant.schema_name):
            response = super().form_valid(form)
        
        print(f'Last query (schema_name: {connection.schema_name}) after response = super().form_valid: ', connection.queries[-1]['sql'])  # Print the last SQL query
        return response

    def form_invalid(self, form):
        response = super().form_invalid(form)
        messages.error(self.request, 'Invalid username or password.')
        return response
    
    def post(self, request, *args, **kwargs):
        self.customer_id = fetch_customer_stripeid_from_url(request)
        return super().post(request, *args, **kwargs)

    

class Index(TemplateView):
    template_name = 'csb_tenant/index.html'

    def dispatch(self, request, *args, **kwargs):
        print(f'request.user.username: {request.user.username}')
        no_redirect = request.user.is_authenticated or request.user.is_superuser or request.user.is_staff
        print(f'no_redirect: {no_redirect}')
        tenant_customer_stripeid = fetch_customer_stripeid_from_url(request)
        if not no_redirect: 
            return HttpResponseRedirect(f'/accounts/login?next=/{TENANT_SUBFOLDER_PREFIX}/{tenant_customer_stripeid}/tenantconfig/')
        return super().dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        tenant_customer_stripeid = fetch_customer_stripeid_from_url(self.request)
        context['csbtenant'] = get_object_or_404(csbtenants, customer_stripeid=tenant_customer_stripeid)
        context['form'] = LoadFilesForm()  
        return context

Here you see the index method that is of course the first page of the model_app, and to have access, the user should have to log in. This logging in happens in the TenantLoginView (you called it the CustomLoginView), inherited and overridden for the classic login view. The secret is indeed, as you supposed already, the use of schema_context.

Having covered the authentication, we’re not there yet, since we now have to cover the third element, the sessions.

Django_tenant sessions management with subfolders.

Let’s make this clear, we’re not talking about the network sessions, but the sessions from Django. These sessions are written in the database, in a table called django_session, and there should be for every tenant and therefore schema a table called Django_session. This is where it goes wrong in the standard Django_tenants, where those tables are generated, but then the records are fetched from the public schema, where the particular records aren’t present.

You do not have to use the database to store the sessions, you can choose not to store them at all, or have them saved differently, but standard is the database.

A Django sessions holds a record of three fields, the first being a unique id, the second the payload and the third a timestamp. The unique Id and the payload (generally a dictionary that can hold a lot of data) are stored in the db fully encrypted. This is to cover for level 4 of confidentiality, where standing data also needs to be encrypted.

I wrote a middleware to manage those sessions, that I enclose to this answer again, that I have put in the general section of my django project and did like this in my settings.py:

‘SessionMiddlewareHans.TenantSessionMiddleware’,
#’django.contrib.sessions.middleware.SessionMiddleware’,

Since the order of these middlewares is important, because if the tenant is not known yet, then you can not authenticate, and if you have not authenticated, you should not go and fetch the session, etc. This is my order of middlewares in settings.py:

MIDDLEWARE = [
    # 'django_tenants.middleware.main.TenantMainMiddleware',
    'subfolderhans.TenantSubfolderMiddlewareHans',
    'django.middleware.security.SecurityMiddleware',
    "whitenoise.middleware.WhiteNoiseMiddleware",
    'SessionMiddlewareHans.TenantSessionMiddleware',
    #'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',    
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware'

This will depend of course of your needs. I give it so you can compare. However, this order works for me.

This is pretty much it.

You will have to adapt a bit my code to your specific situation. Csb_tenants is the main tenant app, csb_tenant is the model app, csbtenants is the main model of the main tenant app, csbtenant is the model of the model app.

If I have forgotten something, or it is not clear, then please feel free to ask.

Django_tenants is extremele valuable software, and I’m happy to be able to contribute in this way.

Hans

Update: since it turns out I cannot upload codefiles, I give them here:

urlresolvers.py:

import re
import sys
from types import ModuleType

from django.db import connection
from django.conf import settings
from django.urls import URLResolver, reverse as reverse_default, path, include
from django.utils.functional import lazy
from django_tenants.utils import (
    get_tenant_domain_model,
    get_subfolder_prefix,
    clean_tenant_url, has_multi_type_tenants, get_tenant_types,
)


def reverse(viewname, urlconf=None, args=None, kwargs=None, current_app=None):
    url = reverse_default(viewname, urlconf, args, kwargs, current_app=current_app)
    return clean_tenant_url(url)


reverse_lazy = lazy(reverse, str)


def get_subfolder_urlconf(tenant):
   if has_multi_type_tenants():
       urlconf = get_tenant_types()[tenant.get_tenant_type()]['URLCONF']
   else:
       urlconf = settings.ROOT_URLCONF

   subfolder_prefix = get_subfolder_prefix()
   class TenantUrlConf(ModuleType):
       urlpatterns = [
           path(f"{subfolder_prefix}/{tenant.domain_subfolder}/", include(urlconf))
       ]

   return TenantUrlConf(tenant.domain_subfolder)

subfolderhans.py:

# Hans Wendel van Hespen 15/08/2023: eigen file van de middleware gemaakt voor een beter debugging op fly.io van de subfolders
from django.conf import settings
from django.core.exceptions import ImproperlyConfigured
from django.db import connection
from django.http import Http404
from django.urls import set_urlconf, clear_url_caches
from django_tenants.middleware import TenantMainMiddleware
from django_tenants.urlresolvers import get_subfolder_urlconf
from django_tenants.utils import (
    get_public_schema_name,
    get_tenant_model,
    get_subfolder_prefix, get_tenant_domain_model,
)


class TenantSubfolderMiddlewareHans(TenantMainMiddleware):
    """
    This middleware should be placed at the very top of the middleware stack.
    Selects the proper tenant using the path subfolder prefix. Can fail in
    various ways which is better than corrupting or revealing data.
    """

    TENANT_NOT_FOUND_EXCEPTION = Http404

    def __init__(self, get_response):
        super().__init__(get_response)
        self.get_response = get_response
        if not get_subfolder_prefix():
            raise ImproperlyConfigured(
                '"TenantSubfolderMiddleware" requires "TENANT_SUBFOLDER_PREFIX" '
                "present and non-empty in settings"
            )

    def process_request(self, request):
        # Short circuit if tenant is already set by another middleware.
        # This allows for multiple tenant-resolving middleware chained together.
        if hasattr(request, "tenant"):
            return
        
        is_public = False
        connection.set_schema_to_public()

        urlconf = None

        tenant_model = get_tenant_model()
        domain_model = get_tenant_domain_model()
        hostname = self.hostname_from_request(request)
        subfolder_prefix_path = "/{}/".format(get_subfolder_prefix())

        # We are in the public tenant
        if not request.path.startswith(subfolder_prefix_path):
            try:
                schema_name = get_public_schema_name()
                tenant = tenant_model.objects.get(schema_name = schema_name)
            except tenant_model.DoesNotExist:
                tenant_subfolder = ""
                is_public = True

            self.setup_url_routing(request, force_public=True)

        # We are in a specific tenant
        else:
            path_chunks = request.path[len(subfolder_prefix_path):].split("/")
            tenant_subfolder = path_chunks[0]
            try:
                tenant = self.get_tenant(domain_model=domain_model, hostname=tenant_subfolder)
            except domain_model.DoesNotExist:
                return self.no_tenant_found(request, hostname, subfolder_prefix_path, tenant_subfolder)

            tenant.domain_subfolder = tenant_subfolder
            urlconf = get_subfolder_urlconf(tenant)

        if is_public is True:
            pass
        else:
            tenant.domain_url = hostname
            request.tenant = tenant
            connection.set_tenant(request.tenant)
            clear_url_caches()  # Required to remove previous tenant prefix from cache, if present

        if urlconf:
            request.urlconf = urlconf
            set_urlconf(urlconf)

    def no_tenant_found(self, request, hostname, subfolder_prefix_path, tenant_subfolder):
        """ What should happen if no tenant is found.
        This makes it easier if you want to override the default behavior """
        raise self.TENANT_NOT_FOUND_EXCEPTION(f"No tenant for subfolder {hostname}. Request: {request}. Subfolder_prefix_path: {subfolder_prefix_path}. tenant_subfolder: {tenant_subfolder}")

SessionMiddlewhareHans.py:

# Hans Wendel van Hespen 01/06/2024: custom version of the standard session middleware, to make them schema_based.
from django.contrib.sessions.middleware import SessionMiddleware
from django_tenants.utils import schema_context
from django.http import HttpResponseServerError
from django.db import connection
from django.conf import settings

class TenantSessionMiddleware(SessionMiddleware):
    def process_request(self, request):
        # Get the session key from the cookies
        session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME)

        # Import csbtenants here, not at the module level
        from csb_tenants.models import csbtenants
        from csb_tenants.csglobals import fetch_customer_stripeid_from_url
        from stripeproject.settings import TENANT_SUBFOLDER_PREFIX

        # Determine the correct schema to use
        request_string = request.get_full_path()
        if TENANT_SUBFOLDER_PREFIX in request_string:
            with schema_context('public'):
                customer_id = fetch_customer_stripeid_from_url(request)
                tenant = csbtenants.objects.get(customer_stripeid=customer_id)
            with schema_context(tenant.schema_name):
                request.session = self.SessionStore(session_key)
        else:
            with schema_context('public'):
                request.session = self.SessionStore(session_key)


    def process_response(self, request, response):
        # Import csbtenants here, not at the module level
        from csb_tenants.models import csbtenants
        from csb_tenants.csglobals import fetch_customer_stripeid_from_url
        from stripeproject.settings import TENANT_SUBFOLDER_PREFIX

        try:
            request_string = request.get_full_path()
            if TENANT_SUBFOLDER_PREFIX in request_string:

                # Try to get the tenant for the current user
                with schema_context('public'):
                    # Get the customer_id from the URL
                    customer_id = fetch_customer_stripeid_from_url(request)
                    tenant = csbtenants.objects.get(customer_stripeid=customer_id)

                # If a tenant was found, use the tenant's schema
                with schema_context(tenant.schema_name):
                    antwoord = super().process_response(request, response)
                    return antwoord
            else:
                # No specific tenant in url: public schema
                with schema_context('public'):
                    return super().process_response(request, response)
            
        except Exception as e:
            print(f'TenantSessionMiddleware: An unexpected error occurred: {e}')
            return HttpResponseServerError('An unexpected error occurred.')

and finally, context_processors.py:

from stripeproject.settings import TENANT_SUBFOLDER_PREFIX
from django.contrib import admin

def get_subfolder_code(request):
    subfolder_code = ""

    if request is None:
        return subfolder_code

    try:
        subfolder_code = "/" + TENANT_SUBFOLDER_PREFIX + "/" + request.tenant.name
    except:
        pass
    return subfolder_code

#for template
def get_program_settings(request):
    subfolder_code = get_subfolder_code(request)
    context = {
        'subfolder_code': subfolder_code
    }
    return {"program_settings": context}

def admin_header_processor(request):
    site_header = getattr(admin.site, 'site_header') 
    site_title = getattr(admin.site, 'site_title')
    index_title = getattr(admin.site, 'index_title')
    return {"site_header": site_header, "site_title": site_title, "index_title": index_title}