Apache - Django - large file upload - moving file from temp folder to final destination fails

Elembivios
October 8, 2024
144 views
0 votes
2 Answers

Stumbled upon this error when upgrading Python, Django and Alpine to latest versions… and it’s a weird one.
The error occurs when I upload a file that is larger than FILE_UPLOAD_MAX_MEMORY_SIZE (set to 5MB).

I have two development environments set up.

One is local dev server (just normal py manage.py runserver),
the other one is running under Docker(Windows) + Alpine linux + Apache2 + mod_wsgi which is also still a ‘development’ environment.

This error occurs only under Docker environment and only when file exceeds max memory size. Under all other circumstances the upload works – even when file exceeds memory limit on local dev server.

The error:

Saving new order file to: customer_name/_web_portal/PendingOrders/7935/zip_03.zip 

Internal Server Error: /b2b_api/orders/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/api/views.py", line 352, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create
    self.perform_create(serializer)
  File "/code/api/views.py", line 375, in perform_create
    serializer.save()
  File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 212, in save
    self.instance = self.create(validated_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/api/serializers.py", line 286, in create
    instance = super().create(validated_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/api/serializers.py", line 125, in create
    db_file.save()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
              ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
    for sql, params in self.as_sql():
                       ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
    value_rows = [
                 ^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
    [
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
    self.prepare_value(field, self.pre_save_val(field, obj))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/files/storage/base.py", line 38, in save
    name = self._save(name, content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/files/storage/filesystem.py", line 130, in _save
    os.chmod(full_path, self.file_permissions_mode)
FileNotFoundError: [Errno 2] No such file or directory: '/sftp/customer_name/_web_portal/PendingOrders/7935/zip_03.zip'

The wierd thing about this is, when I look in to the folder, the file is there. And the user that Apache is running under, can access it.

Tried checking what’s going on with inotifywait and this is the output for upload directory:

Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/sftp/customer_name/_web_portal/PendingOrders/ CREATE,ISDIR 7982
/sftp/customer_name_web_portal/PendingOrders/ OPEN,ISDIR 7982
/sftp/customer_name/_web_portal/PendingOrders/ ACCESS,ISDIR 7982
/sftp/customer_name/_web_portal/PendingOrders/ CLOSE_NOWRITE,CLOSE,ISDIR 7982
# --> between those two records the error is thrown
/sftp/customer_name/_web_portal/PendingOrders/7982/ CLOSE_WRITE,CLOSE zip_02.zip

Also tried editing django files before the error (right before the os.chmod…) and printing if file exists and here is the output:

os.listdir(directory)=['zip_10MB.zip']
os.path.exists(full_path)=False
os.path.isfile(full_path)=False
repr(full_path)="'/sftp/customer_name/_web_portal/PendingOrders/8013/zip_10MB.zip'" repr(directory)="'/sftp/customer_name/_web_portal/PendingOrders/8013'"

Which is unreal to me, because listdir outputs that the folder has a file in it, but os.path.exists outputs False?!
I found out that os.path.exists() and isfile call os.stat() underneath to find the necessary info, … So I’ve tried os.stat(full_path) and it fails with FileNotFoundError.

After the error is thrown, I logged in to Docker with the same user the Apache is running under and tried running os.stat() on said file path, and it returns OK (fileinfo).

I’ve also checked what operation is Django using when moving the file from temp directory to it’s final destination – it’s django/core/files/move.py file_move_safe function – and it’s a plain os.rename()

Next thing I’ve tried is to put time.sleep() before the actual error and cd-ing in upload directory and doing an ls command with each user and here is the output:
root user:

total 10432
drwxrwx---    1 web_port web_port       512 Aug 17 09:48 .
drwxrwx---    1 web_port web_port       512 Aug 17 09:48 ..
-rw-------    1 web_port web_port  10679630 Aug 17 09:48 zip_10MB.zip

web_portal ( apache / application ) user:

ls: ./zip_10MB.zip: No such file or directory
total 0
drwxrwx---    1 web_port web_port       512 Aug 17 09:48 .
drwxrwx---    1 web_port web_port       512 Aug 17 09:48 ..

Ps: I’m using python:3.11-alpine Docker image (Docker Desktop), with Django==4.2.4 and djangorestframework==3.14.0.

Update:
Tried to place my Docker image / container on Linux OS – and the file upload works without an issue. So I’m guessing something is wrong with Docker for Windows, or how "bind" mount is handeled on Windows. Maybe?

Update #2:
Got an idea if maybe the file is being locked by Windows Defender or something.. idk I’m running out of ideas. So I’ve disabled Windows Defender, added exception on folder, .. the error remains. Next I’ve tried to enable Windows audit logging on folder but I have no idea on how to make sense from output of security log.

Has anyone any idea what the problem could be or what can I try next to further limit the cause of this problem?

Answers

- MichaelUshakov
- August 27, 2023 at 7:12 pm
- 0 votes
0
Try to set MAX_UPLOADING_FILE_SIZE in your settings.py file, this solution helps me im my projects:
```
MAX_UPLOADING_FILE_SIZE = 31457280  # 30 MB
```
Login or Signup to reply.

- BriandeHeus
- October 8, 2024 at 7:43 pm
- 0 votes
0
I ran into the same issue myself, and the issue was that I was re-using a TemporaryUploadedFile handler that has already been used for a field. Please note that the behavior is different from the InMemoryUploadedFile.

Lets assume we have this model:
```
class Foo(models.Model):
   file_one = models.FileField()
   file_two = models.FileField()
```
Then if you try to save it as such:
```
my_file = TemporaryUploadedFile()  # Assume it's properly instantiated

Foo.objects.create(file_one=my_file, file_two=my_file)
```
it will fail with an FileNotFoundError exception, because when Django saves the field file_one, file_move_safe is called which moves the file from a temporary location to it’s final destination.

Next up when Django saves the file_two field, it attempts to run file_move_safe again, which won’t work because the file is already moved.

The way I’m solving this, is by creating another instance of TemporaryUploadedFile, which would be an exact copy of my_file. Which isn’t efficient, but at least it resolves the issue without having to resort to loading incredibly large files in memory.

Of course, the best solution would be to ask yourself why you’re saving the same file to two different fields.
Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Apache – Django – large file upload – moving file from temp folder to final destination fails

Answers