I am building a large Python app, and once starting it from console as
python main.py
it wouldn’t take too long for application to start up on my Ubuntu 22.04 OS. Everything was good, and then I moved to build same app on Windows 10 and couldn’t fail to notice how it would take at least 5 seconds for the application to start, and for Ubuntu it was ~1 second. I then started investigating and found that literally every single import statement on Windows runs much slower compared to Ubuntu.
Some examples:
import time
start = time.time()
from PySide6.QtCore import * # I know import * is not the most appropriate way, I am just showing an example
print(time.time() - start)
# 0.21s on Windows, 0.12s on Ubuntu
import time
start = time.time()
import scipy.signal
print(time.time() - start)
# 0.64s on Windows, 0.17s on Ubuntu
import time
start = time.time()
from numba import njit
print(time.time() - start)
# 0.3s on Windows, 0.07s on Ubuntu
And all these differences add up in the app to the point where user opening the app starts noticing the difference. How does it happen so that everything that Windows import is so slow?
2
Answers
First of all check your Python installation and project files, make sure they are excluded from antivirus scans.
Secondly run Python with the -O flag for slight optimizations like this:
python -O main.py
The differences in import times between Windows and Ubuntu can be attributed to several factors:
NTFS vs. EXT4: Windows typically uses the NTFS file system, while Ubuntu commonly uses EXT4. EXT4 is generally faster for file operations than NTFS, which can lead to faster module loading times on Linux.
Antivirus Software: Windows often runs antivirus software that scans files when they are accessed, potentially slowing down file operations. Disabling or configuring the antivirus software to exclude certain directories can help mitigate this.
Path Length and Resolution: Windows has more complex path resolution due to its handling of drive letters, UNC paths, and potentially long path names. This can add overhead to file operations.
Case Sensitivity: Linux file systems are typically case-sensitive, while Windows file systems are not. This can lead to additional checks and slower file handling on Windows.
DLLs vs. Shared Objects: The way Windows handles dynamic link libraries (DLLs) is different from how Linux handles shared objects (.so files). The Windows loader might have more overhead in resolving dependencies compared to Linux.
Python Environment: Differences in the Python environment setup, such as the use of virtual environments or the presence of additional packages and configurations, can affect import times.
Path Variables: The PATH and PYTHONPATH variables might be set differently between the two systems, affecting the time it takes to locate and import modules.
File Fragmentation: NTFS is more prone to file fragmentation compared to EXT4. Fragmented files take longer to read, which can slow down import times.
Disk Speed: Differences in hardware, such as disk type (HDD vs. SSD), can also impact file read times. Ensure that both systems are using comparable hardware for a fair comparison.
Solutions and Workarounds
Profile Imports:
Use a profiling tool to identify which imports are taking the most time and investigate if they can be optimized or deferred.
Optimize File System:
Defragment the disk on Windows.
Exclude Python directories from antivirus scans.
Environment Optimization:
Use a virtual environment to manage dependencies.
Simplify the PYTHONPATH and PATH variables.
Lazy Imports:
Consider using lazy imports for modules that are not immediately needed at startup. This can reduce the initial load time.
Precompile Python Files:
Ensure that .pyc files are being generated and used, as these are faster to load than .py files.
By addressing these factors, you should be able to reduce the import times and improve the startup performance of your Python application on Windows.