I used the following process the generate a numpy array with size = (720, 720, 3). In principle, it should cost 720 * 720 * 3 * 8Byte = 12.3MB. However, in the ans = memory_benchmark(), it costs 188 MB. Why does it cost much more memory than expected? I think it should have same cost as the line m1 = np.ones((720, 720, 3)).
I have following two Environments. Both have same problem.
Environment1: numpy=1.23.4, memory_profiler=0.61.0, python=3.10.6, MacOS 12.6.1(Intel not M1)
Environment2: numpy=1.19.5, memory_profiler=0.61.0, python=3.8.15, MacOS 12.6.1(Intel not M1)
I did memory profile in the following
import numpy as np
from memory_profiler import profile
@profile
def memory_benchmark():
m1 = np.ones((720, 720, 3))
m2 = np.random.randint(128, size=(720, 720, 77, 3))
a = m2[:, :, :, 0].astype(np.uint16)
b = m2[:, :, :, 1].astype(np.uint16)
ans = np.array(m1[b, a].sum(axis=2))
m2 = None
a = None
b = None
m1 = None
return ans
@profile
def f():
ans = memory_benchmark()
print(ans.shape)
print("finished")
if __name__ == '__main__':
f()
(720, 720, 3)
finished
Line # Mem usage Increment Occurrences Line Contents
=============================================================
5 59.3 MiB 59.3 MiB 1 @profile
6 def memory_benchmark():
7 71.2 MiB 11.9 MiB 1 m1 = np.ones((720, 720, 3))
8 984.8 MiB 913.7 MiB 1 m2 = np.random.randint(128, size=(720, 720, 77, 3))
9 1061.0 MiB 76.1 MiB 1 a = m2[:, :, :, 0].astype(np.uint16)
10 1137.1 MiB 76.1 MiB 1 b = m2[:, :, :, 1].astype(np.uint16)
11 1160.9 MiB 23.8 MiB 1 ans = np.array(m1[b, a].sum(axis=2))
12 247.3 MiB -913.6 MiB 1 m2 = None
13 247.3 MiB 0.0 MiB 1 a = None
14 247.3 MiB 0.0 MiB 1 b = None
15 247.3 MiB 0.0 MiB 1 m1 = None
16 247.3 MiB 0.0 MiB 1 return ans
Line # Mem usage Increment Occurrences Line Contents
=============================================================
19 59.3 MiB 59.3 MiB 1 @profile
20 def f():
21 247.3 MiB 188.0 MiB 1 ans = memory_benchmark()
22 247.3 MiB 0.0 MiB 1 print(ans.shape)
23 247.3 MiB 0.0 MiB 1 print("finished")
If I print(type(m1[0, 0, 0])) yields <class 'numpy.float64'>, print(type(m2[0, 0, 0, 0])) yields <class 'numpy.int64'>, print(type(ans[0, 0, 0])) yields <class 'numpy.float64'>
However, in my Ubuntu VM, I don’t have above problem.
2
Answers
I can’t reproduce the results you’re getting. In python 3.7.3, numpy 1.21.4, and memory_profiler 0.61.0, I’m getting the following results
Printing
type(m1[0,0,0,0])yields<class 'numpy.int32'>, so the 457.8 MiB makes sense. On the other hand, your output seems weird, given that assigningm1toNonereports no difference in memory. Which python & library versions are you using?Update: In a different machine, with python 3.10.6, numpy 1.23.5, and memory_profiler 0.61.0, I still cannot reproduce the OP output.
Those numbers look fine to me:
Evidently once you drop it to 247.3 MiB, the interpreter/numpy decides to "hang on" to that memory, rather than return it to the OS. When tracking memory you are dealing the "choices" of several layers – OS, python interpreter, and
numpy'sown memory management. One or more of those layers can maintain a "free space" from which it can allocated new objects or arrays.