Перейти к содержанию

Управление памятью в Python

Как все устроено под капотом

Stack → Heap → Arena (256KB, contains 64 x Pool) → Pool (4KB) → Block (multiple of 8KB) → Object.

Python memory manager components

Полезные команды

Print summary info to stderr about the state of pymalloc's structures:

1
2
3
import sys

sys._debugmallocstats()

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
Small block threshold = 512, in 32 size classes.
class   size   num pools   blocks in use  avail blocks
-----   ----   ---------   -------------  ------------
    0     16           1             278           743
    1     32          11            5106           504
    2     48          56           16580          2460
    3     64         206           52257           273
    4     80         125           25333           167
    5     96          46            7733            87
    6    112          25            3579            46
    7    128          32            3839           225
    8    144          10            1102            28
    9    160          92            9358            26
   10    176           9             809            19
   11    192          13            1055            50
   12    208          28            2120            64
   13    224          20            1383            57
   14    240          21            1375            53
   15    256          17            1039            32
   16    272          15             886            14
   17    288          13             689            39
   18    304          12             594            42
   19    320          10             477            33
   20    336           9             425             7
   21    352           8             352            16
   22    368           8             341            11
   23    384           7             292             2
   24    400          21             820            20
   25    416           7             265             8
   26    432           7             234            25
   27    448           6             188            28
   28    464           6             185            25
   29    480           5             146            24
   30    496           5             157             3
   31    512           7             201            16
# arenas allocated total           =                   14
# arenas reclaimed                 =                    0
# arenas highwater mark            =                   14
# arenas allocated current         =                   14
14 arenas * 1048576 bytes/arena    =           14,680,064
# bytes in allocated blocks        =           13,554,240
# bytes in available blocks        =              406,000
27 unused pools * 16384 bytes      =              442,368
# bytes lost to pool headers       =               41,184
# bytes lost to quantization       =               56,048
# bytes lost to arena alignment    =              180,224
Total                              =           14,680,064
arena map counts
# arena map mid nodes              =                    1
# arena map bot nodes              =                    1
# bytes lost to arena map root     =              262,144
# bytes lost to arena map mid      =              262,144
# bytes lost to arena map bot      =              131,072
Total                              =              655,360
           53 free PyDictObjects * 48 bytes each =                2,544
           4 free PyFloatObjects * 24 bytes each =                   96
           69 free PyListObjects * 40 bytes each =                2,760
  23 free 1-sized PyTupleObjects * 32 bytes each =                  736
1475 free 2-sized PyTupleObjects * 40 bytes each =               59,000
  10 free 3-sized PyTupleObjects * 48 bytes each =                  480
  14 free 4-sized PyTupleObjects * 56 bytes each =                  784
  31 free 5-sized PyTupleObjects * 64 bytes each =                1,984
  13 free 6-sized PyTupleObjects * 72 bytes each =                  936
  10 free 7-sized PyTupleObjects * 80 bytes each =                  800
   9 free 8-sized PyTupleObjects * 88 bytes each =                  792
   7 free 9-sized PyTupleObjects * 96 bytes each =                  672
 0 free 10-sized PyTupleObjects * 104 bytes each =                    0
 1 free 11-sized PyTupleObjects * 112 bytes each =                  112
 2 free 12-sized PyTupleObjects * 120 bytes each =                  240
 5 free 13-sized PyTupleObjects * 128 bytes each =                  640
 2 free 14-sized PyTupleObjects * 136 bytes each =                  272
 1 free 15-sized PyTupleObjects * 144 bytes each =                  144
 6 free 16-sized PyTupleObjects * 152 bytes each =                  912
 3 free 17-sized PyTupleObjects * 160 bytes each =                  480
 1 free 18-sized PyTupleObjects * 168 bytes each =                  168
 3 free 19-sized PyTupleObjects * 176 bytes each =                  528
55 free 20-sized PyTupleObjects * 184 bytes each =               10,120
Small block threshold = 512, in 32 size classes.
class   size   num pools   blocks in use  avail blocks
-----   ----   ---------   -------------  ------------
    0     16           1             278           743
    1     32          11            5106           504
    2     48          56           16581          2459
    3     64         206           52261           269
    4     80         125           25339           161
    5     96          46            7732            88
    6    112          25            3579            46
    7    128          32            3841           223
    8    144          10            1102            28
    9    160          92            9357            27
   10    176           9             810            18
   11    192          13            1055            50
   12    208          28            2120            64
   13    224          20            1383            57
   14    240          21            1377            51
   15    256          17            1033            38
   16    272          15             887            13
   17    288          13             689            39
   18    304          12             593            43
   19    320          10             477            33
   20    336           9             425             7
   21    352           8             353            15
   22    368           8             341            11
   23    384           7             292             2
   24    400          21             820            20
   25    416           7             265             8
   26    432           7             234            25
   27    448           6             188            28
   28    464           6             185            25
   29    480           5             147            23
   30    496           5             157             3
   31    512           7             201            16
# arenas allocated total           =                   14
# arenas reclaimed                 =                    0
# arenas highwater mark            =                   14
# arenas allocated current         =                   14
14 arenas * 1048576 bytes/arena    =           14,680,064
# bytes in allocated blocks        =           13,554,944
# bytes in available blocks        =              405,296
27 unused pools * 16384 bytes      =              442,368
# bytes lost to pool headers       =               41,184
# bytes lost to quantization       =               56,048
# bytes lost to arena alignment    =              180,224
Total                              =           14,680,064
arena map counts
# arena map mid nodes              =                    1
# arena map bot nodes              =                    1
# bytes lost to arena map root     =              262,144
# bytes lost to arena map mid      =              262,144
# bytes lost to arena map bot      =              131,072
Total                              =              655,360
           52 free PyDictObjects * 48 bytes each =                2,496
           2 free PyFloatObjects * 24 bytes each =                   48
           69 free PyListObjects * 40 bytes each =                2,760
  27 free 1-sized PyTupleObjects * 32 bytes each =                  864
1474 free 2-sized PyTupleObjects * 40 bytes each =               58,960
   9 free 3-sized PyTupleObjects * 48 bytes each =                  432
  14 free 4-sized PyTupleObjects * 56 bytes each =                  784
  31 free 5-sized PyTupleObjects * 64 bytes each =                1,984
  13 free 6-sized PyTupleObjects * 72 bytes each =                  936
  10 free 7-sized PyTupleObjects * 80 bytes each =                  800
   9 free 8-sized PyTupleObjects * 88 bytes each =                  792
   7 free 9-sized PyTupleObjects * 96 bytes each =                  672
 0 free 10-sized PyTupleObjects * 104 bytes each =                    0
 1 free 11-sized PyTupleObjects * 112 bytes each =                  112
 2 free 12-sized PyTupleObjects * 120 bytes each =                  240
 5 free 13-sized PyTupleObjects * 128 bytes each =                  640
 2 free 14-sized PyTupleObjects * 136 bytes each =                  272
 1 free 15-sized PyTupleObjects * 144 bytes each =                  144
 6 free 16-sized PyTupleObjects * 152 bytes each =                  912
 3 free 17-sized PyTupleObjects * 160 bytes each =                  480
 1 free 18-sized PyTupleObjects * 168 bytes each =                  168
 3 free 19-sized PyTupleObjects * 176 bytes each =                  528
55 free 20-sized PyTupleObjects * 184 bytes each =               10,120

Документация

  1. Memory Management.
  2. tracemalloc — Trace memory allocations.
  3. gc — Garbage Collector interface.

Ссылки на статьи по теме

  1. Memory management in Python, Artem Golubin.
  2. Garbage collection in Python: things you need to know, Artem Golubin.
  3. Python3: Give unused interpreter memory back to the OS, Stack Overflow.

Ссылки на код

  1. cpython/Objects/obmalloc.c.

Что дальше?

  1. Нашли эту статью полезной? Поделитесь ею и помогите распространить знания!
  2. Нашли ошибку или есть идеи 💡 о том, что и как я могу улучшить? Напишите мне в Telegram.
  3. Хотите узнать обо мне больше? Читайте здесь.