Frequently Asked Questions (FAQ)

DataFrame memory usage

The memory usage of a DataFrame (including the index) is shown when calling the info(). A configuration option, display.memory_usage (see the list of options), specifies if the DataFrame’s memory usage will be displayed when invoking the df.info() method.

For example, the memory usage of the DataFrame below is shown when calling info():

In [1]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
   ...:           'complex128', 'object', 'bool']
   ...: 

In [2]: n = 5000

In [3]: data = dict([(t, np.random.randint(100, size=n).astype(t))
   ...:               for t in dtypes])
   ...: 

In [4]: df = pd.DataFrame(data)

In [5]: df['categorical'] = df['object'].astype('category')

In [6]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
int64              5000 non-null int64
float64            5000 non-null float64
datetime64[ns]     5000 non-null datetime64[ns]
timedelta64[ns]    5000 non-null timedelta64[ns]
complex128         5000 non-null complex128
object             5000 non-null object
bool               5000 non-null bool
categorical        5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 289.1+ KB

The + symbol indicates that the true memory usage could be higher, because pandas does not count the memory used by values in columns with dtype=object.

Passing memory_usage='deep' will enable a more accurate memory usage report, accounting for the full usage of the contained objects. This is optional as it can be expensive to do this deeper introspection.

In [7]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
int64              5000 non-null int64
float64            5000 non-null float64
datetime64[ns]     5000 non-null datetime64[ns]
timedelta64[ns]    5000 non-null timedelta64[ns]
complex128         5000 non-null complex128
object             5000 non-null object
bool               5000 non-null bool
categorical        5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 425.6 KB

By default the display option is set to True but can be explicitly overridden by passing the memory_usage argument when invoking df.info().

The memory usage of each column can be found by calling the memory_usage() method. This returns a Series with an index represented by column names and memory usage of each column shown in bytes. For the DataFrame above, the memory usage of each column and the total memory usage can be found with the memory_usage method:

In [8]: df.memory_usage()
Out[8]: 
Index                 80
int64              40000
float64            40000
datetime64[ns]     40000
timedelta64[ns]    40000
complex128         80000
object             40000
bool                5000
categorical        10920
dtype: int64

# total memory usage of dataframe
In [9]: df.memory_usage().sum()