# Memory Tracking

### Objectives

The purpose of this feature is to provide insight about the memory consumption of each component. This should be useful when debugging memory issues 

### How to enable it?

In order to enable the memory reporting, you should add a compilation flag:

*MEMORY\_REPORTING=1 make -j $(NPROC)* 

### Technical Details 

An arena is a contiguous piece of memory, which Speedb allocates once and use part of the memory to minimize number of allocations requested from the OS

The memory counter is printed to the log by default every 10 minutes. 

* This is a configurable parameter, you can change it under:

       *options.stats\_dump\_period\_sec* (the default of this option is 600, 10 minutes),

In order to get prints more often, the counter should be reduced.

<br>

#### Counters info:

* These metrics have been chosen to be part of the memory reporting feature because they represent most of the memory allocated in Speedb.
* Under Arena stats, there is memory allocated by the Arena, The total usage and per component that allocates using Arena.
* Each line under the Arena Stats describes the distribution of allocations requested (and not deallocated yet) per component in the code.
  * CF Stats - shows Memtable memory allocation per CF and summarizes all CFs together.
  * rocksdb.block-cache-usage - Total block cache usage reporting
  * rocksdb.estimate-table-readers-mem - How much memory is allocated to indexes and filter blocks
  * rocksdb.block-cache-pinned-usage - Pinned block cache usage
  * CacheAllocationUsage - Total memory allocated by BlockFetcher

**Example for general output:**&#x20;

\*\* Memory Reporting \*\*

Arena Stats:

Total: 16M

ArenaWrappedDBIter: 0

FileIndexer::UpdateIndex: 0

MemTable::NewIterator: 0

LevelFileInit: 0

FindLevelFileTest::Add: 0

DoGenerateLevelFilesBrief: 0

TEST\_GetLevelIterator: 0

Version::AddIteratorsForLevel: 0

Version::OverlapWithLevelIterator: 0

LogBuffer::AddLogToBuffer: 0

arena\_test: 0

HashLinkList: 0

HashLinkListIterator: 0

HashLinkListDynamicIterator: 0

HashSkipList: 0

HashSkipListIterator: 0

HashSkipListDynamicIterator: 0

HashSpdb: 16M

HashSpdbIterator: 0

InlineSkipList: 0

SkipList: 0

SkipListIterator: 0

SkipListLookaheadIterator: 0

VectorMemtable: 0

CompactionMergingIterator: 0

NewErrorInternalIterator: 0

NewEmptyInternalIterator: 0

MergingIterator: 0

BlockBasedTableIterator: 0

BlockPrefixIndex::Builder: 0

CuckooTableIterator: 0

PlainTableBloomV1: 0

PlainTableIndexBuilder::FillIndexes: 0

PlainTableReader::NewIterator: 0

DynamicBloom: 0

DefaultMemtableImpl: 0

WriteBatchWithIndex: 0

CF Stats:&#x20;

Total: 16M

\[default]: 16M

rocksdb.block-cache-usage: 96

rocksdb.estimate-table-readers-mem: 659K

rocksdb.block-cache-pinned-usage: 96

Total CacheAllocationUsage: 655K

#### Jmalloc

\
In addition when Jemalloc is being used as an allocator for Speedb, there are more memory reporting metrics, in order to enable this dump please set option.dump\_malloc\_stats to true.

*Allocated: 262920624, active: 264196096, metadata: 7739440 (n\_thp 0), resident: 273063936, mapped: 307642368, retained: 79806464*

**Allocated -** Number of bytes allocated by the Rocksdb

**Active -** Number of bytes allocated actively (includes the entire pages allocated size)<br>

**Metadata** - Jemalloc itself metadata to manage memory allocation

**Resident** - Maximum overall size allocated includes zeroed pages including metadata (which may not reserve actual memory).

**Mapped** - Mapped is the sum of regions of virtual address space currently dedicated (internally) to serving some live allocation. - from jemalloc git

**Retained** - Freed memory that Jemalloc didnt return to the OS

**Source: <https://jemalloc.net/jemalloc.3.html#stats.allocated>**

\ <br>