Memory Tracking
Objectives
The purpose of this feature is to provide insight about the memory consumption of each component. This should be useful when debugging memory issues
How to enable it?
In order to enable the memory reporting, you should add a compilation flag:
MEMORY_REPORTING=1 make -j $(NPROC)
Technical Details
An arena is a contiguous piece of memory, which Speedb allocates once and use part of the memory to minimize number of allocations requested from the OS
The memory counter is printed to the log by default every 10 minutes.
This is a configurable parameter, you can change it under:
options.stats_dump_period_sec (the default of this option is 600, 10 minutes),
In order to get prints more often, the counter should be reduced.
Counters info:
These metrics have been chosen to be part of the memory reporting feature because they represent most of the memory allocated in Speedb.
Under Arena stats, there is memory allocated by the Arena, The total usage and per component that allocates using Arena.
Each line under the Arena Stats describes the distribution of allocations requested (and not deallocated yet) per component in the code.
CF Stats - shows Memtable memory allocation per CF and summarizes all CFs together.
rocksdb.block-cache-usage - Total block cache usage reporting
rocksdb.estimate-table-readers-mem - How much memory is allocated to indexes and filter blocks
rocksdb.block-cache-pinned-usage - Pinned block cache usage
CacheAllocationUsage - Total memory allocated by BlockFetcher
Example for general output:
** Memory Reporting **
Arena Stats:
Total: 16M
ArenaWrappedDBIter: 0
FileIndexer::UpdateIndex: 0
MemTable::NewIterator: 0
LevelFileInit: 0
FindLevelFileTest::Add: 0
DoGenerateLevelFilesBrief: 0
TEST_GetLevelIterator: 0
Version::AddIteratorsForLevel: 0
Version::OverlapWithLevelIterator: 0
LogBuffer::AddLogToBuffer: 0
arena_test: 0
HashLinkList: 0
HashLinkListIterator: 0
HashLinkListDynamicIterator: 0
HashSkipList: 0
HashSkipListIterator: 0
HashSkipListDynamicIterator: 0
HashSpdb: 16M
HashSpdbIterator: 0
InlineSkipList: 0
SkipList: 0
SkipListIterator: 0
SkipListLookaheadIterator: 0
VectorMemtable: 0
CompactionMergingIterator: 0
NewErrorInternalIterator: 0
NewEmptyInternalIterator: 0
MergingIterator: 0
BlockBasedTableIterator: 0
BlockPrefixIndex::Builder: 0
CuckooTableIterator: 0
PlainTableBloomV1: 0
PlainTableIndexBuilder::FillIndexes: 0
PlainTableReader::NewIterator: 0
DynamicBloom: 0
DefaultMemtableImpl: 0
WriteBatchWithIndex: 0
CF Stats:
Total: 16M
[default]: 16M
rocksdb.block-cache-usage: 96
rocksdb.estimate-table-readers-mem: 659K
rocksdb.block-cache-pinned-usage: 96
Total CacheAllocationUsage: 655K
Jmalloc
In addition when Jemalloc is being used as an allocator for Speedb, there are more memory reporting metrics, in order to enable this dump please set option.dump_malloc_stats to true.
Allocated: 262920624, active: 264196096, metadata: 7739440 (n_thp 0), resident: 273063936, mapped: 307642368, retained: 79806464
Allocated - Number of bytes allocated by the Rocksdb
Active - Number of bytes allocated actively (includes the entire pages allocated size)
Metadata - Jemalloc itself metadata to manage memory allocation
Resident - Maximum overall size allocated includes zeroed pages including metadata (which may not reserve actual memory).
Mapped - Mapped is the sum of regions of virtual address space currently dedicated (internally) to serving some live allocation. - from jemalloc git
Retained - Freed memory that Jemalloc didnt return to the OS
Source: https://jemalloc.net/jemalloc.3.html#stats.allocated
Last updated
Was this helpful?