This feature aims to simplify the configuration process for Speedb storage engine by providing an easy way to enable and set Speedb options to optimized configurations. By introducing the SharedOptions class and specific functions, users can effortlessly configure Speedb's features with minimal manual intervention. Also when running multiple Speedb instances, it allows to configure all of the instances using the same method, and same configuration.
Motivation
Speedb users currently configure the database manually. New Speedb users are required to spend time in order to understand how to enable/disable and actually configure all the relevant options to enjoy the capabilities.
Not all of the capabilities are enabled by default and some need different tuning per workload. The user always has the flexibility to tune the database based on their needs, but in case a user would like to enjoy all of Speedb's benefits, we give the option to let us do the work for you and tune the database or databases to a default and improved configuration, following basic system parameters you provide while using this new method.
Technical Details
SharedOptions Class
The SharedOptions class has been introduced to enhance the usability of Speedb in multiple database scenarios. It organizes shared options, making configuration more streamlined and user-friendly.
Configuration Parameters
To benefit from this feature, users need to provide three key configuration parameters:
Total RAM size for Speedb: Specify the amount of RAM allocated for Speedb operations.
Total number of threads for background jobs: Determine the number of threads allocated for handling background tasks.
Delayed write rate: Set the rate at which delayed writes are performed.
By utilizing the provided functions, Speedb's features can be easily configured to default settings, simplifying the process for users. The default configuration includes the following features:
Proactive Flushing
Global Delayed Write
Sorted Hash Memtable
Dynamic Delayed Writes
Paired bloom filter
Implementation
Struct SharedOptions
Contain the shared configuration for multiple databases group
Write buffer manager
Cache
Write controller
Env;
RateLimiter;
SstFileManager;
Logger;
EventListener;
FileChecksumGenFactory;
Contain the methodology for configuring Speedb features
total_threads
total_ram_size_bytes
delayed_write_rate
Write_buffer_size
Paired bloom filter
class SharedOptions
class SharedOptions {
public:
SharedOptions();
SharedOptions(size_t total_ram_size_bytes, size_t total_threads,
size_t delayed_write_rate = 256 * 1024 * 1024ul);
size_t GetTotalThreads() { return total_threads_; }
size_t GetTotalRamSizeBytes() { return total_ram_size_bytes_; }
size_t GetDelayedWriteRate() { return delayed_write_rate_; }
// this function will increase write buffer manager by increased_by amount
// as long as the result is not bigger than the maximum size of
// total_ram_size_ /4
void IncreaseWriteBufferSize(size_t increase_by);
std::shared_ptr<Cache> cache = nullptr;
std::shared_ptr<WriteController> write_controller = nullptr;
std::shared_ptr<WriteBufferManager> write_buffer_manager = nullptr;
Env* env = Env::Default();
std::shared_ptr<RateLimiter> rate_limiter = nullptr;
std::shared_ptr<SstFileManager> sst_file_manager = nullptr;
std::shared_ptr<Logger> info_log = nullptr;
std::vector<std::shared_ptr<EventListener>> listeners;
std::shared_ptr<FileChecksumGenFactory> file_checksum_gen_factory = nullptr;
private:
size_t total_threads_ = 0;
size_t total_ram_size_bytes_ = 0;
size_t delayed_write_rate_ = 0;
};
SharedOptions constructor
Creates a LRU cache
Default values for cache’s construction is:
Capacity = _total_ram_size_bytes
Creates a write buffer manager (with proactive flushes)
Default values for WBM’s construction is:
_write_buffer_size = 1;
kDfltMaxNumParallelFlushes = 4U;
Creates a write_controller
default values for WC’s construction is:
Creates a LRU cache
default values for cache’s construction is:
Capacity = _total_ram_size_bytes
Creates a write buffer manager (with proactive flushes)
default values for WBM’s construction is:
_write_buffer_size = 1;
kDfltMaxNumParallelFlushes = 4U;
Creates a write_controller
Default values for WC’s construction is:
dynamic_delay = true
_delayed_write_rate = 256 * 1024 * 1024ul
Note: The user is still able to configure any non-shared parameter as she is currently doing.
Usage
Below is an example demonstrating the usage of these functions and enabling Speedb features with default configurations. It is also available in the examples directory /enable_speedb_features_example.cc".
#include <cstdio>
#include <iostream>
#include <string>
#include "rocksdb/compression_type.h"
#include "rocksdb/db.h"
#include "rocksdb/options.h"
#include "rocksdb/slice.h"
using namespace ROCKSDB_NAMESPACE;
#if defined(OS_WIN)
std::string kDBPath1 = "C:\\Windows\\TEMP\\enable_speedb_features_example1";
std::string kDBPath2 = "C:\\Windows\\TEMP\\enable_speedb_features_example2";
std::string kDBPath3 = "C:\\Windows\\TEMP\\enable_speedb_features_example3";
std::string kDBPath4 = "C:\\Windows\\TEMP\\enable_speedb_features_example4";
#else
std::string kDBPath1 = "/tmp/enable_speedb_features_example1";
std::string kDBPath2 = "/tmp/enable_speedb_features_example2";
std::string kDBPath3 = "/tmp/enable_speedb_features_example3";
std::string kDBPath4 = "/tmp/enable_speedb_features_example4";
#endif
int main() {
DB *db1 = nullptr;
DB *db2 = nullptr;
DB *db3 = nullptr;
DB *db4 = nullptr;
Options op1;
Options op2;
Options op3;
Options op4;
size_t total_ram_size_bytes = 512 * 1024 * 1024;
size_t delayed_write_rate = 256 * 1024 * 1024;
size_t total_threads = 8;
// define SharedOptions object for each databases group
SharedOptions so1(total_ram_size_bytes, total_threads, delayed_write_rate);
// customize each options file except SpeedbSharedOptiopns members
// as listed in the definition of SpeedbSharedOptiopns in options.h
op1.create_if_missing = true;
op1.compression = rocksdb::kNoCompression;
//...
op1.EnableSpeedbFeatures(so1);
op2.create_if_missing = true;
op2.compression = rocksdb::kZlibCompression;
//...
op2.EnableSpeedbFeatures(so1);
// open the databases
Status s = DB::Open(op1, kDBPath1, &db1);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
s = DB::Open(op2, kDBPath2, &db2);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
std::cout << "DBs group 1 was created" << std::endl;
// do the same for any group of databases
total_ram_size_bytes = 1024 * 1024 * 1024;
delayed_write_rate = 128 * 1024 * 1024;
total_threads = 4;
SharedOptions so2(total_ram_size_bytes, total_threads, delayed_write_rate);
// again customize each options object except SharedOptiopns members
op3.create_if_missing = true;
op3.compaction_style = rocksdb::kCompactionStyleUniversal;
//...
op3.EnableSpeedbFeatures(so2);
op4.create_if_missing = true;
op4.compaction_style = rocksdb::kCompactionStyleLevel;
//...
op4.EnableSpeedbFeatures(so2);
// open the databases
s = DB::Open(op3, kDBPath3, &db3);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
s = DB::Open(op4, kDBPath4, &db4);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
std::cout << "DBs group 2 was created" << std::endl;
// creation of column family
rocksdb::ColumnFamilyOptions cfo3(op3);
rocksdb::ColumnFamilyHandle *cf;
// coustomize it except SpeedbSharedOptiopns members
// call EnableSpeedbFeaturesCF and supply for it the same SharedOptions
// object as the DB, so2 this time.
cfo3.EnableSpeedbFeaturesCF(so2);
// create the cf
s = db3->CreateColumnFamily(cfo3, "new_cf", &cf);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
std::cout << "new_cf was created in db3" << std::endl;
s = db3->DropColumnFamily(cf);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
db3->DestroyColumnFamilyHandle(cf);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return 1;
}
std::cout << "new_cf was destroyed" << std::endl;
s = db1->Close();
assert(s.ok());
s = db2->Close();
assert(s.ok());
s = db3->Close();
assert(s.ok());
s = db4->Close();
assert(s.ok());
delete db1;
delete db2;
delete db3;
delete db4;
return 0;
}
