Global Delayed write

This feature improves write stabilization in multiple databases/CFs environment.

Overview

The usability and management of multiple databases can be significantly improved with the implementation of the Global Delayed Write feature. The Global delayed write feature is designed to address a problem where users running multiple databases on the same disk can experience disk bandwidth starvation, leading to uneven resource distribution. This feature converts the WriteController from a per-database object into a shared object that can record and enforce the live delay requirements of all the databases that are connected to it. In this document, we'll explain the problem, the solution, and how to use the global delayed write feature.

This feature was introduced with Speedb open source version 2.4.0.

Problem description:

When multiple databases are running on the same disk, some databases may experience disk bandwidth starvation. The starvation is caused by the following scenario:

Db1 enters a slowdown/stop condition because the disk can't keep up with the ingest rate.

Other dbs using the same disk are running without delay (since they themselves are not in a delay state) and they are taking up all the disk bandwidth.

In order for Db1 to exit the slowdown/stop condition, it needs to run compactions but the disk bandwidth for these compactions is unavailable due to the other dbs which leaves Db1 in a delayed state indefinitely or until the other dbs release some disk bandwidth.

Solution Description

The global delayed write feature addresses this problem by converting the WriteController from a per-database object into a shared object. This shared object records and enforces the live delay requirements of all the databases that are connected to it. It works by enforcing the most extreme delay (i.e., the minimum rate) on all databases.

In practice, the WriteController is now a shared_ptr option in ImmutableDBOptions. It keeps track of and enforces all the delay requirements of the column families (cfs) in the databases to which those options were passed. The default behavior of having one WriteController per database will remain, but users can also set a WriteController in the options. Note that setting a WriteController in the options is only valid when use_dynamic_delay = true.

How to Use the Global Delayed Write?

To use the global delayed write feature, users should set use_dynamic_delay = true in the options. By default, each database will have one WriteController, but users can set a WriteController in the options if desired..

Example code:

Options options;
options.use_dynamic_delay = true
options.write_controller.reset(new WriteController(
      options.use_dynamic_delay, options.delayed_write_rate));
  DB* db1 = nullptr;
  DB* db2 = nullptr;
  Status s = DB::Open(options, "db_path1", &db1);
  Status s2 = DB::Open(options, "db_path2", &db2);

Test Results

The new global delayed write was tested with db_bench, using the command below.

db_bench --compression_type=None -db=/data/ -num=200000000 -value_size=1000 -key_size=16 --delayed_write_rate=536870912 -report_interval_seconds=1 -max_write_buffer_number=4 -num_column_families=3 -histogram -max_background_compactions=8 -cache_size=8388608 -max_background_flushes=4 -bloom_bits=10 -benchmark_read_rate_limit=0 -benchmark_write_rate_limit=0 -report_file=fillrandom.csv --disable_wal=true --benchmarks=fillrandom,levelstats --column_family_distribution=50,24,26 --level0_slowdown_writes_trigger=8 -write_buffer_size=268435456

The graph shows that with the new global delayed write enabled the results are more stable by 30%.

Without the global delayed write, there are many stalls observed during the test while with the feature enabled there were no stalls.