| Speedb official website | /files/JyL1TYTxTjv0aSiA9yI1 | https://www.speedb.io | ||
| Discord server | News, live discussions and community support | /files/cPqiFdp9wtftadHp3GBQ | https://discord.gg/5fVUUtM2cG | |
| Reporting issues | You can report a bug in Speedb by opening a Github issue in the Speedb repository on GitHub | /files/Bbuw8eF6ooCY0WXgvqhx | https://github.com/speedb-io/speedb/issues |
Writing to a specific address space, parallel writes
| | Checking the triggers |Part of the DB mutex on every write
|Background, write lock only when needed
| |Switch memtable/switch WAL/Trim WAL
Part of the DB mutex on every write
|Background, without any locks
| | Writes rollback |Donât needed (since writing first to the WAL)
|Needed when write to the memtable failed
| To summaries, the new write flow enables parallel writes by the following changes: 1. Speedb changed the DB mutex to read/write-lock 2. Speedb write flow allows parallel writes to the memtable and the WAL. Ack is sent only after the data is written to both, but the writes are no longer serial. 3. The wall in the previous write flow is using append, meaning only single write is allowed at a time. Speedb's new write flow changed the way data is written to the WAL and now writes to a specific address in the file, a fact that enables the parallel writes to the WAL. ### Transactions RocksDB supports both optimistic and pessimistic concurrency controls. The pessimistic transactions make use of locks to provide isolation between the transactions. The default write policy in pessimistic transactions is WriteCommitted, which means that the data is written to the DB, i.e., the memtable, only after the transaction is committed. This policy simplified the implementation but came with some limitations in throughput, transaction size, and variety in supported isolation levels. This approach actually doesn't require any special treatment in the new spdb write flow since the txn (transaction) batches are kept in memory and are written to the WAL/memtable only as part of commit. What is interesting for us is the WritePrepared and WriteUnprepared In this propose we actually have for each txn: **Prepare** **Txn Writes** **Commit** In the prepare phase we write only a wal write that gets a seq as all batch writes does but this id is the identify of the txn Any txn writes will be attached to this txn id. The write flow will consist of a txn data structure that keeps track of the uncomplete txn. The Txn write batches will be inserted to the memtable as the regular batch writes (can be with initial txn id to all writes or increments one - if the second option this seq should be kept in the incomplete txn data) And will be updated in the wal. The version seq will be update as well. Any read from the txn will try find first in the txn data and then from the memtable and levels . New regular writes will proceed with no effect Read from the db and not txn read will skip incomplete txn writes, by checking if the memtable read result is part of the txn WF data. When txn is complete a commit write executed and the relevant txn id is being removed from the txn WF data and the txn data is part of the DB and is accessible to all. This approach is simplify the rollback since any read will ignore the txn data, Memtable switch will be made ONLY when no open txn. As part of flush, if we have failed txn , the data is ignored and not part of the SST.
100% Overwrite workload
50% Random Read performance results
100% random seek workload
| Paired Bloom | Local Bloom | |
|---|---|---|
| Bits per key | 40 | 29 |
| Total memory consumption | 6200MB | 7700MB |
100% random read workload
| Paired Bloom | Local Bloom | |
|---|---|---|
| Bits per key | 20 | 20 |
| Total memory consumption | 4900B | 4800MB |
| Field Name | Meaning | Comments | |
| 1 | Title (âParsing of:âŠâ) | Parsed Log full path | |
| 2 | Name | Parsed Log full path | |
| 3 | Start Time | Time of first log entry | |
| 4 | End Time | Time of last log entry | |
| 5 | Log Time Span | The time difference between the Start Time and the End Time | The value is expressed in number of days, hours, minutes, and seconds. |
| 6 | Creator | The creator of the library that generated the parsed log | Currently may be either Speedb or RocksDB |
| 7 | Version | The libraryâs version [Git Hash] | |
| 8 | DB Size (*) | The total size of all of the SST files in the database | The point in time in which the value was calculated is given below the per-cf table: â(*) Data is calculated âŠ.â |
| 9 | Num Keys Written | The total number of keys written to the DB | The value of the otherwise, extracted from cumulative writes information (DB Stats) If none of the above is available: âData Unavailableâ |
| 10 | Avg. Written Key Size | The average size of a written key | Calculated from information in If there are no such events, âData Unavailableâ |
| 11 | Avg. Written Value Size | The average size of a written value | Calculated from information in If there are no such events, âData Unavailableâ |
| 12 | Error Messages | The messages in severity âERRORâ as they appear in the log | âNo Error Messagesâ if there are no errors in the log |
| 13 | Fatal Messages | The messages in severity âFATALâ as they appear in the log | âNo Fatal Messagesâ if there are no errors in the log |
| 14 | Ingest (*) | Total ingest | Calculated from cumulative writes information. The point in time in which the value was calculated is given below the per-cf table: â(*) Data is calculated âŠ.â If data is not available: âNo Ingest Info Availableâ |
| 15 | Ingest Rate | Ingest Rate in Mega-Bytes per second | Calculated from cumulative writes information. The point in time in which the value was calculated is given below the per-cf table: â(*) Data is calculated âŠ.â If data is not available: âNo Ingest Info Availableâ |
| 16 | Statistics | Whether statistics are available or not | |
| 17 | Writes | The total number of write operations <Percentage> (<Count> / <Total Ops>) Total Ops: Total number of operations (write + read + seek) Count: Total number of writes Percentage: Percentage of writes out of Total Ops | Calculated from the If statistics are not available: âData Unavailable (No Statistics)â |
| 18 | Reads | The total number of read operations (Same as for the Writes field, but for reads) | Same as for Writes |
| 19 | Seeks | The total number of seek operations (Same as for the Writes field, but for seeks) | Same as for Writes |
| 20 | Deletes | <Percentage> (<Num Deletes> / <Num Entries>) Num Entries: The total number of flushed entries Num Deletes: The total number of deletes in Percentage: Percentage of deletes out of Num Entries | Information is gathered from Flush events in the log. If no such events exist: âData Unavailable (No Flush Started Events)â |
| 21 | Num CF-s | Number of cf-s or âCan't be accurately determinedâ | The log parser may not be able to know for sure the number of cf-s in the DB using the parsed log file. In that case it will show the text âCan't be accurately determinedâ. Please see the âAbility to determine the number of cf-sâ for more information. |
| 22 | |||
| 23 | Column Families Information Table | ||
| 24 | Column Family | The name of the CF | Will display any CF for which there was information in the log (TBD - See XXXX) |
| 25 | Size | The total size of all of the SST files of the CF | If no data available to calculate the size: âData Unavailableâ |
| 26 | Avg. Key Size | Average value of a key in newly created SST-s of the CF | Gathered from table_file_creation events in the log |
| 27 | Avg. Value Size | Average value of a value in newly created SST-s of the CF | Gathered from table_file_creation events in the log |
| 28 | Compaction Style | The compaction style used in this CF | Taken from the options for the CF. The values are as they appear in the log. If not known: âUNKNOWNâ |
| 29 | Compression | The compression type used in this CF | Taken from the options for the CF. The values are as they appear in the log. If not known: âUNKNOWNâ |
| 30 | Filter-Policy (<BPK>) | The filter policy used in this CF (If any) <BPK> - The average BPK for the filter. | Taken from the options for the CF. The values are as they appear in the log. If not known: âUNKNOWNâ |
Sub-object per flush reason in the flushes for this CF
All of the fields below are in the context of a CF and flush reason.
| Flush Events | | Sizes Histogram |A histogram of the number of flushes per total data size range (bucket) in a flush.
The numbers are the number of flushes of all flushes for this cf and flush reason.
For example, in the snapshot above, there were 31 flushes whose total data size was more than 31 MB).
| | | Num Flushes | The number of flushes | | | Min Duration | The minimum duration that a single flush took to complete | | | Max Duration | The maximum duration that a single flush took to complete | | | Min Num Memtables | The minimum number of memtables that were part of a single flush. | | | Max Num Memtables | The maximum number of memtables that were part of a single flush. | | | Min Total Data Size | The minimum total data size in a single flush. | | | Max Total Data Size | The maximum total data size in a single flush. | | #### Compactions ObjectSub-object per flush reason in the flushes for this CF
All of the fields below are in the context of a CF and flush reason.
| | Num Compactions | The number of compactions | | Min Compactions BW | Minimum write rate of a compaction | | Max Compactions BW | Maximum write rate of a compaction | | Comp | The elapsed time of all compactions | | Comp Merge CPU | The elapsed CPU time of all compactions | | Per Level Write-Amp | Write amplication per level, and their total (SUM) | #### Reads Object This object contains information about read operations performed by the user (Get and Multi-Get) and associated aspects.histograms\_human\_readable.csv /
histogram\_tools.csv