基于Facebook开源的 rocksdb 分布式数据库模块 [迭代]

Buei

由于之前帖子内容太多,这里重新发布一篇,之前帖子地址--> 地址


什么是rocksdb?

• RocksDB是一个 持久的键值存储库 ，它是用C++编写的，适合在快速、低延迟的存储设备上存储数据。它是由Facebook数据库工程团队开发和维护。
  

LevelDB 有什么区别?

• RocksDB和LevelDB都是基于LSM-Tree的嵌入式键值存储库，但RocksDB是在LevelDB的基础上进行了优化和增强
  
  
  

• RocksDB可以支持 多线程 合并文件，而LevelDB是 单线程
• RocksDB可以根据需要开辟 多个Memtable ，而LevelDB只有 一个Memtable
• RocksDB可支持多种压缩算法，而LevelDB只支持snappy
  
• 单线程模式下 LevelDB 可能稍微快一点,而在多线程下 RocksDB 就会发挥出它的优势了
  
  

rocksdb的优点

• 高性能：RocksDB 使用了很多优化技术，如多线程、高效的数据结构等，因此具有非常高的读写性能。
• 可扩展性：RocksDB 可以处理大规模的数据，并支持自动分片和负载均衡等功能，因此可以很好地应对高并发访问。
• 可靠性：RocksDB 支持 ACID 事务，保证数据的一致性和可靠性。
• 灵活性：RocksDB 支持多种数据格式，包括内存映射文件、纯内存等，让用户可以灵活选择适合自己的存储方式。
• RocksDB在存储数据时是按照键的排序方式进行存储的，它并没有明确的容量限制，可以存储非常大的数据 [理论上无限制容量]。而类似MMKV框架限制容量的方式是使用了一种固定大小的映射文件，即在创建MMKV实例时就已经确定了最大容量，超过容量时就不能再写入数据[大概在 4GB 左右]
  

• 另一款开源 leveldb的帖子
  

下图为 rocksdb和leveldb  单/多线程 写入对比
理论上在单线程下 RocksDB应该比levldb稍微略慢一点

图中可看到300W多线程写入RocksDB一瞬间完成

解压密码:neko

另外发布了一份火山版rocksdb

更新日志  2023/09/21 18:00  - V1.5  本次更新内容有点多

• 本次使用了C++ MT 模式编译
• 增加 Transaction事务类   (此事务支持在事务内创建迭代器,支持事务读取回滚操作)
• 增加 OptimisticTransactionDB_类
• 增加 TransactionDB类 (TransactionDB(PessimisticTransactionDB)和OptimisticTransactionDB，分别对应并发控制中的悲观锁和乐观锁)
• 增加 Merge 合并功能   (使用此命令前务必在 rocksdb_启动参数 的 合并模式 选择合并模式)
• 修复 压缩模式 无效
  

• 修复 关闭写前日志 无效    (如果想完全启用内存模式,请将缓存调大关闭写前日志开启 即可)
  

• OptimisticTransactionDB，TransactionDB  对应的事务 [Transaction事务类]
• ...
  

• 以下为Facebook官方更新
• -Please note 8.5.1 includes a fix for a persisted database corruption in an unlikely edge case. Upgrading to a version including this fix, like this one, is highly recommended!
  

• Fixed a race condition in GenericRateLimiter that could cause it to stop granting requests
• Fix a bug where iterator may return incorrect result for DeleteRange() users if there was an error reading from a file.
• Fix a bug where if there is an error reading from offset 0 of a file from L1+ and that the file is not the first file in the sorted run, data can be lost in compaction and read/scan can return incorrect results.
  

• https://wwqr.lanzouw.com/b09ivrlpc
• 密码:7l4b
  

Buei

问题反馈/交流Q群

Q群

Buei

更新日志  2023/11/28 15:00  - V1.6 [rocksdb内核版本:8.8.1]

增加 设置  预读取缓存大小 [默认未开启
增加 设置  可见性等级  
增加 设置  日志备份次数 [LOG.old 开头的文件每次启动数据库时备份次数]
增加 自定义日志目录
修复 当value为空时写入失败
...



以下为Facebook官方更新日志

8.8.1 (2023-11-17)Bug fixes

• Make the cache memory reservation accounting in Tiered cache (primary and compressed secondary cache) more accurate to avoid over/under charging the secondary cache.
• Allow increasing the compressed_secondary_ratio in the Tiered cache after setting it to 0 to disable.
  

8.8.0 (2023-10-23)New Features

• Introduce AttributeGroup by adding the first AttributeGroup support API, MultiGetEntity(). Through the use of Column Families, AttributeGroup enables users to logically group wide-column entities. More APIs to support AttributeGroup will come soon, including GetEntity, PutEntity, and others.
• Added new tickers rocksdb.fifo.{max.size|ttl}.compactions to count FIFO compactions that drop files for different reasons
• Add an experimental offpeak duration awareness by setting DBOptions::daily_offpeak_time_utc in "HH:mm-HH:mm" format. This information will be used for resource optimization in the future
• Users can now change the max bytes granted in a single refill period (i.e, burst) during runtime by SetSingleBurstBytes() for RocksDB rate limiter
  

Public API Changes

• The default value of DBOptions::fail_if_options_file_error changed from false to true. Operations that set in-memory options (e.g., DB::Open*(), DB::SetOptions(), DB::CreateColumnFamily*(), and DB::DropColumnFamily()) but fail to persist the change will now return a non-OK Status by default.
• Add new Cache APIs GetSecondaryCacheCapacity() and GetSecondaryCachePinnedUsage() to return the configured capacity, and cache reservation charged to the secondary cache.
  

Behavior Changes

• For non direct IO, eliminate the file system prefetching attempt for compaction read when Options::compaction_readahead_size is 0
• During a write stop, writes now block on in-progress recovery attempts
• Deleting stale files upon recovery are delegated to SstFileManger if available so they can be rate limited.
  

Bug Fixes

• Fix a bug in auto_readahead_size where first_internal_key of index blocks wasn't copied properly resulting in corruption error when first_internal_key was used for comparison.
• Fixed a bug where compaction read under non direct IO still falls back to RocksDB internal prefetching after file system's prefetching returns non-OK status other than Status::NotSupported()
• Add bounds check in WBWIIteratorImpl and make BaseDeltaIterator, WriteUnpreparedTxn and WritePreparedTxn respect the upper bound and lower bound in ReadOption. See 11680.
• Fixed the handling of wide-column base values in the max_successive_merges logic.
• Fixed a rare race bug involving a concurrent combination of Create/DropColumnFamily and/or Set(DB)Options that could lead to inconsistency between (a) the DB's reported options state, (b) the DB options in effect, and (c) the latest persisted OPTIONS file.
• Fixed a possible underflow when computing the compressed secondary cache share of memory reservations while updating the compressed secondary to total block cache ratio.
  

Performance Improvements

• Improved the I/O efficiency of DB::Open a new DB with create_missing_column_families=true and many column families.
  

• https://wwqr.lanzouw.com/b09ivrlpc
• 密码:7l4b
  

金叶

这个666666666666666666666

dikexu

        感谢分享，很给力！~

shaokui123

感谢分享，很给力！~

xiao黄鸭

支持支持支持支持支持支持支持支持支持支持支持支持支持支持支持支持支持

一直在沉默

666666666666666666666666666666666

给自己留一片天空

感谢分享，很给力！~

给自己留一片天空

感谢分享，很给力！~

给自己留一片天空

感谢分享，很给力！~

shaoxing0117

支持开源~！感谢分享

胖虎O

感谢分享了