in memory issue and does block cache data has lifetime ?
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
Hi Guys, Is there any ttl available for block cache in rocksdb like it has in db (sst files) or when the ttl expired entries erased from the db does it also erased from block or it persists there until lru remove the least recently used ?
And also i tried with block cache size default 8mb and 100mb. There is no difference in the in memory accessed for my program both takes nearly 20mb of in memory then what is the difference in changing this. Why i cant see my changes.
this is my log file for block cache 100 mb:
2023/12/26-16:49:11.586989 d50 RocksDB version: 4.11.2
2023/12/26-16:49:11.587492 d50 Git sha rocksdb_build_git_sha:98084910a7355eb3447afb8f48549cd47acf55cf
2023/12/26-16:49:11.587492 d50 Compile date Dec 19 2023
2023/12/26-16:49:11.587492 d50 DB SUMMARY
2023/12/26-16:49:11.587492 d50 SST files in AgentDB dir, Total Num: 0, files:
2023/12/26-16:49:11.587492 d50 Write Ahead Log file in AgentDB:
2023/12/26-16:49:11.587492 d50 Options.error_if_exists: 0
2023/12/26-16:49:11.587492 d50 Options.create_if_missing: 1
2023/12/26-16:49:11.587492 d50 Options.paranoid_checks: 1
2023/12/26-16:49:11.587492 d50 Options.env: 0105D738
2023/12/26-16:49:11.601389 d50 Options.info_log: 180E8500
2023/12/26-16:49:11.601389 d50 Options.max_open_files: -1
2023/12/26-16:49:11.601389 d50 Options.max_file_opening_threads: 16
2023/12/26-16:49:11.601389 d50 Options.max_total_wal_size: 0
2023/12/26-16:49:11.601389 d50 Options.disableDataSync: 0
2023/12/26-16:49:11.601389 d50 Options.use_fsync: 0
2023/12/26-16:49:11.601389 d50 Options.max_log_file_size: 0
2023/12/26-16:49:11.601389 d50 Options.max_manifest_file_size: 18446744073709551615
2023/12/26-16:49:11.601389 d50 Options.log_file_time_to_roll: 0
2023/12/26-16:49:11.601389 d50 Options.keep_log_file_num: 5
2023/12/26-16:49:11.601389 d50 Options.recycle_log_file_num: 0
2023/12/26-16:49:11.601389 d50 Options.allow_os_buffer: 1
2023/12/26-16:49:11.601389 d50 Options.allow_mmap_reads: 0
2023/12/26-16:49:11.601389 d50 Options.allow_fallocate: 1
2023/12/26-16:49:11.601389 d50 Options.allow_mmap_writes: 0
2023/12/26-16:49:11.601389 d50 Options.create_missing_column_families: 0
2023/12/26-16:49:11.601389 d50 Options.db_log_dir:
2023/12/26-16:49:11.601389 d50 Options.wal_dir: C:/Program Files (x86)/Log360Cloud_Agent/bin/AgentDB
2023/12/26-16:49:11.601389 d50 Options.table_cache_numshardbits: 6
2023/12/26-16:49:11.601389 d50 Options.delete_obsolete_files_period_micros: 21600000000
2023/12/26-16:49:11.601389 d50 Options.base_background_compactions: 1
2023/12/26-16:49:11.601389 d50 Options.max_background_compactions: 1
2023/12/26-16:49:11.601389 d50 Options.max_subcompactions: 1
2023/12/26-16:49:11.601389 d50 Options.max_background_flushes: 1
2023/12/26-16:49:11.601389 d50 Options.WAL_ttl_seconds: 0
2023/12/26-16:49:11.601389 d50 Options.WAL_size_limit_MB: 0
2023/12/26-16:49:11.601869 d50 Options.manifest_preallocation_size: 4194304
2023/12/26-16:49:11.601869 d50 Options.allow_os_buffer: 1
2023/12/26-16:49:11.601869 d50 Options.allow_mmap_reads: 0
2023/12/26-16:49:11.601869 d50 Options.allow_mmap_writes: 0
2023/12/26-16:49:11.601869 d50 Options.is_fd_close_on_exec: 1
2023/12/26-16:49:11.601869 d50 Options.stats_dump_period_sec: 600
2023/12/26-16:49:11.601869 d50 Options.advise_random_on_open: 1
2023/12/26-16:49:11.601869 d50 Options.db_write_buffer_size: 0d
2023/12/26-16:49:11.601869 d50 Options.access_hint_on_compaction_start: NORMAL
2023/12/26-16:49:11.601869 d50 Options.new_table_reader_for_compaction_inputs: 0
2023/12/26-16:49:11.601869 d50 Options.compaction_readahead_size: 0d
2023/12/26-16:49:11.601869 d50 Options.random_access_max_buffer_size: 1048576d
2023/12/26-16:49:11.601869 d50 Options.writable_file_max_buffer_size: 1048576d
2023/12/26-16:49:11.601869 d50 Options.use_adaptive_mutex: 0
2023/12/26-16:49:11.601869 d50 Options.rate_limiter: 00000000
2023/12/26-16:49:11.601869 d50 Options.sst_file_manager.rate_bytes_per_sec: 0
2023/12/26-16:49:11.601869 d50 Options.bytes_per_sync: 0
2023/12/26-16:49:11.601869 d50 Options.wal_bytes_per_sync: 0
2023/12/26-16:49:11.601869 d50 Options.wal_recovery_mode: 2
2023/12/26-16:49:11.601869 d50 Options.enable_thread_tracking: 0
2023/12/26-16:49:11.601869 d50 Options.allow_concurrent_memtable_write: 0
2023/12/26-16:49:11.601869 d50 Options.enable_write_thread_adaptive_yield: 0
2023/12/26-16:49:11.601869 d50 Options.write_thread_max_yield_usec: 100
2023/12/26-16:49:11.601869 d50 Options.write_thread_slow_yield_usec: 3
2023/12/26-16:49:11.601869 d50 Options.row_cache: None
2023/12/26-16:49:11.601869 d50 Options.wal_filter: None
2023/12/26-16:49:11.601869 d50 Options.avoid_flush_during_recovery: 0
2023/12/26-16:49:11.601869 d50 Compression algorithms supported:
2023/12/26-16:49:11.601869 d50 Snappy supported: 0
2023/12/26-16:49:11.601869 d50 Zlib supported: 0
2023/12/26-16:49:11.601869 d50 Bzip supported: 0
2023/12/26-16:49:11.601869 d50 LZ4 supported: 0
2023/12/26-16:49:11.601869 d50 Fast CRC32 supported: 0
2023/12/26-16:49:11.602365 d50 Creating manifest 1
2023/12/26-16:49:11.666085 d50 Recovering from manifest file: MANIFEST-000001
2023/12/26-16:49:11.667073 d50 --------------- Options for column family [default]:
2023/12/26-16:49:11.667073 d50 Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
2023/12/26-16:49:11.667073 d50 Options.merge_operator: None
2023/12/26-16:49:11.667073 d50 Options.compaction_filter: None
2023/12/26-16:49:11.667073 d50 Options.compaction_filter_factory: TtlCompactionFilterFactory
2023/12/26-16:49:11.667073 d50 Options.memtable_factory: SkipListFactory
2023/12/26-16:49:11.667073 d50 Options.table_factory: BlockBasedTable
2023/12/26-16:49:11.667073 d50 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (195FDF88)
cache_index_and_filter_blocks: 0
pin_l0_filter_and_index_blocks_in_cache: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 19B84A10
block_cache_size: 104857600
block_cache_compressed: 00000000
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 2
2023/12/26-16:49:11.667073 d50 Options.write_buffer_size: 67108864
2023/12/26-16:49:11.667073 d50 Options.max_write_buffer_number: 2
2023/12/26-16:49:11.667073 d50 Options.compression: NoCompression
2023/12/26-16:49:11.667073 d50 Options.bottommost_compression: Disabled
2023/12/26-16:49:11.667073 d50 Options.prefix_extractor: nullptr
2023/12/26-16:49:11.667073 d50 Options.num_levels: 7
2023/12/26-16:49:11.667073 d50 Options.min_write_buffer_number_to_merge: 1
2023/12/26-16:49:11.667073 d50 Options.max_write_buffer_number_to_maintain: 0
2023/12/26-16:49:11.667073 d50 Options.compression_opts.window_bits: -14
2023/12/26-16:49:11.667073 d50 Options.compression_opts.level: -1
2023/12/26-16:49:11.667073 d50 Options.compression_opts.strategy: 0
2023/12/26-16:49:11.667073 d50 Options.compression_opts.max_dict_bytes: 0
2023/12/26-16:49:11.667073 d50 Options.level0_file_num_compaction_trigger: 4
2023/12/26-16:49:11.667073 d50 Options.level0_slowdown_writes_trigger: 20
2023/12/26-16:49:11.667073 d50 Options.level0_stop_writes_trigger: 24
2023/12/26-16:49:11.667073 d50 Options.target_file_size_base: 67108864
2023/12/26-16:49:11.667073 d50 Options.target_file_size_multiplier: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_base: 268435456
2023/12/26-16:49:11.667073 d50 Options.level_compaction_dynamic_level_bytes: 0
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier: 10
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2023/12/26-16:49:11.667073 d50 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2023/12/26-16:49:11.667073 d50 Options.max_sequential_skip_in_iterations: 8
2023/12/26-16:49:11.667073 d50 Options.expanded_compaction_factor: 25
2023/12/26-16:49:11.667073 d50 Options.source_compaction_factor: 1
2023/12/26-16:49:11.667073 d50 Options.max_grandparent_overlap_factor: 10
2023/12/26-16:49:11.667073 d50 Options.arena_block_size: 8388608
2023/12/26-16:49:11.667073 d50 Options.soft_pending_compaction_bytes_limit: 68719476736
2023/12/26-16:49:11.667073 d50 Options.hard_pending_compaction_bytes_limit: 274877906944
2023/12/26-16:49:11.667569 d50 Options.rate_limit_delay_max_milliseconds: 1000
2023/12/26-16:49:11.667569 d50 Options.disable_auto_compactions: 0
2023/12/26-16:49:11.667569 d50 Options.verify_checksums_in_compaction: 1
2023/12/26-16:49:11.667569 d50 Options.compaction_style: 0
2023/12/26-16:49:11.667569 d50 Options.compaction_pri: 0
2023/12/26-16:49:11.667569 d50 Options.compaction_options_universal.size_ratio: 1
2023/12/26-16:49:11.667569 d50 Options.compaction_options_universal.min_merge_width: 2
2023/12/26-16:49:11.667569 d50 Options.compaction_options_universal.max_merge_width: 4294967295
2023/12/26-16:49:11.667569 d50 Options.compaction_options_universal.max_size_amplification_percent: 200
2023/12/26-16:49:11.667569 d50 Options.compaction_options_universal.compression_size_percent: -1
2023/12/26-16:49:11.667569 d50 Options.compaction_options_fifo.max_table_files_size: 1073741824
2023/12/26-16:49:11.667569 d50 Options.table_properties_collectors:
2023/12/26-16:49:11.667569 d50 Options.inplace_update_support: 0
2023/12/26-16:49:11.667569 d50 Options.inplace_update_num_locks: 10000
2023/12/26-16:49:11.667569 d50 Options.min_partial_merge_operands: 2
2023/12/26-16:49:11.667569 d50 Options.memtable_prefix_bloom_size_ratio: 0.000000
2023/12/26-16:49:11.667569 d50 Options.memtable_huge_page_size: 0
2023/12/26-16:49:11.667569 d50 Options.bloom_locality: 0
2023/12/26-16:49:11.667569 d50 Options.max_successive_merges: 0
2023/12/26-16:49:11.667569 d50 Options.optimize_filters_for_hits: 0
2023/12/26-16:49:11.667569 d50 Options.paranoid_file_checks: 0
2023/12/26-16:49:11.667569 d50 Options.report_bg_io_stats: 0
2023/12/26-16:49:11.671537 d50 Recovered from manifest file:C:/Program Files (x86)/Log360Cloud_Agent/bin/AgentDB/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0
2023/12/26-16:49:11.671537 d50 Column family [default] (ID 0), log number is 0
2023/12/26-16:49:11.693361 d50 DB pointer 16F4DC20
2023/12/26-16:49:16.002214 3680 [default] Manual compaction starting
2023/12/26-16:51:09.394442 2bcc [default] New memtable created with log file: #6. Immutable memtables: 0.
2023/12/26-16:51:09.394903 33dc [JOB 2] Syncing log #3
2023/12/26-16:51:09.415330 33dc (Original Log Time 2023/12/26-16:51:09.394903) Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
2023/12/26-16:51:09.415330 33dc [default] [JOB 2] Flushing memtable with next log file: 6
2023/12/26-16:51:09.415330 33dc EVENT_LOG_v1 {"time_micros": 1703638269415330, "job": 2, "event": "flush_started", "num_memtables": 1, "num_entries": 9589, "num_deletes": 0, "memory_usage": 2548916}
2023/12/26-16:51:09.415759 33dc [default] [JOB 2] Level-0 flush table #7: started
2023/12/26-16:51:09.480204 33dc EVENT_LOG_v1 {"time_micros": 1703638269480204, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 7, "file_size": 2302622, "table_properties": {"data_size": 2275036, "index_size": 26745, "filter_size": 0, "raw_key_size": 901728, "raw_average_key_size": 94, "raw_value_size": 1561798, "raw_average_value_size": 162, "num_data_blocks": 576, "num_entries": 9589, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2023/12/26-16:51:09.480204 33dc [default] [JOB 2] Level-0 flush table #7: 2302622 bytes OK
2023/12/26-16:51:09.480204 33dc Creating manifest 8
2023/12/26-16:51:09.513933 33dc (Original Log Time 2023/12/26-16:51:09.480204) [default] Level-0 commit table #7 started
2023/12/26-16:51:09.513933 33dc (Original Log Time 2023/12/26-16:51:09.513933) [default] Level-0 commit table #7: memtable #1 done
2023/12/26-16:51:09.513933 33dc (Original Log Time 2023/12/26-16:51:09.513933) EVENT_LOG_v1 {"time_micros": 1703638269513933, "job": 2, "event": "flush_finished", "lsm_state": [1, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0}
2023/12/26-16:51:09.513933 33dc (Original Log Time 2023/12/26-16:51:09.513933) [default] Level summary: base level 1 max bytes base 268435456 files[1 0 0 0 0 0 0] max score 0.25
2023/12/26-16:51:09.513933 33dc [JOB 2] Try to delete WAL files size 2535935, prev total WAL file size 2535935, number of live WAL files 2.
2023/12/26-16:51:09.514429 33dc [DEBUG] [JOB 2] Delete C:/Program Files (x86)/Log360Cloud_Agent/bin/AgentDB/000003.log type=0 #3 -- OK
2023/12/26-16:51:09.514429 33dc [DEBUG] [JOB 2] Delete C:/Program Files (x86)/Log360Cloud_Agent/bin/AgentDB//MANIFEST-000001 type=3 #1 -- OK
**
**
Suggest some options or configuration that can be used to improve the read performance.
I can't find answer to this. Kindly help me with this.
Contributor guide
Assessment
This issue has not been assessed yet.