OOM in count queries

[wey-gu]

I quoted DM session with @wdroste

I'm running a cluster with 5 nodes which are 4cpu 8gb ram and 250GB of nvme
create space vid int64 w/ replication 3
5 nodes are c6id.xlarge and pushing data w/ t3.small
8 threads writing random data w/ indexes
and trying to write 10m vertexes
i'm kind of getting through the write but it seem slike i can break the cluster pretty easily
i change storage to the following

# Network IO threads number
--num_io_threads=2
# Worker threads number to handle request
--num_worker_threads=4
# Maximum subtasks to run admin jobs concurrently
--max_concurrent_subtasks=1
--rocksdb_rate_limit=20

and the graphd dies w/ the lookup query

LOOKUP ON tagFoo YIELD id(vertex) | YIELD COUNT(*);

This was just a bit of a test, we do a lot of count queries for reports, so i was trying to see where the system would go. We do lots of traversals and well which i assume will go faster because there's a starting node..

the memory profile was great <1GB of ram but rocksdb killed storaged on compaction and i don't know if that's because of the memory? There was 7GB in buffer cache..

[wey-gu]

and the graphd dies w/ the lookup query

LOOKUP ON tagFoo YIELD id(vertex) | YIELD COUNT(*);

This is due to the count is not yet pushdown to the storage side, which will end up lookup to do a full index scan(as no filter condition here) and then count, this is not in hi-priority due to in full scan case we have other ways to get count, and in other cases, the scanned data will be filtered.

In this case(whole tag count), it’s recommended to use this async approach instead: SHOW STATS/SUBMIT JOB STATS

[wey-gu]

the memory profile was great <1GB of ram but rocksdb killed storaged on compaction and i don't know if that's because of the memory? There was 7GB in buffer cache.

@wdroste regarding this, do you meant the storaged got killed during LOOKUP, too, or during a compaction only?

[wdroste]

I was doing the LOOKUP while loading the data to see where things might break, but it was definitely during a compaction that it core dumped.

[critical27]

You probably need to lower the configurations related to rocksdb, such as size of memtable, sst, thread num of background job.

[wey-gu]

such as size of memtable, sst, thread num of background job

rocksdb related:

• memtable: ColumnFamilyOptions::write_buffer_size size for each memtable(default 64MiB), AdvancedColumnFamilyOptions::max_write_buffer_number max number of memtable, DBOptions::db_write_buffer_size all memtable size (default unlimited)
• sst: target_file_size_base size of sst file for level 1, max_bytes_for_level_base all level sst size
• num of background job: max_background_jobs

@wdroste also I put nebula-storaged configuration docs here and quote related chapter:

https://docs.nebula-graph.io/3.2.1/5.configurations-and-logs/1.configurations/4.storage-config/#rocksdb_options

RocksDB options¶

Name	Predefined value	Description
rocksdb_db_options	{}	Specifies the RocksDB database options.
rocksdb_column_family_options	{"write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}	Specifies the RocksDB column family options.
rocksdb_block_based_table_options	{"block_size":"8192"}	Specifies the RocksDB block based table options.

The format of the RocksDB option is {"<option_name>":"<option_value>"}. Multiple options are separated with commas.

Supported options of rocksdb_db_options and rocksdb_column_family_options are listed as follows.

• rocksdb_db_options

  max_total_wal_size
  delete_obsolete_files_period_micros
  max_background_jobs
  stats_dump_period_sec
  compaction_readahead_size
  writable_file_max_buffer_size
  bytes_per_sync
  wal_bytes_per_sync
  delayed_write_rate
  avoid_flush_during_shutdown
  max_open_files
  stats_persist_period_sec
  stats_history_buffer_size
  strict_bytes_per_sync
  enable_rocksdb_prefix_filtering
  enable_rocksdb_whole_key_filtering
  rocksdb_filtering_prefix_length
  num_compaction_threads
  rate_limit
  

• rocksdb_column_family_options

  write_buffer_size
  max_write_buffer_number
  level0_file_num_compaction_trigger
  level0_slowdown_writes_trigger
  level0_stop_writes_trigger
  target_file_size_base
  target_file_size_multiplier
  max_bytes_for_level_base
  max_bytes_for_level_multiplier
  disable_auto_compactions 
  

For more information, see RocksDB official documentation.

[wey-gu]

Also two URLs worthy to be shared:

• https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
• https://github.com/facebook/rocksdb/wiki/RocksDB-Troubleshooting-Guide

[wdroste]

For the default configurations is an template system size the team uses? Any specs along w/ the size of the database.

[wey-gu]

Thanks @wdroste , this is indeed missing.
@whitewum we could prepare more of this kind of reference in the Documentation/ Prod conf template?

[whitewum]

nothing can do with the configuration in this situation. Lookup just oom for a large dataset.

[wdroste]

Ok I up'd the specs quite a bit, 5 r6id.8xlarge

• 16 CPU
• 128 GB RAM
• 950 GB NVMe

Pushed 10m vertices with index on 5/10 properties. All the nodes are up and functioning and no crashes. The only thing is this LOOKUP query does not complete.

(root@nebula) [test]> LOOKUP ON Vulnerability YIELD id(vertex) | YIELD count(*);
[ERROR (-1005)]: Storage Error: part: 18, error: E_RPC_FAILURE(-3).

Thu, 27 Oct 2022 21:22:57 UTC

[wey-gu]

@critical27 correct me if wrong.

This RPC failure is not about RPC time out(where we could enlarge storage_client_timeout_ms to mitigate long queries triggered graphd --> storaged RPC time out), right? What could be the cause?

@wdroste would you mind sharing storaged logs at this time?