Why I cant write sst on hdfs
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev
### Expected behavior
sst is on hdfs,Now I can use my own code to run rocksDB on HDFS, but the write speed is slow,when it begins,its speed is about 11MB/s,then it will fall to 600KB。Have you ever encountered such a situation 。Is it my memory? My memory setting is 8g。
### Actual behavior
sst still on my local Filesystem
### Steps to reproduce the behavior
This is my code,I can use db_bench to write data on HDFS, but when I use my own code, I cant do it
########################################################
```
#include "cassert"
#include "strings"
#include "iostream"
#include "chrono"
#include "rocksdb/db.h"
#include "rocksdb/table.h"
#include "rocksdb/options.h"
#include "rocksdb/filter_policy.h"
#include "rocksdb/cache.h"
#include "rocksdb/iostats_context.h"
#include "rocksdb/perf_context.h"
#include "rocksdb/statistics.h"
//#include "hdfs/env_hdfs.h"
#include "rocksdb/env.h"
#define TEST_FREQUENCY (10000)
using namespace std;
const int MAX = 10;
char* randomstr(int n )
{
static char buf[1024];
for (int i = 0; i < n; ++i) {
buf[i] = 'A' + rand() % 26;
}
buf[n]='\0';
return buf;
}
std::string gen_random(int n){
char alphabet[MAX] = {
'0','1','2','3','4','5','6','7','8','9'
};
string res = "";
for(int i =0;i<8;i++) res=res+'0';
for (int i = 8; i < n; i++)
res = res + alphabet[rand() % MAX];
return res;
}
int main()
{
rocksdb::SetPerfLevel(rocksdb::PerfLevel::kEnableTimeExceptForMutex);
rocksdb::get_perf_context()->Reset();
rocksdb::get_iostats_context()->Reset();
rocksdb::DB* db;
rocksdb::Options options;
options.create_if_missing = true;
options.write_buffer_size = 4145152;
options.max_bytes_for_level_base=41451520;
options.target_file_size_multiplier=1;
options.target_file_size_base=4145152*25;
options.max_bytes_for_level_multiplier=5;
// open
rocksdb::Env* hdfs;
options.env=hdfs;
rocksdb::NewHdfsEnv(&hdfs,"hdfs://master:9000");
rocksdb::Status status = rocksdb::DB::Open(options,"/opt/db", &db);
assert(status.ok());
srand(time(NULL));
options.statistics = rocksdb::CreateDBStatistics();
std::string k[TEST_FREQUENCY];
std::string v[TEST_FREQUENCY];
char key[17];
char value[1009];
// ADD
{
auto start = std::chrono::system_clock::now();
for(int i=0;i<10;++i){
for (int j = 0; j < 10000;++j){
//key = (gen_random(16));
//value = (gen_random(1008));
strcpy (key, gen_random(16).c_str());
strcpy (value, gen_random(1008).c_str());
status = db->Put(rocksdb::WriteOptions(), key, value);
// assert(status.ok());
}
memset(k,'\0',sizeof(k));
memset(v,'\0',sizeof(v));
}
std::cout << rocksdb::get_perf_context()->ToString() << std::endl; //获取所有的perf 状态
std::cout << rocksdb::get_iostats_context()->ToString() << std::endl; //获取所有的iostats状态
auto end = std::chrono::system_clock::now();
auto duration = std::chrono::duration_cast(end - start);
std::cout << TEST_FREQUENCY <<"times: "
<< double(duration.count()) * std::chrono::microseconds::period::num / std::chrono::microseconds::period::den
<< "s" << std::endl;
}
assert(it->status().ok()); // Check for any errors found during the scan
delete it;*/
std::cout << "get_from_memtable_time: "
<< rocksdb::get_perf_context()->get_from_memtable_time
<< "write_nanos: "
<< rocksdb::get_iostats_context()->write_nanos
<< std::endl;
rocksdb::SetPerfLevel(rocksdb::PerfLevel::kDisable); // close profling
delete db;
return 0;
}
```
########################################################
Contributor guide
Research direction
Start by tracing the HDFS environment setup around rocksdb::NewHdfsEnv, options.env, and DB::Open in the provided program. Compare that setup with the db_bench HDFS invocation and inspect where the generated SST files are written. Done means identifying why the custom program uses the local filesystem and explaining the reported write-speed change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100