dotnetcore / dotnetcore/FreeSql
高性能 BulkCopy流式写入需求
- Dominant language
- C#
- Stars
- 4.4k
- Forks
- 910
- PR merge metrics
- No merged PRs in 30d
Description
#### Feature 特性
高性能 BulkCopy方法, 如ExecuteMySqlBulkCopy/ExecutePgCopy之类, 我简略的读了下源码是先将数据转成DataTable然后再执行BulkCopy, 但实际应用中可能存在上万甚至几十万数据的批次写入需求, 如果转成DataTable, 内存溢出的风险是很大的. 能否支持流式写入? 伪代码如下:
```c#
// 流式读取返回数据
private IEnumerable Read(){
// 初始化连接 和reader
await using var reader = await pool.DbConn.ExecuteReaderAsync(sql);
while(reader.Read()){
// 转换实体 yield 流式返回
yield return read.Read2Entity()
}
}
// 流式写入
private async Task Write(IEnumerable items){
// 流式写入 ,ExecutePgCopyAsync会把items的数据一条一条的写,而不是内存中转成DataTable
await fSql.Insert().ExecutePgCopyAsync(items);
}
```
#### 简要描述原因
流式BulkCopy
#### 使用场景
海量数据的写入或者同步
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the existing ExecuteMySqlBulkCopy and ExecutePgCopy implementations and the DataTable conversion described in the issue. Compare their current input and write paths with the proposed ExecutePgCopyAsync(IEnumerable) usage. Done means large batches can be written incrementally without first materializing all rows in a DataTable, with coverage for the affected bulk-copy paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, mysql, postgresql
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100