dotnetcore / dotnetcore/FreeSql

高性能 BulkCopy流式写入需求

Closed
#2,275 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
4.4k
Forks
910
PR merge metrics
No merged PRs in 30d

Description

#### Feature 特性

高性能 BulkCopy方法, 如ExecuteMySqlBulkCopy/ExecutePgCopy之类, 我简略的读了下源码是先将数据转成DataTable然后再执行BulkCopy, 但实际应用中可能存在上万甚至几十万数据的批次写入需求, 如果转成DataTable, 内存溢出的风险是很大的. 能否支持流式写入? 伪代码如下:

```c#

// 流式读取返回数据
private IEnumerable Read(){
// 初始化连接 和reader
await using var reader = await pool.DbConn.ExecuteReaderAsync(sql);
while(reader.Read()){
// 转换实体 yield 流式返回
yield return read.Read2Entity()
}
}

// 流式写入
private async Task Write(IEnumerable items){
// 流式写入 ,ExecutePgCopyAsync会把items的数据一条一条的写,而不是内存中转成DataTable
await fSql.Insert().ExecutePgCopyAsync(items);
}
```

#### 简要描述原因
流式BulkCopy

#### 使用场景
海量数据的写入或者同步

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing ExecuteMySqlBulkCopy and ExecutePgCopy implementations and the DataTable conversion described in the issue. Compare their current input and write paths with the proposed ExecutePgCopyAsync(IEnumerable) usage. Done means large batches can be written incrementally without first materializing all rows in a DataTable, with coverage for the affected bulk-copy paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, mysql, postgresql
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.