dotnet / dotnet/SqlClient

Cannot populate TVP containing lob fields, from a streamed content

Open
#1,143 11 comments 0 reactions 1 assignee Claimed by @apoorvdeshmukh View on GitHub
Dominant language
C#
Stars
989
Forks
340
Avg merge
4d 18h
Merged PRs (30d)
69

Description

There appears to be some...*glue*...missing to allow streaming content into a table valued parameter. Consider the following extremely contrived scenario:

````
CREATE TABLE dbo.test (clob VARCHAR(MAX), blob VARBINARY(MAX));
CREATE TYPE dbo.test_tt AS TABLE(clob VARCHAR(MAX), blob VARBINARY(MAX));
````

````
// works...
using (var stream1 = IO.File.OpenRead("x:\\foo.bin"))
using (var stream2 = IO.File.OpenRead("x:\\bar.bin"))
using (var reader1 = IO.File.OpenText("x:\\foo.txt"))
using (var reader2 = IO.File.OpenText("x:\\bar.txt"))
{
var cmd1 = sqlconn.CreateCommand();
cmd1.CommandType = CommandType.Text;
cmd1.CommandText = "INSERT INTO dbo.test(clob, blob) VALUES (@pclob, @pblob);"
cmd1.Parameters.Add("@pblob", SqlDbType.VarBinary, -1);
cmd1.Parameters.Add("@pclob", SqlDbType.VarChar, -1);

cmd1.Parameters["@pblob"].Value = stream1;
cmd1.Parameters["@pclob"].Value = reader1;
cmd1.ExecuteNonQuery();

cmd1.Parameters["@pblob"].Value = stream2;
cmd1.Parameters["@pclob"].Value = reader2;
cmd1.ExecuteNonQuery();
}

// fails...
using (var stream1 = IO.File.OpenRead("x:\\foo.bin"))
using (var stream2 = IO.File.OpenRead("x:\\bar.bin"))
using (var reader1 = IO.File.OpenText("x:\\foo.txt"))
using (var reader2 = IO.File.OpenText("x:\\bar.txt"))
{
var meta = new[] {
new SqlMetaData("clob", SqlDbType.VarChar, -1),
new SqlMetaData("blob", SqlDbType.VarBinary, -1)
};
var records = new List();
records.Add(new SqlDataRecord(meta);
records.Add(new SqlDataRecord(meta);
records[0].SetValue(0, reader1);
records[0].SetValue(1, stream1);
records[1].SetValue(0, reader2);
records[1].SetValue(1, stream2);

var cmd2 = sqlconn.CreateCommand();
cmd2.CommandType = CommandType.Text;
cmd2.CommandText = "INSERT INTO dbo.test(clob, blob) SELECT clob, blob FROM @test_tt;"
var par = cmd2.Parameters.Add("@test_tt", SqlDbType.Structured, -1)'
par.TypeName = "dbo.test_tt";
par.Value = records;
cmd2.ExecuteNonQuery();
}
````

*(Assuming I've written the example code above correctly, or at least its intent can be reasonably inferred)*, `cmd1` should execute successfully twice, loading each row one-by-one. `cmd2` will never run, as the calls to `SqlDataRecord.SetValue` don't accept either the `Stream` or `TextReader` objects, and throw an `Invalid cast exception`.

To that end, it does not appear that I can support *both* streaming multiple rows via TVP *and* streaming lobs via `Stream` and/or `TextReader`. I either *have* to load row-by-row (foregoing any decent set-based logic I might have in the SQL); or I *have* to load the whole content into memory (e.g. as `String` and `Byte[]`), to pass them. Neither approach seems particularly suitable for lots of input rows.

It's possible that `Stream` support might already come via `SqlTypes.SqlBytes`, which has a constructor which takes a `Stream` (I haven't tried); but the equivalent `SqlTypes.SqlChars` does not offer a constructor which takes `TextReader`. I'm not entirely convinced the SqlTypes don't automatically load the streamed data into contiguous memory, beforehand, anyway.

Is there anything specific preventing either `SqlDataRecord` accepting `Stream` and `TextReader`, or for `SqlChars` to accept `TextReader`, so there is at least *some* combination that would accept populating a TVP via entirely streamed data?

I suppose `IDisposable` lifetime has to be considered. Even though the above is contrived, I would expect that you would have to `Close`/`Dispose` each lob source only *after* each row was streamed in. I imagine normally streaming each row via a `yield`-based `IEnumerable` method, e.g.

````
private IEnumerable GetFileLobRecords()
{
var meta = new[] {
new SqlMetaData("clob", SqlDbType.VarChar, -1),
new SqlMetaData("blob", SqlDbType.VarBinary, -1)
};
var files = IO.Directory.EnumerateFiles("X:\\");
foreach (var f in files)
{
using (var s1 = IO.File.Open(f, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read))
using (var s2 = IO.File.Open(f, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read))
using (var t = new IO.StreamReader(s2))
{
var record = new SqlDataRecord(meta);
record.SetValue(0, t);
record.SetValue(0, s1);
yield return record;
}
}
}
````

...which, on the surface, at least *looks* like it would behave, with `ExecuteNonQuery` streaming each TVP row up to the server in turn, and then cleaning up the lob source after each row has been streamed.

(I note I haven't made any `async` considerations -- the old-style, non-async pattern is fine for my immediate needs, but others' expectations that it might also work in those situations as well may not be unreasonable).

Anyone with some deeper knowledge of the internals care to comment?

(Possibly related: https://github.com/dotnet/SqlClient/issues/982)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.