googleapis / googleapis/google-cloud-php
Streaming data to BQ using Storage Write API not possible
- Dominant language
- PHP
- Stars
- 1.2k
- Forks
- 463
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 145
Description
#### Environment details
- OS: Windows/Mac
- PHP version: 8.2/8.3
- Package name and version: google/cloud-bigquery-storage v2.1.3
#### Steps to reproduce
It seems to be impossible write data because of [descriptor_fix.php](https://github.com/googleapis/google-cloud-php/blob/main/BigQueryStorage/metadata/descriptor_fix.php) (though it might be solving other issues).
I couldn't find any other way to use `AppendRows` other than using `Proto`-stuffs functionality.
The [provided sample](https://github.com/googleapis/google-cloud-php/blob/main/BigQueryStorage/samples/V1/BigQueryWriteClient/append_rows.php#L111) does not relieve the pain of explaining how data is sent.
[There are mentions of fixes and `AppendsRows`, mainly by @bshaffer ](https://github.com/googleapis/google-cloud-php/pull/5957), but I could not find any information on how to move forward without using `Proto`-stuffs.
#### Code example
```php
// prerequisites
$bigQueryWriteClient = new BigQueryWriteClient();
$parent = 'pathhere';
// setup the structure
$row = new Struct();
$row->setFields([
'impressions' => (new Value())->setNumberValue(213),
]);
$protoRows = new ProtoRows();
$protoRows->setSerializedRows([$row->serializeToString()]);
// add the data
$protoData = new ProtoData([
'rows' => $protoRows,
]);
// set the schema, which requires the descriptor
$protoData->setWriterSchema(new ProtoSchema([
'proto_descriptor' => new \Google\Protobuf\DescriptorProto(),
]));
// create the append rows request
$appendRowsRequest = new AppendRowsRequest([
'write_stream' => $parent . '/streams/_default',
'proto_rows' => $protoData
]);
// do the writing
$stream = $bigQueryWriteClient->appendRows();
$stream->write($appendRowsRequest);
// finalize
foreach ($stream->closeWriteAndReadAll() as $element) {
printf('Element data: %s' . PHP_EOL, $element->serializeToJsonString());
}
```
Contributor guide
Assessment
This issue has not been assessed yet.