Access issues using Azurite with Bulk Insert in SQL Server in same Docker Container
- Dominant language
- TypeScript
- Stars
- 2.3k
- Forks
- 393
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 36
Description
### Which service(blob, file, queue, table) does this issue concern?
Blob
### Which version of the Azurite was used?
`3.17.1`
### Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)
`npm`
### What's the Node.js version?
node: `v14.19.1`
npm: `6.14.16`
### What problem was encountered?
SQL Bulk Insert isn't working with Azurite:
https://docs.microsoft.com/en-us/answers/questions/242043/sql-server-bulk-insert-does-not-work-with-azure-bl.html
### Steps to reproduce the issue?
1. Setup a custom docker container using `mcr.microsoft.com/mssql/server:2019-latest` as the base image, and install azurite through npm (`npm install -g azurite`):
```Dockerfile
FROM mcr.microsoft.com/mssql/server:2019-latest
USER root
# Install node / npm
RUN apt-get -y update && \
apt-get install -y curl && \
curl -sL https://deb.nodesource.com/setup_14.x | bash - && \
apt-get install -y nodejs && \
apt-get install -y dos2unix
# Install azurite
RUN npm install -g azurite
# Create app directory
RUN mkdir -p /usr/src/app
RUN mkdir -p /usr/src/app/data
WORKDIR /usr/src/app
# Copy over entrypoint script and ensure it has proper line endings
COPY entrypoint.sh /usr/src/app/
RUN dos2unix *
# Grant permissions
RUN chmod 777 /usr/src/app/entrypoint.sh && \
chmod 777 -R /usr/src/app
# Expose port for SQL
EXPOSE 1433
# Add Azurite Ports
# Blob Storage Port
EXPOSE 10000
# Queue Storage Port
EXPOSE 10001
# Table Storage Port
EXPOSE 10002
USER mssql
ENTRYPOINT /bin/bash ./entrypoint.sh
```
Where the contents of entrypoint.sh is:
```console
/opt/mssql/bin/sqlservr & azurite --cert /workspace/cert.pem --key /workspace/key.pem -l /usr/src/app/data --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 --silent --debug debug.log
```
2. Generate a self-signed certificate using [openssl instructions](https://github.com/Azure/Azurite#openssl)
```console
mkdir -p azurite
cd azurite
openssl req -newkey rsa:2048 -x509 -nodes -keyout key.pem -new -out cert.pem -sha256 -days 365 -addext "subjectAltName=IP:127.0.0.1" -subj "/C=CO/ST=ST/L=LO/O=OR/OU=OU/CN=CN"
```
3. Run the container
```console
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=My-sql-password." -p 1433:1433 -p 10000:10000 -p 10001:10001 -p 10002:10002 -v $(PWD)/azurite:/workspace --name sql -h sql -d "name-of-custom-sql-with-azurite-container"
```
4. Use Azure Storage Explorer to connect to Azurite
* You can use import your self-signed certificate in [Azure Storage Explorer](https://docs.microsoft.com/en-us/azure/storage/common/storage-explorer-troubleshooting?tabs=Windows%2C2004#import-ssl-certificates)
5. Create a blob container (e.g. named `containername`) in your Azurite Storage Account `devstoreaccount1`
6. Upload a CSV file under `devstoreaccount1/containername/path/some.csv`
7. Get a SAS token for your Azurite Blob Container using [Azure Storage Explorer](https://docs.microsoft.com/en-us/azure/vs-azure-tools-storage-explorer-blobs#get-the-sas-for-a-blob-container)
8. Connect to your SQL container and run the following:
* Bad or inaccessible location (occurs using `http`)
```console
Error message: Msg 12704 Bad or inaccessible location specified in external data source "DataSourceAzurite"
```
```sql
CREATE MASTER KEY ENCRYPTION BY PASSWORD='';
CREATE DATABASE SCOPED CREDENTIAL AzuriteSASCredential
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = '';
-- Create External Data Source
CREATE EXTERNAL DATA SOURCE DataSourceAzurite
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'http://127.0.0.1:10000/devstoreaccount1/containername',
CREDENTIAL = DSCAzureSqlServerSAS)
-- Get Error message: Msg 12704 Bad or inaccessible location specified in external data source "DataSourceAzurite"
-- some table will have matching columns for some.csv
BULK INSERT some_table
FROM 'path/some.csv'
WITH (DATA_SOURCE = 'DataSourceAzurite',
FIRSTROW = 2,
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '0x0a',
TABLOCK);
-- Get Error message: Msg 12704 Bad or inaccessible location specified in external data source "DataSourceAzurite"
SELECT * FROM OPENROWSET(
BULK 'containername/path/some.csv',
DATA_SOURCE = 'DataSourceAzurite',
SINGLE_CLOB) AS DataFile;
```
* Referenced external data source "DataSourceAzurite" not found
> This approach uses [https](https://github.com/Azure/Azurite#https-setup) with a SAS token
```sql
CREATE MASTER KEY ENCRYPTION BY PASSWORD='';
CREATE DATABASE SCOPED CREDENTIAL AzuriteSASCredential
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = '';
-- Create External Data Source
CREATE EXTERNAL DATA SOURCE DataSourceAzurite
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'https://127.0.0.1:10000/devstoreaccount1/containername',
CREDENTIAL = DSCAzureSqlServerSAS)
-- Get Error message: Msg 12703 Referenced external data source "DataSourceAzurite" not found.
-- some table will have matching columns for some.csv
BULK INSERT some_table
FROM 'path/some.csv'
WITH (DATA_SOURCE = 'DataSourceAzurite',
FIRSTROW = 2,
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '0x0a',
TABLOCK);
-- Get Error message: Msg 12703 Referenced external data source "DataSourceAzurite" not found.
SELECT * FROM OPENROWSET(
BULK 'containername/path/some.csv',
DATA_SOURCE = 'DataSourceAzurite',
SINGLE_CLOB) AS DataFile;
```
* Also attempting with using the [default storage account key](https://github.com/Azure/Azurite#default-storage-account) and [https](https://github.com/Azure/Azurite#https-setup), but this gives a different error message
```console
Cannot bulk load because the file "path/some.csv" could not be opened. Operating system error code 86(The specified network password is not correct.)
```
```sql
-- Create a cred to try with account key instead
CREATE DATABASE SCOPED CREDENTIAL AzuriteCred
WITH IDENTITY = 'devstoreaccount1',
SECRET = 'defaultStorageAccountKey'; -- https://github.com/Azure/Azurite#default-storage-account
CREATE EXTERNAL DATA SOURCE DSAzuriteWithCred
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'https://127.0.0.1:10000/devstoreaccount1/containername',
CREDENTIAL = AzuriteCred)
-- Msg 4861
-- Cannot bulk load because the file "path/some.csv" could not be opened. Operating system error code 86(The specified network password is not correct.).
BULK INSERT some_table
FROM 'path/some.csv'
WITH (DATA_SOURCE = 'DSAzuriteWithCred',
FIRSTROW = 2,
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '0x0a',
TABLOCK);
-- Msg 4861
-- Cannot bulk load because the file "path/some.csv" could not be opened. Operating system error code 86(The specified network
SELECT * FROM OPENROWSET(
BULK 'path/some.csv',
DATA_SOURCE = 'DataSourceAzurite',
SINGLE_CLOB) AS DataFile;
```
If possible, please provide the debug log using the -d parameter, replacing \ with an appropriate path for your OS, or review the instructions for docker containers:
```
-d ""
```
Please be sure to remove any PII or sensitive information before sharing!
The debug log will log raw request headers and bodies, so that we can replay these against Azurite using REST and create tests to validate resolution.
```console
info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
info: QueueGCManager:markSweepLoop() Start new mark and sweep.
info: QueueGCManger:markSweep() Get all extents.
info: QueueGCManager:marksweep() Get 0 extents.
info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 0ms.
info: QueueGCManager:markSweepLoop() Sleep for 60000
info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
```
### Have you found a mitigation/solution?
Workaround: local file loading
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.