aws / aws/amazon-redshift-odbc-driver

AccessViolationException and Heap Corruption in SQLGetData during ETL operations on Windows

Open
#37 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
25
Forks
17
PR merge metrics
No merged PRs in 30d

Description

# AccessViolationException and Heap Corruption in SQLGetData during ETL operations on Windows

## Environment

| Component | Version |
|-----------|---------|
| **ODBC Driver** | Amazon Redshift ODBC **v2.1.8.0** and **v2.1.12.0** (64-bit) - tested both |
| **aws-sdk-cpp** | 1.11.598 (v2.1.8) / 1.11.711 (v2.1.12) |
| **OpenSSL** | 1.1.1zb / 1.1.1zd |
| **OS** | Windows Server 2019 Standard |
| **.NET Runtime** | .NET 8.0.11, CoreCLR 8.0.1124.51707 |
| **Redshift endpoint** | `pennylane-external.csqwamh5pldr.eu-west-1.redshift.amazonaws.com:5439` |

### DSN Configuration

```
Database: Prod
Auth Type: Standard
SSL Mode: verify-ca
Min TLS: 1.2
Proxy: None
Cursor Size: 20000 (also tested with 1000 - same issue)
Advanced Options:
- Autofetch Refcursors: Enabled
- Rollback on Error: Enabled
- Database Metadata Current Database: Enabled
Retry Delay: 3
```

---

## Description

**Expected behavior:**
ODBC driver should reliably return data from Redshift queries without memory corruption during sustained ETL operations.

**Observed behavior:**
The ODBC driver causes the host application to crash with memory corruption errors:
1. `System.AccessViolationException` in `SQLGetData` native interop
2. `STATUS_HEAP_CORRUPTION` (0xc0000374) in `ntdll.dll`

The crashes are **intermittent** - ETL jobs sometimes complete successfully, other times crash mid-extraction on different tables. This suggests a race condition or memory management issue in the native driver code.

**Impact:**
Production ETL services crash unexpectedly, blocking data pipelines. Multiple independent customers are affected.

---

## Steps to reproduce

1. Configure ODBC DSN with Amazon Redshift driver v2.1.12.0 on Windows Server 2019
2. Connect to Redshift cluster using Standard authentication with SSL (verify-ca)
3. Execute multiple sequential SELECT queries extracting data from various tables
4. Process results using `SQLGetData` to retrieve column values (especially strings)
5. Run ETL jobs on schedule (e.g., every hour) with cursor size 20000

The crash typically occurs:
- During multi-table extraction workflows
- When reading string column data via `SQLGetData`
- Intermittently - not always on the same table

---

## Error messages / Stack traces

### Error 1: AccessViolationException (Windows Event Log)

```
Application: ServerMyReport.exe
CoreCLR Version: 8.0.1124.51707
.NET Version: 8.0.11
Description: The process was terminated due to an unhandled exception.

Exception Info: System.AccessViolationException: Attempted to read or write
protected memory. This is often an indication that other memory is corrupt.

Stack:
at Interop+Odbc.SQLGetData(OdbcStatementHandle, UInt16, SQL_C, CNativeBuffer, IntPtr, IntPtr ByRef)
at System.Data.Odbc.OdbcDataReader.GetData(Int32, SQL_C, Int32, Int32 ByRef)
at System.Data.Odbc.OdbcDataReader.internalGetString(Int32)
at System.Data.Odbc.OdbcDataReader.GetValue(Int32, TypeMap)
at System.Data.Odbc.OdbcDataReader.GetValue(Int32)
[...application code...]
```

### Error 2: Heap Corruption (Windows Event Log)

```
Exception Code: 0xc0000374 (STATUS_HEAP_CORRUPTION)
Faulting Module: ntdll.dll
```

### ODBC Driver Log (redshift_odbc.log)

Recurring empty error messages observed across 6+ months:

```
[ERROR] 2026-01-27 16:39:59.496 [RSLIBPQ:rslibpq.c:472] [11896] pError=
[ERROR] 2026-01-27 16:39:41.242 [RSLIBPQ:rslibpq.c:472] [6540] pError=
[ERROR] 2026-01-27 15:43:50.676 [RSLIBPQ:rslibpq.c:472] [6244] pError=
...
```

Note: The error message after `pError=` is empty, which appears to be a logging issue in the driver.

Also observed streaming cursor operations before crashes:

```
[INFO] 2026-01-27 11:18:33.614 [RSUTIL:rsutil.c:10574] Skiping current result for streaming cursor...
[INFO] 2026-01-27 11:18:33.614 [RSUTIL:rsutil.c:10584] Skiping current result for streaming cursor done.iSocketError=0
```

(Note: "Skiping" appears to be a typo in the driver source code)

---

## Workarounds tried (all unsuccessful)

| Workaround | Result |
|------------|--------|
| Upgraded driver from v2.1.8.0 to v2.1.12.0 | Still crashes |
| Reduced cursor size from 20000 to 1000 | Still crashes |
| Limited to 1 CPU / 1 concurrent ETL query | Still crashes |
| Ensured proper Dispose of ODBC resources in .NET code | Still crashes |

---

## Additional context

- **Two independent customers** experiencing identical symptoms on different infrastructures
- The crash occurs in **native ODBC driver code** (`SQLGetData`), not in the managed .NET layer
- Both `AccessViolationException` and `STATUS_HEAP_CORRUPTION` indicate memory corruption in native code
- Issue persists across driver versions (v2.1.8.0 and v2.1.12.0)
- The intermittent nature and different failing tables suggest a race condition, buffer overflow, or use-after-free bug
- ODBC log shows empty `pError=` messages - possible logging bug masking the actual error

---

## Attachments

- `redshift_odbc.log` - Driver trace log spanning July 2025 to January 2026

[redshift_odbc.log](https://github.com/user-attachments/files/24961862/redshift_odbc.log)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the intermittent SQLGetData failure during sequential string extraction on Windows, then inspect the native paths referenced in redshift_odbc.log, including rslibpq.c:472 and rsutil.c:10574-10584. Done means identifying the memory-corruption cause, preventing the AccessViolationException and heap corruption, and ensuring the relevant driver errors are no longer empty.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c
Domain
database
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.