HTTP Storage Plugin OversizedAllocationException when field present in page 1 is absent in page 2
- Dominant language
- Java
- Stars
- 2k
- Forks
- 990
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 5
Description
**Describe the bug**
When Apache Drill is configured with the HTTP storage plugin, and INDEX pagination mode is enabled, and the HTTP service configured with that plugin includes a field in all records from page 1, but that field is absent in page 2, a query that spans into page 2 results in an OversizedAllocationException.
**To Reproduce**
Steps to reproduce the behavior:
1. Start Drill.
2. Start the Python3 HTTP server that demonstrates the issue:
https://gist.github.com/hyperbolix/24b1696dbb03c6032fb7af1a06b75145
3. Enable the http storage plugin and configure it using this configuration: https://gist.github.com/hyperbolix/7df108c8169cf54bcc72fdc499be1dce
4. Open the Drill query interface and issue this query:
`select * from http.a_bug_test where total_results_val=300 and page_size_val=10 limit 15`
5. See OversizedAllocationException error. Here is an example error:
https://gist.github.com/hyperbolix/5e2d1e4e10eb92d5a6fa9604acd5cac8
6. Modify the Python3 HTTP server, commenting out line 46 and fixing indentation on line 47.
7. Restart the Python3 HTTP server and issue the query again from step 4.
8. Observe that now the error does not occur and result span into page 2.
**Expected behavior**
Drill is supposed to be resilient to flexible schemas with certain limitations. It should be reasonable for a field to be absent on some pages and present on others, or documentation should indicate the contrary. This would result in the response including the data from page 2 and not producing an exception. If this will not be supported the exception should indicate the issue with the result schema, not indicate that a buffer cannot be expanded.
**Error detail, log output or screenshots**
https://gist.github.com/hyperbolix/5e2d1e4e10eb92d5a6fa9604acd5cac8
`org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: OversizedAllocationException: Unable to expand the buffer. Max allowed buffer size is reached.`
**Drill version**
Observed in 1.21.1 and also in the latest commit as of 2024-03-26: 749772cb0bd83c1a8fe455410ec80b1e5a9bf239
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.