aws-samples / aws-samples/amazon-redshift-udfs

Redshift can't open shared object file

Open
#19 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
PLpgSQL
Stars
244
Forks
72
PR merge metrics
No merged PRs in 30d

Description

I'm trying to install the [h3 library ](https://github.com/uber/h3-py) in Redshift using Pip and the utility install script `installPipModuleAsRedshiftLibrary.sh`.

I am building `h3` on Ubuntu 16.04. I am able to successfully upload the h3.zip file to my S3 bucket.

However, when I try to run the UDF, I get this error

`OSError: /rdsdbdata/user_lib/0/0/288461.zip/h3/out/libh3.so.1: cannot open shared object file: Not a directory`

Full stack trace from `svl_udf_log`:

```line 2, in get_hexbin\n File "__init__.py", line 1, in
File "h3.py", line 39, in
File "__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)```

The UDF I'm trying to use:

```CREATE OR REPLACE function get_hexbin(coord_lat double precision, coord_lon double precision, res INTEGER)
RETURNS VARCHAR IMMUTABLE
AS $$
from h3 import h3
return 'hello world'
$$ LANGUAGE plpythonu;
```

While installing `h3` locally, I see this message about skipping wheel but then it uploads a zip file to S3 anyway:

```
Collecting h3
Saved /home/bhavika/Desktop/.h3/h3-3.1.0-cp36-cp36m-linux_x86_64.whl
Skipping h3, due to already being wheel.
```

Contributor guide

Open the contributing guide

Research direction

Start by reading installPipModuleAsRedshiftLibrary.sh and reviewing how the h3 package is imported by the shown Redshift UDF. Reproduce the call and inspect svl_udf_log, comparing the uploaded h3.zip contents with the shared-library path in the traceback. Done means the UDF can import h3 without the libh3.so.1 "Not a directory" error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.