aws-samples / aws-samples/amazon-redshift-udfs

import fails for murmur2 package

Open
#17 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PLpgSQL
Stars
244
Forks
72
PR merge metrics
No merged PRs in 30d

Description

Hello,

I am trying to import murmur2 package as a library in Redshift database. I did following steps

1. Run the module packer
`$ ./installPipModuleAsRedshiftLibrary.sh -m murmur2 -s s3://path/to/murmur2/lib`

2. Create library on redshift
```
CREATE OR REPLACE LIBRARY murmur2
LANGUAGE plpythonu
from 's3://path/to/murmur2/lib/murmur2.zip'
WITH CREDENTIALS AS 'aws_access_key_id=AAAAAAAAAAAAAAAAAAAA;aws_secret_access_key=SSSSSSSSSSSSSSSSS'
region 'us-east-1';

```

3. Create function and query
```
create OR REPLACE function f_py_kafka_partitioner (s varchar, ps int)
returns int
stable
as $$
import murmur2
m2 = murmur2.murmur64a(s, len(s), 0x9747b28c)
return m2 % ps

$$ language plpythonu;

SELECT f_py_kafka_partitioner('jiimit', 100);

```

This gives following error :
```
[Amazon](500310) Invalid operation: ImportError: No module named murmur2. Please look at svl_udf_log for more information
Details:
-----------------------------------------------
error: ImportError: No module named murmur2. Please look at svl_udf_log for more information
code: 10000
context: UDF
query: 0
location: udf_client.cpp:366
process: padbmaster [pid=31381]
-----------------------------------------------;

```

Contributor guide

Open the contributing guide

Research direction

Start with installPipModuleAsRedshiftLibrary.sh and the CREATE LIBRARY, function, and query steps in the report. Inspect the murmur2.zip location and svl_udf_log first to trace why the Redshift PL/Python import cannot find murmur2. Done means the import succeeds and the sample query returns a result.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.