blaze / blaze/odo

OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8 column

Open
#424 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

I am new to odo. I have a simple function for migrating databases:

```
from blaze import Data, odo, like

def migrate(source, target, table_names=None):
src_db = Data(source)
tbl_names = (src_db.fields, table_names)[table_names is not None]
for tbl in tbl_names:
display("Migrating Tablename ::" + tbl)
odo?
odo('{0}::{1}'.format(source, tbl), '{0}::{1}'.format(target, tbl))

migrate(url_config, url_config_bcp)
```

Upon running this on a SQLite database.

```
url_config = 'sqlite:///DB_CONFIG.db'
url_config_bcp = 'sqlite:///DB_CONFIG_BCP.db'
migrate(url_config, url_config_bcp)
```

I get following error.

How can I resolve this ?

> OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8
> column 'V_WIMS_WELLS$COMMENTS' with text ' 29/12/2013 : SAP issue
> related , WO# was closed but still apprear in WIMS for confirmation .
>
> WPH-41 Cut Surface CSG 9-5/8¿ on b

P.S :- I am aware that we can set textfactory expilicitly in sqlalchemy

```
db = create_engine('sqlite:///{0}'.format(sqlite_name))
conn = db.raw_connection()
conn.connection.text_factory = str
```

I don't know how to fix it within odo function

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the SQLite migration through migrate() and the odo(...) call, using the sample source and target URLs. Inspect how odo opens the SQLite connection and compare it with the SQLAlchemy raw_connection/text_factory setup described in the issue. Done means the migration handles the column's non-UTF-8 value without raising OperationalError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.