OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8 column
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
I am new to odo. I have a simple function for migrating databases:
```
from blaze import Data, odo, like
def migrate(source, target, table_names=None):
src_db = Data(source)
tbl_names = (src_db.fields, table_names)[table_names is not None]
for tbl in tbl_names:
display("Migrating Tablename ::" + tbl)
odo?
odo('{0}::{1}'.format(source, tbl), '{0}::{1}'.format(target, tbl))
migrate(url_config, url_config_bcp)
```
Upon running this on a SQLite database.
```
url_config = 'sqlite:///DB_CONFIG.db'
url_config_bcp = 'sqlite:///DB_CONFIG_BCP.db'
migrate(url_config, url_config_bcp)
```
I get following error.
How can I resolve this ?
> OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8
> column 'V_WIMS_WELLS$COMMENTS' with text ' 29/12/2013 : SAP issue
> related , WO# was closed but still apprear in WIMS for confirmation .
>
> WPH-41 Cut Surface CSG 9-5/8¿ on b
P.S :- I am aware that we can set textfactory expilicitly in sqlalchemy
```
db = create_engine('sqlite:///{0}'.format(sqlite_name))
conn = db.raw_connection()
conn.connection.text_factory = str
```
I don't know how to fix it within odo function
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the SQLite migration through migrate() and the odo(...) call, using the sample source and target URLs. Inspect how odo opens the SQLite connection and compare it with the SQLAlchemy raw_connection/text_factory setup described in the issue. Done means the migration handles the column's non-UTF-8 value without raising OperationalError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100