pydata / pydata/patsy

how to deal with unicode?

Open
#34 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
990
Forks
106
Avg merge
7d 34m
Merged PRs (30d)
1

Description

Dealing with just Python 2 for now, I understand that patsy expects strings. But the data containers might not have this design. So what's the recommended way for handling this? Should we be messing with the data keys under the hood, or should patsy? The only way I can think to handle this (other than statsmodels doing it under the hood) is for patsy to accept unicode but also an encoding so the formula and they data keys can both be encoded correctly. E.g., this fails

import pandas as pd 
import patsy

data = pd.DataFrame({
    u'àèéòù' : np.random.randn(100),
    'x' : np.random.randn(100)})

formula = u"Q('àèéòù') ~ x".encode('utf-8')
dmatrices = patsy.dmatrices(formula, data=data)

But if we also encode the data keys, it's fine. So should dmatrices and whatever other entry points also take an encoding? Am I missing something?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported failure at the patsy.dmatrices entry point with the Unicode formula and DataFrame keys shown in the issue. Inspect how the formula and data keys are handled, then establish whether the issue is resolved by documenting a recommendation or by defining encoding support for the affected entry points.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.