How to use mutli-column functions in Patsy which return categorical output?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 990
- Forks
- 106
- Avg merge
- 7d 34m
- Merged PRs (30d)
- 1
Description
I am dealing with a situation where each item can have 1 or 2 labels both of which come from the same set. In this case I only want to have 1 set of categorical variables which encode if the values of each level are present or absent with respect to the reference. I also want to inversely weight the values of these categorical variables based on how many labels are present.
E.g. Let my categories be {A, B, C}
data = pd.DataFrame({"col1": ["A", "A", "B"], "col2": ["O", "B", "C"], "num_vals": [1,2,2]})
def inverse_val(x):
return 1.0/x
# Using categorical coding will not give the correct output
X = patsy.dmatrix("(C(col1) + C(col2)):inverse_val(num_vals)", data, return_type="dataframe")
I tried to solve this issue using the following code:
def multival(*x, **kwargs):
#raise Exception("Not Implemented")
levels, reference = kwargs.get("levels", None), kwargs.get("reference", None)
weights = kwargs.get("weights", None)
if len(x[0].shape) != 1:
raise Exception("Mismatching Shapes. All arrays should be 1d and should have the same shape")
for k in x:
if k.shape != x[0].shape:
raise Exception("Mismatching Shapes. All arrays should be 1d and should have the same shape")
if levels is None:
levels = np.sort(np.unique(np.hstack(x))) # Sort the unique values and then use this ordering as levels
if reference is None:
reference = levels[0]
#print "Levels: %s, reference: %s" % (levels, reference)
levels = levels[levels != reference] # Remove reference from levels
level_len = len(levels)
#print x[0].shape[0], level_len
out = np.zeros((x[0].shape[0], level_len))
for i, v in enumerate(levels):
# print i, v
for col in x:
out[np.where(np.array(col) == v), i] = 1
#print "Created matrix with shape: ", out.shape
colnames = ["T.%s" % k for k in levels]
if weights is not None:
weights = weights.values
return pd.DataFrame(out, columns=colnames).divide(weights, axis=0)
return pd.DataFrame(out, columns=colnames)
X = patsy.dmatrix("multival(col1, col2, weights=num_vals)", data, return_type="dataframe")
There is no way of identifying the column names generated by patsy:
E.g. The names are like this:
multival(col1, col2, weights=num_vals, reference='O')[0]
multival(col1, col2, weights=num_vals, reference='O')[1]
multival(col1, col2, weights=num_vals, reference='O')[2]
I would much rather prefer column names like the following:
multival(col1, col2, weights=num_vals, reference='O')[T.0]
multival(col1, col2, weights=num_vals, reference='O')[T.1]
multival(col1, col2, weights=num_vals, reference='O')[T.2]
OR if I am passing the levels variable:
multival(col1, col2, weights=num_vals, reference='O')[T.A]
multival(col1, col2, weights=num_vals, reference='O')[T.B]
multival(col1, col2, weights=num_vals, reference='O')[T.C]
Is there a way to achieve this in patsy ?
Duplicate of statsmodels/statsmodels#2843
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the patsy.dmatrix call and the custom multival function shown in the issue, then review the linked duplicate in statsmodels/statsmodels#2843. The work is complete when custom multi-column categorical output exposes meaningful level-based column names instead of numeric indices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100