Maximum recursion depth error for formulas with more than 485 terms
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 990
- Forks
- 106
- Avg merge
- 7d 34m
- Merged PRs (30d)
- 1
Description
I am working with a dataframe which has 7000 columns and it turns out that once you go beyond 485 terms, patsy throws a recursion error when going from a formula to a design matrix. Is there a better way of doing this?
Thanks!
In [282]: df = pd.DataFrame(dict(('a' + str(i), np.random.randn(5)) for i in xrange(500)))
In [283]: formula = " + ".join(df.columns)
In [284]: dmatrices(formula, df)
....
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
452 "'%s' operator" % (tree.type,),
453 tree.token)
--> 454 result = self._evaluators[key](self, tree)
455 if require_evalexpr and not isinstance(result, IntermediateExpr):
456 if isinstance(result, ModelDesc):
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in _eval_binary_plus(evaluator, tree)
283
284 def _eval_binary_plus(evaluator, tree):
--> 285 left_expr = evaluator.eval(tree.args[0])
286 if tree.args[1].type == "ZERO":
287 return IntermediateExpr(False, None, True, left_expr.terms)
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
452 "'%s' operator" % (tree.type,),
453 tree.token)
--> 454 result = self._evaluators[key](self, tree)
455 if require_evalexpr and not isinstance(result, IntermediateExpr):
456 if isinstance(result, ModelDesc):
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in _eval_binary_plus(evaluator, tree)
283
284 def _eval_binary_plus(evaluator, tree):
--> 285 left_expr = evaluator.eval(tree.args[0])
286 if tree.args[1].type == "ZERO":
287 return IntermediateExpr(False, None, True,
left_expr.terms)
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
448 assert isinstance(tree, ParseNode)
449 key = (tree.type, len(tree.args))
--> 450 if key not in self._evaluators:
451 raise PatsyError("I don't know how to evaluate this "
452 "'%s' operator" % (tree.type,),
RuntimeError: maximum recursion depth exceeded in cmp
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the 500-term formula shown in the issue, then inspect the formula evaluation path in patsy/desc.py, especially eval and _eval_binary_plus. Compare behavior below and above the recursion threshold and add a regression test showing that formulas with more than 485 terms no longer raise the recursion error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100