pydata / pydata/patsy

Maximum recursion depth error for formulas with more than 485 terms

Open
#18 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

fixed in formulaic
Dominant language
Python
Stars
990
Forks
106
Avg merge
7d 34m
Merged PRs (30d)
1

Description

I am working with a dataframe which has 7000 columns and it turns out that once you go beyond 485 terms, patsy throws a recursion error when going from a formula to a design matrix. Is there a better way of doing this?

Thanks!

In [282]: df = pd.DataFrame(dict(('a' + str(i), np.random.randn(5)) for i in xrange(500)))

In [283]: formula = " + ".join(df.columns)

In [284]: dmatrices(formula, df)

....

/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
    452                                 "'%s' operator" % (tree.type,),
    453                                 tree.token)
--> 454         result = self._evaluators[key](self, tree)
    455         if require_evalexpr and not isinstance(result, IntermediateExpr):
    456             if isinstance(result, ModelDesc):

/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in _eval_binary_plus(evaluator, tree)
    283
    284 def _eval_binary_plus(evaluator, tree):
--> 285     left_expr = evaluator.eval(tree.args[0])
    286     if tree.args[1].type == "ZERO":
    287         return IntermediateExpr(False, None, True, left_expr.terms)

/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
    452                                 "'%s' operator" % (tree.type,),
    453                                 tree.token)
--> 454         result = self._evaluators[key](self, tree)
    455         if require_evalexpr and not isinstance(result, IntermediateExpr):
    456             if isinstance(result, ModelDesc):

/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in _eval_binary_plus(evaluator, tree)
    283
    284 def _eval_binary_plus(evaluator, tree):
--> 285     left_expr = evaluator.eval(tree.args[0])
    286     if tree.args[1].type == "ZERO":
    287         return IntermediateExpr(False, None, True,
    left_expr.terms)

/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
    448         assert isinstance(tree, ParseNode)
    449         key = (tree.type, len(tree.args))
--> 450         if key not in self._evaluators:
    451             raise PatsyError("I don't know how to evaluate this "
    452                                 "'%s' operator" % (tree.type,),

RuntimeError: maximum recursion depth exceeded in cmp

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the 500-term formula shown in the issue, then inspect the formula evaluation path in patsy/desc.py, especially eval and _eval_binary_plus. Compare behavior below and above the recursion threshold and add a regression test showing that formulas with more than 485 terms no longer raise the recursion error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.