Deferring Evaluation of Terms
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 990
- Forks
- 106
- Avg merge
- 7d 34m
- Merged PRs (30d)
- 1
Description
I have written a function called AbsorbingLS that can absorb a large number (millions) of categorical variables or categorical interactions. It is implemented using a Frisch-Waugh-Lovell step where the categoricals are handled using scipy sparse matrices. I would like to add a formula interface. Suppose I have a function A() that indicates that a variable should be absorbed, is there any place to intervene in the formula parsing for a formula that looks like y ~ 1 + x + A(cat) + A(cat*x)?
I can use another syntax. In an instrumental variable regression I use the syntax y ~ 1 + x1 + x2 + [x3 ~ z1 + z2] which is used to determine the configuration of the 2 required regressions. This works fine since it is easy to parse the [] and then it is a couple of standard calls. This approach doesn't obviously work here since I must avoid creating any arrays. I could use a similar structure here, so something like y ~ 1 + x + {cat +cat*x} ({} for simplicity in parsing) which I would then need to find a good way to turn cat +cat*x into usable terms (w/o populating dense arrays).
Any suggestions on how I could write a formula where I could intercept it using something like the pseudocode
1. Patsy parses to terms
2. I remove and terms that are absorbed, which are too large to express as dense arrays
3. Patsy parses the non-absorbed terms, which have reasonable sizes as dense matrices
Any suggestions are appreciated.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files or tests are named. Start by reviewing Patsy's formula parsing and term-handling entry points, then determine whether absorbed terms can be intercepted before dense matrix construction; done would require an agreed syntax and design for parsing those terms without materializing them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100