Nexus: Remove `obj` and `dotdict` container types
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 82
Description
# Motivation
A significant amount of Nexus is built using `obj`/`dotdict`, which are custom container classes that essentially act as dictionaries but with dot access. This provides a host of problems regarding their use, and generally makes long-term support of Nexus significantly more challenging.
## Problems with `obj`/`dotdict`
- They blur the distinction between a mapping and an object.
- `obj` supports attribute access for arbitrary keys, which creates collisions with normal Python methods and attributes.
- They allow implicit dynamic state instead of explicit data structures.
- They make code harder to read, harder to reason about, and harder to refactor safely.
- They are not idiomatic Python and do not follow the conventions of the larger Python community.
- They make static analysis and IDE support worse by hiding the real structure of the data.
- They essentially prohibit type hinting.
- They create subtle runtime traps that are especially problematic in a scientific workflow library where correctness and maintainability matter.
- Attribute access is behind a function call, making them slower than regular `dict`
- `dict`-like key access for `obj` is also behind a function call, which again makes it markedly slower than a `dict`. (roughly twice as slow)
- There is essentially zero ability to type-hint the contents of an `obj`
## Additional problems with LLM-assisted code
An additional problem that I have faced recently is that LLMs seem to have no problem using `obj` and/or `dotdict`. With the burden of reviewing LLM-assisted PRs being significantly higher than non-LLM PRs, the additional problems with regards to `obj` and `dotdict` are just extra work that really shouldn't be added.
# Removal Roadmap
Python already contains many data structures that perform the same task as `obj` and `dotdict`. These will be the replacements for any existing `obj` or `dotdict` in the codebase.
- `dict` for generic mappings
- `dataclass` for anything with a known set of attributes
The first move is to start removing its use from the core of Nexus, and the second is to prevent users from accessing it with a migration guide in place.
The already-accrued technical debt from letting so many uses of `obj` in the codebase is so high that it simply does not make sense to let it get used in any new code, so I think it should additionally be removed from new PRs.
Contributor guide
Research direction
The issue does not name files, tests, or an entry point. Start by locating the obj and dotdict definitions and their uses in Nexus, then determine which sites can use dict and which need dataclass. Done means the migration scope, user-facing migration guidance, and a way to prevent new uses are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100