apache / apache/hamilton

Dagster comparison is not correct

Open
#1,289 3 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

Hi!

I was curious about Hamilton because I was looking for a lightweight DAG library.

Coming from Dagster, I got naturally interested in the Hamilton vs Dagster comparison and found this page in the docs: https://hamilton.dagworks.io/en/latest/code-comparisons/dagster/

I noticed it does not provide accurate information about Dagster and the code examples are not using some of Dagster's main features. More concrete:

- issues with the [first](https://hamilton.dagworks.io/en/latest/code-comparisons/dagster/#dataflow-definition) example:
- It does not utilize the `IOManager` to decouple I/O from computations
- It incorrectly states that asset descriptions have to be defined via metadata, which is not correct (they can be defined in native function docstrings or via the `@asset(description=...` argument).
- issues with the [second](https://hamilton.dagworks.io/en/latest/code-comparisons/dagster/#dataflow-execution) example:
- it incorrectly states that the Dagster job can't be executed in a local Python process
- it incorrectly states that I/O and computations are coupled (duplicate)
- the comparison between loading environment variables at runtime and providing *configuration* time references like `dagster.EnvVar` does not make much sense. Dagster's configuration purposely enables deferring the setting of the exact configuration parameters (since Dagster runs can be executed remotely, e.g. in a Kubernetes pod, and the env var might not be available outside of the remote system). But nothing is preventing the user from setting the value with `os.getenv` directly if needed.

Minor (in the main comparison table):
- important and unique Dagster features such as [Declarative Materialization](https://docs.dagster.io/guides/automate/declarative-automation/) and [Pipes](https://docs.dagster.io/guides/build/external-pipelines/) are not mentioned
- data versioning comparison is a bit strange: it's not very clear how does Hamilton automatically identify code versions (e.g. how does it distinguish between refactoring-like changes and changes in the actual business logic). Dagster's data versioning system enforces explicit code version management to avoid unwanted expensive materializations of the entire asset graph (see: declarative automation).
- important Dagster integrations such as [dagster-dbt](https://docs.dagster.io/integrations/libraries/dbt/) are not mentioned

# Current behavior

The Dagster example is not using relevant Dagster features and provides inaccurate information.

# Expected behavior

The comparison between Hamilton and Dagster should use analogous features in both frameworks to be fair. In particular, it should use the `IOManager` as it's one of the main selling points of Dagster:

```python
import dagster as dg
import pandas as pd

@dg.asset
def topstory_ids() -> pd.DataFrame: ...

@dg.asset
def topstories(topstory_ids: pd.DataFrame): ...
```

Note that some of the popular IOManagers for Pandas and Polars also support loading a subset of the dataframe columns: `@asset(metadata={"columns": ["title"]})`.

It should also provide accurate information on other topics mentioned above.

# Additional context
Technically, this is not a bug, but I couldn't find a better label for this issue.

I am willing to help with improving these docs if my help is considered welcome!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.