mahmoud / mahmoud/awesome-python-applications
[suggestion] Awareness — local-first public-text ingestion engine
- Dominant language
- Jupyter Notebook
- Stars
- 18k
- Forks
- 2.7k
- PR merge metrics
- No merged PRs in 30d
Description
## Project name
Awareness
## Repo URL
https://github.com/nazmiefearmutcu/awareness
## Description (description-only, per CONTRIBUTING)
Local-first public-text ingestion engine. Backfills any historical date range from Common Crawl + FineWeb and live-tails RSS / sitemaps / GDELT into Apache Iceberg, queryable from DuckDB. Single process, no cluster required.
## Suggested tags
`internet`, `dev.storage` (data-pipeline / storage)
## Why it fits
- Pure Python (FastAPI + pyiceberg + duckdb + httpx)
- Self-contained application (web UI ships with the FastAPI server at `/`)
- Long-running (live tail) + one-shot (backfill) modes
- Not a library / not a notebook / not a tutorial
## Status
- v0.1.0 released, MIT licensed
- README has a built-in 'Research Workbench' UI section with 4 screenshots
- Architecture diagram in README
Happy to add to projects.yaml directly via PR if preferred — let me know.
Contributor guide
Research direction
Start by reading CONTRIBUTING and locating projects.yaml in the repository. Add the Awareness project using the details and repository URL from this issue, then open a pull request; done means the entry follows the file's existing format and includes the requested application information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, python
- Domain
- content, documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100