mahmoud / mahmoud/awesome-python-applications

[suggestion] Awareness — local-first public-text ingestion engine

Open Beginner friendly
#225 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
18k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

## Project name
Awareness

## Repo URL
https://github.com/nazmiefearmutcu/awareness

## Description (description-only, per CONTRIBUTING)
Local-first public-text ingestion engine. Backfills any historical date range from Common Crawl + FineWeb and live-tails RSS / sitemaps / GDELT into Apache Iceberg, queryable from DuckDB. Single process, no cluster required.

## Suggested tags
`internet`, `dev.storage` (data-pipeline / storage)

## Why it fits
- Pure Python (FastAPI + pyiceberg + duckdb + httpx)
- Self-contained application (web UI ships with the FastAPI server at `/`)
- Long-running (live tail) + one-shot (backfill) modes
- Not a library / not a notebook / not a tutorial

## Status
- v0.1.0 released, MIT licensed
- README has a built-in 'Research Workbench' UI section with 4 screenshots
- Architecture diagram in README

Happy to add to projects.yaml directly via PR if preferred — let me know.

Contributor guide

Open the contributing guide

Research direction

Start by reading CONTRIBUTING and locating projects.yaml in the repository. Add the Awareness project using the details and repository URL from this issue, then open a pull request; done means the entry follows the file's existing format and includes the requested application information.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
content, documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.