Refactor FastAPI Application Startup to Include Migrations and Seeding
- Dominant language
- Python
- Stars
- 6
- Forks
- 6
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 80
Description
**Describe the Task**
Refactor the FastAPI application's startup process to perform database migrations and seeding during the application's startup lifecycle events, instead of relying on external shell scripts. This involves integrating the migration and seeding logic into the FastAPI `startup` event handlers, ensuring that the application can start and initialize its database dependencies seamlessly.
**Purpose**
- **Simplify Deployment:** Eliminate the need for external shell scripts (`start.sh`, `prestart.sh` and `wait-for-it.sh`), simplifying the deployment and startup process.
- **Improve Maintainability:** Centralize the startup logic within the application codebase, making it easier to manage and maintain.
- **Enhance Reliability:** Ensure that migrations and seeding are tightly integrated with the application's lifecycle, reducing potential points of failure and improving consistency across different environments.
**Acceptance Criteria**
- [ ] **Database Migrations Executed on Startup:** The FastAPI application should automatically execute all pending Alembic database migrations during the startup event.
- [ ] **Database Seeding Performed on Startup:** The application should run the `seed_database.py` script (or equivalent logic) to seed the database with initial data during the startup event.
- [ ] **Removal of External Shell Scripts:** Deprecate and remove the `start.sh`, `prestart.sh` and `wait-for-it.sh` shell scripts from the project. The application should no longer depend on these scripts for startup.
- [ ] **Update Dockerfile:** Update the backend dockerfile to use the python startup method and CMD rather than the shell script
- [ ] **Successful Application Startup:** The application should start successfully without any external scripts, and all migrations and seedings should be applied without errors.
- [ ] **Error Handling and Logging:** Implement appropriate exception handling to catch and log errors that occur during migrations or seeding, preventing the application from starting if critical initialization fails.
- [ ] **Update Documentation:** Update all relevant documentation, including the README and deployment guides, to reflect the new startup process.
- [ ] **Update Deployment Configurations:** Modify any deployment scripts or configurations (e.g., Dockerfiles, CI/CD pipelines) to accommodate the changes in the startup process.
**Additional Context**
- **Current Startup Process:**
- The application currently uses `wait-for-it.sh` to wait for the database to become available.
- `prestart.sh` is used to apply database migrations using Alembic and to run the `seed_database.py` script for seeding initial data.
- The FastAPI application is then started separately after these scripts complete.
- **Proposed Changes:**
- Integrate the logic from `prestart.sh` into the FastAPI application's `startup` event handler in `lifetime.py`.
- Use asynchronous database session handling as appropriate for the migrations and seeding within the event handler.
- Ensure that the application waits for the database to be ready before attempting migrations and seeding.
- **Code References:**
- **Shell Script (`prestart.sh`):**
```bash
#!/usr/bin/env bash
echo "running prestart.sh from $(pwd)"
# Apply base database migrations
echo "Applying base migrations."
poetry run alembic upgrade head
# Check for errors in migrations
if [ $? -ne 0 ]; then
echo "Base database migrations failed."
exit 1
fi
# Run Python seed script using Poetry
echo "Running Python seed script."
poetry run python /app/lcfs/db/seeders/seed_database.py $APP_ENVIRONMENT
# Check for errors in the seed script
if [ $? -ne 0 ]; then
echo "Python seed script failed."
exit 1
fi
echo "Migrations and seeding completed successfully."
```
- **FastAPI Startup Event (`lifetime.py`):**
```python
def register_startup_event(app: FastAPI) -> Callable[[], Awaitable[None]]:
@app.on_event("startup")
async def _startup() -> None:
# Existing setup code...
# TODO: Integrate migrations and seeding here
pass
return _startup
```
- **Considerations:**
- **Database Connectivity:** Ensure the application waits for the database to be ready before running migrations and seeding. This may involve implementing a retry mechanism or checking the database connection within the startup event.
- **Asynchronous Operations:** Since FastAPI supports asynchronous operations, consider using asynchronous methods for running migrations and seeding to prevent blocking the event loop.
- **Error Handling:** If migrations or seeding fail, the application should log the error and exit gracefully to prevent running in an inconsistent state.
- **Environment Variables:** Ensure that environment variables like `$APP_ENVIRONMENT` are accessible within the application context.
- **Benefits:**
- **Consistency:** Having migrations and seeding within the application ensures that all environments (development, testing, production) initialize the database in the same way.
- **Efficiency:** Reduces the overhead of managing separate scripts and potential issues with script execution permissions or paths.
- **Maintainability:** Centralizes the startup logic, making it easier for developers to understand and modify the startup process.
Contributor guide
Research direction
Start by reading lifetime.py and comparing its existing startup setup with prestart.sh, wait-for-it.sh, and seed_database.py. Then inspect the backend Dockerfile and deployment documentation to identify the startup and configuration changes required. Done means migrations and seeding run during FastAPI startup, the external scripts are removed from use, and the README and deployment configurations describe the new process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, fastapi, python, shell
- Domain
- backend, database, devops, documentation
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100