[ENHANCEMENT] Enable Use In Snowflake
- Lingua principale
- Python
- Stelle
- 8
- Fork
- 0
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
@austinwarner-8451 and I have discussed this a bit in a Teams thread, but will recount the general issues here:
### Overview
My current use of `confectioner` is fine in Azure and local python environments:
```python
# prepare baker by pointing to where yml is
home = os.path.abspath(os.path.dirname(__file__))
baker = Baker(home)
def read_sourcing_yaml(sourcing_type: str) -> Dict:
"""
Reads and processes a sourcing YAML file based on the specified sourcing type.
Utilizes the confectioner library to bake the various the recipe from the YAML file into a dictionary.
Args:
sourcing_type (str): The type of sourcing to be used for processing the YAML file (e.g. "new_item_full", "new_item_partial", etc.)
Returns:
dict: A dictionary representation of the processed YAML file.
"""
return baker.bake('recipe.yml', sourcing_type=sourcing_type)
```
However, I am running into issues when I try to use the same approach in Snowflake.
### Snowflake Issues:
I think this is due to how snowflake adds packages, specifically I have to upload them to a stage as a `.zip`. From my cursory review, I believe this means that packages are unzipped in a manner that doesn't allow things you'd do in a normal file system (such as using `__file__`).
@austinwarner-8451 and I have tried a few approaches, and it sounds like there could be a benefit in using something like `importlib.resources` within `confectioner`.
### Potential Solution:
As mentioned by @austinwarner-8451, it seems like allowing `Baker` to accept a `Traversable` might resolve the issues here.
Below is an example of how I think I would use it:
```python
from importlib.resources import files
from confectioner import Baker
# this would point to the main module, since in my example `recipe.yml` falls in there:
package_root = files('stratum_sourcing_8451')
baker = Baker(package_root)
baker.bake('recipe.yml', sourcing_type=sourcing_type)
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Look at the Baker class initialization in confectioner to understand how it currently uses file paths. Investigate importlib.resources and the Traversable protocol to adapt Baker to accept a Traversable object. Check existing tests for Baker to see how to test the new functionality in a simulated environment.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- backend-api-design
- Tipo di issue
- Funzionalità
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100