[ENHANCEMENT] Enable Use In Snowflake
- Vorherrschende Sprache
- Python
- Sterne
- 8
- Forks
- 0
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
@austinwarner-8451 and I have discussed this a bit in a Teams thread, but will recount the general issues here:
### Overview
My current use of `confectioner` is fine in Azure and local python environments:
```python
# prepare baker by pointing to where yml is
home = os.path.abspath(os.path.dirname(__file__))
baker = Baker(home)
def read_sourcing_yaml(sourcing_type: str) -> Dict:
"""
Reads and processes a sourcing YAML file based on the specified sourcing type.
Utilizes the confectioner library to bake the various the recipe from the YAML file into a dictionary.
Args:
sourcing_type (str): The type of sourcing to be used for processing the YAML file (e.g. "new_item_full", "new_item_partial", etc.)
Returns:
dict: A dictionary representation of the processed YAML file.
"""
return baker.bake('recipe.yml', sourcing_type=sourcing_type)
```
However, I am running into issues when I try to use the same approach in Snowflake.
### Snowflake Issues:
I think this is due to how snowflake adds packages, specifically I have to upload them to a stage as a `.zip`. From my cursory review, I believe this means that packages are unzipped in a manner that doesn't allow things you'd do in a normal file system (such as using `__file__`).
@austinwarner-8451 and I have tried a few approaches, and it sounds like there could be a benefit in using something like `importlib.resources` within `confectioner`.
### Potential Solution:
As mentioned by @austinwarner-8451, it seems like allowing `Baker` to accept a `Traversable` might resolve the issues here.
Below is an example of how I think I would use it:
```python
from importlib.resources import files
from confectioner import Baker
# this would point to the main module, since in my example `recipe.yml` falls in there:
package_root = files('stratum_sourcing_8451')
baker = Baker(package_root)
baker.bake('recipe.yml', sourcing_type=sourcing_type)
```
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.