8451 / 8451/confectioner

[ENHANCEMENT] Enable Use In Snowflake

Offen
#2 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
8
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

@austinwarner-8451 and I have discussed this a bit in a Teams thread, but will recount the general issues here:

### Overview

My current use of `confectioner` is fine in Azure and local python environments:
```python

# prepare baker by pointing to where yml is
home = os.path.abspath(os.path.dirname(__file__))
baker = Baker(home)

def read_sourcing_yaml(sourcing_type: str) -> Dict:
"""
Reads and processes a sourcing YAML file based on the specified sourcing type.
Utilizes the confectioner library to bake the various the recipe from the YAML file into a dictionary.

Args:
sourcing_type (str): The type of sourcing to be used for processing the YAML file (e.g. "new_item_full", "new_item_partial", etc.)

Returns:
dict: A dictionary representation of the processed YAML file.
"""
return baker.bake('recipe.yml', sourcing_type=sourcing_type)
```

However, I am running into issues when I try to use the same approach in Snowflake.

### Snowflake Issues:

I think this is due to how snowflake adds packages, specifically I have to upload them to a stage as a `.zip`. From my cursory review, I believe this means that packages are unzipped in a manner that doesn't allow things you'd do in a normal file system (such as using `__file__`).

@austinwarner-8451 and I have tried a few approaches, and it sounds like there could be a benefit in using something like `importlib.resources` within `confectioner`.

### Potential Solution:

As mentioned by @austinwarner-8451, it seems like allowing `Baker` to accept a `Traversable` might resolve the issues here.

Below is an example of how I think I would use it:
```python
from importlib.resources import files
from confectioner import Baker

# this would point to the main module, since in my example `recipe.yml` falls in there:
package_root = files('stratum_sourcing_8451')

baker = Baker(package_root)

baker.bake('recipe.yml', sourcing_type=sourcing_type)
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.