8451 / 8451/confectioner

[ENHANCEMENT] Enable Use In Snowflake

Aperta
#2 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
8
Fork
0
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

@austinwarner-8451 and I have discussed this a bit in a Teams thread, but will recount the general issues here:

### Overview

My current use of `confectioner` is fine in Azure and local python environments:
```python

# prepare baker by pointing to where yml is
home = os.path.abspath(os.path.dirname(__file__))
baker = Baker(home)

def read_sourcing_yaml(sourcing_type: str) -> Dict:
"""
Reads and processes a sourcing YAML file based on the specified sourcing type.
Utilizes the confectioner library to bake the various the recipe from the YAML file into a dictionary.

Args:
sourcing_type (str): The type of sourcing to be used for processing the YAML file (e.g. "new_item_full", "new_item_partial", etc.)

Returns:
dict: A dictionary representation of the processed YAML file.
"""
return baker.bake('recipe.yml', sourcing_type=sourcing_type)
```

However, I am running into issues when I try to use the same approach in Snowflake.

### Snowflake Issues:

I think this is due to how snowflake adds packages, specifically I have to upload them to a stage as a `.zip`. From my cursory review, I believe this means that packages are unzipped in a manner that doesn't allow things you'd do in a normal file system (such as using `__file__`).

@austinwarner-8451 and I have tried a few approaches, and it sounds like there could be a benefit in using something like `importlib.resources` within `confectioner`.

### Potential Solution:

As mentioned by @austinwarner-8451, it seems like allowing `Baker` to accept a `Traversable` might resolve the issues here.

Below is an example of how I think I would use it:
```python
from importlib.resources import files
from confectioner import Baker

# this would point to the main module, since in my example `recipe.yml` falls in there:
package_root = files('stratum_sourcing_8451')

baker = Baker(package_root)

baker.bake('recipe.yml', sourcing_type=sourcing_type)
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Look at the Baker class initialization in confectioner to understand how it currently uses file paths. Investigate importlib.resources and the Traversable protocol to adapt Baker to accept a Traversable object. Check existing tests for Baker to see how to test the new functionality in a simulated environment.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
backend-api-design
Tipo di issue
Funzionalità
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.