cioos-siooc / cioos-siooc/ocean-data-parser
Move away from eval in vocabulary mapping
- Dominant language
- Jupyter Notebook
- Stars
- 5
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
# Issue
For some variable vocabularies, an `apply_func` parameter is defined to derive a new variable from the existing one. The defined apply_func value is then evaluated via `eval()` to generate a new variable. This can create some security risks which should ideally be mitigated.
This has for objectif to help standardizing the different variables variance to a same standard. Here's a quick list of the different functions applied:
- `lambda x: x/10`: convert from S/m to mS/cm
- `lambda x: x*22.319/31.998` convert oxygen from mg/l to umol/l
- `lambda x: x*0.022391` convert form umol/l to ml/l
- `lambda x: -1*gsw.z_from_p(x,ds.attrs['latitude'])` derive depth from sea pressure
- `lambda x: gsw.SP_from_SK(x)` convert Knut salinity (g/kg) to practica salinity
- `lambda x: gsw.t90_from_t68(x)` convert temperature in IPTS-68 to ITS-90
- `lambda x: 42.814*x/10` convert conductivity ration CRAT to S/m
The related vocabularies are the different DFO offices related ones:
- [ocean_data_parser/vocabularies/dfo_ios_vocabulary.csv](https://github.com/cioos-siooc/ocean-data-parser/blob/development/ocean_data_parser/vocabularies/dfo_ios_vocabulary.csv)
- [ocean_data_parser/vocabularies/dfo_nafc_p_files_vocabulary.csv](https://github.com/cioos-siooc/ocean-data-parser/blob/development/ocean_data_parser/vocabularies/dfo_nafc_p_files_vocabulary.csv)
- [ocean_data_parser/vocabularies/dfo_odf_vocabulary.csv](https://github.com/cioos-siooc/ocean-data-parser/blob/development/ocean_data_parser/vocabularies/dfo_odf_vocabulary.csv)
# Fix
A few possibilities regarding how to fix this issue:
1. Create a dedicated function that essentially identify those problematic variables and apply a correction. The function can be either specific to each parser or more global. Likely the first may be easier to maintain in the future.
2. Attempt to still use eval or an equivalent while fixing the security issues.
@sjbruce feel free to add anything in there.
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace where the apply_func values from the DFO vocabulary CSVs are evaluated, then compare the listed conversion and derivation expressions with the parser code that handles them. Decide on a maintained replacement that avoids unsafe evaluation while preserving these conversions. Done means the three vocabulary files still produce the intended standardized variables without relying on eval().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, security
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100