CGI-FR / CGI-FR/PIMO

[PROPOSAL] External masks library

Open
#240 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
43
Forks
16
PR merge metrics
No merged PRs in 30d

Description

# Definitions

A masking definition contains the following parts :
- the generator : describe the process to generate a new value
- the coherence context : describe the level of coherence expected for the new value (consistency with other current values or previous values)
- the location : where the value will be written in the json data

The generator is usually defined by the `mask` part of the masking.yml, except for "hash" and "hashInUri" masks which contains a coherence element.

The coherence is usually defined by some properties added to the mask : `seed`, `cache` or the `hash` part in "hash" and "hashInUri" masks.

The location is defined by the selector part.

~~What we need to store in a masking library, is only the **generator** part.~~ When applied in a given context, we can choose where we apply it (selector) and how we handle consistency (cache, seed, hash + what source field is used).

Note: we **can** allow coherence information in some dedicated masks.
Note: we **can** allow selector information in case of multiple fields output.

## Examples

This generator :

```yaml
- randomChoiceInUri: "pimo://nameFR"
```

Can be used in differnt contexts :

```yaml
# synthesize new data :
- selector:
jsonpath: "name1"
masks:
- add: ""
- randomChoiceInUri: "pimo://nameFR"

# synthesize new data consistently with another field:
- selector:
jsonpath: "name2"
masks:
- add: ""
- randomChoiceInUri: "pimo://nameFR"
seed:
field: "id"

# pseudonymize consistently with another field:
- selector:
jsonpath: "name3"
mask:
randomChoiceInUri: "pimo://nameFR"
seed:
field: "id"

...
```

# How to define a mask library

The library should expose a variety of data types
- how to generate a french familly name (locale fr_FR)
- how to generate a french siret
- how to generate a birth date
- etc ...

This can be done by storing a single file for each data type, that contains the list of masks to apply.

filename : **person_name_fr_FR.yml**
```yaml
version: "1":
masking:
- selector:
jsonpath: "."
mask:
randomChoiceInUri: "pimo://nameFR"
```

It's similar to a normal masking. Except for the "." jsonpath that allow to write on the current location in the json stream (where the mask is applied).

Some generators can take parameters

filename : **nir.yml**
```yaml
masking:
- selector:
jsonpath: "gender" #if present then gender is used a parameter
masks:
- add: true #add parameter if not present
- randomChoice: [1, 2]
preserve: "value" #preserve parameter value if present
# other parameters ...
- selector:
jsonpath: "nir"
masks:
- add: true #in this example, the result will be created in a new subfield
- template: '{{if eq .gender "M" }}1{{else}}2{{end}}{{.birth_date | substr 8 10}}{{.birth_date | substr 3 5}}{{.department_code | printf "%02d"}}{{.city_code | printf "%03d"}}{{.order | printf "%03d"}}'
- template: '{{ sub 97 (mod (int64 .nir_start) 97)}}'
```

# How to use masks library

The library can be a folder, a git repository, a website, ...

A new property need to be created to load the library, in the masking.yml

```yaml
version: "1"
librairies:
- "http://domain.org/mylibrary"
- "pimo://internal-library"
- "https+git://github.com/repo/library.git@v0.1.0"
- "file://mylocalibrary"
```

Then a mask from library can be used via a new type of mask

```yaml
- selector:
jsonpath: "nir"
mask:
generate:
using: "nir" # name of the yaml file in the library
```

## Passing parameters : option 1

```yaml
- selector:
jsonpath: "nir"
mask:
generate:
using: "nir" # name of the yaml file in the library
with:
gender: "M"
```

or, if we want to use an existing field as parameter

```yaml
- selector:
jsonpath: "nir"
mask:
generate:
using: "nir"
with:
gender: { from: "gender" }
```

## Passing parameters : option 2

```yaml
# precreate a param with a value
- selector:
jsonpath: "gender"
mask:
constant: "M"
# call mask on the current document (selector: ".")
- selector:
jsonpath: "."
mask:
generate:
using: "nir" # name of the yaml file in the library
```

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing masking.yml schema and the code that loads and applies masks. The proposal mentions external library sources, generate masks, and parameter passing, but does not name implementation files or tests. Done would require an agreed design and working support for the selected library format and loading sources.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.