Enrich translated dataset

Hi,

it would be a great improvement, if the translated dataset can be enriched with more data or fields:

• The Original data (instruction, input and output) can be included to have a better comparison of original and translated data.
• Add support for different translation models.
• Field that is e.g. named review_needed should be added. Problematic or wrong examples can be detected (automatically or manual) and can then be flagged.

On Slack we had the discussion about markdown tables. So one could easily write a markdown table detection script and flag the found examples with the review_needed option, so that these examples can be reviewed later.

Another issue to be discussed: do we want to "override" the existing translated_german_alpaca.json? Or should we introduce a new file for that? But is more than one "dataset" confusing?

Concrete implementation

Concrete implementation steps would be to introduce the following new keys for each example in the dataset:

• instruction
• input
• output
• review_needed (Boolean, default: false).
• translations with instruction, input and output as keys

Proof of concept

One example entry of that enriched dataset could look like:

{
    "instruction": "Give three tips for staying healthy.",
    "input": "",
    "output": "1. Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule.",
    "translations": {
        "instruction": [
            {"facebook/wmt19-en-de": "Geben Sie drei Tipps, um gesund zu bleiben."},
            {"deepl/formality.less": "Gib drei Tipps, um gesund zu bleiben."}
        ],
        "input": [
            {"facebook/wmt19-en-de": ""},
            {"deepl/formality.less": ""}
        ],
        "output": [
            {"facebook/wmt19-en-de": "1. Ern\u00e4hren Sie sich ausgewogen und sorgen Sie f\u00fcr reichlich Obst und Gem\u00fcse. \n2. Sport treiben Sie regelm\u00e4\u00dfig, um Ihren K\u00f6rper aktiv und stark zu halten. \n3. Erhalten Sie gen\u00fcgend Schlaf und halten Sie einen gleichm\u00e4\u00dfigen Schlafplan."},
            {"deepl/formality.less": "1. ern\u00e4hre dich ausgewogen und achte darauf, dass du viel Obst und Gem\u00fcse isst. \n2. Treibe regelm\u00e4\u00dfig Sport, um deinen K\u00f6rper aktiv und stark zu halten. \n3. Schlafe ausreichend und halte dich an einen festen Schlafrhythmus."}
        ]
    },
    "review_needed": false
},

Start by inspecting translated_german_alpaca.json and compare its current entries with the proposed enriched schema in the issue. Decide whether the existing file or a new dataset should be used, then verify that each example preserves the original fields, records translations by model, and includes review_needed with a default of false.