github / github/copilot-cli

Factually incorrect and suggested scripts and files not existent

Aperta
#4,815 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

triage
Lingua principale
Shell
Stelle
11.2k
Fork
1.9k
Merge medio
14h 16m
PR unite (30g)
6

Descrizione

Describe the bug

The model described steps to perform the data discovery and alert building based upon the current files existing in the folder. However, although the steps sound logical and positively feasible, the files such as render.py, compare.py. etc are not truly present. They are purely made up by the model, in addition, when assessing the details about the crosswalk from index to the datastream in elastic, although the actual file was there in .csv format, the suggested transition from the index to the stream mentioned by the model was not existing in reality. The name of the dataset is not real either.

Affected version

No response

Steps to reproduce the behavior

Here is the step-by-step practical guide using the (redacted)/  as our working example.

────────────────────

End-to-End Workflow Map

[ Step 1: Data Discovery & Crosswalk ]
│ (Find source index, target datastream & ECS fields)

[ Step 2: ES|QL Query Formulation & Data Evidence ]
│ (Convert SPL → ES|QL & test in Jupyter / Dev Tools)

[ Step 3: YAML Specification & JSON Rendering ]
│ (Write spec-*.yaml → run render_alert.py)

[ Step 4: Dry-Run Diff & Push to Kibana ]
│ (compare_alerts.py → render_alert.py --push)

[ Step 5: Master Index Confirmation ]
(Update ALERTS_INDEX.md status)

────────────────────

Step 1: Data Discovery & Crosswalk Mapping

Goal: Determine where the Splunk data lives in Elasticsearch.

  1. Inspect the original Splunk query (from  redacted):
    index=redacted apiName="inf-bnkng-party-physaccsprof-accsprofvecos-v1"
    | where apiPath="redacted"
    | where responseCode >= 400
  2. Find the target Data Stream:
    Look up in  data/Kibana_dashboard_objects/...CROSSWALK.csv :
    • Splunk  redacted redacted logs $\rightarrow$  redacted 
    • Dataset:  redacted 
  3. Map the Fields:
    •  apiPath  $\rightarrow$  redacted
    •  redacted  $\rightarrow$  redacted
    ────────────────────

Step 2: ES|QL Formulation & Data Evidence

Goal: Formulate and test the query against live data to verify hits.

  1. ES|QL Query Formulation:
    redacted
  2. Test & Verify:
    • Run in Kibana Discover $\rightarrow$ ES|QL tab or in a verification notebook ( redacted.md ).
    • Confirmation Criteria: Ensure no field type errors and that the count calculation matches expected failure counts.

────────────────────

Step 3: Create redacted

Goal: Avoid writing bulky Kibana JSON by using human-readable YAML.

  1. Create or edit the YAML spec ( redacted):
    rule_id: "auto"
    name: "redacted"
    tags: ["redacted"]
    schedule_interval: "5m"
    time_window:
    size: 5
    unit: "m"
    esql: |
    redacted
    email:
    to: ["redacted]
    subject: "{{context.hits.0._source.labels.environment}} - redacted."
    snow:
    node: "redacted"
    resource: "/"
    metric_name: "redacted"
    short_description: "redacted"
  2. Execute Python Rendering Script:


────────────────────

Step 4: Diff & Push to Kibana


### Expected behavior

Accurate to the individual details mentioned in the response from the file names, contents and inferred information (at least logical instead of making it up). The model should have suggested to create the files, instead of making them up to mislead.

### Additional context

_No response_

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia riproducendo il workflow segnalato in copilot-cli e confronta i riferimenti generati con data/Kibana_dashboard_objects/...CROSSWALK.csv e il notebook di verifica disponibile. Controlla come il modello presenta render.py, compare.py, render_alert.py, compare_alerts.py e i nomi dei dataset. Il lavoro è completato quando la guida generata distingue i file e i dati esistenti dai file suggeriti oppure contrassegna chiaramente i passaggi di creazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
elasticsearch, python, shell, yaml
Ambito
ai, cli
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.