Kaszanas / Kaszanas/SC2_Datasets
Missing Parser Fields and Classes
- Dominant language
- Python
- Stars
- 17
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Several fields present in the JSON produced by SC2InfoExtractorGo are silently discarded during parsing, and three game event types have no parser class at all. The parsed objects are well-formed, so the loss is invisible to callers — the data simply never arrives.
The most consequential case is `Cmd`, where the entire command payload is dropped. What survives tells you *that* a command was issued; what is lost is *what the command was*. That makes abilities, spell casts, attack targets and command queueing unavailable from the parsed API, even though all of it is present in the source JSON.
## Method
Audited by comparing the raw JSON key set of every event type against `dataclasses.fields()` of its parser class, over replays sampled uniformly across the SC2EGSet merged dataset (23,476 replays). Field names and example values below are quoted from the raw JSON.
## Dropped fields
| Stream | Event | Dropped fields |
|---|---|---|
| `gameEvents` | `Cmd` | `abil`, `cmdFlags`, `data` |
| `gameEvents` | `ControlGroupUpdate` | `mask` |
| `trackerEvents` | `UnitBorn` | `creatorAbilityName`, `creatorUnitTagIndex`, `creatorUnitTagRecycle` |
## Missing parser classes
| Stream | Event | Notes |
|---|---|---|
| `gameEvents` | `TriggerPing` | Player map pings — attention and team-communication signal |
| `gameEvents` | `DecrementGameTimeRemaining` | |
| `gameEvents` | `TriggerDialogControl` | |
## Detail
### `Cmd.abil`, `Cmd.data`, `Cmd.cmdFlags`
`Cmd` currently parses `id`, `loop`, `otherUnit`, `sequence`, `unitGroup`, `userid`. The raw event also carries:
```json
{
"abil": {"abilLink": 147, "abilCmdIndex": 0, "abilCmdData": null},
"cmdFlags": ["User", "Queued", "SmartClick", "Minimap"],
"data": {"TargetPoint": {"x": 40.305, "y": 65.806, "z": 3.996}}
}
```
- **`abil`** identifies the ability. Present on ~57% of `Cmd` events; `null` denotes a smart-click (right-click move/attack). Note that `abilLink` is indexed per game version, so consumers need `header.version` alongside it to interpret the value.
- **`data`** is one of `{"TargetPoint": {x,y,z}}` (~62%), `{"TargetUnit": {...}, "snapshotPoint": {...}}` (~17%), or `{"None": null}`. The `TargetUnit` form includes `snapshotUnitLink` (the target's unit type), `tag`, `snapshotControlPlayerId` and `targetUnitFlags`.
- **`cmdFlags`** is a list of string flags — observed values include `User`, `Queued`, `SmartClick`, `Minimap` — which distinguish a queued (shift) order from an immediate one, and a minimap click from a main-view click.
Without these three, `Cmd` cannot answer what was cast, where it was aimed, or whether it was queued.
### `UnitBorn.creatorAbilityName`
A human-readable ability name attached to the unit it created, e.g. `BarracksTrain`, `MorphZerglingToBaneling`, `CalldownMULE`, `ForceField`, `SpawnChangeling`, `PurificationNovaTargeted`, `HallucinationPhoenix`.
Over a 61-replay sample: 43,844 `UnitBorn` events, of which **19,633 (44.8%) carry a non-null `creatorAbilityName`**, spanning **27 distinct names**. The remainder are units not created by an ability.
This is valuable precisely because it is a *string* rather than a version-indexed integer, so unlike `abilLink` it is directly comparable across game versions. It is also the natural ground truth for resolving `Cmd.abil.abilLink` to a name within a version.
`creatorUnitTagIndex` / `creatorUnitTagRecycle` identify the producing structure and join against the existing `UnitBorn`/`UnitInit` tag keys, enabling per-structure production tracking.
### `ControlGroupUpdate.mask`
Identifies which units the control group operation applied to. Without it, only the fact and index of a control group action are available, not its contents.
## Suggested change
Additive, and backwards compatible if the new fields default to `None`:
1. Add `abil`, `cmdFlags` and `data` to `Cmd`, with small nested dataclasses for the ability (`abilLink`, `abilCmdIndex`, `abilCmdData`) and the target union. `Target2D` / `Target3D` / `TargetUnit` already exist under `game_events/events/nested/` and cover most of `data`.
2. Add `creatorAbilityName`, `creatorUnitTagIndex`, `creatorUnitTagRecycle` to `UnitBorn`.
3. Add `mask` to `ControlGroupUpdate`.
4. Add parser classes for `TriggerPing`, `DecrementGameTimeRemaining` and `TriggerDialogControl`, and confirm the event dispatcher does not silently skip unknown `evtTypeName` values — a warning on an unrecognised type would have surfaced these three earlier.
Happy to open a PR for any or all of the above.
Contributor guide
Research direction
Start by inspecting the parser classes for Cmd, UnitBorn, and ControlGroupUpdate, the nested dataclasses under game_events/events/nested/, and the event dispatcher. Compare those classes with the raw JSON fields listed in the issue, then examine how unknown evtTypeName values are handled. Done means all listed fields and three event types are represented without discarding data, with unknown event types surfaced rather than silently skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100