bigscience-workshop / bigscience-workshop/interpretability-ideas

Integrate existing tools: Captum, LIT, AllenNLP Interpret, NeuroX into the HF pipeline

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

**duration**: scalable, can be both 175 and 350 hours
**mentor**: @oserikov
**difficulty**: medium
**requirements**:
1. pytorch
2. sklearn
3. python engineering code, with OOP and patterns
4. experience with Transformer Language models

**useful links**:
- [Captum](https://captum.ai/)
- [AllenNLP Interpret](https://allenai.github.io/allennlp-website/interpret)
- [NeuroX](https://neurox.qcri.org/)

## Idea Description:
There exist lots of interpretability tools, both for Industry and Academia users.
While some of them are general-purpose, and the others are very field-specific, all of them have several things in common.
One would typically apply them to HuggingFace models. All of these methods try to explain the black-boxes we have.

What we propose is, shortly, to put together the existing popular models interpretation stack. We've made a survey of interpretability for LLMs and now have both scientific and engineering vision of what we should implement in order to maximize the interpretability of the existing LLMs.

You need to implement the HF-compatible interpretability aggregation API. The exact **tasks** to accomplish are:
1. choose the most important methods provided by Captum, Interpret and NeuroX (which ones? to better understand the task, try to figure it out yourself. having done this, reach out to us ASAP and we will discuss your vision)
2. implement the all-in-one interpret method to run all the chosen ones
3. perform the initial analysis of the BigScience models checkpoints
4. ensure the codebase is easy to cover the new methods

### Coding Challenge
see task 1.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.