asyml / asyml/forte

add profiler for data packs and processors

Open
#549 0 comments 0 reactions 1 assignee Claimed by @wanglec View on GitHub
Dominant language
Python
Stars
253
Forks
59
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Profilers could test whether our new data structure has improvements and measure the time from each processor.

**Describe the solution you'd like**
Add profilers for data packs and processors and mimic specific usage scenarios (such as adding pos tag and tokenization). Add another profiler with pure python and try to do similar actions as the data pack profiler. Then, compare what part of the data pack takes the most time and what part has heavy overheads. Some examples are [here.](https://github.com/asyml/forte/tree/new_data_structure/examples/profiler)

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.