python / python/pyperformance

Reorganize tags on benchmarks

Open
#210 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
203
Avg merge
1h 20m
Merged PRs (30d)
2

Description

Once https://github.com/python/pyperformance/issues/208 is complete, we will probably want to reorganize the benchmarks into tags so that they are more useful and meaningful. This issue can hopefully provide a place for discussion.

Most importantly: Are there other places where the tags are used such that changing them would be an issue? How can I identify these folks other than posting here?

The currently assigned tags are:

apps: {'2to3', 'tornado_http', 'html5lib', 'chameleon'}
math: {'pidigits', 'float', 'nbody'}
regex: {'regex_v8', 'regex_compile', 'regex_effbot', 'regex_dna'}
serialize: {'xml_etree_generate', 'pickle_dict', 'json_dumps', 'unpickle_pure_python', 'unpickle', 'xml_etree_process', 'json_loads', 'pickle', 'xml_etree_parse', 'pickle_list', 'xml_etree_iterparse', 'unpickle_list', 'pickle_pure_python'}
startup: {'python_startup_no_site', 'python_startup'}
template: {'genshi_xml', 'mako', 'genshi_text', 'django_template'}

For the most part, I think the existing tags are fine, though apps is perhaps a little vague and perhaps should be removed.

I would propose adding the following tags (each benchmark can have multiple tags):

  • Size:
    • workload: This would be for benchmarks that represent real world workloads. These would roll up into "one big number" that we report in places like the CPython release notes. I'm not crazy about the name of this tag. Suggestions?
    • feature: The opposite of a macrobenchmark, for benchmarks that test a very specific feature.
  • Domain:
    • web: Typical tasks used in server-side web development: for example, serializing/deserializing HTML, JSON, XML, l10n and i18n related things

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing issue 208, then investigate where the existing benchmark tags are used and whether other consumers would be affected by changing them. Done means reaching agreement on the tag taxonomy, including proposed workload, feature, and web tags, and documenting the impact of any changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.