Document compliance differences between different sdks

Building off of #254, I've started to put together a script to pull together the differences the SDKs have. In working on parsing the different repositories, I noticed that some tests are easier to parse than others. I'm reaching out to better understand what folks think the right approach is for addressing this.

The spec_finder utility does a few things:

1. Looks in a source repository to find annotations of some sort about which specs are implemented and the language used.
2. Compare the numbers that SDK covers and the text they use to determine if there is a match.
3. Output a report (both human-friendly and json)

Do we complicate the spec_finder utility to better understand the semantics of the underlying languages? Or do we recommend some sort of easily-parsable convention like comments on each test? Or maybe we scrap this idea and have each language output their coverage to some known place that we can more easily deal with?

These were the ones that were a bit difficult to parse

• ruby
• js
• go
• php

Janky first attempt

Code:

import json
from collections import defaultdict

status_map = {
    'missing': '💔',
    'good': '✅',
    'extra': '❓',
    'different-text': '🎭',
    'unknown': ''
}

def main(files):
    reports = {
        # e.g.
        # 'cs': <json-file-contents>
    }
    output = defaultdict(lambda: {})
    # for example..
    #     '1.1.2': {
    #         'cs': 'extra',
    #         'rs': 'missed',
    #     }
    for filename in files:
        with open(filename) as f:
            report = json.loads(''.join(f.readlines()))

        ext = filename.split('-', 1)[0]
        reports[ext] = report

        for status, nums in report.items():
            for num in nums:
                output[num][ext] = status
    # yes this is gross
    print('''
    <style>
    thead {
      background-color: #333;
      color: white;
    }

    th, td {
      padding: .5em 2em;
    }


    tbody tr:nth-child(odd) {
      background-color: #fff;
    }

    tbody tr:nth-child(even) {
      background-color: #eee;
    }
    </style>
    ''')
    print('<table cellspacing="0" cellpadding="0">')
    print('<thead>')
    print('<tr>')
    print("<th>Number</th>") # account for number
    stable_langs = reports.keys()
    for ext in stable_langs:
        print('<th>%s</th>' % (ext))
    print('</tr>')
    print('</thead>')

    print('<tbody>')
    for num, lang_status in sorted(output.items()):
        print("<tr>")
        print("<td>%s</td>" % num)

        for lang in stable_langs:
            status = lang_status.get(lang, "unknown")
            print("<td>%s</td>" % status_map[status])

        print("</tr>")

    print('</tbody>')
    print('</table>')
if __name__ == '__main__':
    import sys
    main(sys.argv[1:])

Result:

Start with the spec_finder utility and the linked Ruby, JavaScript, Go, and PHP SDK test examples to compare how coverage annotations are represented. Review the existing Python report-generation attempt, then determine which parsing or output convention the project should adopt. Done means an agreed approach can consistently produce human-friendly and JSON compliance reports.