🎯 Python Experience Improvements

## Preamble

This is a tracking issue summarizing the existing pain points in **Python** front-ends generated by `jsii-pacmak`. Members of the community are more than welcome to contribute to this, for example by sharing their pet peeves in the communication thread below.

The initial post will be periodically updated to include a summary of the items that are accepted as **needed improvements** to our **Python** offering.

## Known Pain Points

- [x] aws/aws-cdk#474 - Incomplete [CDK documentation] due to quirks in `sphinx`
* Nested types are missing
* Many parameters are showing typed as `Forwardref`
- [x] Incorrectly transliterated sample code in the [cdk documentation]
* Type information is necessary in many cases, and `jsii-rosetta` does not _require_ that examples compile
* In many cases, it's only a matter of adding a [rosetta fixture]
- [ ] Un-pythonic reliance on a custom [metaclass]
* [Metaclasses][metaclass] complicate extension: all sub-classes in a class family must eventually derive from the *same* [metaclass]
* Python doesn't really have interfaces, the idiom is using [Protocols][protocol]
+ Protocols can be implicitly implemented (structural typing)
+ Explicit implementation uses the extension syntax, but this cannot work due to [metaclass] dissonance
- [ ] Passing `dicts` in lieu of *jsii structs* does not consistently work
* Case adjustments of values is not always consistently done, resulting in runtime errors or worse (silently ignored properties, incorrect/unexpected side-effects, ...)
- [ ] Type-checking errors from `@jsii/kernel` are obscure

[cdk documentation]: https://docs.aws.amazon.com/cdk/api/latest/python
[metaclass]: https://docs.python.org/3/reference/datamodel.html#metaclasses
[protocol]: https://www.python.org/dev/peps/pep-0544/
[rosetta fixture]: https://github.com/aws/jsii/tree/master/packages/jsii-rosetta#fixtures

## Discussion

### Incomplete CDK Documentation

The AWS CDK API reference documentation is currently generated using a *private* tool, which leverages the idiomatic documentation generator for each supported language (`docfx`, `javadoc`, `sphinx`, ...). While this has the advantage of giving users the same documentation experience that they're used to for their language of choice, it has several disadvantages:

- it is an awkward experience for users building polyglot applications, since each language's documentation set is entirely separate from the others
- we have to suffer the limitations of the idiomatic tools
+ customizing templates is often a difficult process
+ certain *jsii* features are broken or weird in some languages (i.e: `sphinx` and nested types)

Additionally, the current generator being private means other users of `jsii` are not able to produce a documentation site that looks and feels like the AWS CDK reference. One of the goals of `jsii` being that developers need not learn all about the standard tools of front-end languages, providing a `jsiidoc` tool would greatly reduce friction around documenting polyglot APIs. This is however a significant project, and shorter-term solutions (such as using `autodoc` for **Python** with generated `.rst` files).

### Incorrectly Transliterated Sample Code

The `jsii-rosetta` tool is able to collect code example snippets from various locations in the `jsii` assemblies, and attempts to automatically transliterate those to other front-end languages. It delivers the best fidelity when it is able to obtain complete type information around the example (i.e: it is valid and compiling **TypeScript** code); and must resort to broad assumptions otherwise.

In the context of AWS CDK, many example snippets lack context information necessary in order to obtain complete type information, which often leads to incorrectly transliterated code. In the **Python** context, this can mean the transliterated code uses keyword arguments at a location where they are not actually supported.

There is a `--fail` option to `jsii-rosetta` which will cause it to abort if an example is not compiling (the corollary of this being there cannot be sufficient type information). Enabling this behavior has two benefits:

- It guarantees example code is up-to-date with the current library's API
- Complete type information should be available, enabling the best possible transliteration result

This feature is however not enabled by default on the AWS CDK, as many examples there do not have the full context needed for them to compile. In many cases, all that is needed is the introduction of [fixtures][rosetta fixture] that provide the necessary context while not weighting example code shown in the documentations with repetitive boilerplate.

### Un-pythonic reliance on a custom metaclass

The *jsii* runtime for Python provides a `JSIIMeta` [metaclass] that is used by all generated bindings for classes (classes from the *jsii assembly*). As a consequence, attempting to leverage **Python**'s multiple inheritance (for example, to implement an interface explicitly) may lead to a runtime error, as all bases of a **Python** class must be rooted on the same [metaclass].

Additionally, the `JSIIMeta` [metaclass] is merely used to record *jsii-specific* metadata on generated classes; but since those are code-generated, it would be trivial to register the exact same metadata as part of the generated declarations, rendering the [metaclass] useless.

Not relying on the [metaclass] would mean users are able to implement *interfaces* explicitly in the idiomatic way, which would provide better information to static analyzers such as `mypy`, to help ensure developers have correctly implemented an interface, and are correctly passing values that conform to interfaces to method parameters.

```py
# The slightly awkward:
@jsii.implements(lib.IInterface)
class Implementor():
...

# Could become:
class Implementor(lib.IInterface):
...
```

### Passing `dict` in lieu of *jsii structs* does not consistently work

Similar to the **Javascript** idioms, **Python** developers will often intuitively use `dict` literals in places where the API expects some instance of a *jsii struct*. The generated **Python** code has specific code paths to convert a `dict` to the correct *struct* instance, however there seem to be many cases where this conversion is not happening, leading to:

- type-checking errors in the `@jsii/kernel` module
- property keys not being translated from `snake_case` to `camelCase`
- properties being dropped entirely by the `@jsii/kernel`

The workaround forces users to resort to relatively verbose boilerplate code, which in turns looks very un-pythonic.

```py
# The somewhat heavy-handed
s3.Bucket(self, 'Gold', lifecycle_rules=[s3.LifecycleRule(expiration=cdk.Duration.days(15)])

# Could become:
s3.Bucket(self, 'Gold', lifecycle_rules=[{ 'expiration': cdk.Duration.days(15) }])
```

### Type-checking errors from `@jsii/kernel` are obscure

When a user makes a programming mistake and passes the wrong kind of value to an API (be it a parameter of some method or constructor, or the value being set to some property), the errors generated by the `@jsii/kernel` are not very helpful to developers as:

- They no not relate to the user's native language type names
- They may lack sufficient context to be parseable by the user (for example, when the error occurs in a deeply-nested value)

This tracking issue covers several unchecked areas, including generated Python bindings, @jsii/kernel errors, and jsii-rosetta fixtures. Start by selecting one pain point and reproducing it in the relevant entry point; done requires a narrowly scoped improvement with focused coverage, but the issue names no specific files or tests.

Written by the indexing model from the issue text.