Converting TensorFlow markdown to JavaDoc text in op_generator

@karllessard @Craigacp

I have been experimenting with converting the TF Markdown text to JavaDoc format in the op_generator code.
I did this by creating another c++ class, that calls out to Python using the Python C library. This runs the Python marko package with my own marko renderer class javadoc_renderer.JavaDocRenderer that converts markdown to JavaDoc.
In the C++ class, SourceWriter, I call out to the python code to convert the Markdown text to JavaDoc. The converted JavaDoc code is then written out to the class.

Here is an example of the old and new generated JavaDoc for org.tensorflow.op.math.Abs:

Current JavaDoc:

/**
 * Computes the absolute value of a tensor.
 * <p>
 * Given a tensor `x`, this operation returns a tensor containing the absolute
 * value of each element in `x`. For example, if x is an input element and y is
 * an output element, this operation computes \\(y = |x|\\).
 * 
 * @param <T> data type for {@code y()} output
 */

New JavaDoc:

/**
 * <p>Computes the absolute value of a tensor.</p>
 * <p>
 * <p>Given a tensor <code>x</code>, this operation returns a tensor containing the absolute
 * value of each element in <code>x</code>. For example, if x is an input element and y is
 * an output element, this operation computes \(y = |x|\).</p>
 * 
 * @param <T> data type for {@code y()} output
 */

There still needs some tweaks to JavaDoc output, like <p> on a single line.
Also, I am still chasing down an infrequent error where the conversion string gets garbled.

I have made several design decision that should probably be discussed. For example, I put my Python module in bazel-bin and point the PYTHONPATH to it in build.sh.

env PYTHONPATH=:$BAZEL_BIN/markdown_javadoc $BAZEL_BIN/java_op_generator \
    --output_dir=$GEN_SRCS_DIR \
    --api_dirs=$BAZEL_SRCS/external/org_tensorflow/tensorflow/core/api_def/base_api,src/bazel/api_def \
    $TENSORFLOW_LIB

Also, I cannot figure out how to bring in the python library from the framework into the BUILD file.
For now, I have it hard coded.

tf_cc_binary(
    name = "java_op_generator",
    linkopts = select({
        "@org_tensorflow//tensorflow:windows": [],
        "//conditions:default": [
            "-lm",
            "-L/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin",
            "-lpython3.7"
          ],
        }),
    deps = [
        ":java_op_gen_lib",
    ],
)

Any help on setting the bazel rules for include the python library would be appreciated.

I did find @org_tensorflow//third_party/python_runtime:headers, which I added as a dependency in the cc_library section of BUILD. This allowed me to compile the c++ code with the Python.h header.

cc_library(
    name = "java_op_gen_lib",
    srcs = [
        "src/bazel/op_generator/op_gen_main.cc",
        "src/bazel/op_generator/op_generator.cc",
        "src/bazel/op_generator/op_specs.cc",
        "src/bazel/op_generator/source_writer.cc",
        "src/bazel/op_generator/markdown_javadoc.cc",
    ],
    hdrs = [
        "src/bazel/op_generator/java_defs.h",
        "src/bazel/op_generator/op_generator.h",
        "src/bazel/op_generator/op_specs.h",
        "src/bazel/op_generator/source_writer.h",
        "src/bazel/op_generator/markdown_javadoc.h",
    ],
    copts = tf_copts(),
    deps = [
        "@org_tensorflow//tensorflow/core:framework",
        "@org_tensorflow//tensorflow/core:lib",
        "@org_tensorflow//tensorflow/core:op_gen_lib",
        "@org_tensorflow//tensorflow/core:protos_all_cc",
        "@org_tensorflow//third_party/python_runtime:headers",
        "@com_googlesource_code_re2//:re2",
    ],
)

I can create a draft PR if you want to look at the whole project, so we can iterate on some of the design decisions, and figure out how to link with the Python C library in a bazel friendly way.

Commencez par lire src/bazel/op_generator/source_writer.cc et markdown_javadoc.cc, puis examinez les extraits de BUILD et de build.sh concernant l’intégration de Python et le chemin JavaDoc généré. La tâche serait terminée lorsqu’un linkage Python compatible avec Bazel aura été convenu, que la sortie JavaDoc aura été corrigée et que la conversion parfois corrompue signalée ici aura été résolue.