ParallelRunStep output file rows clobber each other

Very recently, a process that had been fine for months is failing at a pandas read_csv step, when we try to import the dataset created from a ParallelRunStep. The issue seems to be that two rows are merged in the output file, and so a row is longer than expected.

We expect 70 columns in our data. When this happens, one row in the file has more than 70. The number varies (we've seen 2 failures, one had 137 columns, one 100).

In the bad row, it's clear that the second row data comes in in the middle of a row. The first field from the next row is squished onto some field in the middle of the previous row. This is like the following (if our dataset had 3 fields)

we expect:
1 blue balloon
2 yellow car
3 red shirt

but we see instead:
1 blue ballo2 yellow car
3 red shirt

In both of our 2 failures so far the row position and specific case differed (no not the same rows each time). But we saw only one bad row per run (and we had very large files).

We can work around by ignoring bad rows when we do the read_csv, but this is definitely an Azure bug.

Comece pela geração do arquivo de saída de ParallelRunStep descrita no relatório e, em seguida, reproduza a falha importando o dataset dele com pandas read_csv. Compare as linhas de saída com o formato esperado de 70 colunas; considera-se concluído quando a saída concorrente não mesclar mais dados de linhas adjacentes e o arquivo gerado puder ser lido sem ignorar linhas inválidas.

Escrita pelo modelo de indexação a partir do texto da issue.