pathwaycom / pathwaycom/pathway

[Bug]: Simple processing of 1M rows takes more than 10 minutes

Open
#21 12 comments 0 reactions 1 assignee View on GitHub

@KamilPiechowiak is already working on this.

Since Mar 20, 2024.

bug
Dominant language
Python
Stars
62.3k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Steps to reproduce

I have a simple Pathway function that takes more than 30 seconds to run. I am curious if this is expected or there is something wrong? How long does this take for you?

The code is:

import pathway as pw

class MySchema(pw.Schema):
    datetime: str
    flag1: bool
    val1: float
    val2: str
    val3: int
    
def run():
    data = pw.io.csv.read('data.csv', schema=MySchema, mode='static')
    clean_data = data.select(val1=pw.this.val1, val2=pw.this.val2, datetime=pw.this.datetime.dt.strptime(fmt='%Y-%m-%dT%H:%M:%S.%f'))
    pw.debug.compute_and_print(clean_data, n_rows=5)
  
run()

Just create a CSV file with 1M rows to test this.

Now, I am trying to understand why is this taking so long. What is the best way to profile Pathway performance? Also, what is the best way to load the data with the DateTimeNaive datatype from CSV? The logs from previous runs are telling me parsing DateTimeNaive from an external datasource is not supported.

Relevant log output
There are no errors, just a Pathway run to completion
What did you expect to happen?

I expected that this operation would take a few hundred ms at best? Or maybe a second.

Version

0.8.3

Docker Versions (if used)

No response

OS

Linux

On which CPU architecture did you run Pathway?

x86-64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.