pathwaycom / pathwaycom/pathway
[Bug]: Simple processing of 1M rows takes more than 10 minutes
@KamilPiechowiak is already working on this.
Since Mar 20, 2024.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Steps to reproduce
I have a simple Pathway function that takes more than 30 seconds to run. I am curious if this is expected or there is something wrong? How long does this take for you?
The code is:
import pathway as pw
class MySchema(pw.Schema):
datetime: str
flag1: bool
val1: float
val2: str
val3: int
def run():
data = pw.io.csv.read('data.csv', schema=MySchema, mode='static')
clean_data = data.select(val1=pw.this.val1, val2=pw.this.val2, datetime=pw.this.datetime.dt.strptime(fmt='%Y-%m-%dT%H:%M:%S.%f'))
pw.debug.compute_and_print(clean_data, n_rows=5)
run()
Just create a CSV file with 1M rows to test this.
Now, I am trying to understand why is this taking so long. What is the best way to profile Pathway performance? Also, what is the best way to load the data with the DateTimeNaive datatype from CSV? The logs from previous runs are telling me parsing DateTimeNaive from an external datasource is not supported.
Relevant log output
There are no errors, just a Pathway run to completion
What did you expect to happen?
I expected that this operation would take a few hundred ms at best? Or maybe a second.
Version
0.8.3
Docker Versions (if used)
No response
OS
Linux
On which CPU architecture did you run Pathway?
x86-64
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.