twosigma / twosigma/flint

"timeColumn" option not respected in a "read.dataframe" call

Open
#56 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.2k
Forks
199
PR merge metrics
No merged PRs in 30d

Description

@icexelloss hi there!

I'm glad the issues are being (pro)actively monitored and attended to, I wasn't expecting that.

Here is one issue I'm facing, it's not a big one, but an inconvenient one:
```
print( sc.version )
print( tm )

n = df.filter( df['Container'] == 'dbc94d4e3af6' ).select( tm, 'MemPercentG', 'CpuPercentG' )
n.show( truncate = False )
n.printSchema()

from ts.flint import FlintContext, clocks
from ts.flint import utils

fc = FlintContext( sqlContext )

r = fc.read \
.option('isSorted', False) \
.option('timeUnit', 's') \
.option('timeColumn', tm) \
.dataframe( n )
```

The output is:
```
2.3.1
TimeStamp
+-------------------+---------------+------------+
|TimeStamp |MemPercentG |CpuPercentG |
+-------------------+---------------+------------+
|2018-08-01 05:55:35|0.0030517578125|0.002331024 |
|2018-08-01 05:58:05|0.0030517578125|0.0031538776|
|2018-08-01 05:59:05|0.0030517578125|0.0030176123|
+-------------------+---------------+------------+

root
|-- TimeStamp: timestamp (nullable = true)
|-- MemPercentG: double (nullable = true)
|-- CpuPercentG: float (nullable = true)

IllegalArgumentException: 'Field "time" does not exist.\nAvailable fields: TimeStamp, MemPercentG, CpuPercentG'
---------------------------------------------------------------------------
IllegalArgumentException Traceback (most recent call last)
in ()
14 fc = FlintContext( sqlContext )
15
---> 16 r = fc.read .option('isSorted', False) .option('timeUnit', 's') .option('timeColumn', tm) .dataframe( n )
17
18

/databricks/python/lib/python3.5/site-packages/ts/flint/readwriter.py in dataframe(self, df, begin, end, timezone, is_sorted, time_column, unit)
362 time_column=time_column,
363 is_sorted=is_sorted,
--> 364 unit=self._parameters.timeUnitString())
365
366 def parquet(self, *paths):

/databricks/python/lib/python3.5/site-packages/ts/flint/dataframe.py in _from_df(df, time_column, is_sorted, unit)
248 time_column=time_column,
249 is_sorted=is_sorted,
--> 250 unit=unit)
251
252 @staticmethod

/databricks/python/lib/python3.5/site-packages/ts/flint/dataframe.py in __init__(self, df, sql_ctx, time_column, is_sorted, unit, tsrdd_part_info)
133 # throw exception
134 if time_column in df.columns:
--> 135 self._jdf = self._jpkg.TimeSeriesRDD.canonizeTime(self._jdf, self._junit)
136
137 if tsrdd_part_info:

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
77 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
78 if s.startswith('java.lang.IllegalArgumentException: '):
---> 79 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
80 raise
81 return deco

IllegalArgumentException: 'Field "time" does not exist.\nAvailable fields: TimeStamp, MemPercentG, CpuPercentG'
```

Is this reproducible for you?

Please, advise, if I'm not using/calling it correctly or if it's a bug.

The flint libraries (the Scala and the Python ones) I installed on DataBricks via its UI (from the respective online repos, which might be dated) - I can try and install the latest builds from the freshest source code, if you think that will help.

Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.