4paradigm / 4paradigm/OpenMLDB

After inserting data with precompiled SQL, when queried, certain fields have additional null characters ('\x00') appended at the end

Open
#3,779 2 comments 0 reactions 1 assignee Claimed by @yht520100 View on GitHub
bug python
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

**Bug Description**
Service Version: Code from the main branch after version 0.8.4, compiled and deployed on macOS 13.5.1.
Client Version: The current project branch python-sdk, accessing the database using the dbapi method.

After inserting data with precompiled SQL, when queried, certain fields have additional null characters ('\x00') appended at the end, as shown in the fourth column:
`
[(1001, '2022-05-01', 'province1', 'city1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 1, 1590738991000),
(1002, '2022-05-02', 'province2', 'city2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 2, 1590738992000),
(1003, '2022-05-03', 'province3', 'city3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 3, 1590738993000),
(1004, '2022-05-04', 'province0', 'city4\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 4, 1590738994000)]
`

**Expected Behavior**
The field contents should not have additional null characters appended and should match the inserted content.

**Relation Case**

**Steps to Reproduce**

1. Use the python-sdk code from the main branch after version 0.8.4.
2. Run the test cases below to reproduce the corresponding error.

```
import sys
import os
from openmldb.dbapi import connect
import pytest

from .case_conf import OpenMLDB_ZK_CLUSTER, OpenMLDB_ZK_PATH

class TestOpenmldbDBAPI:
cursor_without_db = None
cursor = None
db_name = "dbapi_test"

@classmethod
def setup_class(cls):
# connect without db to create it
db = connect(zk=OpenMLDB_ZK_CLUSTER, zkPath=OpenMLDB_ZK_PATH)
cls.cursor_without_db = db.cursor()
cls.cursor_without_db.execute(
f"create database if not exists {cls.db_name};")
# then we can use the db to connect
db = connect(database=cls.db_name,
zk=OpenMLDB_ZK_CLUSTER,
zkPath=OpenMLDB_ZK_PATH)
cls.cursor = db.cursor()
cls.cursor.execute("create database if not exists {};".format(cls.db_name))
assert cls.db_name in cls.cursor.get_databases()

def recreate_table(self, table, schema):
if table in self.cursor.get_tables(self.db_name):
self.cursor.execute("drop table {}".format(table))
assert table not in self.cursor.get_tables(self.db_name)

self.cursor.execute("create table {}({}) OPTIONS(partitionnum=1);".format(table, schema))
assert table in self.cursor.get_tables(self.db_name)

def test_more_parameterized_query(self):
table = "test_param_query"
schema_str = "col1 bigint, col2 date, col3 string, col4 string, col5 int, col6 timestamp, " \
"index(key=col3, ts=col1), index(key=col3, ts=col6)"
self.recreate_table(table, schema_str)
test_rows = [(1000 + i, '2022-05-0' + str(i), 'province' + str(i % 4),
'city' + str(i), i, (1590738990 + i) * 1000)
for i in range(1, 5)]
self.cursor.executemany('insert into {} values (?, ?, ?, ?, ?, ?);'.format(table), test_rows)

result = self.cursor.execute("select * from {};".format(table)).fetchall()
result = sorted(list(result), key=lambda x: x[0])
assert result == test_rows

if __name__ == "__main__":
sys.exit(pytest.main(["-vv", os.path.abspath(__file__)]))
```

Contributor guide

Open the contributing guide

Research direction

The issue includes a failing Python test case that inserts and queries data via the OpenMLDB Python SDK. Start by running the provided test to reproduce the null character padding. Examine the C++ database engine's handling of string fields in precompiled SQL, particularly memory allocation and padding logic. Check the Python SDK's dbapi layer for any string encoding/decoding issues. The goal is to ensure strings are returned without extra null characters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
backend-api-design, databases, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.