spring-projects / spring-projects/spring-batch
The table field type in the SQL Server database is ntext and synchronized to the Oracle database, resulting in invalid column types in the corresponding NCLOB
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 2.5k
- Avg merge
- 6d 53m
- Merged PRs (30d)
- 3
Description
Please do a quick search on Github issues first, there might be already a duplicate issue for the one you are about to create.
If the bug is trivial, just go ahead and create the issue. Otherwise, please take a few moments and fill in the following sections:
Bug description
The table field type in the SQL Server database is ntext and synchronized to the Oracle database, resulting in invalid column types in the corresponding NCLOB,resulting in data insertion failure
Environment
source DB Microsoft SQL Server 2008 R2 (RTM) - 10.50.1600.1 (X64) Apr 2 2010 15:48:46
target DB Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
Synchronization Tool : spring-batch-core-4.2.2.RELEASE
Steps to reproduce
Steps to reproduce the issue.
GfDatasource sourceDs = (GfDatasource) this.sTdbMap.get("sourceDs");
GfDatasource targetDs = (GfDatasource) this.sTdbMap.get("targetDs");
Boolean isSqlServer = false;
Boolean isOracle = false;
if (sourceDs != null) {
if (StringUtils.isNotBlank(sourceDs.getUrl())) {
isSqlServer = sourceDs.getUrl().contains("sqlserver");
}
}
if (targetDs != null) {
if (StringUtils.isNotBlank(targetDs.getUrl())) {
isOracle = targetDs.getUrl().contains("oracle");
}
}
if (isSqlServer && isOracle) {
for (Map.Entry<String, Object> entry : item.entrySet()) {
Object value = entry.getValue();
if (value instanceof ClobImpl) {
ClobImpl clob = (ClobImpl) value;
Connection conn = (Connection) this.sTdbMap.get("oracleConn");
OracleConnection oracleConn = null;
if (conn instanceof OracleConnection) {
oracleConn = (OracleConnection) conn;
} else {
throw new RuntimeException("conn不是OracleConnection");
}
Reader reader = clob.getCharacterStream();
// 创建字符缓冲区
StringBuilder stringBuilder = new StringBuilder();
// 读取字符流并存储到字符缓冲区中
char[] buffer = new char[1024];
int length;
while ((length = reader.read(buffer)) != -1) {
stringBuilder.append(buffer, 0, length);
}
// 将字符缓冲区中的字符转换为字节数组
String str = stringBuilder.toString();
byte[] bytes = str.getBytes(“UTF-16LE”);
NCLOB nClob = new NCLOB(oracleConn, bytes);
char[] charArray = str.toCharArray();
nClob.setPrefetchedData(charArray);
nClob.setChunkSize(8132);
NClob nClob1 = oracleConn.createNClob();
int i = nClob1.setString(1, str);
// 关闭 Reader
reader.close();
entry.setValue(nClob);
}
}
}
If I use Oracle to synchronize to Oracle, then the field corresponding to the NCLOB type is successfully synchronized, and it is a byte array,the byte array like this :
byte[] tmpBytes = {0, 84, 0, 1, 4, 76, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 10, 34, -108, -77, 0, 8, -39, -34, 0, 8, -39, -35, 0, 2, 0, 2, 7, -48, 0, 1, 78, 10, 109, 119, 94, 2, 0, 0, -23, 122, -38, 54, 0, 0, 0, 0, 54, -32, 70, -63, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, -39, -35, 1, -117, -21, -113, 0, 0};
But when using SQL Server to synchronize to Oracle, the field type is ntext to NCLOB, and the corresponding type object for ntext is ClobImpl. Therefore, it is necessary to save the byte array stored in this object to the NCLOB object.The byte array obtained from the ClobImpl object looks like this:
byte[] bytes = {10, 78, 119, 109, 2, 94};
Expected behavior
My goal is to save NCLOB type fields in tables in the SQL Server database that can be synchronized to the Oracle database, such as nvarchar (max) or ntext. I hope to be able to save the field values corresponding to this type normally, or the byte array mentioned above can be successfully saved.
Minimal Complete Reproducible example
Please provide a failing test or a minimal complete verifiable example that reproduces the issue.
Bug reports that are reproducible will take priority in resolution over reports that are not reproducible.
step = stepBuilderFactory.get(row.getConfName())
.listener(new StepExecutionListener(row.getConfName()))
.startLimit(1)
.allowStartIfComplete(false)
.chunk(10000)
.reader(initJdbcPagingItemReader(row))
.processor(initItemProcessor(map, sTdbMap))
.writer(initJdbcBatchItemWriter(row))
.exceptionHandler(new DefaultExceptionHandler())
.build();
here is
@Override
public Map process(Map<String, Object> item) throws Exception {
GfDatasource sourceDs = (GfDatasource) this.sTdbMap.get("sourceDs");
GfDatasource targetDs = (GfDatasource) this.sTdbMap.get("targetDs");
Boolean isSqlServer = false;
Boolean isOracle = false;
if (sourceDs != null) {
if (StringUtils.isNotBlank(sourceDs.getUrl())) {
isSqlServer = sourceDs.getUrl().contains("sqlserver");
}
}
if (targetDs != null) {
if (StringUtils.isNotBlank(targetDs.getUrl())) {
isOracle = targetDs.getUrl().contains("oracle");
}
}
if (isSqlServer && isOracle) {
for (Map.Entry<String, Object> entry : item.entrySet()) {
Object value = entry.getValue();
if (value instanceof ClobImpl) {
ClobImpl clob = (ClobImpl) value;
Connection conn = (Connection) this.sTdbMap.get("oracleConn");
OracleConnection oracleConn = null;
if (conn instanceof OracleConnection) {
oracleConn = (OracleConnection) conn;
} else {
throw new RuntimeException("conn不是OracleConnection");
}
Reader reader = clob.getCharacterStream();
StringBuilder stringBuilder = new StringBuilder();
char[] buffer = new char[1024];
int length;
while ((length = reader.read(buffer)) != -1) {
stringBuilder.append(buffer, 0, length);
}
String str = stringBuilder.toString();
byte[] bytes = str.getBytes("UTF-16LE");
NCLOB nClob = new NCLOB(oracleConn, tmpBytes);
char[] charArray = str.toCharArray();
nClob.setPrefetchedData(charArray);
nClob.setChunkSize(8132);
reader.close();
entry.setValue(nClob);
}
}
}
item.put("SEND_BATCH_NO",map.get("SEND_BATCH_NO"));
return item;
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported process method and the initJdbcPagingItemReader/initJdbcBatchItemWriter flow, then reproduce synchronization from SQL Server to Oracle using the versions and configuration described. Compare the ClobImpl-to-NCLOB handling with Oracle-to-Oracle synchronization; done means ntext or nvarchar(max) values are inserted into Oracle NCLOB columns without failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100