testcontainers / testcontainers/testcontainers-java

[Bug]: init script UTF8 conversion breaks encoding used in database when not in UTF8

Open
#8,776 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Java
Stars
8.7k
Forks
1.9k
Avg merge
2d 17h
Merged PRs (30d)
9

Description

Module

Core

Testcontainers version

1.19.8

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host Arch

x86

Docker version
Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:56 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
What happened?

I am using the init script feature to create my tables and insert data after my container has been started and before my application is started.
It is a very convenient way to init my Oracle database for testing purpose.

My database is very old and use the CP1252 encoding to store data.

To respect this encoding requirement my init script is encoded using the windows-1252 .

And in fact it fails to start because my accents encoding is not respected and take more spaces than allowed by the column definition.
In CP1252 the accent character is stored in one byte. And when I run the script I suspect it to occupy more than one byte.

I guess it occurs in this part:

    public static void runInitScript(DatabaseDelegate databaseDelegate, String initScriptPath) {
        try {
            URL resource = Thread.currentThread().getContextClassLoader().getResource(initScriptPath);
            if (resource == null) {
                resource = ScriptUtils.class.getClassLoader().getResource(initScriptPath);
                if (resource == null) {
                    LOGGER.warn("Could not load classpath init script: {}", initScriptPath);
                    throw new ScriptLoadException(
                        "Could not load classpath init script: " + initScriptPath + ". Resource not found."
                    );
                }
            }
            String scripts = IOUtils.toString(resource, StandardCharsets.UTF_8);
            executeDatabaseScript(databaseDelegate, initScriptPath, scripts);
        } catch (IOException e) {
            LOGGER.warn("Could not load classpath init script: {}", initScriptPath);
            throw new ScriptLoadException("Could not load classpath init script: " + initScriptPath, e);
        } catch (ScriptException e) {
            LOGGER.error("Error while executing init script: {}", initScriptPath, e);
            throw new UncategorizedScriptException("Error while executing init script: " + initScriptPath, e);
        }
    }

and specifically here

String scripts = IOUtils.toString(resource, StandardCharsets.UTF_8);

The scripts encoding has been converted using the UTF_8 charset. And now my accent will be stored using 2 bytes when running my insert sql command.

Is it possible to change the code by avoiding enforcing the Charset ?

Regards,

Damien

Relevant log output

No response

Additional Information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at ScriptUtils.runInitScript and inspect the IOUtils.toString(resource, StandardCharsets.UTF_8) call shown in the issue. Reproduce the init-script failure with a Windows-1252-encoded script and accented data in the Oracle database. Done means init scripts preserve the encoding required by the database without breaking existing UTF-8 behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.