microsoft / microsoft/go-sqlcmd

GO version of sqlcmd does not parse ANSI text files correctly

Open
#494 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Localization
Dominant language
Go
Stars
595
Forks
91
Avg merge
9h 35m
Merged PRs (30d)
1

Description

If a sql text file is encoded as ANSI (as opposed to UTF-8 or similar) the newer Go version of sqlcmd will not correctly parse non-ASCII characters.

For example, if a file contains non-breaking spaces (character 160), which in T-SQL is generally treated identically to a normal space. In ANSI Windows-1252, this is encoded as a single-byte hex A0.

The Go version of sqlcmd appears to assume all files are UTF encoded, for it treats such a character as unknown and replaces it with unicode character 65533, which would be consistent with assuming UTF-8 encoded, for the single byte A0 is not valid UTF-8.

The attached file is a simple example txt file encoded using the Windows notepad as ANSI, containing "SELECT{Non-breaking-space}CURRENT_TIMESTAMP"

testfile.txt

It can be run in sqlcmd with a command like:
sqlcmd -i testfile.txt

The original ODBC version of sqlcmd has no problem running the above file, returning the expected timestamp.

The GO version however fails:
"Could not find stored procedure 'SELECT�CURRENT_TIMESTAMP'."

The behavior of the GO sqlcmd should either match the ODBC behavior, or this should be documented as one of the "Breaking changes from sqlcmd (ODBC)" that ANSI-encoded text files are not supported.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached testfile.txt, which contains a Windows-1252 non-breaking space, and run it with sqlcmd -i testfile.txt. Compare the Go sqlcmd result with the expected timestamp and the ODBC behavior; done means the ANSI input is handled correctly or the limitation is documented as a breaking change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
cli, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.