microsoft / microsoft/STL

<locale>: Empty locale name and UTF-8 issues

Open
#1,033 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
11.1k
Forks
1.7k
Avg merge
4d 15h
Merged PRs (30d)
22

Description

Describe the bug
This was reported at Developer Community where several issues were mentioned.

  • Issue number 1. The locale name shouldn't be empty according to the reporter of these issues. I don't know what's correct but other compilers returns "C".
  • Issue number 2. Microsoft's implementation throws "bad locale name" for valid locales that ends with ".utf8" or "UTF-8". According to the original reporter, issue number 2 was a regression in your libraries and worked in version 15.8.0,
  • Issue number 3. codecvt_byname::in is reported to return an error and so the unicode conversion fails. I couldn't reproduce this, but the result from FromNarrowString doesn't compare equal with wideStr if you initialise deletable_facet with "en_US.UTF-8", so the test still fails.

Command-line test case

C:\Temp>type repro.cpp
#include <iostream>
#include <locale>
#include <type_traits>
#include <exception>

using namespace std;
namespace
{
    struct deletable_facet : public codecvt_byname<wchar_t, char, mbstate_t>
    {
        deletable_facet(const std::string& name) : codecvt_byname<wchar_t, char, mbstate_t>(name.c_str()) { }
        ~deletable_facet() = default;
    };
}

wstring FromNarrowString(const char* from, const char* to, const locale& l)
{
    const deletable_facet cvt{ l.name() };
    mbstate_t mbstate{};
    const size_t externalSize = to - from;
    wstring resultWStr(externalSize, '\0');
    const char* from_next; wchar_t* to_next;

    // Issue number 3, cvt.in returns an error.
    codecvt_base::result result = cvt.in(mbstate, from, to, from_next, &resultWStr[0], &resultWStr[resultWStr.size()], to_next);
    if (result != codecvt_base::ok)
    {
        throw std::runtime_error("Error converting locale multibyte string to UNICODE");
    }
    resultWStr.resize(to_next - &resultWStr[0]);
    return resultWStr;
}

int main()
{
    // Issue number 1. The locale name should not be empty according to the reporter. Other compilers returns "C".
    if (std::locale("").name().empty())
    {
        std::cout << "locale name should not be empty\n";
        return -1;
    }

    // Issue number 2. Microsoft's STL throws "bad locale name" for valid locales that ends with ".utf8" or "UTF-8".
    for (const char* localName : { "en_US.utf8", "en_US.UTF-8" })
    {
        try
        {
            const string localMBString = "\x7a\xc3\x9f\xe6\xb0\xb4\xf0\x9d\x84\x8b";
            const wstring wideStr = L"zß水𝄋";

            locale l{ localName };
            if (FromNarrowString(localMBString.c_str(), localMBString.c_str() + localMBString.length(), l) != wideStr)
                return -1;
        }
        catch (const std::exception& ex)
        {
            std::cout << ex.what() << std::endl;
            return -1;
        }
    }

    std::cout << "success\n";
    return 0;
}

C:\temp>cl /EHsc /W4 /WX .\repro.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.27.29009.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

repro.cpp
Microsoft (R) Incremental Linker Version 14.27.29009.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:repro.exe
repro.obj

C:\temp>repro
locale name should not be empty

Expected behavior
It should print success. That's what GCC does. Click to run with compiler explorer

STL version
Microsoft Visual Studio Community 2019 Preview
Version 16.7.0 Preview 3.1

Additional context
Also tracked by DevCom-330322 and VSO-679264 / AB#679264."

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the repro.cpp command-line test case and the locale and codecvt_byname::in entry points described in the issue. Run it on the reported MSVC configuration and compare the results for an empty locale name and the en_US.utf8 and en_US.UTF-8 names. Done means the test prints success and the UTF-8 conversion matches the expected wide string.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
localization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.