microsoft / microsoft/STL

`<xloctime>`: `time_put_byname` doesn't encode the output in UTF-8 even if the locale name contains ".UTF-8"

Open
#6,135 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
11.1k
Forks
1.7k
Avg merge
4d 15h
Merged PRs (30d)
22

Description

Describe the bug

time_put_byname seems to use the system code page even if the locale name contains .UTF-8.

For example, in the ja_JP.UTF-8 locale, "土曜日" (which means "Saturday") is encoded as "\xCD\xC1\xEA\xD7\xC8\xD5" on my computer, but it's UTF-8 representation is "\xE5\x9C\x9F\xE6\x9B\x9C\xE6\x97\xA5".

The libcxx test std/localization/locale.categories/category.time/locale.time.put.byname/put1.pass.cpp expects the output to be encoded in UTF-8 in this case.

time_put_byname is super old and rarely used, and fixing this would be a behavior change. I don't know if it's worth the trouble.

Command-line test case

D:\test>type test-time-put-byname.cpp
#include <cassert>
#include <ctime>
#include <iostream>
#include <locale>

class my_facet : public std::time_put_byname<char, char*>
{
public:
    explicit my_facet(const std::string& nm, std::size_t refs = 0)
        : time_put_byname(nm, refs) {}
};

int main() {
    const my_facet f("ja_JP.UTF-8", 1);
    std::tm t = {};
    t.tm_wday = 6;
    std::ios str(nullptr);

    {
        char buf[200];
        char* iter = f.put(buf, str, '*', &t, 'A');
        std::cout.setf(std::ios::hex, std::ios::basefield);
        for (char* p = buf; p != iter; ++p) {
            std::cout << (int)(unsigned char)*p << ' ';
        }
        std::cout << '\n';
    }
}
D:\test>cl /EHsc test-time-put-byname.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.50.35725 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

test-time-put-byname.cpp
Microsoft (R) Incremental Linker Version 14.50.35725.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test-time-put-byname.exe
test-time-put-byname.obj
D:\test>test-time-put-byname.exe
cd c1 ea d7 c8 d5

Expected behavior

The output is e5 9c 9f e6 9b 9c e6 97 a5.

STL version

https://github.com/microsoft/STL/commit/117ca965fe2f7157540edd9df5a62b0e42d7ece2

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the time_put_byname implementation and compare its behavior with std/localization/locale.categories/category.time/locale.time.put.byname/put1.pass.cpp, which expects UTF-8 output. Reproduce the ja_JP.UTF-8 command-line case, then verify that the emitted bytes match e5 9c 9f e6 9b 9c e6 97 a5.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
localization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.