Visual Studio Code – C++20 UTF-8 String Literals With Octal Sequences

While porting legacy code to C++20, I replaced string literals (with expected UTF-8 encoded text) to UTF-8 string literals (the one prefixed with u8).

Thereby, I ran into an issue with octal sequences which I used in the past to encode UTF-8 sequences byte for byte:

While
"303274" was the proper encoding of ü,
u8"303274" ended up in ü.

I investigated into this further and found on cppreference.com:

2. For each numeric escape sequence, given v as the integer value represented by the octal or hexadecimal number comprising the sequence of digits in the escape sequence, and T as the string literal’s array element type (see the table above):

   • If v does not exceed the range of representable values of T, then the escape sequence contributes a single code unit with value v.

(Emphasis mine)

In my own words: In UTF-8 string literals, octal (ooo) and hex (xXX) escape sequences are interpreted as Unicode code points, similar like Unicode sequences (uXXXX and UXXXXXXXX).

Hence, this appeared reasonable to me: For UTF-8 string literals, Unicode escape sequences should be favored over the byte-wise octal sequences (I used in the past).

Out of curiosity (and for the purpose of demonstration), I made a small test on coliru and was surprised to see that with g++ -std=c++20, the octal sequences still are interpreted as single bytes. The above cite in mind, I came to the conclusion:

MSVC seems to be correct, and g++ wrong.

I made an MCVE which I ran in my local Visual Studio 2019:

#include <iostream>
#include <string_view>

void dump(std::string_view text)
{
  const char digits[] = "0123456789abcdef";
  for (unsigned char c : text) {
    std::cout << ' '
      << digits[c >> 4]
      << digits[c & 0xf];
  }
}

#define DEBUG(...) std::cout << #__VA_ARGS__ << ";n"; __VA_ARGS__ 

int main()
{
  DEBUG(const char* const text = "344270255");
  DEBUG(dump(text));
  std::cout << 'n';
  DEBUG(const char8_t* const u8text = u8"344270255");
  DEBUG(dump((const char*)u8text));
  std::cout << 'n';
  DEBUG(const char8_t* const u8textU = u8"u4e2d");
  DEBUG(dump((const char*)u8textU));
  std::cout << 'n';
}

Output for MSVC:

const char* const text = "344270255";
dump(text);
 e4 b8 ad
const char8_t* const u8text = u8"344270255";
dump((const char*)u8text);
 c3 a4 c2 b8 c2 ad
const char8_t* const u8textU = u8"u4e2d";
dump((const char*)u8textU);
 e4 b8 ad

(Please, note that the dump that the dump of the 1^st and 3^rd literal are identical while the second results in UTF-8 sequences by interpreting each octal sequence as Unicode code point.)

The same code run in Compiler Explorer, compiled with g++ (13.2):

const char* const text = "344270255";
dump(text);
 e4 b8 ad
const char8_t* const u8text = u8"344270255";
dump((const char*)u8text);
 e4 b8 ad
const char8_t* const u8textU = u8"u4e2d";
dump((const char*)u8textU);
 e4 b8 ad

The same code run in Compiler Explorer, compiled with clang (17.0.1):

const char* const text = "344270255";
dump(text);
 e4 b8 ad
const char8_t* const u8text = u8"344270255";
dump((const char*)u8text);
 e4 b8 ad
const char8_t* const u8textU = u8"u4e2d";
dump((const char*)u8textU);
 e4 b8 ad

Demo on Compiler Explorer

Is my conclusion correct that MSVC does it correct according to the C++ standard, in opposition to g++ and clang?

What I found by web search before:

• C++20 with u8, char8_t and std::string
• Using UTF-8 string-literal prefixes portably between C++17 and C++20

Using hex escape sequences instead of octal sequences doesn’t change anything: Demo on Compiler Explorer.

I preferred the somehow unusual octal sequences as they are limited to 3 digits, no unrelated character may extend them unintendedly — in opposition to hex sequences.

Update:

When I was about to file a bug for MSVC, I realized that this was already done:
escape sequences in unicode string literals are overencoded (non conforming => compiler bug)