WordPress / WordPress/sqlite-database-integration

Limit character-set support to UTF-8, ASCII, and binary

Open
#456 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
363
Forks
67
Avg merge
2d 15h
Merged PRs (30d)
7

Description

Problem

The SQLite driver currently accepts MySQL character sets that it cannot represent safely. The SQL lexer expects UTF-8 input, and SQLite text values are stored as UTF-8.

This becomes visible in WordPress when column metadata reports a non-UTF-8 character set. wpdb::strip_invalid_text() then generates CONVERT(... USING charset) queries containing the original encoded bytes, which the driver cannot parse or convert correctly.

Supported character sets

Limit text support to character sets whose encoded values are valid UTF-8:

  • utf8mb4
  • utf8mb3 / utf8
  • ascii
  • binary values and types

Other character sets, including Big5, KOI8-R, Latin-1, CP1251, Hebrew, and TIS-620, are unsupported. They should not be accepted for columns or CONVERT(... USING charset) expressions.

Expected behavior

  • Reject unsupported character sets in CREATE TABLE and ALTER TABLE with a clear driver error.
  • Reject unsupported character sets in CONVERT(... USING charset).
  • Implement correct behavior for supported UTF-8, ASCII, and binary conversions instead of treating every conversion as a no-op.
  • Ensure column metadata only exposes supported character sets.
  • Define the behavior for existing SQLite databases whose emulated metadata contains unsupported character sets.
  • Add coverage for DDL, metadata, CONVERT, and WordPress CRUD behavior at the supported-character-set boundary.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the SQLite driver's character-set handling in CREATE TABLE, ALTER TABLE, metadata, and CONVERT(... USING charset), then review the WordPress wpdb::strip_invalid_text() path and related CRUD behavior. Done means supported UTF-8, ASCII, and binary cases are covered, unsupported sets are rejected clearly, metadata behavior is defined for existing databases, and tests cover the listed boundaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, sqlite
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.