nextcloud / nextcloud/server

Allow combined UTF-8 characters for filenames on Nextcloud External local storage files

Open
#40,546 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage enhancement feature: external storage hotspot: filename handling
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

On Nextcloud v 27.1.0 I have an external storage of type local.
Having file names with utf-8 character encoding and combined character with combining diaeresis fails to be shown.

The error is shown like the following:

php occ files:scan -p /myPath/myExternalStorageFolder
Starting scan for user 1 out of 1 (me)
        Entry "klaus/Hans hätte besser aufgepasst.mp4" will not be accessible due to incompatible encoding
.
.

For example the letter 'ä' can be represented in two utf-8 complaint forms:

  • (U+00E4), hexadecimal representation: c3a4
  • or a combined form of the character "a" (U+0061) followed by combining diaeresis (U+0308), which is used to add two dots above a character. Hexadecimal representation: 61cc88.

Using the latter method for filenames results in an error due to "wrong" character encoding, even though it is valid UTF-8 character encoding.

Consider a scenario where you have a Spanish keyboard as a German tourist in Mallorca. You may produce a combined UTF-8 character for the letter 'ä' using such a keyboard to name your holiday videos and store them on your memory stick right there.

This report highlights the improvement request.

a workaround:

A tool called 'convmv' is available for Linux and can be installed with the following command: sudo apt install convmv

To correct filenames on your holiday memory stick, following my example, use the following command for a dry run:

convmv -r -f utf-8 -t utf-8 --nfc  ./myPhantisticHolydayMemories/*

This command will show what files would be renamed without actually renaming them. After inspecting the actions it would take, you can perform the renaming with the following command:

convmv -r -f utf-8 -t utf-8 --nfc --notest ./myPhantisticHolydayMemories/*

Once you have completed this process, trigger a scan again using:

php occ files:scan -p /myPath/myExternalStorageFolder

You should no longer encounter any "incompatible" encoding errors.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with a decomposed UTF-8 filename in an external local storage and run php occ files:scan -p /myPath/myExternalStorageFolder. Trace the external-storage scan path that reports the filename as having incompatible encoding. Done when valid combined UTF-8 filenames are accepted and shown without that error.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.