MerginMaps / MerginMaps/db-sync
UnicodeDecodeError when decoding GeoDiff stderr on Windows hides the original error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
Environment
• Mergin DB Sync 2.3.0
• GeoDiff 2.3.0
• Windows 11
• Python 3.12
Description
While investigating another issue, I found that Mergin DB Sync crashes with a UnicodeDecodeError when GeoDiff returns an error message containing non-UTF-8 characters on Windows.
Instead of reporting the original GeoDiff error, DB Sync terminates while decoding the stderr output produced by GeoDiff.
The relevant code is:
geodiff_stderr = res.stderr.decode()
This assumes that the error output is UTF-8 encoded. On Windows, this assumption may not hold depending on how GeoDiff emits its error messages.
Actual Result
DB Sync terminates with:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position ...
As a consequence, the original GeoDiff error is hidden, making it difficult to diagnose the underlying problem.
Investigation
To investigate the issue, I modified the run_geodiff() function to preserve the original stdout and stderr output.
Instead of:
geodiff_stderr = res.stderr.decode()
I temporarily changed the code to:
geodiff_stdout = res.stdout.decode(errors="replace")
geodiff_stderr = res.stderr.decode(errors="replace")
This allowed DB Sync to continue running and display the original GeoDiff error.
In my case, the hidden error was:
Error: Missing 'modified' file when opening sqlite driver:
C:/Users/.../camada_mergin_maps—_estacoes_amostragem_teste__estaes_de_amostragem.gpkg
Without this modification, the actual GeoDiff error could not be identified because DB Sync terminated first with the UnicodeDecodeError.
Expected Result
DB Sync should always display the original GeoDiff error, regardless of the encoding used by the underlying process.
It should not terminate while decoding the stderr output.
Possible Improvement
Instead of:
geodiff_stderr = res.stderr.decode()
consider one of the following approaches:
geodiff_stderr = res.stderr.decode(errors="replace")
or
import locale
geodiff_stderr = res.stderr.decode(
locale.getpreferredencoding(False),
errors="replace",
)
Either approach would prevent DB Sync from crashing while still preserving the original GeoDiff error message.
Related Issue
While investigating this behaviour, I identified the underlying GeoDiff issue:
MerginMaps/geodiff#258
DB Sync currently hides that error because of the UTF-8 decoding failure.
I'd be happy to test a fix on Windows if needed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the _run_geodiff() function at the res.stdout.decode() and res.stderr.decode() calls. Reproduce the Windows failure with non-UTF-8 GeoDiff output, then verify that the original GeoDiff error remains visible and decoding no longer raises UnicodeDecodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100