openzim / openzim/devdocs

Wrong exception is raised at keyboard interrupt and application never finishes

Open
#14 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

upstream
Dominant language
Python
Stars
9
Forks
3
PR merge metrics
No merged PRs in 30d

Description

When a keyboard interrupt is sent while the libzim creator has already been created and is procesing entries, wrong exception is raised and the scraper never finishes and stays stuck.

^C[devdocs2zim::2024-09-10 07:12:10,459] ERROR:Traceback (most recent call last):
  File "libzim/libzim.pyx", line 98, in libzim.string_cy_call_fct
  File "libzim/libzim.pyx", line 84, in libzim.call_method
AttributeError: 'StaticItem' object has no attribute 'get_mimetype'
Traceback (most recent call last):
  File "/home/benoit/Repos/openzim/devdocs/src/devdocs2zim/entrypoint.py", line 90, in main
    ).run()
      ^^^^^
  File "/home/benoit/Repos/openzim/devdocs/src/devdocs2zim/generator.py", line 366, in run
    self.generate_zim(
  File "/home/benoit/Repos/openzim/devdocs/src/devdocs2zim/generator.py", line 423, in generate_zim
    self.add_zim_contents(
  File "/home/benoit/Repos/openzim/devdocs/src/devdocs2zim/generator.py", line 499, in add_zim_contents
    creator.add_item_for(  # type: ignore
  File "/home/benoit/Repos/openzim/devdocs/.hatch/devdocs2zim/lib/python3.12/site-packages/zimscraperlib/zim/creator.py", line 367, in add_item_for
    self.add_item(
  File "/home/benoit/Repos/openzim/devdocs/.hatch/devdocs2zim/lib/python3.12/site-packages/zimscraperlib/zim/creator.py", line 404, in add_item
    raise exc
  File "/home/benoit/Repos/openzim/devdocs/.hatch/devdocs2zim/lib/python3.12/site-packages/zimscraperlib/zim/creator.py", line 401, in add_item
    super().add_item(item)
  File "libzim/libzim.pyx", line 358, in libzim._Creator.add_item
RuntimeError: Traceback (most recent call last):
  File "libzim/libzim.pyx", line 98, in libzim.string_cy_call_fct
  File "libzim/libzim.pyx", line 84, in libzim.call_method
AttributeError: 'StaticItem' object has no attribute 'get_mimetype'

I strongly suspect this is linked to both wrong exception handling in pylibzim and/or a problem freeing C++ resources.

I'll open an issue in pylibzim, because I strongly suspect this is an upstream issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/devdocs2zim/entrypoint.py at main, then trace generator.py through run, generate_zim, and add_zim_contents to the creator.add_item_for call shown in the traceback. Reproduce the keyboard interrupt and determine whether the incorrect exception or resource cleanup is handled here or in the suspected upstream pylibzim dependency. Done means interruption raises the correct exception and the scraper exits instead of remaining stuck.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.