nextcloud / nextcloud/recognize

Support for external taggers

Open
#1,062 2 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
PHP
Stars
699
Forks
68
Avg merge
1d 4h
Merged PRs (30d)
5

Description

Describe the feature you'd like to request

Given how limiting the built-in AI models can be it would be nice if it was possible to add external taggers that could be used instead of the built-in ones.

Describe the solution you'd like

A possible solution is using web requests to communicate with external web servers where the image can be sent over and the server can respond with tags that match the image. This will allow to use different models that aren't implemented in the recognize app itself and also stronger models that may not work in docker setups or the host hardware.

For example the https://github.com/AUTOMATIC1111/stable-diffusion-webui project hosts a web server that can generate images locally. Using the extension https://github.com/picobyte/stable-diffusion-webui-wd14-tagger and --api option, a user can send a POST request with any image encoded in base64 and get a json response of all tags in it including weights on how much the image matches set tags.

One the external setup is complete the process is simple:

  1. Convert the image to base64 data
  2. Craft it into a javascript object. (eg. {"image": "data:image/<type>;base64,<image data>", "model": "<model name>", "threshold": 0.35})
  3. Create a POST request to a target link (eg. "http://127.0.0.1:7860/tagger/v1/interrogate")
  4. Get a response like:
{
    "caption": {
        "tag": {
            "tag1": 0.6852931380271912,
            "tag2": 0.6729011535644531,
            "tag3": 0.46537110209465027,
            "tag4": 0.5044106245040894
        }
    }
}

I don't know any other image tagging software that can be interacted using web requests but I think it's a nice start. Even if its only for object tagging.
I wish I could provide a pull request with this option but unfortunately I'm not skilled enough to write clean PHP code with the nextcloud API.

Describe alternatives you've considered

Implementing new models into the app itself is time consuming and slow. And making them work with the limited dependencies of the nextcloud environment and docker instances is even harder.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how recognize currently invokes its built-in taggers and how image data and tags move through the PHP app. Compare that flow with the proposed POST request and JSON response, including configuration for the endpoint, model, and threshold. Done means an external service can be selected and its returned tags can be used without removing built-in taggers.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
api, backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.