nextcloud / nextcloud/recognize
Support for external taggers
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 699
- Forks
- 68
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 5
Description
Describe the feature you'd like to request
Given how limiting the built-in AI models can be it would be nice if it was possible to add external taggers that could be used instead of the built-in ones.
Describe the solution you'd like
A possible solution is using web requests to communicate with external web servers where the image can be sent over and the server can respond with tags that match the image. This will allow to use different models that aren't implemented in the recognize app itself and also stronger models that may not work in docker setups or the host hardware.
For example the https://github.com/AUTOMATIC1111/stable-diffusion-webui project hosts a web server that can generate images locally. Using the extension https://github.com/picobyte/stable-diffusion-webui-wd14-tagger and --api option, a user can send a POST request with any image encoded in base64 and get a json response of all tags in it including weights on how much the image matches set tags.
One the external setup is complete the process is simple:
- Convert the image to base64 data
- Craft it into a javascript object. (eg.
{"image": "data:image/<type>;base64,<image data>", "model": "<model name>", "threshold": 0.35}) - Create a POST request to a target link (eg. "
http://127.0.0.1:7860/tagger/v1/interrogate") - Get a response like:
{
"caption": {
"tag": {
"tag1": 0.6852931380271912,
"tag2": 0.6729011535644531,
"tag3": 0.46537110209465027,
"tag4": 0.5044106245040894
}
}
}
I don't know any other image tagging software that can be interacted using web requests but I think it's a nice start. Even if its only for object tagging.
I wish I could provide a pull request with this option but unfortunately I'm not skilled enough to write clean PHP code with the nextcloud API.
Describe alternatives you've considered
Implementing new models into the app itself is time consuming and slow. And making them work with the limited dependencies of the nextcloud environment and docker instances is even harder.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how recognize currently invokes its built-in taggers and how image data and tags move through the PHP app. Compare that flow with the proposed POST request and JSON response, including configuration for the endpoint, model, and threshold. Done means an external service can be selected and its returned tags can be used without removing built-in taggers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- api, backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100