fulltextsearch:document:provider -c displays wrong c... | nextcloud/fulltextsearch

I'm still in the process of trying to wrap my head around how FTS works. FTS is still in the process of indexing my contents, but I am using Kibana to inspect the current progress and to see if OCR (tesseract) works. Specifically, I have created two files with the same content magicvalue123 as a PDF and a JPEG.

It seems as if php occ fulltextsearch:document:provider -c <user> files <document id> can be used to inspect the index via the FTS plugin (I'm only using the files provider).

Unfortunately, when run with my test document IDs the output is unexpected. For the JPEG the content is as expected, but for the PDF it appears to be the PDF's first couple if bytes formatted as text.

Even more strange, Kibana does not find any documents when running the KQL query _id : "files:1570975" (JPEG) but does for _id : "files:1570976" (PDF). Perhaps this is due to the fact that I added the index with a date field to Kibana, but the documents that were OCR'ed only (the JPEG) do not contain a date field.

PDF

$ docker exec -it -uwww-data nextcloud_app_1 php occ fulltextsearch:document:provider -c agross files 1570976
Document:
{
    "id": "1570976",
    "providerId": "files",
    "access": {
        "ownerId": "agross",
        "viewerId": "",
        "users": [],
        "groups": [],
        "circles": [],
        "links": []
    },
    "modifiedTime": 1568827937,
    "title": "_test\/Untitled-1.pdf",
    "link": "",
    "index": {
        "ownerId": "agross",
        "providerId": "files",
        "source": "files_local",
        "documentId": "1570976",
        "lastIndex": 0,
        "errors": [],
        "errorCount": 0,
        "status": 28,
        "options": {
            "_files_pdf": "1",
            "_files_local": "1"
        }
    },
    "source": "files_local",
    "info": {
        "share_names": {
            "agross": "_test\/Untitled-1.pdf"
        }
    },
    "hash": "",
    "contentSize": 179480,
    "tags": [],
    "metatags": [
        "files_local"
    ],
    "subtags": [],
    "more": [],
    "excerpts": [],
    "score": ""
}
Content:
%����1.4
<</Linearized 1/L 134608/O 6/E 122620/N 1/T 134409/H [ 7
2 Part(s)
'ocr'    (size: 0)
'comments'    (size: 0)

$ docker exec nextcloud_search01_1 curl -XGET http://localhost:9200/nextcloud/_search -d '{ "query": { "terms": { "_id": ["files:1570976"] } } }' -H 'Content-Type: application/json' | jq
{
  "took": 764,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "nextcloud",
        "_type": "standard",
        "_id": "files:1570976",
        "_score": 1,
        "_source": {
          "owner": "agross",
          "groups": [],
          "circles": [],
          "metatags": [
            "files_local"
          ],
          "source": "files_local",
          "title": "_test/Untitled-1.pdf",
          "users": [],
          "content": "magicvalue123",
          "tags": [],
          "attachment": {
            "date": "2019-09-18T17:32:12Z",
            "content_type": "application/pdf",
            "language": "et",
            "content_length": 17
          },
          "provider": "files",
          "subtags": [],
          "parts": {
            "comments": "",
            "ocr": ""
          },
          "links": [],
          "share_names": {
            "agross": "_test/Untitled-1.pdf"
          },
          "hash": "c56703736caa8ca957675ffc842daea1"
        }
      }
    ]
  }
}

JPEG

$ docker exec -it -uwww-data nextcloud_app_1 php occ fulltextsearch:document:provider -c agross files 1570975
Document:
{
    "id": "1570975",
    "providerId": "files",
    "access": {
        "ownerId": "agross",
        "viewerId": "",
        "users": [],
        "groups": [],
        "circles": [],
        "links": []
    },
    "modifiedTime": 1568827947,
    "title": "_test\/Untitled-1.jpg",
    "link": "",
    "index": {
        "ownerId": "agross",
        "providerId": "files",
        "source": "files_local",
        "documentId": "1570975",
        "lastIndex": 0,
        "errors": [],
        "errorCount": 0,
        "status": 28,
        "options": []
    },
    "source": "files_local",
    "info": {
        "share_names": {
            "agross": "_test\/Untitled-1.jpg"
        }
    },
    "hash": "",
    "contentSize": 20,
    "tags": [],
    "metatags": [
        "files_local"
    ],
    "subtags": [],
    "more": [],
    "excerpts": [],
    "score": ""
}
Content:
magicvalue123
1 Part(s)
'comments'    (size: 0)

$ docker exec nextcloud_search01_1 curl -XGET http://localhost:9200/nextcloud/_search -d '{ "query": { "terms": { "_id": ["files:1570975"] } } }' -H 'Content-Type: application/json' | jq
{
  "took": 462,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "nextcloud",
        "_type": "standard",
        "_id": "files:1570975",
        "_score": 1,
        "_source": {
          "owner": "agross",
          "groups": [],
          "circles": [],
          "metatags": [
            "files_local"
          ],
          "source": "files_local",
          "title": "_test/Untitled-1.jpg",
          "users": [],
          "content": "magicvalue123",
          "tags": [],
          "attachment": {
            "content_type": "text/plain; charset=ISO-8859-1",
            "language": "et",
            "content_length": 14
          },
          "provider": "files",
          "subtags": [],
          "parts": {
            "comments": ""
          },
          "links": [],
          "share_names": {
            "agross": "_test/Untitled-1.jpg"
          },
          "hash": "87f4070c15672652a26b56bf86ceca4e"
        }
      }
    ]
  }
}

fulltextsearch:document:provider -c displays wrong content

Description

Contributor guide

First steps

Research direction

Assessment