matchImagesFeatures returns not relevant rectangle, findImageOccurrence failes to find matching
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 752
- Avg merge
- 6d 21h
- Merged PRs (30d)
- 8
Description
Description
Hi,
It's the first time I try to use imageMatching. Something goes wrong. Please, help
I cut from real screenshot (png file) a part with a button and save it as a png. Trying to match the original screenshot with its part as described https://appium.readthedocs.io/en/latest/en/writing-running-appium/image-comparison/
When I try to run matching function matchImagesFeatures() a result with points returns:
[debug] [W3C (7b502ad1)] Calling AppiumDriver.compareImages() with args: ["matchFeatures","iVBORw0KGgoAAAANSUhEUgAAAoAAAARwCAYAAAB6j+UAAACAAElEQVR42uxdB7xcRfV+IbTQQUXsCCh2RQX/9kIvIVQBIbSEXtMLhN4RFRSwN0BpovQO0ntJIKQXSkh/7229dff8zzd3v+XszSPZmAQMzsvv5O7evXdmbpmZb77TOuIoFi9evHjx4sWLl5VFojiWAJJEKqFKIGFSlTiuSC2qiASQqkogtTCUKIqkmsRSrKVSUCmniTs/bJbT2L6dNH9vnOfqq+j+ilTT0JVXSlNXPraVJG2UH7p2RXHVbdFOtDeMo3f9Hnb4F8mLFy9evHjx8t8tSe57qqCupkAqVWCXOAHgihXopQr46kGo4C9yUgtiScJEj08k1PNCd27aKHNpJTsv0TrSqCpJFGhZAHo1rb/ekJorH3UmYeyOTaKwsY3+a+6pB4BevHjx4sWLl5UEAGYAKglrkgQiiqukqqCrnNTdNlRQmISpAkCVaratqaSNrVTfklrwn4uiPC0jbABMlC+uPZnUdV/N1a/IVNsQG8nOBzj0ANCLFy9evHjx4qUNABg1vmdgrq5ASoFfUpNiWpdSmoHAWMEhABgEQDFqsH4AXVJRKWfbWpBIGmbb9iV127pjF0O3zUCdSK2qQvDXBKBx4xhIdn4aegDoxYsXL168ePGyVLZ/2DoGrhI7MFZNYNdXU8mYwDCqS6pADEwcGMFymtn8wd5PEaICwOxcqGczNjFWUBY31LVLkqRxfORsC2th1ACF9Sb4I0iE6rnWOA5q6ez8tKlG9gDQixcvXrx48eKlHQDYAE5g36RaUXBVdU4YxVospVrinDQyFXBNQSDs8gAOE+muxwoCMzAG1S1sBOGIAScSbJdWojhUCdzWtc2xeqljFB3Qc6pqHtNwAtFj6UgSeQDoxYsXL168ePHSLgBMGwAQXr4FqUUFBYAVBX+BA4KwEaw7+7zM1g7sIMDfQomkUM8AWT3InDcy793QbatJY9u24Pyq28K...
[debug] [XCUITest] Executing command 'compareImages'
[debug] [W3C (7b502ad1)] Responding to client with driver.compareImages() result: {"points1":[{"y":540.4632568359375,"x":170.20114135742188},{"y":680.8045654296875,"x":333.23590087890625},{"y":830.103759765625,"x":477.757568359375},{"y":662.888671875,"x":333.23590087890625},{"y":680.8045654296875,"x":304.5704650878906},{"y":97.20000457763672,"x":298.8000183105469},{"y":542.4539184570312,"x":169.20579528808594},{"y":648.555908203125,"x":333.23590087890625},{"y":967.4591064453125,"x":334.4302978515625},{"y":584,"x":222},{"y":437,"x":222},{"y":648.555908203125,"x":304.5704650878906},{"y":641.9867553710938,"x":176.173095703125},{"y":437,"x":533},{"y":498.0622863769531,"x":168.4095458984375},{"y":875,"x":221},{"y":874.800048828125,"x":220.8000030517578},{"y":349.3602294921875,"x":164.22915649414062},{"y":98,"x":299},{"y":523.14453125,"x":490.8959045410156},{"y":69,"x":336},{"y":498.0622863769531,"x":308.15362548828125},{"y":437,"x":105},{"y":728,"x":533},{"y":800.6400146484375,"x":492.4800109863281},{"y":96,"x":318},{"y":437,"x":416},{"y":818.4000244140625,"x":454.8000183105469},{"y":439,"x"...
When I ask for rectangle coordinates I receive strange result:
for Rect1 height=805; width=195; x=336;y=69
for Rect2 height=3; width=15; x=52;y=57
When I try to run matching function findImageOccurrence() a failure result is:
[HTTP] {"mode":"matchTemplate","firstImage":"iVBORw0KGgoAAAANSUhEUgAAAoAAAARwCAYAAAB6j+UAAACAAElEQVR42uxdB7xcRfV+IbTQQUXsCCh2RQX/9kIvIVQBIbSEXtMLhN4RFRSwN0BpovQO0ntJIKQXSkh/7229dff8zzd3v+XszSPZmAQMzsvv5O7evXdmbpmZb77TOuIoFi9evHjx4sWLl5VFojiWAJJEKqFKIGFSlTiuSC2qiASQqkogtTCUKIqkmsRSrKVSUCmniTs/bJbT2L6dNH9vnOfqq+j+ilTT0JVXSlNXPraVJG2UH7p2RXHVbdFOtDeMo3f9Hnb4F8mLFy9evHjx8t8tSe57qqCupkAqVWCXOAHgihXopQr46kGo4C9yUgtiScJEj08k1PNCd27aKHNpJTsv0TrSqCpJFGhZAHo1rb/ekJorH3UmYeyOTaKwsY3+a+6pB4BevHjx4sWLl5UEAGYAKglrkgQiiqukqqCrnNTdNlRQmISpAkCVaratqaSNrVTfklrwn4uiPC0jbABMlC+uPZnUdV/N1a/IVNsQG8nOBzj0ANCLFy9evHjx4qUNABg1vmdgrq5ASoFfUpNiWpdSmoHAWMEhABgEQDFqsH4AXVJRKWfbWpBIGmbb9iV127pjF0O3zUCdSK2qQvDXBKBx4xhIdn4aegDoxYsXL168ePGyVLZ/2DoGrhI7MFZNYNdXU8mYwDCqS6pADEwcGMFymtn8wd5PEaICwOxcqGczNjFWUBY31LVLkqRxfORsC2th1ACF9Sb4I0iE6rnWOA5q6ez8tKlG9gDQixcvXrx48eKlHQDYAE5g36RaUXBVdU4YxVospVrinDQyFXBNQSDs8gAOE+muxwoCMzAG1S1sBOGIAScSbJdWojhUCdzWtc2xeqljFB3Qc6pqHtNwAtFj6UgSeQDoxYsXL168ePHSLgBMGwAQXr4FqUUFBYAVBX+BA4KwEaw7+7zM1g
[debug] [W3C (2641088d)] Calling AppiumDriver.compareImages() with args: ["matchTemplate","iVBORw0KGgoAAAANSUhEUgAAAoAAAARwCAYAAAB6j+UAAACAAElEQVR42uxdB7xcRfV+IbTQQUXsCCh2RQX/9kIvIVQBIbSEXtMLhN4RFRSwN0BpovQO0ntJIKQXSkh/7229dff8zzd3v+XszSPZmAQMzsvv5O7evXdmbpmZb77TOuIoFi9evHjx4sWLl5VFojiWAJJEKqFKIGFSlTiuSC2qiASQqkogtTCUKIqkmsRSrKVSUCmniTs/bJbT2L6dNH9vnOfqq+j+ilTT0JVXSlNXPraVJG2UH7p2RXHVbdFOtDeMo3f9Hnb4F8mLFy9evHjx8t8tSe57qqCupkAqVWCXOAHgihXopQr46kGo4C9yUgtiScJEj08k1PNCd27aKHNpJTsv0TrSqCpJFGhZAHo1rb/ekJorH3UmYeyOTaKwsY3+a+6pB4BevHjx4sWLl5UEAGYAKglrkgQiiqukqqCrnNTdNlRQmISpAkCVaratqaSNrVTfklrwn4uiPC0jbABMlC+uPZnUdV/N1a/IVNsQG8nOBzj0ANCLFy9evHjx4qUNABg1vmdgrq5ASoFfUpNiWpdSmoHAWMEhABgEQDFqsH4AXVJRKWfbWpBIGmbb9iV127pjF0O3zUCdSK2qQvDXBKBx4xhIdn4aegDoxYsXL168ePGyVLZ/2DoGrhI7MFZNYNdXU8mYwDCqS6pADEwcGMFymtn8wd5PEaICwOxcqGczNjFWUBY31LVLkqRxfORsC2th1ACF9Sb4I0iE6rnWOA5q6ez8tKlG9gDQixcvXrx48eKlHQDYAE5g36RaUXBVdU4YxVospVrinDQyFXBNQSDs8gAOE+muxwoCMzAG1S1sBOGIAScSbJdWojhUCdzWtc2xeqljFB3Qc6pqHtNwAtFj6UgSeQDoxYsXL168ePHSLgBMGwAQXr4FqUUFBYAVBX+BA4KwEaw7+7zM1g7sIMDfQomkUM8AWT3InDcy793QbatJY9u24Pyq28K...
[debug] [XCUITest] Executing command 'compareImages'
[debug] [W3C (2641088d)] Encountered internal error running command: Error: Cannot find any occurences of the partial image in the full image. Original error: Error: Cannot find any occurrences of the partial image in the full image above the threshold of 0.5. Highest match value found was 0.3522113859653473
[debug] [W3C (2641088d)] at Object. (/usr/local/lib/node_modules/appium/node_modules/appium-support/lib/image-util.js:443:11)
[debug] [W3C (2641088d)] at Generator.next ()
[debug] [W3C (2641088d)] at asyncGeneratorStep (/usr/local/lib/node_modules/appium/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
[debug] [W3C (2641088d)] at _next (/usr/local/lib/node_modules/appium/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
[debug] [W3C (2641088d)] at
Environment
- java client: 7.0.0
- Appium server version: 1.10
- Desktop OS/version used to run Appium if necessary: both IOS and Anddroid
- Real device or emulator/simulator: Real device
Details
Please provide more details, if necessary.
Code To Reproduce Issue [ Good To Have ]
BufferedImage imageBig = ImageIO.read(new File(originalPNG));
ByteArrayOutputStream outputStreamBig = new ByteArrayOutputStream();
ImageIO.write(imageBig, "png", outputStreamBig);
byte[] imageBig64InByte = Base64.encodeBase64(outputStreamBig.toByteArray());
BufferedImage image = ImageIO.read(new File(buttonPNG));
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ImageIO.write(image, "png", outputStream);
byte[] image64InByte = Base64.encodeBase64(outputStream.toByteArray());
FeaturesMatchingResult result1 = driver
.matchImagesFeatures(imageBig64InByte, image64InByte, new FeaturesMatchingOptions()
.withDetectorName(FeatureDetector.ORB)
.withGoodMatchesFactor(40)
.withMatchFunc(MatchingFunction.BRUTE_FORCE_HAMMING)
.withEnabledVisualization());
Rectangle rec = result1.getRect1(); //match on the BIG
Rectangle rec2 = result1.getRect2();
OccurrenceMatchingResult result = driver
.findImageOccurrence(imageBig64InByte, image64InByte, new OccurrenceMatchingOptions()
.withEnabledVisualization());
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided Java reproduction for matchImagesFeatures() and findImageOccurrence(), comparing the full and cropped PNG inputs and their reported rectangles. Inspect the reported image-util.js:443 failure and determine why the matching result is incorrect; done means the rectangle coordinates are relevant and occurrence matching succeeds for the supplied images.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100