appium / appium/java-client

matchImagesFeatures returns not relevant rectangle, findImageOccurrence failes to find matching

Open
#1,088 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

User side problem/problem of an external tool
Dominant language
Java
Stars
1.3k
Forks
752
Avg merge
6d 21h
Merged PRs (30d)
8

Description

Description

Hi,

It's the first time I try to use imageMatching. Something goes wrong. Please, help

I cut from real screenshot (png file) a part with a button and save it as a png. Trying to match the original screenshot with its part as described https://appium.readthedocs.io/en/latest/en/writing-running-appium/image-comparison/

When I try to run matching function matchImagesFeatures() a result with points returns:

[debug] [W3C (7b502ad1)] Calling AppiumDriver.compareImages() with args: ["matchFeatures","iVBORw0KGgoAAAANSUhEUgAAAoAAAARwCAYAAAB6j+UAAACAAElEQVR42uxdB7xcRfV+IbTQQUXsCCh2RQX/9kIvIVQBIbSEXtMLhN4RFRSwN0BpovQO0ntJIKQXSkh/7229dff8zzd3v+XszSPZmAQMzsvv5O7evXdmbpmZb77TOuIoFi9evHjx4sWLl5VFojiWAJJEKqFKIGFSlTiuSC2qiASQqkogtTCUKIqkmsRSrKVSUCmniTs/bJbT2L6dNH9vnOfqq+j+ilTT0JVXSlNXPraVJG2UH7p2RXHVbdFOtDeMo3f9Hnb4F8mLFy9evHjx8t8tSe57qqCupkAqVWCXOAHgihXopQr46kGo4C9yUgtiScJEj08k1PNCd27aKHNpJTsv0TrSqCpJFGhZAHo1rb/ekJorH3UmYeyOTaKwsY3+a+6pB4BevHjx4sWLl5UEAGYAKglrkgQiiqukqqCrnNTdNlRQmISpAkCVaratqaSNrVTfklrwn4uiPC0jbABMlC+uPZnUdV/N1a/IVNsQG8nOBzj0ANCLFy9evHjx4qUNABg1vmdgrq5ASoFfUpNiWpdSmoHAWMEhABgEQDFqsH4AXVJRKWfbWpBIGmbb9iV127pjF0O3zUCdSK2qQvDXBKBx4xhIdn4aegDoxYsXL168ePGyVLZ/2DoGrhI7MFZNYNdXU8mYwDCqS6pADEwcGMFymtn8wd5PEaICwOxcqGczNjFWUBY31LVLkqRxfORsC2th1ACF9Sb4I0iE6rnWOA5q6ez8tKlG9gDQixcvXrx48eKlHQDYAE5g36RaUXBVdU4YxVospVrinDQyFXBNQSDs8gAOE+muxwoCMzAG1S1sBOGIAScSbJdWojhUCdzWtc2xeqljFB3Qc6pqHtNwAtFj6UgSeQDoxYsXL168ePHSLgBMGwAQXr4FqUUFBYAVBX+BA4KwEaw7+7zM1g7sIMDfQomkUM8AWT3InDcy793QbatJY9u24Pyq28K...
[debug] [XCUITest] Executing command 'compareImages'
[debug] [W3C (7b502ad1)] Responding to client with driver.compareImages() result: {"points1":[{"y":540.4632568359375,"x":170.20114135742188},{"y":680.8045654296875,"x":333.23590087890625},{"y":830.103759765625,"x":477.757568359375},{"y":662.888671875,"x":333.23590087890625},{"y":680.8045654296875,"x":304.5704650878906},{"y":97.20000457763672,"x":298.8000183105469},{"y":542.4539184570312,"x":169.20579528808594},{"y":648.555908203125,"x":333.23590087890625},{"y":967.4591064453125,"x":334.4302978515625},{"y":584,"x":222},{"y":437,"x":222},{"y":648.555908203125,"x":304.5704650878906},{"y":641.9867553710938,"x":176.173095703125},{"y":437,"x":533},{"y":498.0622863769531,"x":168.4095458984375},{"y":875,"x":221},{"y":874.800048828125,"x":220.8000030517578},{"y":349.3602294921875,"x":164.22915649414062},{"y":98,"x":299},{"y":523.14453125,"x":490.8959045410156},{"y":69,"x":336},{"y":498.0622863769531,"x":308.15362548828125},{"y":437,"x":105},{"y":728,"x":533},{"y":800.6400146484375,"x":492.4800109863281},{"y":96,"x":318},{"y":437,"x":416},{"y":818.4000244140625,"x":454.8000183105469},{"y":439,"x"...

When I ask for rectangle coordinates I receive strange result:
for Rect1 height=805; width=195; x=336;y=69
for Rect2 height=3; width=15; x=52;y=57

When I try to run matching function findImageOccurrence() a failure result is:

[HTTP] {"mode":"matchTemplate","firstImage":"iVBORw0KGgoAAAANSUhEUgAAAoAAAARwCAYAAAB6j+UAAACAAElEQVR42uxdB7xcRfV+IbTQQUXsCCh2RQX/9kIvIVQBIbSEXtMLhN4RFRSwN0BpovQO0ntJIKQXSkh/7229dff8zzd3v+XszSPZmAQMzsvv5O7evXdmbpmZb77TOuIoFi9evHjx4sWLl5VFojiWAJJEKqFKIGFSlTiuSC2qiASQqkogtTCUKIqkmsRSrKVSUCmniTs/bJbT2L6dNH9vnOfqq+j+ilTT0JVXSlNXPraVJG2UH7p2RXHVbdFOtDeMo3f9Hnb4F8mLFy9evHjx8t8tSe57qqCupkAqVWCXOAHgihXopQr46kGo4C9yUgtiScJEj08k1PNCd27aKHNpJTsv0TrSqCpJFGhZAHo1rb/ekJorH3UmYeyOTaKwsY3+a+6pB4BevHjx4sWLl5UEAGYAKglrkgQiiqukqqCrnNTdNlRQmISpAkCVaratqaSNrVTfklrwn4uiPC0jbABMlC+uPZnUdV/N1a/IVNsQG8nOBzj0ANCLFy9evHjx4qUNABg1vmdgrq5ASoFfUpNiWpdSmoHAWMEhABgEQDFqsH4AXVJRKWfbWpBIGmbb9iV127pjF0O3zUCdSK2qQvDXBKBx4xhIdn4aegDoxYsXL168ePGyVLZ/2DoGrhI7MFZNYNdXU8mYwDCqS6pADEwcGMFymtn8wd5PEaICwOxcqGczNjFWUBY31LVLkqRxfORsC2th1ACF9Sb4I0iE6rnWOA5q6ez8tKlG9gDQixcvXrx48eKlHQDYAE5g36RaUXBVdU4YxVospVrinDQyFXBNQSDs8gAOE+muxwoCMzAG1S1sBOGIAScSbJdWojhUCdzWtc2xeqljFB3Qc6pqHtNwAtFj6UgSeQDoxYsXL168ePHSLgBMGwAQXr4FqUUFBYAVBX+BA4KwEaw7+7zM1g
[debug] [W3C (2641088d)] Calling AppiumDriver.compareImages() with args: ["matchTemplate","iVBORw0KGgoAAAANSUhEUgAAAoAAAARwCAYAAAB6j+UAAACAAElEQVR42uxdB7xcRfV+IbTQQUXsCCh2RQX/9kIvIVQBIbSEXtMLhN4RFRSwN0BpovQO0ntJIKQXSkh/7229dff8zzd3v+XszSPZmAQMzsvv5O7evXdmbpmZb77TOuIoFi9evHjx4sWLl5VFojiWAJJEKqFKIGFSlTiuSC2qiASQqkogtTCUKIqkmsRSrKVSUCmniTs/bJbT2L6dNH9vnOfqq+j+ilTT0JVXSlNXPraVJG2UH7p2RXHVbdFOtDeMo3f9Hnb4F8mLFy9evHjx8t8tSe57qqCupkAqVWCXOAHgihXopQr46kGo4C9yUgtiScJEj08k1PNCd27aKHNpJTsv0TrSqCpJFGhZAHo1rb/ekJorH3UmYeyOTaKwsY3+a+6pB4BevHjx4sWLl5UEAGYAKglrkgQiiqukqqCrnNTdNlRQmISpAkCVaratqaSNrVTfklrwn4uiPC0jbABMlC+uPZnUdV/N1a/IVNsQG8nOBzj0ANCLFy9evHjx4qUNABg1vmdgrq5ASoFfUpNiWpdSmoHAWMEhABgEQDFqsH4AXVJRKWfbWpBIGmbb9iV127pjF0O3zUCdSK2qQvDXBKBx4xhIdn4aegDoxYsXL168ePGyVLZ/2DoGrhI7MFZNYNdXU8mYwDCqS6pADEwcGMFymtn8wd5PEaICwOxcqGczNjFWUBY31LVLkqRxfORsC2th1ACF9Sb4I0iE6rnWOA5q6ez8tKlG9gDQixcvXrx48eKlHQDYAE5g36RaUXBVdU4YxVospVrinDQyFXBNQSDs8gAOE+muxwoCMzAG1S1sBOGIAScSbJdWojhUCdzWtc2xeqljFB3Qc6pqHtNwAtFj6UgSeQDoxYsXL168ePHSLgBMGwAQXr4FqUUFBYAVBX+BA4KwEaw7+7zM1g7sIMDfQomkUM8AWT3InDcy793QbatJY9u24Pyq28K...
[debug] [XCUITest] Executing command 'compareImages'
[debug] [W3C (2641088d)] Encountered internal error running command: Error: Cannot find any occurences of the partial image in the full image. Original error: Error: Cannot find any occurrences of the partial image in the full image above the threshold of 0.5. Highest match value found was 0.3522113859653473
[debug] [W3C (2641088d)] at Object. (/usr/local/lib/node_modules/appium/node_modules/appium-support/lib/image-util.js:443:11)
[debug] [W3C (2641088d)] at Generator.next ()
[debug] [W3C (2641088d)] at asyncGeneratorStep (/usr/local/lib/node_modules/appium/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
[debug] [W3C (2641088d)] at _next (/usr/local/lib/node_modules/appium/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
[debug] [W3C (2641088d)] at

Environment

  • java client: 7.0.0
  • Appium server version: 1.10
  • Desktop OS/version used to run Appium if necessary: both IOS and Anddroid
  • Real device or emulator/simulator: Real device

Details

Please provide more details, if necessary.

Code To Reproduce Issue [ Good To Have ]

			BufferedImage imageBig = ImageIO.read(new File(originalPNG));
			ByteArrayOutputStream outputStreamBig = new ByteArrayOutputStream();
			ImageIO.write(imageBig, "png", outputStreamBig);
			byte[] imageBig64InByte =	Base64.encodeBase64(outputStreamBig.toByteArray());
			
			BufferedImage image = ImageIO.read(new File(buttonPNG));
			ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
			ImageIO.write(image, "png", outputStream);
			byte[] image64InByte =	Base64.encodeBase64(outputStream.toByteArray());
			
			FeaturesMatchingResult result1 = driver
			        .matchImagesFeatures(imageBig64InByte, image64InByte, new FeaturesMatchingOptions()
			                .withDetectorName(FeatureDetector.ORB)
			                .withGoodMatchesFactor(40)
			                .withMatchFunc(MatchingFunction.BRUTE_FORCE_HAMMING)
			                .withEnabledVisualization());
			Rectangle rec = result1.getRect1(); //match on the BIG
			Rectangle rec2 = result1.getRect2();

			OccurrenceMatchingResult result = driver
			        .findImageOccurrence(imageBig64InByte, image64InByte, new OccurrenceMatchingOptions()
			                .withEnabledVisualization());

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided Java reproduction for matchImagesFeatures() and findImageOccurrence(), comparing the full and cropped PNG inputs and their reported rectangles. Inspect the reported image-util.js:443 failure and determine why the matching result is incorrect; done means the rectangle coordinates are relevant and occurrence matching succeeds for the supplied images.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.