Image Comparison for Dynamic and Multilingual screen

オープン
#2,426 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
静か
技術スタック
java

調査の方向性

SimilarityMatchingOptions と、類似度計算および出現箇所検索に関するリンク先の Appium Images プラグインのドキュメントを確認することから始めます。要求されたテキスト除外、しきい値、座標または要素のマスキング、AI ベースの代替案を比較し、そのうえで、多言語テキストと動的テキストに対して意図した動作を示す、実装可能な範囲とテストを定義します。

索引モデルが issue の本文から書いたものです。

説明

Hi,
I am using Appium Images plugin to perform visual testing, I have found Similarity Calculation and Occurences Lookup to be particularly useful for my use case where I am comparing if a screen matches the baseline, and if a certain element screenshot is present as a subset of screen.
For multilingual apps, I have found one limitation is that it picks up the translation differences which results in reduced value of ComparisonResult. For example,
English Screen

Image

German Screen

Image

Similarity Result

Image

In case, where the baseline screenshot is the same language (English) then the similiarity score comes out to be 0.999995768070221 while when the baseline screenshot is in a different language (German) then the similarity is significantly low 0.8476240038871765 due to the different text on image.

To counter this difference, one approach would be to run tests against different locales in which baseline images stored for comparison correspond to the lanague of app under test. While this approach is helpful, it adds an overhead to run tests on multiple lanaguages and does not scale if no of supported languages is large.

This is also relevant for cases where the text is variable for example, name/address/price of an on-screen element can differ. While mocking the result to have always same values for image comparison can be done, this is not always the possibility and easier to achieve.

For such image comparison, I would like to have a feature that basically ignores any text on the screen when performing image similiarity. Would it be possible to have such an option in SimilarityMatchingOptions where this flag can be passed to either consider or ignore on-screen text together with a threshold value which decided if and how much on-screen text will be taken into account during image similarity check? Another idea is to pass element ID or screen coordinates which should be excluded from image comparison, maybe to mask/blackout them, this way we can manually pass elements which have dynamic text to be excluded for similiarity matching?

Or perhaps if this is not a possibility, would it be possible to achieve this visual testing without relying on external providers and only using Appium MCP with some sort of image comparison tooling either using Images plugin or via LLM native image models. There is a feature request I have also put up for https://github.com/appium/appium-mcp/issues/418. There is a tool available for finding elements using AI but not sure if it can be extnded to also use AI for image comparison.

If there is a rather more suitable approach that already exists, I would be also interested in trying it out. thanks

主要言語
Java
スター
1.3k
フォーク
752
平均マージ
6日 21時間
マージ済み PR(30日)
8

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

appium/java-client のほかの issue

appium/java-client の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。