OCR integrate with Appium, Part 2 - Dilato Innovative Technology (Beijing) Limited

This is the second 2-part series on OCR integration with Appium.
In this article, we’ll look at general OCR libraries that cab achieve image recognition and how this library integrate with Appium.

AirTest Library

It is a cross-platform UI automated testing framework based on image identification. And it is suitable for games and apps, and the supported platforms are Windows, Android and iOS.

Install airtest via pip:

Copy to Clipboard

The module airtest.core.api in Airtest provides a series of cross-platform API for calling, including touch operation touch, drag operation swipe, input text operationtext and other operations.

Copy to Clipboard

Integrate with Appium

Start appium and connect to server:

Copy to Clipboard

Get scale of resolution and set size for image recognition:

Copy to Clipboard

Overwrite findelement function to support image recognition and return ImageElement:

Copy to Clipboard

try:
	self.ios.implicitly_wait(timeout)
	formatted_locator, message, short_message = self._format_which(which, args)

# find image
	if which.by == By.IMAGE:
		class WaitImageFound:
			def __init__(self, instance):
				self.instance = instance

def __call__(self):
				screen = aircv.utils.string_2_img(self.instance.ios.get_screenshot_as_png())
				current_file_path = os.path.split(os.path.realpath(__file__))[0]
				img_path = os.path.abspath(os.path.join(current_file_path, which.value))
				po = Template(img_path).match_in(screen)
				if po:
					return po[0] / self.instance.scale_height, po[1] / self.instance.scale_width
				else:
					return False

timeout = timeout if timeout else settings.IMPLICITLY_WAIT_TIMEOUT
		x, y = Wait(timeout, wait_interval=settings.WAIT_INTERVAL).until(WaitImageFound(self))
		return IOSImageElement(self, (x, y), message, short_message, which)
	# find control
	else:
		if multi:
			elements = self.ios.find_elements(by=which.by, value=formatted_locator)
			return [IOSUIElement(self, x, message, short_message, which) for x in elements]
		else:
			element = self.ios.find_element(by=which.by, value=formatted_locator)
			return IOSUIElement(self, element, message, short_message, which)
finally:
	self.ios.implicitly_wait(settings.IMPLICITLY_WAIT_TIMEOUT)

Appium supports both implicit and explicit wait strategies, to make your tests can robustly wait until your reference UI component exist.
So you need to create your own explicit wait strategies to support find_by_image:

Copy to Clipboard

class Wait:
    """
    Polling wait
    """

def __init__(self, timeout, wait_interval=None):
        """
        :param timeout: Wait timeout
        :param wait_interval: Polling wait interval
        """
        self.timeout = timeout
        self.wait_interval = wait_interval if wait_interval else settings.WAIT_INTERVAL
        self.start_time = time.time()

def until(self, method, message=''):
        """
        Call the waiting step circularly, and judge whether to continue the waiting based on the result returned by the step
        If the wait has timed out, terminate the wait and throw an exception
        For waiting steps that do not do exception capture processing, please configure exception capture logic in step
        """
        end_time = time.time() + self.timeout
        while True:
            value = method()
            if value:
                return value
            time.sleep(self.wait_interval)
            if time.time() > end_time:
                break
        raise TimeoutException(message)

Finnally achieve the operations under ImageElement class:

Copy to Clipboard

After framework encapsulation, write the test script:

Copy to Clipboard

AirTest Library

Integrate with Appium

Share This Story, Choose Your Platform!

About the Author: Dilato

Related Posts

Pytest Fixture: Enhancing Test Maintainability

Pytest Plugins: Extending the Functionality of the Testing Framework

Fault Tolerance and Exception Handling in Test Automation