Automating MacOS Apps Using OCR - Dilato Innovative Technology (Beijing) Limited

AirTest & Appium Mac2Driver

It is a cross-platform UI automated testing framework based on image identification. And it is suitable for games and apps, and the supported platforms are Windows, Android and iOS.

In previous articles we have introduced the Airtest library and appium mac2 driver in detail. Assuming you have gone through all the setup instruction. Now we’ll be able to put them all together to achieve automation MacOS apps using OCR.

Integrate with Appium Mac2Driver

Start appium and connect to server:

Copy to Clipboard

Get scale of resolution and set size for image recognition:

Copy to Clipboard

Overwrite findelement function to support image recognition and return ImageElement:

Copy to Clipboard

try:
	self.mac.implicitly_wait(timeout)
	formatted_locator, message, short_message = self._format_which(which, args)

# find image
	if which.by == By.IMAGE:
		class WaitImageFound:
			def __init__(self, instance):
				self.instance = instance

def __call__(self):
				try:
					screen = aircv.utils.string_2_img(self.instance.mac.get_screenshot_as_png())
					current_file_path = os.path.split(os.path.realpath(__file__))[0]
					po = None
					if isinstance(which.value, list):
						for each in which.value:
							img_path = os.path.abspath(os.path.join(current_file_path, each))
							po = Template(img_path).match_in(screen)
							if po:
								break
					else:
						img_path = os.path.abspath(os.path.join(current_file_path, which.value))
						po = Template(img_path).match_in(screen)
					if po:
						return po[0] / self.instance.scale_height, po[1] / self.instance.scale_width
					else:
						return False
				except:
					return False
		timeout = timeout if timeout else settings.IMPLICITLY_WAIT_TIMEOUT
		x, y = Wait(timeout, wait_interval=settings.WAIT_INTERVAL).until(WaitImageFound(self))
		return MacImageElement(self, (x, y), message, short_message, which)
	# find control
	else:
		if multi:
			class WaitElements:
				def __init__(self, instance):
					self.instance = instance

def __call__(self):
					try:
						return self.instance.find_elements(by=which.by, value=formatted_locator)
					except:
						return False
			elements = Wait(timeout).until(WaitElements(self.mac))
			return [MacUIElement(self, x, message, short_message, which) for x in elements]
		else:
			class WaitElement:
				def __init__(self, instance):
					self.instance = instance

def __call__(self):
					try:
						return self.instance.find_element(by=which.by, value=formatted_locator)
					except:
						return False
			element = Wait(timeout).until(WaitElement(self.mac))
			return MacUIElement(self, element, message, short_message, which)
finally:
	self.mac.implicitly_wait(settings.IMPLICITLY_WAIT_TIMEOUT)

Finnally achieve the operations under ImageElement class:

Copy to Clipboard

After framework encapsulation, write the test script:

Copy to Clipboard

AirTest & Appium Mac2Driver

Integrate with Appium Mac2Driver

Share This Story, Choose Your Platform!

About the Author: Dilato

Related Posts

Beginner’s Guide: Getting Started with Pytest

Continuous Integration and Automated Testing: Building a Seamless Integration Pipeline

Performance Testing and Load Testing: Ensuring Application Reliability and Scalability