In this instalment of the RPA Implementation Lifecycle series, we will be exploring the Development phase. We will discuss User Interface (UI) Object Selectors which should be a key consideration when developing RPA bots that act through the UI interfaces of the underlying applications
Any automation tool that uses an application’s UI or presentation layer to create an automated procedure requires some sort of mechanism to identify and interact with the UI objects. Interestingly, there is more than one way by which this can be achieved, and we can categorise these mechanisms into different maturity levels as below based on their accuracy and the robustness of the resultant automation.
Level 1– Using attribute-based locators – the automation tool uses certain key attributes of UI objects that allow it to uniquely identify these objects for performing actions. These attributes are nothing but the ones that the developer of the application has defined for the UI objects in the presentation layer code. An example of this could be the “HTML ID” defined for a UI object in the case of a web application. It is generally the case that the tool allows you to configure what attributes to capture and use for different object classes.
Level 2– Using structure-based locators– the automation tool uses the position of the object in the UI layer either based on a hierarchy, for example using locators such as XPath when developing automation on XML DOM based HTML pages, or based on an index relative to other objects that have similar attribute values.
Level 3–Â Using Image-based recognition– the automation tool captures and uses an image of the UI object. Image-based recognition in the recent years has advanced especially in the form of Optical Character Recognition (OCR) which has been made available in UI automation tools and is beneficial in cases where you cannot identify objects either using an attribute or structure-based locators. This is specifically used in case of photos or scanned documents containing typed, handwritten, or printed text. There is obviously a given level of tolerance for pixel by pixel matching when working with images within which the technique is useful, although the degree of accuracy of OCR engines has greatly improved over time.
Level 4–Â Using screen coordinates– the automation tool uses the X and Y coordinates relative to a screen or window to identify a UI object. Such a mechanism is dependent on the screen resolution of the machine used for developing automation and is least reliable to use. It is generally not advisable to use this mechanism unless none of the other 3 options are feasible in a situation.
In summation, most RPA tools now support all four of these categories of UI object selector types when developing automation that is through the UI of an application. Ideally, you would like to apply Level 1 selector mechanism given the RPA tool you’ve chosen supports this for the most commonly used UI technologies for enterprise applications (such as Web, SAP, Java and so on.) and to consider Level 2 to 4 capabilities to deal in situations where Level 1 is not suitable. Level 3 is needed, particularly for unstructured data document processing. Specialised third party OCR engines are now readily available that can be integrated into RPA tools.
Read the other articles in this series from here:
The Robotic Process Automation Index
Contact us at SQA Consulting, to see how we may assist you in developing the necessary skills needed for implementing RPA projects.
© SQA Consulting Ltd
