• Journal of Korea Multimedia Society
• /
• v.21 no.11
• /
• pp.1305-1316
• /
• 2018

Graphical user interface(GUI) has a very important role to interact with software users. However, designing and coding of GUI are tedious and pain taking processes. In many studies, the researchers are trying to convert GUI elements or widgets to code or describe formally their structures by help of domain knowledge of stochastic methods. In this paper, we propose the GUI elements detection approach based on object detection strategy using deep neural networks(DNN). Object detection with DNN is the approach that integrates localization and classification techniques. From the experimental result, if we selected the appropriate object detection model, the results can be used for automatic code generation from the sketch or capture images. The successful GUI elements detection can describe the objects as hierarchical structures of elements and transform their information to appropriate code by object description translator that will be studied at future.

• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.7
• /
• pp.319-326
• /
• 2016

Given the attributes of mobile apps that shorten the time to make purchase decisions while enabling easy purchase cancellations, usability can be regarded to be a highly prioritized quality attribute among the diverse quality attributes that must be provided by mobile apps. With that backdrop, mobile app developers have been making great effort to minimize usability hampering elements that degrade the merchantability of apps in many ways. Most elements that hamper the convenience in use of mobile apps stem from those potential errors that occur when GUIs are designed. In our previous study, we have proposed a technique to analyze the usability of mobile apps using user behavior logs. We proposes a technique to detect usability hampering elements lying dormant in mobile apps' GUI models by expressing user behavior logs with finite state models, combining user behavior models extracted from multiple users, and comparing the combined user behavior model with the expected behavior model on which the designer's intention is reflected. In addition, to reduce the burden of the repeated test operations that have been conducted by existing developers to detect usability errors, the present paper also proposes a mobile usability error detection automation tool that enables automatic application of the proposed technique. The utility of the proposed technique and tool is being discussed through comparison between the GUI issue reports presented by actual open source app developers and the symptoms detected by the proposed technique.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.20 no.3
• /
• pp.89-95
• /
• 2020

Recently, studies are being conducted to recognize a sketch image of a GUI (Graphical User Interface) based on a deep learning and to make it into a code implemented in an application. UI / UX designers can communicate with developers through storyboards when developing mobile applications. However, UI / UX designers can create different widgets for ambiguous widgets. In this paper, we propose an automatic UI detection method using symbol markers to improve the accuracy of DNN (Deep Neural Network) based UI identification. In order to evaluate the performance with or without the symbol markers, their accuracy is compared. In order to improve the accuracy according to of the symbol marker, the results are analyzed when the shape is a circle or a parenthesis. The use of symbol markers will reduce feedback between developer and designer, time and cost, and reduce sketch image UI false positives and improve accuracy.