SMARE –– Structure Matching and Recognition Engine for Hand-Drawn Chemical Formulas
Expressing chemical compounds in various representations
is challenging. This is especially true for novices, since the task demands
extensive domain-specific knowledge and spatial visualization skills. To
address this challenge, we propose SMARE, our Structure Matching and
Recognition Engine for chemical formulas. It interprets hand-drawn mo-
lecular structures and identifies and highlights errors and thereby is a
fundamental component of educational applications. SMARE leverages
a YOLO (You Only Look Once) model to recognize fundamental entities
in chemical structures such as atoms or bonds. The dataset for train-
ing, validating, and testing the model consists of 1,844 hand-drawn che-
mical molecular images collected from students. SMARE processes the
identified entities to construct an abstract molecular graph. The engine
compares the identified molecular graph against a database of known
molecules and detects errors such as incorrect bonding, and valency vi-
olations. Our fine-tuned YOLO model achieves an accuracy of 93.8 %
in recognizing chemical entities in hand-drawn molecules. SMARE was
tested on 7,909 hand-drawn chemical structures from 519 school students
under real-world conditions, successfully identifying numerous errors in
their chemical drawings. This demonstrates effectiveness of SMARE as
a powerful and practical tool for chemistry education.
Purandare, M., Rothlin, T., Loch, F., Huwer, J., Thoms, L.-J. (2025). SMARE—Structure Matching and Recognition Engine for Hand-Drawn Chemical Formulas. In: Cristea, A.I., Walker, E., Lu, Y., Santos, O.C., Isotani, S. (eds) Artificial Intelligence in Education. AIED 2025. Lecture Notes in Computer Science, vol 15881. Springer, Cham. https://doi.org/10.1007/978-3-031-98462-4_16
Evaluating Usability and User Experience of the OrChemSTAR Educational App Using Eye Tracking
This paper explores how qualitative eye-tracking data can reveal usability issues in educational applications. We studied OrChemSTAR, a multimodal iPad app for chemistry learning that combines handwriting recognition, adaptive feedback, and augmented reality (AR). Using a small-scale qualitative study with eye tracking glasses, we analyzed how students interact with three core learning modes: scanning, AR exploration, and guided practice. Rather than relying on quantitative gaze metrics, we used eye tracking as a tool to reveal patterns of confusion, hesitation, and misalignment between user expectations and system responses. Gaze recordings were paired with observational annotations and user feedback to identify friction points, including misinterpreted feedback icons, interface misalignments, and unmet gesture expectations. Our findings demonstrate that even simple eye-tracking studies can expose critical micro-interactions that impact usability, user experience, and cognitive load. We argue that qualitative eye tracking is a valuable addition to UX methodologies in early-stage educational technologies, especially those involving AR, gesture input, or adaptive interfaces. The article concludes with design implications for the development of learning applications.
Däullary, L., Loch, F., Syskowski, S., Huwer, J., Thoms, L.-J. (2026). Evaluating Usability and User Experience of the OrChemSTAR Educational App Using Eye Tracking. In: Smith, B.K., Borge, M., Sottilare, R.A., Schwarz, J. (eds) HCI International 2025 – Late Breaking Papers. HCII 2025. Lecture Notes in Computer Science, vol 16344. Springer, Cham. https://doi.org/10.1007/978-3-032-13174-4_2
Supporting Employee Engagement and Knowledge Transfer via Gamification in the Context of Sheltered Workplaces: A Literature Review and Interview Study
Gamification describes the use of game design elements in non-game contexts. It is used in various domains to motivate users to exhibit a desired behavior. Gamification is applied in industrial settings, to motivate employees and improve quality. However, gamification with the aim of fostering knowledge sharing and in user groups with disabilities is underexplored and limited to case studies in specific use cases. This paper investigates the applications of gamification in sheltered workplaces for people with disabilities and identifies research directions. It reports a literature review and a qualitative interview study to identify motivations of employees and suggest matching gamification mechanics. Our findings indicate that gamification mechanics for these environments should avoid performance pressure and focus on motivations such as social relatedness and emphasize the achievement of teams. The paper presents and motivates prototypes of these mechanics.
Loch, F., Federspiel, E. (2024). Supporting Employee Engagement and Knowledge Transfer via Gamification in the Context of Sheltered Workplaces: A Literature Review and Interview Study. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. HCII 2024. Lecture Notes in Computer Science, vol 14698. Springer, Cham. https://doi.org/10.1007/978-3-031-60884-1_7
An Intuitive Interface for Technical Documentation Based on Semantic Knowledge Graphs
Loch, F., Stolze, M. (2023). An Intuitive Interface for Technical Documentation Based on Semantic Knowledge Graphs. In: da Silva, H.P., Cipresso, P. (eds) Computer-Human Interaction Research and Applications. CHIRA 2023. Communications in Computer and Information Science, vol 1996. Springer, Cham. https://doi.org/10.1007/978-3-031-49425-3_19
Maintaining technical documentation is a challenge. Products are becoming more complex, product lifecycles are getting shorter, and the number of product variants is increasing. Manuals that guide personnel in the use and maintenance of products are critical to their efficient and safe operation. Authoring system for technical documentation therefore increasingly apply semantic models to control the cost of maintaining technical documentation. Working with formal semantic structures is challenging for technical writers who usually work with plain, written text. This paper presents an intuitive interface for a semantic knowledge graph to facilitate the adoption and use of semantic models in technical documentation. The interface allows working with unstructured text and to postpone its semantification. Users can add semantic annotations in an iterative and incremental way. The interface was developed using a user-centered design process and subjected to an evaluation with technical writers. The results indicate that technical writers could use the prototype successfully and enjoyed the underlying concepts. Further iterations will extend the system and, for example, use artificial intelligence to suggest semantic links to improve the quality of the knowledge graph.
Using Real-time Feedback in a Training System for Manual Procedures
Human workers remain a crucial part of production environments for conducting manual assembly or maintenance procedures, despite increasing automation. These procedures cannot be automated due to small lot sizes and high product variability. Performing manual procedures requires the application of procedural knowledge and motor skills, such as bimanual coordination and complex hand movements. Many training systems for manual procedures have been proposed. However, these systems focus on declarative knowledge about the sequence of work steps. The inherent haptic characteristics and sense for correct tool and component application gets lost. This paper proposes a training system that introduces haptic components for the training of assembly procedures. The proposed training system instructs the user in mounting two physical components by employing haptic and visual interaction. Augmentations and real-time feedback assist the user during the training and enable the assessment of applying accurate torque on screw connections. An evaluation compared the training system against video-based instructions and indicated advantages for the proposed system in terms of objective measures (time on task, precision) and in terms of subjective measures such as usability.
Loch, F., Ziegler, U., Vogel-Heuser, B. (2019). Using Real-time Feedback in a Training System for Manual Procedures. IFAC-PapersOnLine, 52(19), 241–246. https://doi.org/10.1016/j.ifacol.2019.12.089
Hierarchical Gestures: Gestural
Shortcuts for Touchscreen Devices
This report proposes hierarchical gestures, a technique for constructing gesture sets on touchscreen devices in which commands are composed by chaining primitives — for example, Call and Frieder into Call Frieder — each primitive being assigned a gesture, so that a command is invoked by concatenating the corresponding gestures. The structure of such a vocabulary was expected to yield better learnability, higher expressivity, greater efficiency, and higher satisfaction than the standard smartphone interface, while retaining the general advantages of gesture-based interaction for mobile use, namely one-handed operation and low demand on visual attention. To test these hypotheses, an Android application supporting contacts, SMS, and calls was designed and implemented, then evaluated in a controlled laboratory study (n = 14), which supported the claims regarding learnability, efficiency, and satisfaction. The application was revised on the basis of these results and deployed in a field study to assess practical use; the study proceeded successfully, indicated the validity of the hypothesis, and showed the critical components to be robust, although remaining defects impaired interaction and the sample was small (n = 3). A further revision and a field study with more participants are therefore proposed.
Loch, F.(2012) Hierarchical Gestures: Gestural Shortcuts for Touchscreen Devices, Master’s thesis. University of Twente.