Key-important Considerations for HF Studies and Training Assessment

Introduction

Last week started the test phase of a US-based formative human factors study we're managing on behalf of our client, and for which we've developed the study design, Instructions for Use (IFU), and on-device labels. This study is particularly interesting because it assesses, amongst others, the impact on use-related safety and usability of different training levels, the presence or absence of a training device, and the impact of different forms of presenting the training device. Because the identification, evaluation, and specification of training requirements is a common challenge for many manufacturers of medical devices and drug-device combination products, we'd like to share how HF studies can support this process including our most important considerations (which are applicable to any HF study and not only the ones that include the assessment of training requirements).


Understanding the impact of different training measures is essential

Understanding the impact of different training measures such as the provision of user training programs and the availability of training devices or training kits on the use-related safety and usability is a essential for the success of many medical devices and drug-device combination products because:

  • It is of critical importance for optimising the use-related safety and usability of the product, because effective training measures significantly support the systematic and successful mitigation of many use-related risks.

  • It impacts on the definition of the product presentation. Typical examples of product presentation-related aspects that are impacted by training requirements are the design of instructions for use, the need for supplying quick reference cards and demonstration videos, whether formal trainings are to be established, as well as decisions on the number of product units that shall be provided to the user, and whether training devices or training kits need to be made available, how they should be made available, and to whom they should be made available.

  • Depending on the type of product, health authorities may require use-related safety data for the expected instructional and training conditions, especially if similar products are on the market which make use specific training measures.

  • An additional challenge for many manufacturers is the transition from the clinical phase 3 product to the commercial product. This is because commercial settings usually do not allow the implementation of the same level of risk control as in a clinical phase 3 study. In particular, commercial settings usually do not allow the same level of control over the way product-related information is presented to the intended users as in a clinical phase 3 study setting. Furthermore, there may be no possibility to perform any formal training on how to use the product with each user, even though such training has been provided in the clinical phase 3 study setting. Ultimately, such differences lead to the question which measures are feasible for a commercial setting to keep the use-related risks at comparable levels to the clinical phase 3 product. For example, can the provision of a training device for self-guided practice in a commercial setting replace a use-demonstration that has been provided in a clinical phase 3 setting?

Human factors studies provide, alongside other methods such as subject matter expert reviews and competitor analysis, a feasible method for identifying and evaluating the effectiveness of different training measures. The key advantages of human factors studies, if carefully designed and executed, are the validity of the results for the product in question and its intended users and, thus, their predictive value. Furthermore, HF studies generate empirical data that inform the risk management process and support health authority submissions.


Critical aspects for designing successful HF studies

When designing human factors studies, all critical aspect must to be carefully and systematically defined to ensure the validity of the study results. These critical aspects are summarised in Table 1 along with examples of how we are considering these aspects in our HF studies.


Table 1: Critical study design aspects for successful HF studies

Study design aspects

Examples for the consideration of study design aspects

Study participant characteristics: Study participants must be carefully recruited such that they best represent the intended user groups in terms of key important characteristics (and their variation within this group) that have the potential to impact on the use of the product.

Age, gender, handedness, hand size, grip strength, comorbidities, level of experience with similar marketed devices, educational background, certifications, etc.

Environmental characteristics upon product introduction and training: The environment in which the product is introduced and where training takes place must represent the expected use contexts and their variability with regard to all aspects that may impact on the use of the product.

Multiple trainees at a time, duration of the session, form of introduction and training (e.g. online vs. face-to-face), etc.

Study materials: The representativeness of the study materials for the final product design with regard to look, feel, and functionality is essential.

Fully functional devices incl. accompanying materials and packaging vs. non-functional, or partly functional mockups that allow the assessment of specific product aspects to guide design decisions. Standalone instructions for use vs. complete prescribing information leaflets, etc.

Product introduction: The way the product is introduced must be representative of the way the it is expected to be introduced in the real world.

Provision of background information on the product, demonstration of how the product is intended to be used, outlining how the product compares to similar marketed products, etc.

Familiarisation and training procedure: The possibilities to familiarise, the training procedure, and the variability that is expected in the real world must be representative of the expected real world setting.

​Allow to unpack the product and read the printed instructions, watch a demonstration or a training video, prepare the product, perform specific tasks with respect to the intended use of the product, demonstrate correct use to complete training, etc.

Memory decay period between product introduction and training on the one hand, and product use test on the other hand: Though usually largely different from reality, must mimic real world memory decay as realistically as possible.

Allowing participants to take the product home in order to familiarise and practice in their own environment and at their own time, returning the product to include a second period without access to the product to simulate the fact that the point in time the product shall be used may be unpredictable, etc.

Test tasks: Must be selected carefully to include all task that are realistic and relevant for meeting the purpose of the study.

Selection of different product options and alternatives, unpack the product, prepare the product, perform a specific set of the (critical) tasks with respect to the intended use of the product, etc.

Data collection methods: Shall allow the assessment of all use-related risks and user requirements that have been identified, and shall allow the identification of any unanticipated use-related risks and user requirements.

Completion of a root cause analysis for any observation, knowledge task assessments incl. root cause analysis for all aspects that cannot be observed, preference questions and open ended questions to complete user needs and user requirements assessment, etc.

Environmental characteristics upon product use: The environment in which the product is used must represent the expected use contexts and their variability with regard to all aspects that may impact on the use of the product.

Presence of time pressure, bystanders, audio-visual distractions, lighting conditions, furniture, access to materials, etc.

As you may have noticed already, particularly the study design aspects relating to environmental characteristics, study materials, product introduction, familiarisation and training, as well as memory decay period, and test tasks must be systematically defined and thoroughly implemented for identifying and evaluating the effectiveness of different training measures and to support the identification of the most suitable training measures for your product.


Summary

Because a thorough understanding of the impact of training measures such as training programs, training devices, and training kits on the use-related safety and usability is essential, we recommend manufacturers of medical devices and drug-device combination products to identify the possible means for systematically providing patients and users with guidance on how to use their product safely and effectively, and to assess the effectiveness of these means as early as possible during the design and development process through carefully and specifically designed subject matter expert reviews, competitor analysis, and HF studies.


Get in touch with us if you like to find out how we can support your project(s). And while you are here, feel free to visit our website to learn more about us and what we do.


Ergonomics Factory Study Services

This publication is part of our 'EFTY News & Insights' series. This series is intended to provide our clients, partners, and anyone who is curious about us and our work with updates on our journey to help improving lives.


Want to stay up-to-date? Sign-up above or follow us on LinkedIn.