This article was produced and financed by Norwegian centre for E-health research - read more
How do we know that a diabetes app works?
The answer is that it is almost impossible to know, because apps are evaluated in different ways. Some researchers in Norway have started to work on these criteria.
The number of mobile apps and online resources for people with diabetes are increasing steadily.
Studies have shown that those who use apps and other technology, often get health benefits in the short and long term. Patients gain more knowledge about their health. For example, people with diabetes learn more about the causes of fluctuations in their blood glucose and about the effect of their diet, which allows them to better cope with the disease.
Sharing one’s own health data with the doctor means that individuals could have treatments better tailored to them. However, to find out if a mobile app has a positive effect on the patient, the app must be evaluated.
The question is: how should these apps be evaluated?
“Researchers definitely do not have a standard way of doing this. They use different methods and criteria, which makes it difficult to compare the apps. Some researchers are concerned with what people think about using apps in general, that is, usability and similar criteria. Others look at the clinical impact, whether they can see changes in the overall health or health outcomes,” says Dillys Larbi, an adviser at the Norwegian Centre for E-health Research.
She and her colleagues have conducted a systematic review of scientific publications from 2015 to 2018, which describes the evaluation of diabetes apps and online resources.
The article was recently published in the Journal of Medical Internet Research (JMIR).
The overall aim of their project, funded by Helse Nord, is to create a user-friendly list of criteria for assessing the effectiveness of apps and online resources for people with diabetes in Norway.
Patients are not always involved
The 31 international studies included in the review had obtained responses from a total of 3689 people. One of the things Larbi and her colleagues wanted to find out was whether people with diabetes were involved in the evaluation process.
How did they feel about using these technologies, and what did they appreciate the most?
“Researchers rarely asked the users about it. But this is very important. It is the people with diabetes who use the technology and know their self-management needs best. Their opinions must be made clearer in research and development work. People with diabetes should not only be observed by researchers, but be actively involved,” says Larbi.
In addition to people with diabetes, who accounted for only 12 % of those surveyed, the studies included feedback from health professionals (26 %), IT developers (13 %), researchers (10 %), parents and other family members (13 %) and others including experts and unspecified persons (26 %).
More than half of the studies dealt with diabetes apps. The other studies had evaluated websites or other content on the internet. In 68 % of the studies, the researchers used questionnaires, and they conducted interviews and user group meetings in the others.
“The studies looked mostly at the cognitive and clinical impact of using these technologies in people with diabetes. In addition, they gathered information about usability. But patients were not asked to evaluate security or privacy,” Larbi points out.
“Privacy-related questions are clearly something we should ask the population about. This is a critical knowledge gap that we must help fill.”
In the review, they also found that the researchers were not particularly interested in topics such as sustainability and interoperability. Interoperability means that a product's interface is standardized, so that it can work with other products or systems.
WHO checklist
In 2016, the WHO gathered several experts who agreed on some common criteria for evaluating technological tools. They then made a checklist for evaluating mobile apps.
The included criteria were:
- Sustainability and scalability
- Available to all users
- Ability to interact with other ICT solutions
- Possibility of customization
- Opportunity to recreate method and development
Dillys Larbi and her colleagues believe the criteria from the WHO are a good starting point.
“In addition, we must work to counteract socio-economic health inequalities, and take into account that patients' gender can play a role when they use mobile technology. We need standardization so that organizations and individuals who want to use the tools can understand the technology,” she says.
Should use mixed methods
The researchers found that studies where the authors had a technology-related background were most focused on the interfaces of computer systems and devices. Health researchers, for their part, wanted to find out what people with diabetes and health workers thought about the technology and what they gained from using it.
“Clinical trials involving people with diabetes were mostly used to find out what effect apps had. Researchers often used questionnaires to achieve this. It is important to remember that there are a number of other methods for studying the effect of technology use,” says Larbi.
If researchers collect both quantitative and qualitative data, it can give a more complete picture of what work best for the end users.
“The best way to evaluate the usefulness of apps and online resources is to involve patients all the way. In addition, we should use objective and subjective methods to dig deep into each question.
Then we can get a comprehensive overview of the effect and potential. Our goal is to contribute to better health tools and services, and patients can and should contribute to this development,” says Dillys Larbi.
Reference:
Dillys Larbi et al.: Methods and Evaluation Criteria for Apps and Digital Interventions for Diabetes Self-Management: Systematic Review. J Med Internet Res. July 2020. DOI: 10.2196/18480
WHO checklist for evaluation of digital tools: Classification of digital health interventions: a shared language to describe the uses of digital technology for health.
