Is This Health App Any Good?

There are no shortage of evaluation frameworks and review processes for digital mental health apps.  In the UK, the digital health landscape often seems unwieldy and difficult to navigate.  How do we know if a health app is really any good? 

In terms of evaluation, we have the NICE Evidence Standards Framework for Digital Health Technologies [1]; the Digital Technology and Assessment Criteria overseen by  NHS Digital [2]; the ORCHA independent review of health apps [3]; the medical device regulatory framework maintained by the Medicines and Healthcare products Regulatory Authority [4]…to name a few.

We had been running randomised control trials (RCTs) of our apps long before the NICE framework for Digital Health was published in 2018; we’ve been building digital mental health apps since 2009.  But it was still positive news for us when the accumulated evidence for our apps slotted easily into the most stringent requirements of the NICE framework.  The evidence from our RCTs was more than we needed for the NICE framework, but it reassured us that we had been on the right track all along, collecting the right data and evidence to support our patients and clinicians.

In 2020, we submitted our apps to ORCHA for a review of usability, clinical assurance and data security.  We found it a straightforward and well-supported review process and we are proud to have produced the two most highly rated apps on ORCHA for supporting people with psychosis.  We hope this gives our patients and clinicians further peace of mind that our apps are usable and effective, and that it assures them that we take data security and privacy very seriously.

The starting point for any evaluation is to identify and clarify the aims and intended purpose of the product.  For the foreseeable future, the MHRA process is a good and necessary early step for any software that will be used as a medical device, while the NICE standards are a helpful guide to understand and assess the evidence required to justify the product claims.   The ORCHA and DTAC processes provide helpful checklists and reviews against the important areas of usability, clinical assurance, accessibility, interoperability and data security.  More evidence standards and evaluation frameworks will surely come and go over the next 10 years, as digital mental health continues to expand and more apps enter the marketplace.  The current MHRA, NICE, ORCHA and DTAC frameworks and processes are useful guides for assessing digital mental health products for now.

---