This report presents additional selected results from our analysis of data provided by the Department of Health (DoH) in relation to the delivery of State-managed adult mental health services from 2013 to 2017. It provides further detail about the analysis that was used in the Auditor General’s Access to State-Managed Adult Mental Health Services report (Report 4 of 2019-20) and includes some analysis that was not included.
As part of that performance audit, we undertook an extensive data analytics exercise. As well as providing further information to Parliament, the results documented in this report may be of interest to a range of stakeholders including mental health service providers, advocacy bodies, consumers, carers, clinicians and research analysts.
The data analysis covered a 5 year period from 1 January 2013 to 31 December 2017. The DoH extracted the data from its systems and provided it to us in September 2018. The data covers mental health services delivered by public Health Service Providers (HSPs). This data includes admissions to hospitals, presentations to emergency departments (EDs) and contacts with community treatment services.
Our analysis was conducted on the specific data set at a point in time and consequently should not be seen as a complete or definitive representation of all aspects of Western Australia’s (WA’s) mental health services. The analysis and results were designed to answer a specific set of questions that related to the audit objective and scope. The analysis also tested the feasibility of using the DoH’s mental health data to provide new insights into how people access mental health services in WA.
In conducting the audit, the audit team engaged in an extensive consultation process that included staff from the DoH, the HSPs, the Mental Health Commission (MHC) and the Office of the Chief Psychiatrist. The consultation process provided assurance that the information, analytical approach and results were appropriate and soundly based.
The analysis also involved a significant contribution from the managers of the DoH Data Collections, their analysts and the Data Linkage Branch within the DoH. Their assistance in refining the approach and their role in extracting and linking the correct data was critical, and underpinned our analysis. In addition, their continued engagement in providing clarification and explanations helped ensure the data and analysis was appropriately interpreted. A reference group of psychiatrists, analysts and administrators within the HSPs also provided explanations and information to ensure we understood and appropriately interpreted the data.
This supporting data report was produced at an estimated cost of $175,000, which is in addition to the costs reported in Access to State-Managed Adult Mental Health Services (Report 4 of 2019-20). The data warehouse has now been shared with the DoH.
We engaged a data analytics expert to assist us in adopting a systems engineering approach to:
- execute the data integration
- transform the data to create linked episodes of care or stays
- calculate the measures utilised to support the performance audit report.
This approach ensured the analysis was repeatable, transparent, auditable, validated and documented. It involved developing a data warehouse to which we then applied specific queries. These were designed to provide audit evidence to support our Report 4 of 2019-20, and they also produced the additional information provided in this report. Specifically, our approach involved:
- The DoH identifying people who accessed mental health care in hospital beds, EDs and through Community Mental Health Teams (CMHTs). The DoH then ensured the data was anonymous by allocating a unique, encrypted reference number or identifier to each individual, and removing any other identifying information. The identifier linked the mental health care events across the 3 services we looked at as part of this data analysis. This allowed us to use individual instances of care across different care settings to map journeys of people over the 5 year period from 1 January 2013 to 31 December 2017.
- In consultation with our stakeholders, developing tests or measures that allowed us to identify instances where the patterns or pathways did not reflect the intended models of care for the mental health services being delivered.
- Verifying the results we produced with the data analysts, psychiatrists, psychologists, mental health nurses, advocacy bodies and administrators from the MHC, the DoH and the HSPs.
The data provided by the DoH was sourced from 4 discrete, standalone DoH data collections plus data from the Department of Justice which allowed us to ascertain if and when there was a death. As the State does not have a single unique identifier for people across the data sources, the Data Linkage Branch at the DoH routinely links the data using probabilistic matching. The process of probabilistic matching is robust however it is not 100% accurate. When we validated the data to ensure accuracy and completeness, we identified, through feedback from the Data Linkage Branch, a very small number of instances where the unique identifier had linked events for 2 different people. This variance was not material to the overall system-level analysis.
Child and adolescent mental health services were excluded from the analysis, however the data included a small number of children and adolescents that accessed State-managed mental health services. Given the small numbers, this did not materially impact the analysis.
This data analysis exercise was limited by the inability to include data on mental health community support services provided by non-government organisations (NGOs) funded by the MHC. Data for the services comparable to that provided by the DoH on the services delivered by HSPs was not available at the time the analysis was conducted.
In addition to the data around the delivery of mental health services, CMHTs also collect data on the people referred to the service. We requested this data and it was provided by the DoH. However, multiple referrals can be generated via the system’s referral and triage process. This made it difficult to correctly link the first referral within the community treatment service to the next appointment or contact. The data included duplicates that prevented us from merging the referral data into the data warehouse with any degree of confidence. As a result, the referrals data was excluded from the analysis. The DoH currently has a project underway to improve the quality of this referral data and develop performance measurements to report this data.
Our analysis did not include primary health care data (including from general practitioner consultation and treatment) because, as a Commonwealth funded service, the services and data are outside the Auditor General for Western Australia’s access powers and were outside the scope of our audit.
Figures in the audit report Access to State-Managed Adult Mental Health Services were rounded for ease of understanding. In this report, the exact number have been included as appropriate. As a result, the number may differ slightly to the original report.