The focus of critical care outcomes has shifted from survival to long-term functional status and quality of life [1]. Although individual recovery trajectories for critical illness vary, intensive care unit (ICU) survivors often do not return to their previous state for a long period post-discharge [2]. This gap can be described as Post-Intensive Care Syndrome (PICS), referring to cognitive, physical, or mental health problems that develop or worsen related to ICU treatment [3]. Over 50% of ICU survivors experience varying degrees of PICS, and PICS is known to last for many years [4].

As PICS received more research attention, another issue appeared, in that the measures for PICS evaluation were too diverse [5, 6]. A scoping review of 425 studies on ICU survivorship reported that approximately 250 different tools were used to measure post-ICU outcomes [7]. Excessive diversity of tools to measure a variable can hinder the integration of study findings; therefore, standardization of PICS measurement is necessary. Needham et al. [8] proposed a core outcome set using the Delphi technique for assessing PICS in acute respiratory failure survivors. This set comprises existing tools for evaluating physical, mental, and cognitive functions such as the Hospital Anxiety and Depression Scale (HADS), Impact of Event Scale-Revised (IES-R), Montreal Cognitive Assessment (MoCA), 36-Item Short Form Survey (SF-36), and EuroQol-5 Dimension (EQ-5D). However, the types of tools in the core set differ, and the total item number is still large. In addition, even though they were not developed for critically ill patients, they are commonly used in critical care research. This may lead to measurement validity or sensitivity issues.

Contrastingly, a new PICS questionnaire (PICSq) was developed through in-depth interviews and surveys with 536 ICU survivors, and appropriate reliability and validity were reported [9]. Because this tool was developed based on the experience of ICU survivors, it may be more appropriate to assess PICS than other tools. Additionally, the scope of application is wider, because it is targeted at the general ICU population, rather than a specific group of patients. Clinicians can use PICSq scores to screen for PICS in ICU survivors, evaluate intervention effectiveness, or track long-term outcomes [9]. However, PICSq cut-off values for ICU survivor prognosis such as unplanned hospital readmissions have not been investigated. Thus, defined optimal cut-off values will further improve the applicability and usability of the PICSq.

ICU survivorship is associated with significant hospital resource utilization and costs [10, 11]. According to a United Kingdom national data analysis, 81.7% of survivors experienced more than one hospital readmission within five years, of which 54% were unplanned [11]. Unplanned hospital readmissions are an indicator of cost and quality improvements, placing strain on families, society, and individual survivors [12]. Internationally, ICU survivors spend 18,847 to 148,454 US dollars annually on medical expenses [13]. The complex health and psychosocial issues of ICU survivors, namely, PICS, contribute to early readmission after discharge [14]. Experts also noted that early intervention for PICS is important to lower unplanned hospital readmissions [15]. Providing optimal cut-off values for the PICSq for screening patients at high risk of unplanned hospital readmission may help initiate early intervention.

Additionally, since the PICSq is a newly developed tool, expert consensus is needed on whether all items have the same weight. Weights can be imposed through a subjective method of assessing the importance of each item or through a statistical analysis of responses [16]. The analytic hierarchy process (AHP) is a method of mathematically integrating subjective weights. It allows experts to assign relative weights to each item by repeated ranking and comparisons between pairs of items in a tool based on their experience [17]. The AHP compensates for the risk that statistical analysis results may not be consistent with the weights in the real world [18]. Therefore, it is an appropriate method for evaluating the weight of the PICSq, which is still lacking in clinical and research applications. This study aimed to (1) assign AHP weights for the PICSq items and subscales, and (2) suggest optimal cut-off values for raw and weighted PICSq scores for the screening of the unplanned readmission of ICU survivors.

We analysed AHP weights and cut-off values for each subscale and item to validate the PICSq, a measure of PICS in ICU survivors. The optimal cut-off values for screening unplanned readmissions of ICU survivors were 23.00 for the raw score and 23.73 for the weighted score, with AUCs of .933 and .929, respectively, corresponding to excellent accuracy [23]. There was no difference in accuracy between cut-off values for raw and weighted scores; thus, it may not be necessary to use weighted scores when assessing PICS. The AHP is a multi-criteria decision-making method in which experts assign weights through pairwise comparisons between items in each category [17, 26]. Therefore, the weights in this study could be used by healthcare professionals to determine priorities of the needs or preferences of survivors.

The highest weight was assigned to the cognitive subscale of the PICSq. This can be interpreted that critical care experts perceive cognitive impairment as the most important factor of PICS. In addition, it is consistent with previous studies that reported that cognitive impairment experienced by critical care survivors had a higher incidence and longer duration than other areas of PICS [27, 28, 29]. The major risk factor for cognitive impairment is known as delirium [30], so ICU medical staff apply various interventions to prevent delirium. Recently, the use of ABCDEF Bundle has been recommended [31]. It has been reported that periodic assessment of consciousness and the use of appropriate analgesic and sedatives can reduce delirium [32].

Our expert AHP placed a high weight on items that differed from existing measurement tools commonly used (i.g., MoCA, HADS, measures of activities of daily living [ADL]). For example, the cognitive subscale item with the highest weight was “I am confused with date or time.” This item could be more comprehensive and consistent than the items of the MoCA [8], which is recommended as a post-ICU cognitive evaluation tool, and gives 1 point each for orientation to the current year, month, and date. Most previous studies evaluated survivors' physical impairment using a 6-Minuite Walk Test (6MWT) or assessing ADL [7]. Unlike these, in our study, the item with the highest weight on the physical subscale was “I get tired easily,” corresponding to a symptom survivors experience. The item with the highest weight on the mental subscale, “I have no hope,” is also missing on the HADS, which experts recommend most to measure survivors' anxiety and depression [8]. The cause of these differences may be that the MoCA, ADL assessments, and HADS were not developed for ICU survivors. The weights in this study, along with the PICSq based on ICU survivors' experiences, can help to more appropriately assess survivors' cognitive, physical, and mental health.

Optimal cut-off values for measurement tools need to be determined by considering not only statistical accuracy but also clinical significance and usefulness [23]. In this study, sensitivity and specificity of 23.00 and 23.73 points, respectively, suggested optimal PICS_total and wPICS_total cut-off values were 80.9~85.5%. If we had raised cut-off values, sensitivity could have been higher than 90%; however, specificity would have been lowered to 70%. We determined the optimal cut-off values by considering positive LR, which is the ratio of those who did and did not readmit among survivors above the cut-off. The high sensitivity of the PICSq can make it easier to identify survivors who are at-risk for re-admission, allowing for measures to be taken to reduce clinical and financial burdens [33]. However, we should also pay attention to the relatively low LR of our cut-off values. LR is the best indicator for ruling-in diagnoses, and the criteria for good diagnostic tests is LR > 10 [23]. Compared to the sensitivity, LR is related to the usefulness of diagnostic tests [33]; thus, our findings suggested that the practical usefulness of the PICSq may be low. However, the PICSq is not a diagnostic test, rather it is a screening tool. It is a self-report measure that is free, non-invasive, and requires minimal time and effort. Therefore, low LR is unlikely to interfere with clinical use of the PICSq.

Healthcare providers can use our cut-off values to identify survivors likely to be readmitted and provide them with appropriate interventions. A quarter of our survivors reported unplanned readmissions at the time of the survey. We excluded planned hospitalization, such as chemotherapy or rehabilitation; therefore, readmission incidence appears lower than 70%, as reported in a previous study [34]. Until 12 months post-discharge, ICU survivors' physical and mental health is at low levels and recovers slowly [35]. According to a relevant qualitative study [36], readmission breaks the trajectory of post-ICU recovery. Therefore, it is important to proactively assess unplanned readmission risk and provide appropriate interventions.

One intervention for ICU survivors is post-ICU clinics [27, 37]. In post-ICU clinics, the 6MWT, MoCA, HADS, and assessment of ADL have been used for screenings and follow-up evaluations of survivors [38]. Using the PICSq cut-off values instead of these measures, which require space and assessor training, will increase the effectiveness of screenings, evaluations of interventions, and follow-up management. Specifically, using the cut-off values, nurses can easily screen PICS risk and help provide timely interventions to survivors in need. In addition, cut-off values can be used to evaluate various interventions such as therapeutic communication, family support, and the ICU diary provided for PICS prevention or reduction [39]. Because unplanned hospital readmission is a major long-term outcome indicator of critical care [15, 40], our cut-off values could also contribute to the evaluation and improvement of medical treatment and nursing care in the ICU. Additionally, PICS affects cognitive, physical, and mental health, and survivors may experience impairments in some or all areas [29]. We proposed the cut-off values for each subscale, as well as the PICSq total score, to provide a basis for more individualized assessments and interventions for PICS.

This study is meaningful in that it increased the usability of the PICSq by presenting cut-off values for the risk of unplanned readmission of ICU survivors. We proposed more valid cut-off values for screening hospital readmissions, for which various prevalence rates have been reported for different populations, by considering positive LR, as well as sensitivity and specificity. It is expected that our cut-off values for raw and weighted scores can be used for PICS screenings and intervention evaluations.

This study has some limitations. First, because participants were recruited voluntarily, survivors in relatively poor health would have been excluded. Thus, it is possible that overall PICS and readmission rates were underestimated. Second, participants self-reported their planned/unplanned hospital readmissions, and the accuracy of the reports cannot be guaranteed. In future studies, it is necessary to evaluate readmission by objective methods such as reviewing the participant's medical record. In addition, it is necessary to re-analyze the accuracy of the cut-off values by conducting a prospective study from the time the patient transferred from the ICU to the general inpatient unit. Third, this study was conducted at four university hospitals in three cities in Korea and did not include survivors who could not be discharged from the hospital after the ICU treatment or who had been transferred to other hospitals. Therefore, there may be limitations on the generalization of study results. As measurement accuracy is population dependent [23], repeated studies with different post-intensive care populations and settings are needed.

We calculated the weights of PICSq subscales and items through an expert group AHP, and suggested cut-off values for raw and weighted scores for unplanned readmission risk. The optimal cut-off values considering sensitivity, specificity, and positive LR were 23.00 for the raw score and 23.73 for the weighted score, with AUCs of .933 and .929, respectively, corresponding to excellent accuracy. Based on these findings, we recommend healthcare providers use the PICSq to screen for the risk of unplanned readmission and evaluate relevant interventions for ICU survivors.

Supplementary data to this article can be found online at https://doi.org/10.4040/jkan.20233.

Table 1

Differences between AUCs of the Raw and Weighted Cut-off Values (N = 240)