Pandemics that caused great and miscellaneous damage to the whole world have been encountered throughout history [1]. The last pandemic that humanity has faced is the coronavirus disease (COVID-19) pandemic. This virus, which first appeared in China (Wuhan region), spread rapidly worldwide. The situation was declared a pandemic by the World Health Organization (WHO) on March 11, 2020. Our country, Turkey, was ranked sixth in the world among countries with the most confirmed cases. There have been 6 million confirmed cases of COVID-19 with 60 thousand deaths, reported to WHO according to current data in Turkey. The number of people who have received two doses of vaccine in Turkey is only around 20 million (25% of the total population) [2].

COVID-19 is a disease with high morbidity and mortality rates [3, 4, 5]. At the same time, during the pandemic, people can experience many negative emotions, such as the risk of infection, getting sick, fear of death, feeling of helplessness, and stigma. On the other hand, lockdowns on social life, such as the closure of schools and the restrictions on transportation, have caused constant and high levels of stress [4, 5, 6].

In our country, there were many factors affecting stress levels in this situation. Inconsistent quarantine rules applied during the COVID-19 pandemic period, such as no restrictions during weekdays, lockdown during weekends, restrictions on older persons, and no restrictions on middle-aged persons. Most of these inconsistencies were due to social and economic problems. The public did not have sufficient financial resources during this quarantine period. Therefore, there were hundreds of suicides [7]. Inconsistent statements were made by politicians about the use of masks, the ways of reducing the spread of the virus, and the application of the vaccine. Due to the speculations in social media, extreme anxiety prevailed among the public. People began to experience communication problems, increased anger, irritability, depression, and anxiety. The lack of opportunities to engage in social activities outside the home, families were forced to spend more time indoors at home, which led to more tension and irritability. For all these reasons, stress and anxiety have increased in our population [7, 8].

It is known that stress and anxiety play an important role in shaping the behavioral responses of people [9]. People with too little anxiety about a viral outbreak are less likely to be hygienic, less likely to adhere to physical distance, and are less likely to get themselves vaccinated. On the other hand, people with excessive stress and anxiety can interpret mild infections as serious infections, and they may go to hospitals and clinics unnecessarily and also exhibit socially destructive behaviors [10]. Given t he role of a nxiety a nd stress in shaping our behavior, it is critical for healthcare professionals to comprehend this situation. However, it has been reported worldwide that COVID-19 anxiety and stress assessment scales are mostly one-dimensional and have limited psychometric properties [11, 12, 13]. In our country, only a short form of the Coronavirus Anxiety Scale Turkish version is available to evaluate this anxiety level. The scale is quite short, consisting of only five items [14]. A more detailed tool is needed to measure the COVID-related stress. It is predicted that this need can be met by the scale developed by Taylor et al. [11]. In this context, the aim of this study is to adapt the COVID Stress Scale (CSS), which is a comprehensive and multidimensional measurement tool, for Turkish society.

It was determined that the mean age of the participants was 28.4 ± 6.93 (min: 18, max: 55) a nd their duration of education was 15.4 ± 2.33 (5~22) years. Moreover, a great majority of the participants (86.4%) were single and unemployed (75.8%), and more than half of them (62.5%) had an income equal to their expenses. More than half of the participants (67.4%) were living in a city center, 10.6% had at least one child, 16.9% were smokers, 8.5% had chronic diseases such as asthma, hypertension, and heart disease, and 7.2% received support from a psychologist or psychiatrist during the COVID-19 pandemic.

1. Reliability analysis

1) Item analysis

The item-total score correlations of the Turkish version of the CSS are given in Table 1. When the item-total score correlations of 36 items were examined for the reliability study of the Turkish version of the CSS, it was determined that the reliability coefficient varied between .51 and .89 and the correlation between item scores and total scale scores was positive and statistically significant (p < .001) (Table 1).

Table 1
COVID Stress Scale Subscales Item-Subscale Total Score Correlations (N = 360)

• Download Table

When the item-subscale total score correlations of each subscale of the Turkish version of the CSS were examined, it was determined that the reliability coefficient of the 12 items in the COVID danger and contamination subscale ranged between r = .62 and .80, the reliability coefficient of six items in the socioeconomic consequences of COVID subscale ranged between r = .81 and .90, the reliability coefficient of six items in the COVID xenophobia subscale ranged between r = .73 and .93; the reliability coefficient of six items in the traumatic stress due to COVID subscale ranged between r = .67 and .87, and the reliability coefficient of six items in the compulsive checking for COVID subscale ranged between r = .76 and .80, and the correlation coefficients of all items were positive and statistically significant (p < .001) (Table 1).

2) Reliability coefficient of internal consistency

In the analysis made for internal consistency in the reliability study of the Turkish version of the CSS, the Cronbach's alpha reliability coefficients were determined as α = .91 for the COVID danger and contamination subscale, α = .93 for the socioeconomic consequences of COVID subscale, α = .92 for the COVID xenophobia subscale; α = .90 for the traumatic stress due to COVID subscale, α = .86 for the compulsive checking for COVID subscale (Table 1), and α = .93 for the whole scale.

3) Test and retest

In testing the time-invariance of the Turkish version of the CSS, which was adapted to Turkish, test-retest measurements conducted with 24 participants with a two-week interval were evaluated with Pearson product-moment correlation and t-test. When the correlation between the scores obtained from the first and second applications of the Turkish version of the CSS and its subscales was examined with Pearson correlation analysis, the reliability coefficient between the scores of two measurements made for the scale and five subscales varied between .04 and .97, and no correlation was found between the subscales except for the total scores of the scale (Table 2). When the test-retest mean scores of the participants were compared with the independent groups t-test, no statistically significant difference was found between the mean scores (p > .05) (Table 2).

Table 2
Comparison of Test and Re-Test Mean Scores of COVID Stress Scale and Sub-Dimensions, and Correlations (N = 30)

• Download Table

2. Validity analysis

The language validity and content validity of the scale were determined and are stated in the data collection and procedure section.

1) Exploratory factor analysis

First, an EFA was performed to determine the construct validity of the Turkish version of the CSS. In this study, the factor analysis was performed by obtaining data as 10 times the number of items (n = 360) for the construct validity of the Turkish version of the CSS with 36 items. In addition, the Kaiser-Meyer-Olkin (KMO) test was conducted to determine the suitability of the data for factor analysis, and Bartlett's test was applied to determine the significance and zero difference between the variables to be analyzed. The KMO coefficient was found to be .83, the chi-square value for Bartlett's test (χ² = 2749.62; df = 66; p < .001) was highly significant (p < .001), and the data were suitable and sufficient for the factor analysis. The maximum likelihood method and oblique (oblimin) rotation method were used in the factor analysis. It was found that the eigenvalue of the scale with a total of 36 items in the Turkish version of CSS and its subscales was above 1.00, with five factors accounting for 65% of the total variance (Table 3).

Table 3
Exploratory Factor Analysis: Factor Loadings (N = 360)

• Download Table

The items remained in the original and recommended form of the scale, and the factors were named as follows.

(1) COVID danger and contamination

This factor group consists of 12 items, including items 1, 2, 3, 4, 5, 6 (danger) and 19, 20, 21, 22, 23, and 24 (contamination).

(2) Socioeconomic consequences of COVID

This factor group consists of a total of six items including items 7, 8,9,10,11, and 12.

(3) COVID xenophobia

This factor group consists of six items, including items 13, 14, 15, 16, 17, and 18.

(4) Traumatic stress due to COVID

This factor group consists of six items, including items 25, 26, 27, 28, 29, and 30.

(5) Compulsive checking for COVID

This factor group consists of six items, including items 31, 32, 33, 34, and 36 (Table 3).

2) Confirmatory factor analysis

The recommended CFA standard goodness of fit index measures [17] are shown in Table 4. As a result of the five-factor CFA, the fit indices were found to be Chi-square = 546.29 (p < .001), degree of freedom = 156 (χ² = 546.29; df = 156, χ²/df = 3.50), root mean square error of approximation (RMSEA) = .08 (p < .05) standardized root-mean-square residual (SRMR) = .04, comparative fit index (CFI) = .92, non-normed fit index (NNFI) = .95, goodness of fit index (GFI) = .95, and adjusted goodness of fit index (AGFI) = .94. The factor loads obtained as a result of the CFA of all items were determined to be between .50 and .91. Figure 1 shows a CFA diagram.

Figure 1
COVID stress scale five-factor confirmatory factor analysis diagram.
C = Item no; e = Residual covariance matrix; F1 = COVID Danger and contamination; F2 = Socioeconomic consequences of COVID; F3 = COVID xenophobia; F4 = Traumatic stress due to COVID; F5 = Compulsive checking for COVID.

• Download Figure

Table 4
Recommended Standard Goodness of Fit Criteria

• Download Table

In this study, a validity and reliability study of the CSS was conducted, and the Turkish version of the CSS was found to have good psychometric properties. There are a limited number of studies on the stress caused by COVID-19 in Turkey.

In the reliability analysis of the Turkish version of the CSS, test-retest, internal consistency, and item analysis were performed. Test-retest reliability is the power of a measurement tool to provide consistent results from application to application and to show time invariance. The test-retest correlation analysis results of the Turkish version of the CSS showed that there was no statistically significant correlation between the subscales, but there was a strong correlation for the overall scale [15, 16]. Retest reliability is recommended if the measured property has a structure that shows continuity [18]. The fact that there was no statistically significant difference between the subscales in this study showed that it was not affected by time. Another supporting result for the reliability of the scale was that the internal consistency coefficient was statistically significant. To evaluate the internal consistency of the scale, Cronbach's alpha technique, which is suitable for Likert-type scales, was used. It is assumed that the higher the alpha coefficient of the scale, the more consistent the items in the scale, and it is composed of items predicting the components of the same property. The alpha coefficient, which lies between 0 and 1 with the average of the total variances of the items with the overall variance, determines whether the questions in the scale form a whole to explain a homogeneous structure [17, 19]. In the analysis of internal consistency in the reliability study of the Turkish version of the CSS, the Cronbach's alpha reliability coefficient was determined to be at the desired level in five subscales and was highly similar to the internal consistency coefficients of the original scale [11]. The internal consistency coefficients of the subscales of the original scale, which were found to be highly reliable, were found to be between .86~.93 in this study. In addition, they were between .85 and .94, and.86 and .95 in the Canadian and the United States of America samples, respectively [11].

If the items in a scale have equal weights and are in the form of independent units, the correlation coefficient between each item and the total values is expected to be high. The higher the correlation coefficient, the higher the correlation of that item with the property to be measured [17, 19]. Although there is no specific standard regarding when the reliability will be deemed insufficient for the criterion below which the item total score correlation coefficient falls, the correlations are recommended not to be negative and to be above .25 and .30 [20]. It should be noted that the higher the correlation coefficient, the better the reliability of the items [17, 19]. It was observed in the item analysis performed for the item reliability of the Turkish version of CSS that the correlation of the subscale score of all items in the scale with the scale total score was above the value of .30, as stated in the literature. This result means that all items measured the same attitude [20].

However, a high level of agreement among experts is an important result of the content validity of the scale [21]. In conclusion, it can be asserted that the scale has an understandable language structure and content.

As a result of the EFA conducted to determine the construct validity of the Turkish adaptation of the scale, the items having factor loads of < .40, and the items that were involved in more than one factor simultaneously and had the difference between loads in two factors below 0.20, were eliminated and a scale consisting of 36 items as in the original scale was obtained. As a result of the analysis performed in accordance with the literature, a structure with an eigenvalue greater than 1 and a total variance of over 50% was obtained [22, 23]. In EFA, the adequacy of the sample is determined based on the KMO value [20]. KMO value between.90 and .99 is considered as perfect, between .80 and .89 as very good, between .70 and .79 as good, between .60 and .69 as moderate, between .50 and .59 as weak, and below .50 as unacceptable [17]. In this study, the KMO and Bartlett test results are consistent with the EFA results reported in the literature.

In the literature, the use of EFA and CFA analyses in the same sample remains controversial. It is suggested that if the sample size is adequate (> 300), both EFA and CFA analyses can be performed on the same population [24]. Consistent with the literature [14, 25], analyses were performed on the same sample used in this study. Using CFA, the representation level of the items in the determined subscale and their adequacy in explaining the structure were tested [15, 16]. For this purpose, the commonly used goodness of fit tests, chi-square fit statistics, RMSEA, SRMR, CFI, NNFI, GFI, and AGFI were used [15, 16]. The goodness of fit statistics should be at the desired level in the CFA [15, 16]. On the other hand, the fit indices of the 5-item short form of the Coronavirus Anxiety Scale, which was studied in Turkey, are RMSEA = .09, GFI = .98, AGFI = .94, CFI = .98, and NNFI = .97, which were determined to be within acceptable values. This was in agreement with the results of a previous study [14]. The results of the study indicated that all goodness of fit values of the scale adapted χ²/df, SRMR, RMSEA, GFI, AGFI, NNFI, and CFI were above the acceptable limit values. From a holistic framework, the results obtained showed that the adapted model is acceptable.

It is very important to detect symptoms and intervene early before any psychopathology develops in individuals. The Turkish version of the CSS is a valid and reliable measurement tool for assessing the COVID-19 stress scale. It can be a valuable tool for assessing the stress caused by the pandemic (Appendix 1).

However, the Turkish version of the CSS is not aimed at establishing a diagnosis of stress disorder. It should not be ignored that the scale has the function of referring to psychotherapist/psychiatrist experts only in accordance with the obtained scores. Since all data collected for the stress levels caused by COVID-19 are based on personal statements, the error margin should be taken into account. In addition, the data were not collected face-to-face. Individuals with psychopathologies were not identified. There could be a possibility of bias in the answers, and of reliability problems. Furthermore, older individuals with lower socioeconomic levels could not be reached during the online data collection process.

The results of the study showed that the Turkish version of the CSS had sufficient validity and reliability. The internal consistency and validity values were compatible with the values obtained in the original study. Therefore, it can be asserted that the scale has sufficient psychometric properties to evaluate the stress associated with the COVID-19 pandemic. In line with these results, the use of the Turkish version of CSS in determining the stress level for the COVID-19 pandemic can be recommended by considering that it is easy, understandable, and can be filled quickly.

Researchers should focus on the effect of COVID-19 on the vulnerable population living in rural areas and on the lower socioeconomic strata, who face barriers in accessing health care. There is a need to develop mental health interventions that health professionals can use effectively.