Psychometric Properties of the Vocational Ability Scale in Individuals with Intellectual Disabilities

Abstract

The purpose of this study was to identify the psychometric properties of the vocational ability scale used in the 8th Panel Survey of Employment for the Disabled in Korea by using the Rasch model. The sample data was collected from 398 individuals with intellectual disabilities. Item fitness, item difficulty, the appropriateness of the rating scale, and the separation index of the vocational ability scale were evaluated. All 15 items show an appropriate fitness level. The analysis of item difficulty indicate that modifications are required. Specifically, the need for the addition of less difficult question items is identified. The use of a 5-point rating scale is shown to decrease the test difficulty in terms of clarity and readability when appropriate and a 4-point modification is also determined as appropriate. With respect to the outcomes of the analysis, the person separation reliability value and separation index are high, and the reliability of the items is also high.

1. INTRODUCTION

Employment for people with disabilities has been a major social issue for a long time: jobs provide people with a sence of purpose, including individuals with intellectual disabilities (IDs). The issue has become particularly prominent with the advancement of relevant legislation [1]. For example, in South Korea, the Special Education Act for Disabled Persons, which was enacted in 2007, addressed the issue by supporting career and vocational education as well as the establishment and progression of the careers of people with IDs. The Act on Employment Promotion and Vocational Rehabilitation of Disabled Persons continues these efforts to promote the employment of persons with disabilities [2].

Despite these efforts to increase employment of individuals with IDs, various studies have reported low employment rates across different countries. It has been revealed that individuals with IDs have employment rates 3–4 lower times than their peers without disabilities, and a large number of individuals with IDs still work in sheltered workshops or segregated settings compared to those with other disabilities [3]. The quality of employment also seems low compared to that of their non-disabled peers: individuals with IDs tend to work in entry level positions, earn lower wages, and work fewer hours [4], [5], [6].

The situation is no different in South Korea. In December 2017, the average employment rate for the general population aged 15 to 64 was reported as 66.5%. On the other hand, the employment rate of individuals with disabilities was 36.5%. Among the 15 types of disability, the employment rates for people with IDs is one of the lowest at 22.9%, representing the most vulnerable type of economic activity when compared with 36.5%, the employment rate of individuals with all types of disabilities [7]. In the second quarter of 2018, the percentage of job seekers with IDs was 2,977 (26.5%) and the number of employed persons was 2,169 (23.0%), ranking only second after persons with physical disabilities, at 36.2% and 34.4%, respectively [8]. The mean duration of maintaining jobs for individuals with IDs was 60 months, which was the fourth shortest duration among the 15 types of disabilities [9].

Previous papers have reported a number of factors that could affect the employment of people with IDs such as the general characteristics of individuals with IDs; psychological factors including the acceptance of disability, self-esteem, and job motivation; family factors such as support from family; and vocational ability. Gender, education level, severity of disability, and basic living security benefits have been reported most frequently as having the biggest impact on factors affecting employment [10], [11], [12]. Self-esteem has been found to be an essential component of employment [13]. Choi and Shin [14] have reported that the higher the family support for work,the higher the probability of employment, which is further supported by Reid and Bray [15] who have reported a similar relationship between the level of family support and employment and job retention. Studies have also shown that the vocational ability of people with IDs is the main variable affecting their employment [16], [17]. Researchers have also pointed out that self-confidence in the ability of people with IDs increases their job motivation [16]. Li [18] has discovered that individuals with IDs with higher-than-average job skills have more employment opportunities than those with belowaverage skills. 

The results of an evaluation on vocational ability provide an important baseline for the development of training programs and necessary services for unemployed persons with disabilities. Although the most accurate method of evaluating vocational abilities is a direct individual assessment using evaluation tools, the results of a survey conducted through systematic and large population data collection are still necessary for policy making In South Korea, the vocational ability scale is being used actively in the panel data of the 8th Panel Survey of Employment for the Disabled (PSED). Despite the importance of vocational abilities in creating services and programs, the psychometric properties of the vocational ability scale used in PSED has not been studied. It is crucial that these studies usereliable and valid measurement tools so that any shortcomings can be identified.

The purpose of this study was to verify the psychometric properties of the vocational ability scale of the PSED using Rasch analysis, which has the advantage of analyzing tool characteristics at the item level and providing information for the elimination or modification of questions in the specific target population and for the revision of the rating scale. Analysis of the items using the classical test such as factor analysis [23] is easy to understand and easy to use. The Rasch analysis is one of the item response theories used for evaluating the suitability of items and the appropriateness of item difficulty [24]. The item response theory can calculate the degree of difficulty and discrimination of the item irrespective of the group in which the test was conducted, and has the field that the student can estimate the number of his/her own ability. The item response theory is that unlike the classical test theory in which tests or items are analyzed by the total test score, it is analyzed by item characteristic curve with unique characteristics of each item. The most frequently used method of item response theory is the Rasch model, which evaluates the appropriateness of the item's suitability and item difficulty [24].

This process helps in obtaining more reliable data. The specific research questions were as follows. First, is the item fitness of the vocational ability scale appropriate for individuals with intellectual disabilities? Second, is the item difficulty of the vocational ability scale appropriate for individuals with intellectual disabilities? Third, are the response categories of the vocational ability scale appropriate for individual with intellectual disabilities? Fourth, is the vocational ability scale reliable when used with individual with intellectual disabilities?

 

2. METHODS

2.1 Data

Data was obtained from the 8th PSED. This is a representative panel survey that identifies the current status and characteristics of the employment of persons with disabilities, as well as their state of economic activity. The contents included in the survey are data that can be used to answer the research questions of this study, including information on disability, economic activity, daily life, gender, age, personal income, and other household information about the disabled persons. Personal interviews were performed by a study investigator, in which interviews were conducted at previously scheduled times. The total number of respondents was 4,577, of whom 398 were individuals with IDs.

 

2.2 Measures 

The vocational ability scale consisted of 15 items. The reliability of this study was Cronbach's α = .947. Responses to items were scored as very low = 1, low = 2, normal = 3, high = 4, and very high = 5. The specific contents of the items were as follows: power or physical strength; movement ability; standing ability; hand movement ability; computer application ability; internet application ability; cooperation ability with coworkers or superiors; customer response ability; technical skills; mathematical computing ability; ability to utilize mechanical equipment; foreign language ability; time management ability; concentration and attention; adaptability to new situations and locations.

 

2.3 Statistical analysis 

The rating scale model in Rasch was selected for calculating the parameters. Rating scale model was followed [29]:

 

\(\operatorname{Pr}\left(X_{n i}=x\right\}=\frac{\exp \sum_{k=0}^{x}\left(\beta_{n}-\left(\delta_{i}+\tau_{k}\right)\right)}{\sum_{j=0}^{m} \exp \sum_{k=1}^{J}\left(\beta_{n}-\left(\delta_{i}+\tau_{k}\right)\right)}\)

 

Where \(𝛿_i\) is the difficulty of item i and \(𝜏_k\)is the kth threshold location of rating scale which is in common to all the items. m is the maximum score and is identical for all the items. \({\tau}_o\) is chosen for computational convenience.

When the infit mean square statistic (infit MNSQ) of each item was smaller than .5 or larger than 1.7, the item fitness was determined as unacceptable [19]. The comparison in connection to the individual ability scores and the item difficulties was analyzed by using the distributions of the items and individuals with IDs. The distributions of the items and persons were included in one grap according to each respective attribute score so as to enable direct comparisons. Because the individual ability scores and the item difficulties were converted equally using the logit scale, it was possible to make direct comparisons, and it was therefore possible to evaluate whether the item difficulties are appropriate for the individuals with IDs to be analyzed. When the ranges of two different distributions are consistent, that is, when the distribution ranges of the item difficulties are similar so that the item difficulties can measure all ranges of the individual attribute scores and difficulties, it can be said that the distribution is adequate [20]. The distributions of item difficulties and personal ability was provided through the person and item map. 

The rating scale was analyzed using a fit index for each rating score. In general, the proficiency estimates and threshold of each rating scale increased with increases in the rating scores. Individual fit values for each rating over 1.5 with a 1.0 standard point suggest the rating scale is not functioning effectively. Criteria suggested by Linacre [22] was used the determining the rating scale appropriateness. The criteria was (1) scale had at least 10 cases per rating category, (2) monotonically increasing average measures across each category, (3) category outfit mean square values less than 2, monotonically increasingstep calibrations, and (4) step calibration differences greaterthan 1.4.

In Rasch analysis, the standard error of measurement is calculated according to all proficiency levels apart from the sample group. This concept is displayed in two criteria: the subject separation index and the item separation index. When these two separation indices are larger, the measurement function level is considered more accurate [21]. Inadequately responding subjects and misfit items were removed based on these criteria.

We used the WINSTEPS 3.6 [22] statistics software to complete the Rasch analysis and to verify the psychometric properties of the vocational ability scale in individuals with IDs.

 

3.  RESULTS

General characteristics of individuals with ID, item fitness, item difficulties using person and item map, rating scale analysis, and separation index were analyzed and presented.

 

3.1 General characteristics of participants

The general characteristics of individuals with ID are shown in Table 1.

 

Table 1. General characteristics of participants

 

A total of 398 participants were included in this study. 253 participants were male, and 145 were female. 203 members were between 15-29 years old, 105 were between 30–39, 49 were between 40–49, 36 were between 50–59, and 5 were over 60 years old. 79 participants had below elementary school education, 93 were middle school graduates, 204 were high school graduates, and 22 had college degree or above. 41 lived with a spouse and 357 lived without a spouse. 150 participants received basic living security, while 245 did not. There were three missing data. 

 

3.2 Item fitness 

The results of estimating the fitness of the 15 items of the vocational ability scale are shown in Table 2. There were no items with an infit MNSQ value greater than 1.7 or less than 0.5. The lowest infit value was for item 9, regarding technical skill, and the highest value was for item 1, regarding strength.

 

Table 2. Item fit statistics according to entry order

 

3.3 Item difficulties 

The results of item difficulties were presented in Fig. 1. When comparing the individual attribute scores and item difficulties, it appeared that item 12 had the highest difficulty, whereas item 1 had the lowest difficulty. No participants exhibited proficiencies higher than the item difficulties. Only92 persons exhibited proficiencies that were between the item difficulties.

 

Fig.1. Person and item map Note. M = mean of item difficulties; T = mean of person abilities; S = Standard deviation

 

3.4 Rating scale analysis 

The results of estimating the 5-point rating scale of the vocational ability scale are shown in Table 3.

 

Table 3. Rating scale analysis of the 5 point scale

 

The vocational ability scale showed that the more the scaling scores increased, the more the average proficiency estimate of the subjects increased. However, the fit index for the fifth point showed a misfit and small observed count (number = 49, 1%). Because the results of rating scale analysis showed a misfit category (level 5), the rescaling was completed. The results are presented in Table 4.

 

Table 4. Rating scale analysis of the 4 point scale 

 

The fit index of the individual scale scores showed values over 1.5 or more based on the value 1.0, which indicated that the applicable scale did not function properly. The analysis of the results showed that there were no misfit scale items. Each scale showed at least 10 case per category. When each estimate monotonically increased, in a similar way to that for the average proficiency estimate of the subjects, the threshold also showed an increasing tendency. Analyzing the scale threshold showed that the threshold was proportional to the increase of the scale scores in all subscales and was greater than 1.4.

 

3.5 Separation index

The separation reliability was shown in Table 5.

 

Table 5. Separation reliability 

 

For the 5-point scale, the separation index was 2.83 and reliability was .89 for persons, and 7.01 and .98 for items, respectively. When a 4-point response scale was used, the separation index was 2.91 and reliability was .89 in persons, and 7.02 and .99 for items, respectively. This was result of changing the category to 4-point scale to show increase in reliability.

 

4. DISCUSSION

Participation of individuals with IDs in studies on employment is a key issue in improving their employment. Employment contributes to their quality of life. Vocational abilities have been reported as one of the major factors affecting employment for individuals with IDs. This study examined the psychometric properties of the vocational ability scale using the Rasch analysis. Validity testing according to the item response theory can help identify the difficulty and the discrimination of each item and can estimate the real ability of the subject based on the analysis results; the advantage of the item characteristic estimation is not affected by the characteristics of the target group [25]. The main results of this study were as follows.

The first research question addressed whether items of the vocational ability scale fit a unidimensional model. All 15 items generated a unidimensional trait based on the results from item-fit statistics. Although the fit index indicated the need to pay attention to item 1 about strength, it was not necessary to revise the item. For clinical observations, a range from 0.5 to 1.7 was recommended for reasonable item MNSQ for infit and outfit [19]. The infit MNSQ of strength showed a slight overfit. Although overfit could result in a misinterpretation that the model worked better than expected, it does not always have this effect on the model [26]. The criteria which determine the item fit based on MNSQ was not absolute and the criteria may vary depending on the purpose of scale. The fit index between 1.5 and 2.0 means that unproductive for construction of measurement, but not degrading [21].

The second research question investigated the item difficulty of the vocational ability scale. The results indicated that the items have higher difficulty levels than the ability level of individuals with IDs. The person means of the vocational ability scale were lower than the item means. This means that the items were relatively difficult for individuals with IDs. Difficult items were those regarding technical skills, mathematics, equipment utilization, and foreign languages. The question of whether these items are appropriate for assessing the vocational abilities of people with IDs can be assessed from the contents of the items with high difficulty levels. Another problem is that many individuals with ID have a lower ability than the difficulty of the item. The results of item difficulty mean that easier items should be inserted into the vocational ability scale for more exact assessment. 

The third research question addressed the suitability of the rating scale of the vocational ability scale. The response scale used when constructing the measure should have clear response levels as well as the potential to measure the desired variable [27]. If the fit index of an individual scale shows a value of 1.5 or higher based on a standard unit of 1.0, it indicates that the scale is not functioning properly and provides information on whether to the scale scores need to be merged into a new response scale [22]. The results showed that the 5-point response scale of the test was inappropriate. When the appropriateness of the response categories was re-evaluated after converting to a 4-point scale, the results suggested that the new rating scale was more appropriate. The 4-point vocational ability scale had at least 10 cases per rating category, monotonically increasing average measures across each category, category outfit mean square values less than 2, monotonically increasing step calibrations, and step calibration differences greater than 1.4 [22].

The final research question verified the person and item separation index and was related to the reliability of scale. The person separation index could provide information about the ability of the scale to classify people. Person separation reliability of 0.9 means that the test could classify 3 or 4 levels of the sample, 0.8 could capture 2 or 3 levels, and 0.5 could capture 1 or 2 levels [28]. Based on the results, the vocational ability scale captured 3 or 4 strata of vocational ability in individuals with IDs

It is very important to assess the vocational ability of people with ID through panel data because panel data collects information on the employment status and the job retention simultaneously, providing indications about the vocational abilities that could improve the employment of people with IDs. Because a panel survey collects data through systematic sampling, social confidence in the results derived through panels is high. Therefore, it is important to look at psychometric properties such as the reliability and validity of the tools used in the panel survey, which is also necessary for reliable data collection. This study is meaningful in that it examined these properties for the vocational ability scale in individuals with IDs and suggested appropriate amendments through the Rasch analysis.

 

5. CONCLUSION

In this study, the panel survey data of assessment performed using the vocational ability scale were analyzed using the Rasch model to determine the psychometric properties of the scale for vocational ability measurement in individuals with IDs. Results for item fit index, item difficulty, rating scale appropriateness, and reliability outcomes were addressed. The results of this study showed that item fit of the vocational ability scale was good. However, modification of item difficulty was required; specifically, easy items should be developed. Modification of the response scale was also needed. Modifying the 5-point scale into a 4-point scale improved the appropriateness of the response scale to an acceptable level. The reliability of vocational ability was good. Further studiesshould be completed to verify the psychometric properties of the scales used in the PSED.