• Communications of Mathematical Education
• /
• v.28 no.4
• /
• pp.431-455
• /
• 2014

The purpose of this study was to suggest methods to apply generalizability theory to mathematics teacher evaluation using classroom observations in Korea by analysing mathematics teachers in the U.S. using the instructional quality of assessment instrument as an illustrative example. The subjects were 96 teachers participating in Year 3 and Year 4 from the Middle-school Mathematics and the Institutional Setting of Teaching (MIST) project funded by the National Science Foundation since 2007. The MIST project investigates the following question: What does it takes to support mathematics teachers' development of ambitious and equitable instructional practices on a large scale (MIST, 2007). This study examined data based on both the univariate generalizability analysis using GENOVA program and the multivariate generalizability analysis using mGENOVA program. Specifically, this study determined the relative effects of each error source and investigated optimal measuring conditions to obtain the suitable generalizability coefficients. The methodology applied in this study can be utilized to find effective optimal measurement conditions for the mathematics teacher evaluation for professional development in Korea. Finally, this study discussed limitations of the results and suggested directions for future research.

• Journal of Gifted/Talented Education
• /
• v.23 no.5
• /
• pp.671-695
• /
• 2013

This study provides an illustrative example of using the multivariate generalizability theory. Specifically, it investigates relative effects of each error source, and finds optimal measurement conditions for the number of items within each content domain that maximizes the reliability-like coefficients, such as a generalizability coefficient and an index of dependability. The method is based on teacher recommendation letters and self-introduction letters, using an analytic scoring method in the context of selection of mathematically gifted students by observation and nomination. This study analyzed data from the 2011 academic year in the science education institute for the gifted, which is attached to the university located in the Seoul metropolitan area. It should be noted that the optimal scoring structures of this study are not generalizable to other selection instruments. However, the methodology applied in this study can be utilized to find optimal measurement conditions for the number of raters, the number of content domains, and the number of items in other selection instruments self-developed by many institutions including: the education institutes for the gifted at provincial offices of education, gifted classes, and the science education institutes for the gifted attached to universities in general. In addition, the methodology will provide bases for making informed decisions in selection instruments of the gifted based on measurement traits.

• The Mathematical Education
• /
• v.57 no.2
• /
• pp.179-196
• /
• 2018

The purpose of this study was to investigate the measurement equivalence and to suggest application ways in teaching aptitude and personality test results for pre-service mathematics teachers between a graduate school of education and a college of education. This study analyzed the scores of the teaching aptitude and personality test of 36 pre-service mathematics teachers enrolled in a graduate school of education and 111 pre-service mathematics teachers in a college of education by performing a multivariate generalizability analysis. The main results were as follows. First, graduate's pre-service mathematics teachers had a higher level of teaching aptitude and personality than that of college's pre-service mathematics teachers based on the total scores. In addition, graduate's pre-service mathematics teachers had higher levels of teaching aptitude and personality than those of college's pre-service mathematics teachers except for a creativity application domain based on the sub-domain scores. Second, cognitive domains were measured more precisely but affective domains were measured less precisely for graduate's pre-service mathematics teachers than for college's pre-service mathematics teachers. Third, regardless of school levels, Cronbach's ${\alpha}$ values, which might be overestimated by applying the classical test theory, were higher than dependability coefficients. Fourth, this study showed a somewhat negative result in ensuring the measurement equivalence for a problem solving exploration domain. However, regardless of school levels, this study indicated that the overall measurement was generally reliable on composite scores. Based on these results, it was confirmed that multivariate generalizability methodologies' approach can be useful for exploring the measurement equivalence issues. Finally, this study suggests how to utilize the results of the test, how to apply a multivariate generalizability analysis for detecting the measurement equivalence, and how to develop future research based on limitations.

• Journal of Science Education
• /
• v.47 no.2
• /
• pp.211-224
• /
• 2023

The purpose of this study is to suggest how to investigate the reliability of the assessment, which consists of items generated by automatic item generation using empirical example data. To achieve this, we analyzed the illustrative assessment data by applying the multivariate generalizability theory, which can reflect the design of responding to different items for each student and multiple error sources in the assessment score. The result of the G-study showed that, in most designs, the student effect corresponding to the true score of the classical test theory was relatively large after residual effects. In addition, in the design where the content domain was fixed, the ranking of students did not change depending on the item types or items. Similarly, in the design where the item format was fixed, the difficulty showed little variation depending on the content domains. The result of the D-study indicated that the original assessment data achieved a sufficient level of reliability. It was also found that higher reliability than the original assessment data could be obtained by reducing the number of items in the content domains of operation, geometry, and probability and statistics, or by assigning higher weights to the domains of letters and formulas, and function. The efficient measurement conditions presented in this study are limited to the illustrative assessment data. However, the method applied in this study can be utilized to determine the reliability and to find efficient measurement conditions for the various assessment situations using automatic item generation based on measurement traits.