International research consortium proposes methodological advancements in the management sciences

Horst Treiblmaier, Head of the Department of International Management, was part of an international group of scholars who recently published a study titled ‘Breaking free from the limitations of classical test theory: Developing and measuring information systems scales using item response theory’’ in the Journal of Information and Management.

Over the past decades researchers have developed sophisticated procedures to build and validate multi-item scales that measure latent constructs, which cannot be measured directly. Examples include an individual’s intelligence or the level of satisfaction with a company’s service. The vast majority of studies use classical test theory (CTT), but this approach suffers from three major theoretical shortcomings: (1) it assumes a linear relationship between the latent variable and observed scores, which rarely represents the empirical reality of behavioral constructs; (2) the true score can either not be estimated directly or only by making assumptions that are difficult to be met; and (3) parameters such as reliability, discrimination, location, or factor loadings depend on the sample being used. To address these issues, a research consortium consisting of Thomas Rusch (WU Vienna), Patrick Mair (Harvard University), Paul Benjamin Lowry (University of Hong Kong) and Horst Treiblmaier (MODUL University) investigated item response theory (IRT) as a collection of viable alternatives for measuring continuous latent variables.

As Dr. Treiblmaier states “it was our goal to demonstrate that alternative forms of measurement exist, which are tested and highly underestimated in the management sciences. I am happy to say that MODUL University is active in the field of methodological research, which builds the foundation for all applied research projects”. The results are now published in the international journal “Information and Management”.

IRT offers several advantages: over CTT: (1) it assumes nonlinear relationships; (2) it allows more appropriate estimation of the true score; (3) it can estimate item parameters independently of the sample being used; (4) it allows the researcher to select items that are in accordance with a desired model; and (5) it applies and generalizes concepts such as reliability and internal consistency, and thus allows researchers to derive more information about the measurement process. In their paper the authors demonstrate how a scale for measuring hedonic aspects of websites is developed under both approaches. The results illustrate how IRT can be successfully applied in research projects and provide better scale results than CTT. The authors conclude by explaining the most appropriate circumstances for applying IRT, as well as the limitations of IRT. A short video which outlines the content of this publication can be found on the publisher's website.