Course Outline
COURSE NUMBER:
|
APD1292H
|
COURSE TITLE:
|
Instrument Design and Analysis [RM]
|
SECTION/COHORT:
|
7130 / Summer 2024
|
2. Course Description
Various quality indicators, assessment tools, and survey instruments are widely used in professional, academic, clinical, and research settings. They are crucial for informing theories and advancing practice. Poor instrument quality and misuse have profound impact on research, policy, practice, and most importantly, the lives of people. This course is designed to introduce core concepts and principles required for high-quality instrument design and evaluation. It is intended to provide a sound foundation for developing instruments for one’s own purpose or evaluating instruments developed by others. The course brings together theoretical foundations and measurement approaches. Building on the key concepts of measurement, students will have the opportunity to gain hands-on experiences with instrument design, analysis, and interpretations. There are four learning modules, each of which consists of two parts: 1) knowledge building and 2) skill building through hands-on data application.
3. Course Delivery Modes
This course was designed to be delivered in an Online Synchronous mode. Classes will be held on Tuesdays and Thursdays ( 18:00 – 21:00).
A. Quercus
Quercus is the primary online learning space for the course. All course activities such as class announcements, course contents, assignment uploads, and feedback sharing, will be through Quercus. The Zoom link for class meetings is available on Quercus.
To access Quercus:
. Go tohttps://q.utoronto.ca
. Use your UTORid and password to log in
. Once you've logged in, you'll be in your "Dashboard"
. You should see a "Course Card" for each of the courses you are or enrolled in
. Click on a "Course Card" to access the course.
. If you do not see your courses listed, tryactivating your UTORidand/orverifying your UTORid
B. Zoom
Online synchronous classes will be held through Zoom. Click the invitation link posted on Quercus. Below are some useful tutorials for our zoom meetings:
Joining a zoom meeting
Main tool bar during a zoom meeting(meeting info, audio/video, screen share, breakout rooms, etc.)
In-meeting chat and reaction
More tutorials videos here for other functions:https://learn-zoom.us/show-me
C. Succeeding and Excelling in Online Learning
Please seeoiseonline.org, a resource put together by members of the OISE community to help students succeed and excel in online learning.
Also, resources available from the School of Graduate Studies:
. Guide to Working from Home for Graduate/Postdoctoral Researchers:
https://studentlife.utoronto.ca/wp-content/uploads/Strategies-for-Online-Learning.pdf
. SGS Graduate Wellness Portal:
https://www.sgs.utoronto.ca/gradhub/resources-supports/#health-wellness
4. Learning Outcomes
By the end of this course, students will be able to:
1. Identify and describe core concepts and principles required for high-quality instrument design and evaluation;
2. Interpret and communicate research about instrument development and validation;
3. Plan ways to develop and/or evaluate data collection instruments and measures used in education and psychology;
4. Evaluate instruments developed by themselves or others by applying analysis techniques based on Critical Test Theory, Factor Analysis, and Item Response Theory.
5. Scope and Sequence of Course Topics
Class Topic Readings Assignments
|
Class 1
Tues. May 7
|
Introduction to
Educational and
Psychological
Measurement
|
Furr Ch. 2 Core Principles, Best Practices, and an Overview of Scale Construction
|
|
Class 2
Thurs. May 9
|
Instrument Design: Questionnaire
|
Furr Ch. 3 Response format and item writing
Kline Ch. 2 Designing and writing items
Pekrun et al. (2011)
|
|
Class 3
Tues. May 14
|
Instrument Design: Tests and Rubrics
|
Spaan (2007)
Moskal & Leydens (2000)
|
|
Class 4
Thurs. May 16
|
Application I
|
Kline Ch. 3 Designing and scoring responses
|
Your Online Instrument
|
Class 5
Tues. May 21
|
Introducing R and R studio
|
Albano Ch. 1Introduction
Online course:Getting Started with R
|
A1 Due
|
Class 6
Thurs. May 23
|
Data Collection and Data Preparation
|
Kline Ch. 4 Collecting data : Sampling and screening
Furr Ch. 6 Threats to psychometric quality
|
|
Class 7
Tues. May 28
|
Classical Test Theory
|
Kline Ch. 5 Classical test theory:
Assumptions, equations, imitations, and item analyses
Kline Ch. 7 Reliability of test scores and test
|
|
6. Course Assignments
While taking this course, the achievement of learning outcomes will be accomplished through the following assignments:
Learning Outcomes Assignment 1 Assignment 2 Assignment 3 Assignment 4
|
1. Identify and describe core
concepts and principles required for high-quality instrument design and evaluation
|
V
|
V
|
V
|
V
|
2. Interpret and communicate research about instrument
development and validation
|
V
|
V
|
V
|
V
|
3. Plan ways to develop and/or evaluate data collection
instruments and measures used in education and psychology
|
V
|
V
|
V
|
|
4. Evaluate instruments developed by themselves or others by
applying analysis techniques based on Critical Test Theory, Factor
Analysis, and Item Response
Theory
|
|
V
|
V
|
V
|
Class participation (20%)
Students are required to attend classes and participate in class discussions and small group activities. Complete attendance and active participation are required.
Assignment 1: Instrument Development (20%)
Deadline : Tues. May 21, 2024 (5pm)
Length: No more than 3 pages (tables and figures included; references and your instrument excluded) The paper should include the following:
1) Description of the target construct, its definition, operationalization, intended use, and structure of the instrument
2) A diagram of the instrument structure
3) The instrument (including intro/instructions page and items) in the appendix (NOT as a link to the online instrument)
Assignment 2: Application of Classical Test Theory (20%)
Deadline : Tues. June 4, 2024 (5pm)
Length: No more than 3 pages (tables and figures included; references excluded)
Students will analyze one of the datasets provided using classical test theory and report their results. The paper should include the following:
1) A brief introduction to the data source
2) Formulate 3 questions requiring answers based on CTT and provide empirical evidence for each question.
3) Use tables and graphs to present empirical evidence and provide your interpretations.
Assignment 3: Application of Factor Analysis (20%)
Deadline : Thurs. June 13, 2024 (5pm)
Length: No more than 4 pages (tables and figures included; references excluded)
Students will perform. factor analysis using one of the datasets provided to investigate the internal structure (dimensionality) of the instrument. The paper should include the following:
1) A brief introduction to the data source
2) Formulate 3 questions requiring answers based on EFA/CFA and provide empirical evidence for each question. These questions should be the building blocks that are linearly linked in order to achieve the assignment’s goal (i.e., instrument validation and, if applicable, suggestions for
improvement).
3) Use tables and graphs to present empirical evidence and provide your interpretations.
Assignment 4: A Critical Review of An Instrument Validation Study (20%)
Deadline : Thurs. June 20, 2024 (5pm)
Length: No more than 4 pages (tables and figures included; references excluded)
Students will identify a widely used instrument of interest and a peer-reviewed published study of its development and/or validation that used either factor analysis or item response theory (or both). Then, prepare a critical review of the selected study with a focus on its method and result interpretation. Use the course reading materials and/or relevant literature to support your opinions. The paper should
include the following:
1) A brief introduction to the instrument (e.g., purpose, context, operationalization) 2) Analytical approaches used to validate the instrument
3) Critical evaluation of the validation study in terms of the concepts and principles covered in the course
4) The paper reviewed (attach as a separate PDF file)
*Note all assignments: please follow the APA 7th style (e.g., double-spaced, pagination, reference list). Do not include your name in the header. No title page required.