The final section of exercise 4 provides you with an opportunity to “fix” a survey, and improve the quality of data collected and the study validity.
Description of the exercise and background information
On the next page, you will find a short, fictitious survey that needs improvement. Drawing from our discussion in class and from textbook and notes, you have to rewrite the questions and response options to minimize respondent error. As you redesign the survey, please consider the following information regarding the study’s framework:
1. The researchers want to understand teachers’ perceptions of self-efficacy and effectiveness of their practices. Thus, they have developed a scale to assess one specific aspect of their construct of interest: Teacher perception of success with students. They conceptualized “success with students” very broadly, including academic achievement as well as other motivational and affective outcomes.
2. The survey questionnaire includes a few questions about teachers’ demographic characteristics, as the researchers think that age and sex of the respondents as well as their time in service may explain variability in teachers’ perceptions.
Your task
1. Give a short description of the “problems” with the question or response options. Be specific on whether the problem is for respondents, who may struggle understanding or deciding how to answer the question, or for the researchers, who may not be able to use or interpret the results obtained from such answers.
2. Provide a remedy for each question- this includes various things that you may do, such as rewriting the question or response options, breaking up one question into multiple questions, or proposing altogether an alternative and better way to ask and collect the same information.
3. Indicate whether you would eventually suggest eliminating the item, as it may be irrelevant to the scope of the survey, and create simply too much noise, or distraction for the respondents
An example for you before you start the exercise
I provided you with an example of how to approach your task. Here is a mock question, an attempted response, and the feedback received from the course instructor.
How easy is it for you to teach complicated material to your students?
Extremely hard
1 2 3 Neutral 4 5 6 Very easy
7
Student response about the problems: “The question is posing a biased perspective (I am not sure I am using the right word here), as it starts by suggesting that it ought to be easy for a teacher to do something. Also, within the same sentence there is the word “easy” for the teacher and “complicated” for the material taught to the students. He response options are lined up so that they use the word “hard” rather than complicated, so they don’t match the terms in the question. Also, I would expect that on this side of the world it would look smoother to present a continuum that goes from easy-left, to complicated-right. It is also strange that while the teachers are asked “how easy”, the response scale starts with “extremely hard” as the first thing that a respondent reads.
The response options are only partially labelled and they don’t always match the numbers – both of which increase confusion and measurement error. Finally, the act of “teaching complicated material” seems less connected to the construct of success of the student. It seems more to relate to the success of the teachers. A better way to nail the student success would be to assess or to report on how students seem to understand complicated material. In essence, this question, as is, does not relate directly to student success.”
Remedy (or suggested change) proposed by the student:
How confident are you that you can help your student understand complex contents?
Not at all confident Slightly confident Somewhat confident Quite confident Extremely confident
Instructor feedback:
I am very impressed with your ability to identify problems. This is an excellent account of the various problems in the question and in the response options. Definitively you have developed great analytical skills. I have one concern, and is related to the new proposed question: you changed completely the construct to measure! There is somewhat of a distinction to be made between teaching efficacy (i.e., how confident you are in your teaching abilities) and perception of success. For instance, if, as a teacher, I feel that success is all about high student scores on standardized tests (perhaps because my district evaluates me that way) I might be very successful despite not being particularly confident as a teacher (if I teach at a school where students are well prepared to be great test-takers).
Grade: 95%
*****Exercise: Survey on Teacher Perceptions of Student Success*****
Section 1: We’d like to better understand how successful you feel that you are in several different aspects of your teaching. Please indicate the answer that best represents your opinion by filling in the appropriate bubble for each item.
1. I rarely feel like I make a difference in the lives of my students or that my work is important.
Strongly Slightly Slightly Strongly
Agree Neutral Disagree
Agree Agree Disagree Disagree
Problem:
Remedy:
2. To what extent do you have a growth mind-set and feel that you are continually open to becoming a better teacher?
A tremendous
Not at all A little bit Somewhat Quite a bit
amount
Problem:
Remedy:
3. What percentage of your students enjoys learning from you most of the time? ____ % Problem:
Remedy:
4. To what extent can you motivate your students to become life-long learners?
Not at all A little bit Somewhat Quite a bit A tremendous amount
Problem:
Remedy:
5. Leading up to a unit test, how well can you prepare students without “teaching to the test”?
A tremendous
Not at all A little bit Somewhat Quite a bit
amount
Problem:
Remedy:
6. Do you believe your students _________ after spending a year with you?
A Learn more than they would learn with an “average teacher”
B. Learn what they would learn with an “average teacher”
C. Learn less than they would learn with an “average teacher”
Problem:
Remedy:
Please make any additional comments or notes about the scale as a whole which are important to note:
Section 2: Please tell us a bit about yourself so that we have a better understanding of the different people who participated in our study.
7. How long have you been a teacher at this school for? How long have you taught for overall? ________
Problem:
Remedy:
8. During that time, how many families or students have asked to be transferred from your class to another teacher?
A 0-2% of students B. 2-5% of students C. 5-10% of students
D. 10-20% of students E. I don’t know if any have requested a transfer
Problem:
Remedy
9. What is your date of birth?
___ day ___ month ___ year
Problem:
Remedy
10. What grade do you teach primarily (please check all that apply)?
___ K-6 ___ 6-8
___ K-8 ___ 9-12
___ K-12 ___ Other: ____________________
Problem:
Remedy
12. What is your sex?
___ Female
___ Male
___ Other
Problem:
Remedy
