By Tim Wadsworth
Introduction and Project Goals
In my years of teaching I have often wondered about the relationship between student effort and academic achievement. While I have always assumed that students who put more time and energy into my courses would earn higher grades I have never had much in the way of empirical evidence. I have occasionally looked at the relationship between attendance and final grade but my attendance taking has been more spot checking than a regular role call and thus includes significant measurement error which hinders any analysis. Relying on attendance as a measure of effort also does not measure whether students are doing the readings, watching the outside of class videos, participating in discussions, etc. In addition to significant measurement error there is also the issue of selection bias鈥攎aybe the students who are attending more regularly also share other characteristics that are associated with better grades (e.g. stronger academic background, higher IQ, better study habits, etc). If this is the case it is difficult to know whether it is their attendance, or other characteristics associated with attendance that more strongly influence their grades. Would the 鈥済ood鈥� students do just as well with less effort or participation?
While teaching my first online course in the summer of 2011 I learned that I could track how much time a student is spending participating in various activities on the course website. While this is certainly not a perfect measure of effort, it is an improvement over attendance measures (certainly the type of attendance measures that I have used in the past) as it is more precise and includes a wider variety of different types of course participation. The current project utilizes data that are readily available from the CULearn website to evaluate the relationship between student effort and academic achievement while beginning to control for the possibility of selection bias.
Data and Methods
The main dependent variables I include in the analysis are the final course grade that each student received (Final Grade) as well as their average score on the ten quizzes that they took throughout the course (Average Quiz Grade) and their total assignment score based on the four written assignments they were required to complete (Total Assignment Score). The primary independent variable is time spent logged into the course website (Total Time). This measure includes time spent watching video lectures, participating in online discussions鈥攂oth posting comments and reading the comments of others, and taking and reviewing quizzes. I also explore the influence of other measures of effort that include the number of discussion posts posted by other students that each student read (Messages Read). Lastly, to begin to address the influence of selection I include the student鈥檚 second quiz grade (Quiz 2 Grade) and first assignment grade (Assignment 1 Grade) as initial measures of student success. The purpose of doing so is to be able to examine the influence of effort throughout the course on final grades after controlling for initial achievement or aptitude. Descriptive statistics for each of the variables discussed above can be seen in Table 1.
Table 1: Descriptive Statistics
| 听 |
N |
Minimum |
Maximum |
Mean |
Std. 听听 Deviation |
|---|---|---|---|---|---|
| Final Grade |
29 |
54 |
94 |
77.38 |
11.011 |
| Total Time |
29 |
12:55:20.000 |
77:00:12.000 |
27:32:00.862 |
14:27:22.93930 |
| Messages read |
29 |
22 |
646 |
192.55 |
159.552 |
| Average Quiz Grade |
29 |
41.32 |
97.64 |
73.0352 |
16.16719 |
| Total Assignment Score |
29 |
56.00 |
93.00 |
81.9770 |
9.94067 |
| Quiz 2 Grade |
29 |
0 |
105 |
75.31 |
25.663 |
| Assignment 1 Grade |
29 |
0 |
98 |
72.90 |
26.748 |
| Valid N (listwise) |
29 |
听 | 听 | 听 | 听 |
In order to evaluate the influence of effort on grades I use multiple regression. Multiple regression is an approach to statistical modeling which allows the researcher to evaluate the influence of various independent variables while controlling for the potential influence of all of the other variables that are included in the model. In the first set of models I examine the independent influence of Total Time and Messages Read on Final Grade, Average Quiz Grade, and Total Assignment Score without controlling for other characteristics. In subsequent models I begin to control for the possibility of selection by including measures of initial achievement鈥�Quiz 2 Grade and Assignment 1 Grade.
Findings
Model 1 demonstrates that without controlling for other characteristics there is a statistically significant positive relationship between the total amount of time a student spent on the website and their final grade. 乐播传媒 who spent more time on the website received higher grades.
Model 1
| 听 | Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
|
|---|---|---|---|---|---|---|
| 听 | 听 |
B |
Std. 听听 Error |
Beta |
听 | 听 |
| 1 | (Constant) |
66.463 |
3.878 |
听 |
17.140 |
.000 |
| Total Time |
.000 |
.000 |
.521 |
3.168 |
.004 |
|
a. Dependent Variable: Final Grade
Model 2 suggests that while the relationship is in the expected direction鈥攕tudents who read more of their classmates鈥� comments received higher grades, the relationship is not strong enough to reach conventional levels of statistical significance and thus we cannot be very confident in the finding.
Model 2
| 听 | Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
|
|---|---|---|---|---|---|---|
| 听 | 听 |
B |
Std. 听听 Error |
Beta |
听 | 听 |
| 1 | (Constant) |
73.036 |
3.117 |
听 |
23.434 |
.000 |
| Messages read |
.023 |
.013 |
.327 |
1.797 |
.084 |
|
a. Dependent Variable: Final Grade
Models 3 and 4 suggest that Total Time also influences Average Quiz Grade and Total Assignment Score. This is not too surprising as these indicators are two of the main components that comprise the Final Grade measure. However one of my concerns has been that students who are stronger writers may be doing better regardless of their comprehension of the material. Model 4 suggests that success on written assignments is significantly related to the amount of time spent with the course material.
Model 3
| Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
||
|
B |
Std. 听听 Error |
Beta |
||||
| 1 | (Constant) |
61.357 |
6.170 |
听 |
9.945 |
.000 |
| Total Time |
.000 |
.000 |
.379 |
2.130 |
.042 |
|
| a. Dependent Variable: Average Quiz Grade | ||||||
Model 4
| 听 | Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
|
|---|---|---|---|---|---|---|
| 听 | 听 |
B |
Std. 听听 Error |
Beta |
听 | 听 |
| 1 | (Constant) |
72.697 |
3.574 |
听 |
20.342 |
.000 |
| Total Time |
9.362E-5 |
.000 |
.490 |
2.922 |
.007 |
|
a. Dependent Variable: Total Assignment Score
In the next set of models I examine the relationship between Total Time and three primary measures of student success (Final Grade, Average Quiz Grade, and Total Assignment Score) while controlling for measures of initial success (Second Quiz Grade and First Assignment Grade). This approach begins to address for the possibility of selection bias as discussed above.
Model 5 suggests that Total Time is significantly positively related to Final Grade even after controlling for two indicators of initial success. Looking at the standardized coefficient of .475 suggests that how much time a student spent on the course website was a better predictor of their final grade that their first assignment grade.
Model 5
| Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
||
|
B |
Std. 听听 Error |
Beta |
||||
| 1 | (Constant) |
56.331 |
7.593 |
听 |
7.419 |
.000 |
| Total Time |
.000 |
.000 |
.475 |
3.003 |
.006 |
|
| Quiz 2 Grade |
-.012 |
.068 |
-.029 |
-.180 |
.858 |
|
| Assignment 1 Grade |
.165 |
.064 |
.400 |
2.569 |
.017 |
|
a. Dependent Variable: Final Grade
Model 6 suggests that the same cannot be confidently said when examining average quiz grades. While the relationship between total time and average quiz grade is in the expected direction it does not reach conventional levels of statistical significance.
Model 6
| Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
||
|
B |
Std. 听听 Error |
Beta |
||||
| 1 | (Constant) |
41.394 |
12.759 |
听 |
3.244 |
.003 |
| Total Time |
9.158E-5 |
.000 |
.295 |
1.628 |
.116 |
|
| Quiz 2 Grade |
.099 |
.115 |
.157 |
.858 |
.399 |
|
| Assignment 1 Grade |
.208 |
.108 |
.343 |
1.925 |
.066 |
|
a. Dependent Variable: Average Quiz Grade
However, Model 7 suggests that total time is significantly and positively associated with overall success on writing assignments even after controlling for initial success on both writing assignments and the second quiz.
Model 7
| Model |
Unstandardized 听听 Coefficients |
Standardized 听听 Coefficients |
t |
Sig. |
||
|
B |
Std. 听听 Error |
Beta |
||||
| 1 | (Constant) |
59.375 |
6.113 |
听 |
9.712 |
.000 |
| Total Time |
8.038E-5 |
.000 |
.421 |
2.983 |
.006 |
|
| Quiz 2 Grade |
-.006 |
.055 |
-.016 |
-.114 |
.910 |
|
| Assignment 1 Grade |
.207 |
.052 |
.558 |
4.012 |
.000 |
|
a. Dependent Variable: Total Assignment Score
Discussion
My intention in this project was to make a first cut at using data available on CULearn (and hopefully D2L) to examine the relationship between the amount of effort students put into an online course and their success in that course. Initial analyses suggest that students who spend more time watching online lectures, participating in discussions, and taking and reviewing quizzes earn higher grades in the class. While the possibility of selection bias is always present these findings hold even after controlling for initial class success by including measures of the students鈥� second quiz and first assignment grades. These findings were statistically significant despite having a very small sample (N=29) and including multiple variables in the models.
