I also have a Dutch version of this blog
Sometimes you do not succeed in getting one of your articles published in a scientific journal. Even if you are convinced that the article is a valuable contribution to the field, it sometimes happens that your manuscript is not seen as publishable by editor and reviewers. In the first place you have to seek the cause in yourself. Apparently you did not succeed in presenting your ideas in a convincing way. Sometimes, however, it is the case that your article goes against an idea, and reviewers keep defending that idea. That can be frustrating. This is what happened to one of my articles on the concept of “Instructional Efficiency”. That is a measure that has frequently been used within the context of Cognitive Load theory, which has as a central thesis that in instruction you should take in mind that learners have a limited working memory capacity. In principle this is a plausible idea, but the way the proponents of the theory measure Cognitive load and the conclusions they draw are sometimes disputable.
One of the concepts used within CLT is that of “Instructional Efficiency”, also referred to as “Relative Condition Efficiency”. I will plainly refer to this as “Efficiency”. It is supposed to be a measure for the quality of someones knowledge. The higher the efficiency, the better the knowledge. Efficiency is a trade-off between performance on a test and “Mental Effort”. If you perform well on a test with little effort your knowledge is more efficient than of someone with the same performance, using more effort. This line of reasoning can be critiqued, but that is not my point here. I addressed the mathematical form of the efficiency measure.
Efficiency concerns both performance and effort and therefore both must be measured. Performance is usually measured by the score on a test, expressed as a percentage of a maximum score. Cognitive Load researchers usually measure mental effort by letting subjects indicate the amount of effort they spent on a scale from 1-9. Also on this measure it is possible to expres some critical notes. To what extent can people indicate their effort themselves and shouldn’t you involve time on task as well? This discussion is indeed performed elsewhere. The curious thing, however is the way efficiency is computed mathematically. That is done using the following formula:
The reasoning behind the formula is the following: Someone with an average performance(P) and an average effort(R) has efficiency 0. To compute the efficiency of another person, we compute how much he or she deviates from the averages by computing z-scores. A z-score is the difference with the average, divided by the standard deviation, which is a measure for the amount the data points are spread. The point is plotted in a graph and the distance between the point and the line where for which both z-scores are the same is seen as the efficiency. To make that clear I depicted this in a simulation.
In the graph below, two or three conditions can be compared, the conditions may be groups of students following different kinds of instruction. Efficiency is the length of the line between the red points and the line that is drawn under a 45 degree angle. By pressing Simulate, a new data set is generated.
Now for my problems with this way of computing. There are two: first, a graph is confused with a geometrical plane. In a graph the x and y axes have different units. In a plane they both have dimension length and you can measure the distance between points in any direction. In a graph this is impossible because of the differing units. A line under an angle does not have a meaning, it is like adding meters to liters.
The second objection is the fact that the measure is dependent on the way the data is distributed. If you vary the standard deviation with the same averages you see that efficiency also changes and even can change its sign! For instance, try for performance standard deviations of 20, 10 and 1 and see what happens to the efficiency. A measure that depends on distribution cannot be a good measure (unless it is a measure of the distribution itself of course). Also adding a third condition changes the efficiencies of the original two conditions.
Here is the manuscript that has been sent to five journals and was rejected everytime. If your are interested, especially if you have mathematical or statistical background, I would appreciate your comments.