Sometime ago, I fount the paper Learning to “See” Less Than Nothing: Putting Perceptual Skills to Work for Learning Numerical Structure by Jessica M. Tsang, Kristen P. Blair, Laura Bofferding & Daniel L. Schwartz. The paper is not directly related to my research, but I think it is one of the best papers in Mathematics Education I have ever read. Below, I will comment a bit about it.
This is the starting point of the paper:
“Our proposal [...] is that people recruit the distinct perceptuo-motor system of symmetry to make meaning of and to work with integer structure. If true, how can we use this knowledge to help children learn?” (pp. 157).
To answer that question, the authors utilized an experimental design: three groups, each learning integers with a different approach (two common in American textbooks and one emphasizing symmetry). They took a series of measures to ensure the basic premisses of experimental designs (something that is not common in educational research), but what I think makes this paper particularly good is how they defined the null hypothesis. Instead of only comparing the results in a pre and post-test, they used two measures: regular pre and post-tests + a post-test composed of generative questions.
If the results in the regular post-test showed differences between the groups, it would not mean that one of the approaches was better or worst than the others, but it would mean that the quality of the instruction received by the groups varied and this would be a problem in terms of their research question. Therefore, they were expecting similar results in the regular post-test and better performance in the generative questions by the group taught using symmetry.
This was the first time I saw a research using this approach. I think this is very distinctive and improves greatly the quality of the results because it neutralizes the interference of unexpected changes in engagement, excitement, expectations and instruction quality due to simply "being involved in a research project" due to the requirement of similar results in the regular post-test.
Every time I read a paper of a researcher or teacher trying two different approaches in a classroom and simply comparing pre and post-test results, I think: how can I know if the teacher was equally engaged in the lessons? I wouldn't! It is natural to expect that the involvement of the teacher in the research would affect his expectations and performance in the classroom. That is why I think the requirement of similar results in the regular post-test and a second measure to indicate the success of the intervention (generative questions, for instance) sounds very appropriate.
In fact, there are some issues related to the validity of the generative questions, but it is already a step towards more convincing experimental approaches in educational researches directly connected to classrooms.
PS: the paper has other merits beyond what was discussed here. It really worth reading.
Tsang, J. M., Blair, K. P., Bofferding, L., & Schwartz, D. L. (2015). Learning to “See” Less Than Nothing: Putting Perceptual Skills to Work for Learning Numerical Structure. Cognition and Instruction, 33(2), 154–197. http://doi.org/10.1080/07370008.2015.1038539