The problem with levels- gaps in basic numeracy skills identified by rigorous diagnostic testing
I have felt for a long time that one of the disadvantages of the current levelling system is that it encourages teachers to constantly teach students mathematical concepts and ideas only at levels equal to or above that which they have recently scored on an assessment. The data-focused, level-centric system only rewards teachers if their students can score marks on higher-level content; there is no explicit incentive for filling gaps in students’ knowledge at lower levels.
Good maths teachers know the importance of their students having a strong knowledge in the foundations of the subject and the long-term benefit of plugging any gaps students have. This year I wanted to be much more systematic in identifying any key gaps students had in their knowledge of the foundations of secondary maths as soon as possible when they arrived in year 7.
What we did
We diagnostically tested every year 7 student in three areas: mental numeracy calculation strategies, timestables and key nodes. For each student, we assessed 30 mental calculation strategies such as: number bonds, reversing an addition sum to make it easier and counting from the smallest number to the largest in subtraction etc. We assessed 30 of the timestables. The top 30 key nodes topics identified from my previous work on understanding the most important topics for students to master prior to studying GCSE maths were also assessed. In total we collected 90 data points for each of 203 year 7 students shortly after they arrived with us in September 2014.
The testing was carried out in a single 50 minute lesson using the QuickKey app available in the App Store. Students were shown a PowerPoint presentation that contained the 90 questions, each with a specified time limit. Teachers click ‘start’ on the presentation and students had to identify the correct answer for each question from five multiple choice answers. We ensured ‘distractor’ answers were placed alongside the correct answer in each question based on common misconceptions. Students each had a grid on a piece of A3 paper in which they had to colour in the correct circle corresponding to their chosen answer for each question. After the assessment the test papers were scanned using the QuickKey app on my iPhone which automatically marked them and recorded the whole cohort’s results in a single spreadsheet file. The scanning took just 3 hours and we obtained over 18,000 data points for the whole cohort from just a 50 minute lesson!
We tested the following number of students in each KS2 sub-level group:
3.0- 4, 3.3- 5, 3.7- 9, 4.0- 23, 4.3- 28, 4.7- 32, 5.0- 30, 5.3- 35, 5.7- 16, 6.0- 21
This graph shows the performance of different KS2 sub-level groups in the three main areas assessed.
The Pearson correlation coefficient for the mental strategies, timestables and key nodes assessments versus KS2 level were 0.72, 0.58 and 0.71 respectively. These correlations were also checked against a KS3 SAT assessment sat by all students during the first half term with us and found to be almost identical. The weakness in the mental calculation strategies of level 3 students is clear to see. Level 4 students were far from being secure in many of the basic mental calculation strategies we would take for granted that they would know. For example, some did not automatically reverse an addition to make it easier, could not calculate a number bond to 100 or tell the time. It is important to state the timing of the questions was reasonably swift during the mental calculation strategies and timestables section of the assessment as we wanted to assess what students could do fluently through recall and fast strategies, rather than what they could do with written calculations if they had a lot of time. During the key nodes assessment we gave students a bit more time and allowed them to use written calculations, but again set the timing such that if they did not show good fluency in choosing and executing the correct strategy they would not have had time to answer the question. So when I claim that some students could not do number bonds to 100, I am saying that they could not do this mentally within approximately 10 seconds. The thinking behind this assessment approach was that in order for these skills not to become barriers to learning and working-memory-consuming difficulties when studying more challenging topics on the GCSE course, we want to assess whether these skills are fluent and can be executed quickly, almost without thinking.
The results of the timestables assessment were better than I expected, particularly for the lower attaining KS2 students. However, when looking into the data it was quite apparent how many of these students who knew their timestables, perhaps did not understand the concepts behind them. For example, they got the questions on understanding multiplication by its link to repeated addition wrong.
To delve a little deeper, the following diagrams show the mean average proportion of students within each KS2 sub-level group that got each of the 90 questions correct.
Mental numeracy calculation strategies
There are many interesting interpretations and observations to be made from these results. I will leave it to you to delve into this as deeply as you wish. I think the diagrams summarise nicely the differences between what level 3, 4, 5 and 6 KS2 to students can do fluently across the topic of number. Broadly speaking: level 3 students have mastery of none of the three areas; level 4 students have some proficiency with mental calculation strategies and timestables, but not the key nodes; level 5 students have reasonably secure mental calculation strategies and timestables and have some proficiency with the key nodes; and level 6 students are broadly secure in mental calculation strategies, timestables and can do many of the key node topics already. It is an obvious statement, but with the strong correlation coefficients already cited, it would appear that what a student can do in number is a strong indicator for what they could do across the curriculum.
These results add further support to my belief about levels hiding gaps in the foundation knowledge of some students. A significant proportion of 4 students struggled to fluently identify number bonds to 100, a level 3 skill. Even many of the level 5 and 6 students did not answer the questions on understanding multiplication as repeated addition or division as the inverse of multiplication correctly. It goes to show- giving a ‘level 6 learning objectives’ sheet to a ‘level 5′ student is not good enough. Perhaps they could already do some level 6 topics, but may have gaps at levels 3 and 4. It must be personalised on a individual student basis.
Has growing up in a levels-based system, where teachers and students are only rewarded for achieving success on content at higher levels in the subject, resulted in oversight/ignorance of the gaps in the students’ foundation knowledge at lower levels? How much easier would learning the more advanced topics be for students if they had comprehensive fluency with these basic skills? The hidden gears need oiling once in a while.
To be clear, this is in no way a reflection on our excellent primary colleagues. They do a brilliant job. They are constrained by a single-minded, level-incentivised, high-stakes system, just like we are in secondary, and they act accordingly to meet the external pressures placed on them. We do the same in Year 11.
A change in mindset is required. If levels are going (and they are!) we must not replace it with a system that has the same flaws. I am certainly not suggesting that we shouldn’t teach students higher level content than they can currently attain; but this must not be just a single-minded focus either. From the diagnostic testing we did this year I have learned that KS3 lessons across the ability spectrum still require systematic, planned, regular practice in building students’ fluency in the foundation topics of number. Fluency (speed and accuracy) is a fitness, it is not binary. Even if you are 100% accurate, you can always be faster then you are at present. If two students can calculate number bonds to 100, but one of them takes five times as long to do it as the other, their learning of higher-level concepts will be all the more difficult for them later in their maths education. We must not fall into the trap of confusing instantaneous performance for retention and transfer – learning. I have written about this extensively before. Learning must not be seen as a checklist of visit-once objectives. Even the highest attaining students need to occassionally revisit some of these elementary topics for which their “fluency fitness” has fallen. A green cell in a spreadsheet indicating that they can do a skill today should not be taken as a proxy that they will be equally as fluent in this skill in six months’ time.
In our KS3 lessons at Wyvern College from September 2015 we will ensure that not only do we strive to raise students’ proficiency with higher-level concepts, but we will also provide short daily exercises that build students’ speed and accuracy in all three areas assessed in our diagnostic trial. It is with great excitement and anticipation that I am going to launch a five-minute-every-lesson, fluency-building product on Great Maths Teaching Ideas this summer. It will provide systematic, rigorous coverage of every topic assessed during our diagnostic trial. Watch out for that! As a consequence of what we learned from the diagnostic testing of our year 7 arrivals this year, I feel so passionately about the need for this product, that it will be made available free for all schools via this website in the next month or two.
Keep an eye out!