# The problem with levels- gaps in basic numeracy skills identified by rigorous diagnostic testing

I have felt for a long time that one of the disadvantages of the current levelling system is that it encourages teachers to constantly teach students mathematical concepts and ideas only at levels equal to or above that which they have recently scored on an assessment. The data-focused, level-centric system only rewards teachers if their students can score marks on higher-level content; there is no explicit incentive for filling gaps in students’ knowledge at lower levels.

Good maths teachers know the importance of their students having a strong knowledge in the foundations of the subject and the long-term benefit of plugging any gaps students have. This year I wanted to be much more systematic in identifying any key gaps students had in their knowledge of the foundations of secondary maths as soon as possible when they arrived in year 7.

**What we did**

We diagnostically tested every year 7 student in three areas: mental numeracy calculation strategies, timestables and key nodes. For each student, we assessed 30 mental calculation strategies such as: number bonds, reversing an addition sum to make it easier and counting from the smallest number to the largest in subtraction etc. We assessed 30 of the timestables. The top 30 key nodes topics identified from my previous work on understanding the most important topics for students to master prior to studying GCSE maths were also assessed. In total we collected 90 data points for each of 203 year 7 students shortly after they arrived with us in September 2014.

The testing was carried out in a single 50 minute lesson using the QuickKey app available in the App Store. Students were shown a PowerPoint presentation that contained the 90 questions, each with a specified time limit. Teachers click ‘start’ on the presentation and students had to identify the correct answer for each question from five multiple choice answers. We ensured ‘distractor’ answers were placed alongside the correct answer in each question based on common misconceptions. Students each had a grid on a piece of A3 paper in which they had to colour in the correct circle corresponding to their chosen answer for each question. After the assessment the test papers were scanned using the QuickKey app on my iPhone which automatically marked them and recorded the whole cohort’s results in a single spreadsheet file. The scanning took just 3 hours and we obtained over 18,000 data points for the whole cohort from just a 50 minute lesson!

We tested the following number of students in each KS2 sub-level group:

3.0- 4, 3.3- 5, 3.7- 9, 4.0- 23, 4.3- 28, 4.7- 32, 5.0- 30, 5.3- 35, 5.7- 16, 6.0- 21

**The results**

This graph shows the performance of different KS2 sub-level groups in the three main areas assessed.

The Pearson correlation coefficient for the mental strategies, timestables and key nodes assessments versus KS2 level were 0.72, 0.58 and 0.71 respectively. These correlations were also checked against a KS3 SAT assessment sat by all students during the first half term with us and found to be almost identical. The weakness in the mental calculation strategies of level 3 students is clear to see. Level 4 students were far from being secure in many of the basic mental calculation strategies we would take for granted that they would know. For example, some did not automatically reverse an addition to make it easier, could not calculate a number bond to 100 or tell the time. It is important to state the timing of the questions was reasonably swift during the mental calculation strategies and timestables section of the assessment as we wanted to assess what students could do fluently through recall and fast strategies, rather than what they could do with written calculations if they had a lot of time. During the key nodes assessment we gave students a bit more time and allowed them to use written calculations, but again set the timing such that if they did not show good fluency in choosing and executing the correct strategy they would not have had time to answer the question. So when I claim that some students could not do number bonds to 100, I am saying that they could not do this mentally within approximately 10 seconds. The thinking behind this assessment approach was that in order for these skills not to become barriers to learning and working-memory-consuming difficulties when studying more challenging topics on the GCSE course, we want to assess whether these skills are fluent and can be executed quickly, almost without thinking.

The results of the timestables assessment were better than I expected, particularly for the lower attaining KS2 students. However, when looking into the data it was quite apparent how many of these students who knew their timestables, perhaps did not understand the concepts behind them. For example, they got the questions on understanding multiplication by its link to repeated addition wrong.

To delve a little deeper, the following diagrams show the mean average proportion of students within each KS2 sub-level group that got each of the 90 questions correct.

**Mental numeracy calculation strategies**

**Timestables**

**Key nodes**

There are many interesting interpretations and observations to be made from these results. I will leave it to you to delve into this as deeply as you wish. I think the diagrams summarise nicely the differences between what level 3, 4, 5 and 6 KS2 to students can do fluently across the topic of number. Broadly speaking: level 3 students have mastery of none of the three areas; level 4 students have some proficiency with mental calculation strategies and timestables, but not the key nodes; level 5 students have reasonably secure mental calculation strategies and timestables and have some proficiency with the key nodes; and level 6 students are broadly secure in mental calculation strategies, timestables and can do many of the key node topics already. It is an obvious statement, but with the strong correlation coefficients already cited, it would appear that what a student can do in number is a strong indicator for what they could do across the curriculum.

These results add further support to my belief about levels hiding gaps in the foundation knowledge of some students. A significant proportion of 4 students struggled to fluently identify number bonds to 100, a level 3 skill. Even many of the level 5 and 6 students did not answer the questions on understanding multiplication as repeated addition or division as the inverse of multiplication correctly. It goes to show- giving a ‘level 6 learning objectives’ sheet to a ‘level 5′ student is not good enough. Perhaps they could already do some level 6 topics, but may have gaps at levels 3 and 4. It must be personalised on a individual student basis.

Has growing up in a levels-based system, where teachers and students are only rewarded for achieving success on content at higher levels in the subject, resulted in oversight/ignorance of the gaps in the students’ foundation knowledge at lower levels? How much easier would learning the more advanced topics be for students if they had comprehensive fluency with these basic skills? The hidden gears need oiling once in a while.

To be clear, this is in no way a reflection on our excellent primary colleagues. They do a brilliant job. They are constrained by a single-minded, level-incentivised, high-stakes system, just like we are in secondary, and they act accordingly to meet the external pressures placed on them. We do the same in Year 11.

A change in mindset is required. If levels are going (and they are!) we must not replace it with a system that has the same flaws. I am certainly not suggesting that we shouldn’t teach students higher level content than they can currently attain; but this must not be just a single-minded focus either. From the diagnostic testing we did this year I have learned that KS3 lessons across the ability spectrum still require systematic, planned, regular practice in building students’ fluency in the foundation topics of number. Fluency (speed and accuracy) is a ** fitness**, it is not binary. Even if you are 100% accurate, you can always be faster then you are at present. If two students can calculate number bonds to 100, but one of them takes five times as long to do it as the other, their learning of higher-level concepts will be all the more difficult for them later in their maths education. We must not fall into the trap of confusing instantaneous performance for retention and transfer – learning. I have written about this extensively before. Learning must not be seen as a checklist of visit-once objectives. Even the highest attaining students need to occassionally revisit some of these elementary topics for which their “fluency fitness” has fallen. A green cell in a spreadsheet indicating that they can do a skill today should not be taken as a proxy that they will be equally as fluent in this skill in six months’ time.

In our KS3 lessons at Wyvern College from September 2015 we will ensure that not only do we strive to raise students’ proficiency with higher-level concepts, but we will also provide short daily exercises that build students’ speed and accuracy in all three areas assessed in our diagnostic trial. It is with great excitement and anticipation that I am going to launch a five-minute-every-lesson, fluency-building product on Great Maths Teaching Ideas this summer. It will provide systematic, rigorous coverage of every topic assessed during our diagnostic trial. Watch out for that! As a consequence of what we learned from the diagnostic testing of our year 7 arrivals this year, I feel so passionately about the need for this product, that it will be made available * free* for all schools via this website in the next month or two.

Keep an eye out!

I really enjoyed reading what you have done and discovered. MathsBox now has weekly skills checks for KS3 based on the new curriculum, and have a skills checklist for the pupils to monitor what they need to improve. If you haven’t already it might be beneficial to look at that while your creating your ‘five-minute-every-lesson, fluency-building product’.

I cant wait to seeing what you produce, and THANK YOU for wanting to share it with all schools!

A fantastic article thank you for sharing. I had not heard of quickKey app before so I will be looking into this sounds great for diagnostic testing. I totally agree the gaps students have when we get them in year 7 are overlooked in our quest to move them up the levels and show “progress”. I will definitely be using your diagnostic test on our new intake of year 7 students in September and possibly year 8 too, and I look forward to your 5 minutes fluency builder. Once again thank you for taking the time to write and share such an interesting article and resource.

Thanks for your comments, Ruth. Do let me know how you get on in Sept and watch out for the fluency-building free product I’ll be launching on this website this summer 🙂

Hi,

Is there a way for you to share the marking grid across quickkey?

Love the idea, going to use it 🙂

Hi,

The marking grid is in the PP at the end of the blog article.

Hope that’s useful and thanks for getting in touch!

Will

Sorry Will I cannot find the marking grid. Please can you sent to me. Have you also got a date when the fluency building product will be available? I loved this power point. Excellent, thank you.

Hi Deb,

The fluency practice product is live, see http://www.numeracyninjas.org. Over 170 schools signed up so far! Hope it’s useful for you.

As for the marking grid, I’m not 100% sure what you mean. Could you explain a bit more please and I’ll do my best to help you out.

Thanks,

Will

If you mean the grid that students mark their answers on, you download it from the Quickkey site; they’ve updated it recently so using the one I used a year ago would now be out of date. Was that what you meant?

The answer grid, for example q1- D q2 – A

To save time going through the test and doing the answers ourselves. We’re very lazy!

Really interesting approach. 2 (quite different!) questions:

1) 3 hours sounds like a long time to scan in score sheets – is there no way you can just feed them into a photocopier scanner and get a multi-page pdf document rather than “photographing” each sheet individually.

2) Was the test demoralising for the level 3 students? I can imaging my students panicking and complaining loudly as the questions disappeared and moved on to the next one before they had chance to work out an answer.

Hi Mark,

1) 3 hours wasn’t so bad! One evening’s work and we did get a 18 000 data points out of it! Good idea re the scanning via photocopiers. QuickKey doesn’t offer this option, but others might. Do you know of any?

2) We briefed the students on what to expect, told them the reasons why we are doing it and made sure they knew it was low-stakes. We told them we wouldn’t be setting classes off it or even reporting their individual results to them. I think the key to keeping them engaged was about being honest and up front with them and managing their expectations.

I love QuickKey and the way that you have applied it.

Thanks, Marc!

In primary school we are very aware that old SATs could be guilty of giving false impressions of understanding and skills of our Y6’s levels, marking fundamental gaps. If secondaries are to be given levels for new Y7’s, then exact marks for each paper, at the very least, should be handed up so that levels are given some context. I prefer a more detailed document of a breakdown of the papers from the ‘marker’, so that strengths and weaknesses, when children are performing at ‘best’, are reported on. When children arrive with secondary school they can re test to see what has been retained etc and move on from there. Having 2 sets of basic data can then highlight areas to work on.

Yes, some very well made points here; thank you Leigh. I fully agree with you that more in-depth data is needed about what students’ strength and weaknesses are. I’m glad that levels are going, but hope we’re given enough time to develop something better and another govt doesn’t bring back a levelling system in the mean time…

Thank you so much for sharing this Will, this is just the kind of thing I wish I had invented. I am eagerly awaiting the resources that you are working on to address the issues. I have been doing a similar thing in terms of addressing addition, subtraction, multiplication and division skills and measuring progress but this is so much more!

Love this, trialled it yesterday with 20 low ability year 10 pupils (I know not the target audience) but a valuable insight into their abilities, plus they loved the instant feedback, I had time to scan in lesson.

If anyone is planning on doing it in Sept I would recommend a trial as I ironed out many bugs and learnt a lot about the system. Pupils have to be very careful about filling out boxes, for example, I tried pencils but thick whiteboard pens worked much better.

Would you consider making your data analysis sheet available so I can use it to analyse the data I have?

How did you scan it in? I looked at QuickKey and it appears you need to upload both class and answers. Can you guide me through how to do this so can scan our students answers please.

Thank you

This article was really interesting to read and I have since downloaded the quickkey app and am thinking of trialling it in September!

Thanks for sharing all these ideas, any tips for a first time user of this?

Thank you so much for creating this resource. It is really great. Will be using this with Years 7 to 11 as part of their first lessons back. Cannot get the app to work so team will be marking them unless I can work out how to use the app.

Thankyou for sharing this wonderful work!

Will be using this powerpoint to assess our incoming Yr.7’s next summer

(Australia!)

Will also be accessing Numeracy Ninjas

Many thanks!!

This is awesome! Thank you for sharing 🙂