SQA results and racial bias: how data is manipulated to brush racism under the carpet.

Today, pupils across Scotland received their SQA national qualification results based on teachers’ estimated grades and the SQA’s moderation process. As soon as exam cancellations were announced due to Covid-19, I explained why the process of estimating grades by teachers, with the SQA's moderation according to schools’ previous performances, was entrenching privilege and exacerbating inequalities (see my article). As predicted, those from lower socioeconomic backgrounds faced the most disadvantage because of their schools’ previous performance: rather than trusting individual merit and teachers' judgement, the pupils from the most deprived areas were much more likely to have the grades estimated by their teachers brought down during the moderation process (see table A13 below).

Table from the SQA Equality Impact Assessment

At least, the SQA did take the time to carry out and publish an Equality Impact Assessment. I was curious to see what the report said about bias and racial inequality in particular. You can find the report here.

Data Manipulation for Racist Discourses

When it comes to examining racial bias in previous estimates from 2019 as an equality impact exercise, page 20 of the report states that the ethnicity data is unreliable for analysis because of its sample size:

“Race: around 90% of candidate entries were either ‘White – British’ or ‘White – Other’, with the largest other ethnicity (Asian – Pakistani) being 2.5%. Thus, each non-white ethnicity is a small dataset — and small datasets are difficult to analyse and draw firm conclusions from as the data tends to be variable, meaning it is often not possible to distinguish the natural variation found in small datasets from meaningful signals.” (2020 Alternative Certification Model: Equality Impact Assessment, SQA)

Following QuantCrit and Critical Race Theory, such a statement is just a lazy cop-out that makes it easy to sweep racism under carpet. Considering that we are looking at ethnic minority populations, it’s pretty obvious that the percentages are going to be small. In fact, having 10% of the general pupil population coming from non-white backgrounds is probably the highest rate we've ever had in Scotland. From the last Scottish census data in 2011, ethnic minorities made up only 4% of the general population.

Numbers aren't neutral. Especially when they are used to present the argument that there's nae race problem here. The 2010 Count Us In: Success For All report on equality in Scottish education had no problem in using ethnicity data with small sample sizes to put forward the discourse that white pupils had the highest attainment gap and Asian pupils had the narrowest attainment gap. In other words, there couldn't possibly be a race problem in Scottish education because Asian pupils do so well and white pupils are being left behind. That report was based on government data in which Asians made up only 2.11% of the total population and the largest minority ethnic group, Asian Pakistani, made up only 1.19%. There was no mention of small datasets being difficult to analyse and unreliable back then, even though they were smaller than the ones in the SQA Equality Impact Assessment.

Figure from the 2010 Count Us In: Success for All report

Drawing on Critical Race Theory, such discourses (like the one used in the 2010 Count Us In table above) are racist because they are used to shut down any claims of racial inequality in Scottish education. Note that the low attainment of Black-Caribbean pupils, for instance, was never even mentioned in the 2010 report. Some numbers are emphasised, others are overlooked and data is manipulated to fuel racist discourses. The data is only good enough when it can be used to support a policy-maker's agenda.

Analysing the SQA 2020 Equality Impact Assessment Data

With that in mind, let’s take a closer look at the data presented in the 2020 SQA Equality Impact Assessment. As part of an equality impact exercise, the SQA used data from 2019 to assess how likely teachers were to underestimate or overestimate candidates according to different protected characteristics. The data includes entries from all candidates with estimates who were on roll at a publicly-funded mainstream school. Table A1 shows the proportion of candidate entries for different protected characteristics in Diet 2019 National 5 qualifications.

Table taken from the SQA 2020 Equality Impact Assessment

Tables A2 and A3 show the distribution and the percentage point difference of estimated grades and resulted grades in Diet 2019 for each characteristic at National 5.

Table taken from the SQA 2020 Equality Impact Assessment

Table taken from the SQA 2020 Equality Impact Assessment

Looking at tables A2 and A3, it is clear that under-estimation is common for A grades at National 5 in 2019. It does happen to white Scottish pupils and it would be interesting to see how many of them come from lower socioeconomic backgrounds. In the teacher estimates, pupils from lower socioeconomic backgrounds appeared to be penalised the most. However, we need to bear in mind the intersectionality of these issues. There will be many pupils of colour who also come from lower socioeconomic backgrounds who may face a heightened level of disadvantage because of racial bias.

So, which groups have the highest levels of under-estimation of A grades from their teachers in 2019? Asian-Pakistani, the biggest group of all the ethnic minorities, with 8.9% of pupils being under-estimated for A grades. This group also happens to be over-represented in social deprivation and lower socioeconomic categories. Another ethnic minority group that tends to be over-represented in poverty and social deprivation is the Roma Gypsy-Traveller category which does not appear on the table as their numbers are small, or they are hidden in the "All Other Categories." Encouragingly, the SQA Equality Impact Assessment does note that Irish and Roma Gypsy-Travellers tend to face additional barriers to attainment, so they were given additional consideration in the Equality Impact Assessment.

The second group with a higher level of under-estimation is the “Mixed” category, with 8.8%. This category makes it difficult to assess the racial identities within them. Technically, I fit the “mixed” category because my mother is Indian (brown) and my father is white French. But I am racialised as brown or South Asian, never white. Same would go for a pupil with a white parent and a black parent – they would probably be racialised as black. Therefore, there are many pupils in that category who may very well have been under-estimated because of their black or brown skin, which might have added to the numbers in the Asian or Black groups. That is the challenge with ethnicity data – ethnicity does not mean race. Ethnicity refers to a common language, common cultural heritage, identity and practices, whereas race is a social construct whereby society categorises people according to skin colour and other physical features. You get to more of a say about your ethnicity - I could say I'm French or Indian. But race is predetermined for you - people see me and racialise me as brown or South Asian, not white or French. For now at least, we have to work with the ethnicity data we’ve got.

The third group with a higher level of under-estimation is the Asian-Indian group, with 7.7%. I wonder how many teachers actually know whether their South Asian pupils are Indian, Pakistani, Bangladeshi or Sri Lankan. They tend to be racialised in similar ways and so the impact of racial bias may equally be similar. In fact, if we add up those who are typically racialised as brown or South Asian, the numbers would be much higher – hinting at the important role of racial bias in teachers’ estimation of grades.

The fourth group with noticeably higher levels of under-estimation is "African / Black / Caribbean," with 7.6%, followed by Asian-Chinese with 7.3%, White non-Scottish with 7.2% and White Scottish with 7.1%. The prevalence of racial bias in teacher estimates from 2019 is clear and the data that was not analysed in the SQA Impact Assessment would suggest that teachers tend to underestimate the ability of pupils of colour.

Critically Reading Data, and Gathering Better Data, Leads to Better Anti-Racist Action

All the numbers analysed above were taken from the SQA exam results and teacher estimates from 2019, based on Scottish Government data because the SQA does not gather ethnicity data. There was no data from 2020 in the report because it was not available, as far as we're aware. So it's hard to know how much racial bias influenced the SQA results for the 2020 alternative model of assessment since there was no ethnicity data available, or published at least. But if we look at the racial bias in 2019, we can be sure that it's got a pretty big role to play today.

Thankfully, the SQA Equality Impact Assessment does conclude that it needs to explore more ways of gathering data to assess equality impacts more accurately. Institutions need to stop shying away from gathering and analysing ethnicity data if they intend to commit to race equality. You can’t fix the problem unless you see it. Refusal to properly look at the data, and uncover uncomfortable truths about racial inequalities, allows racism and racial inequalities to persist.

The 2020 report does also mention the need for “conscious or unconscious” bias training, although this needs to be much more substantial than the initial training it provided online in May 2020 for it to be useful. Racial bias is definitely not something that will be fixed with a one-off, tick-the-box unconscious bias exercise. We need much deeper, structural change and long-term commitment to anti-racism in every aspect of education. If we are committed to race equality, we need to first identify racial bias as a significant problem in assessments and racism as a pervasive feature in education more broadly. It’s going to take much more than implicit bias training to resolve. Racial literacy, assessment and curriculum changes are a must. For more solutions to fix this problem, The Anti-Racist Educator's resources is a good place to start!

Resources that might inspire anti-racist solutions:

• The Anti-Racist Educator Glossary

• Article on institutional racism and scientific racism in Scottish education

• Article on the manifestations of racism in Scottish education (part 1 and part 2)

• Article on bias and SQA exam cancellations

• Article introducing Critical Race Theory with the podcast to follow

• Podcast episode on Critical Race Theory which includes an explanation of QuantCrit (how data is manipulated to defend racist discourses)

• The Anti-Racist Educator reading list and listening list.

• The Anti-Racist Educator's webinar on Decolonising the English Curriculum with the Scottish Association for Teaching English.