“It is what it is.” That was my first reaction to the PISA 2012 results released last week (Full Canadian Results). PISA (The Programme for International Student Assessment) is designed to provide indicators of the skills and knowledge of 15-year-old students across the world (please see here for more backgrounder information on PISA). While the assessment tool does measure a limited set of skills, there is much PISA doesn’t measure. And, true, PISA continues to tilt toward 20th century over 21st century skills, but it is still the world’s best, widely used assessment tool on how we are doing in education and on providing guidance for education improvement.
Although much attention is given to the ranking part of the tests, as Yong Zhao points out, even those at the top are wondering about their success:
While the East Asian systems may enjoy being at the top of international tests, they are not happy at all with the outcomes of their education. They have recognized the damages of their education for a long time and have taken actions to reform their systems. Recently, the Chinese government again issued orders to lessen student academic burden by reducing standardized tests and written homework in primary schools. The Singaporeans have been working on reforming its curriculum and examination systems. The Koreans are working on implementing a “free semester” for the secondary students. Eastern Asian parents are willing and working hard to spend their life’s savings finding spots outside these “best” education systems. Thus international schools, schools that follow the less successful Western education model, have been in high demand and continue to grow in East Asia. Tens of thousands of Chinese and Korean parents send their children to study in Australia, the U.K., Canada, and the U.S. It is no exaggeration to say that the majority of the parents in China would send their children to an American school instead of keeping them in the “best performing” Chinese system, if they had the choice.
But, if one does want to buy into the assessment, we need to do more than use the results to search for our flaws or accentuate our ideologies. It has been disappointing and discouraging to see some of the commentary in British Columbia, and across the country in response to the results. I suspect most who have commented (for example) on the need to “focus on the basics” to raise scores haven’t looked at the problem-solving questions that PISA asks (not very back-to-basics questions).
So, while acknowledging the limits of using the nation “rankings”, let me share some of the insights I have gleaned from my first look at the results and some stories you may have not seen:
1) British Columbia was the highest performing English-speaking jurisdiction in the world
British Columbia is not only the highest performing province in Canada, but ahead of all other English-speaking participating nations including Australia, United States, United Kingdom, and New Zealand (to name a few). If you look at countries in general, Canada would be first in this category.
2) British Columbia was the highest performing multicultural jurisdiction in the world
One characteristic that other countries at the top of the charts do not share with British Columbia and Canada is its diversity. In language and cultural diversity, BC and Canada stand out as the highest performing on the assessments.
3) British Columbia was the highest-performing province in Canada in science and reading and second to Quebec in Math
British Columbia has typically been among the strongest performing provinces in each area (typically with Alberta, Ontario and Quebec). The most recent results show BC was first in science, ahead of Alberta and Ontario. Reading on, the same three provinces performed at the top level in Canada, and again, all near the top of the International charts. In math, Quebec led the way with BC, Alberta and Ontario following. It is worth noting, of those who completed the digital math assessment, BC was the highest performing province (more on digital below).
4) There was both excellence and equity in British Columbia’s results
The difference between the high and low achievers in BC (those between the 90th and 10th percentile) is lower than in all of Canada, and the OECD, in all three disciplines. The gap is also lower than that in Finland (often cited for its high level of achievement and equity) in both Reading and Science.
5) British Columbia’s results have been steady for the last decade
In absolute terms, since 2006, British Columbia’s results have been fairly steady. It is true that in Mathematics in particular, in relative terms BC (and all of Canada) has declined — in part due to more countries participating, and also because of the improvements in several Asian countries.
And then beyond these headlines, there is other interesting data:
There is a lot to analyze and much more that will come out from the OECD over the next year. One piece of information that was particularly interesting in the first report was how much less the gender gap was in reading when the test was completed on a computer. For those using print reading, the BC gap in scores (in favour of girls) was 26 points, but when completed digitally, the gap was only 14 points. Across Canada there was similar data indicating a shrinking of the gender gap when the reading was digital. This is incredibly interesting given the increase in digital print we currently encounter — and just one of the many pieces of data that is worth taking the time to better understand.
It is also a given that there are many ways in which our system can improve, and those who make the case for more services, new programs and innovative approaches are right. And, yes, socio-economics and issues like poverty matter. It is also true that BC has an amazing education system. It is interesting to see what a more positive view the British seem to have of our results in Canada — having such a quality teaching force in BC is our huge advantage.
Now, let’s get past the rankings part and focus on the learning part — what we can learn from others about how we can improve the experiences for our students both locally and globally. And, let’s not spend our time thinking about how we can get better at the tests, but instead focus our attention and system on how it can help our kids for their world today and for tomorrow.
cultureofyes.ca 
Korean students spend many more hours in school and after school academies. Most students are not happy and are not living a balanced life.
It is not suprising that they do well on these tests.
It is also interesting to read about how Korea wants to transform their system and move away from the high pressure / high stakes characteristics that have long defined the country.
The part of all this that still irks me is the “we’re better than they are” which continues to think of education as a competition. I understand we will always have or deal with rankings but even if we out value in these tests, shouldn’t we be focusing on growth?
I think we are forced to do both – we need to comment on the rankings . . . or simply allow others from outside the system to completely frame the story and then we need to also focus on growth – what can we learn from others to make a better version of us. I think doing one or the other will be ineffective.
I find it interesting that the CMEC report does not comment on socio-economic status [class] differences in the report. Volume II (Equity) of the PISA Report devotes two chapters [pp. 25-63] examining socio-economic differences amongst and in countries.
Some quotes from Volume II follow:
“Across OECD countries, 14.6% of differences in performance among students are explained by disparities in students’ socio-economic status, whether the focus is on mathematics, reading or science (Table II.2.1). In countries and economies where this relationship is strong, students from disadvantaged families are less likely to achieve high levels of performance. Some 39 score points – the equivalent of one year of formal schooling – separate the mathematics performance – and the difference is very similar for reading and science” (Table II.2.1 p.2). Socio-economic status is commonly understood to refer to class differentiation with those in the lower classes living at or near poverty levels.
Table II.2.1 suggests a 9.4% difference in Mathematical achievement scores in Canada can be attributed to socio-economic [class] differences. The OECD interprets socio-economic status with indicators which include “parental education and occupation, the number and type of home possessions that are considered proxies for wealth, and the educational resources available at home” (Box II.2.1 p.37 Volume 2).
“On average, the performance difference between advantaged (the top quarter of socio-economic status) and disadvantaged (the bottom quarter of socio-economic status) students is 90 score points, or the equivalent of more than two years of schooling and more than one PISA proficiency level. Disadvantaged students are, on average, more than twice as likely as students who are not considered disadvantaged to score in the bottom quarter of the performance distribution” (Table II.2.4a p.38). When the report addresses “disadvantaged students” poverty issues surface.
“At the school level, higher socio-economic status is associated with a range of characteristics of a community that might be related to student performance, such as a safe environment, and the availability of quality educational resources, such as public libraries or museums. At the individual level, socio-economic status may be related to parental attitudes towards education, in general, and to their involvement in their child’s education, in particular” (Box 11.2.1 p.37 Volume 2).
Further, Figure II.2.5 (p.42 Volume 2) displays “Mean mathematics score after accounting for country’s/economy’s socio-economic profile.” If socio-economic considerations were not a factor, Canada’s ranking in Mathematics achievement would fall from 13th to 17th in the world.
Clearly there is a problem with socio-economic status class and poverty issues in Canada. Again, why doesn’t the CMEC report address the disparity of socio-economic status amongst 15 year olds in the Country?
Thanks Robert – and important addition to the conversation. I am hopeful that this area gets greater attention in the coming weeks / months. I have seen some interesting charts that show the huge gap in US performance from high to low socio-economic communities. There is a lot that is within the control of the system, but there are also so many other factors that are far bigger than the education system of any jurisdiction.
Thanks for adding this – very important!
Is PISA fundamentally flawed? (http://www.tes.co.uk/article.aspx?storycode=6344672)
Excerpts:
“They are the world’s most trusted education league tables. But academics say the Programme for International Student Assessment rankings are based on a ‘profound conceptual error’. So should countries be basing reforms on them?”
[…]
“What if you learned that Pisa’s comparisons are not based on a common test, but on different students answering different questions? And what if switching these questions around leads to huge variations in the all- important Pisa rankings, with the UK finishing anywhere between 14th and 30th and Denmark between fifth and 37th? What if these rankings – that so many reputations and billions of pounds depend on, that have so much impact on students and teachers around the world – are in fact “useless”?”
[…]
“I was recommending 10 years ago to the OECD that it should try to incorporate longitudinal data, and it simply hasn’t done that,” Goldstein tells TES. He would like to see Pisa follow a cohort over time, as the government pupil database in England does, so that more causal relationships could be studied.”
Thanks Peter – I definitely have seen more articles this time questioning the tool – and it is important that as we talk about how much it means, we understand the tool, and how it is used.
Hey Chris – I had a very interesting conversation with the significant other of my step brother last night. She taught in Shanghai for the past 2 years and asked me if I had heard much about their system. I responded that I knew they had done well on tests and she laughed and said… that is their skill test taking… that is what it is all about. Some highlights from our discussion:
– she said that the test scores are what it is all about there
– students spent hours memorizing test questions and patterns from tests. Sometimes they had the answer keys and spent the majority of the time memorizing the multiple choice answers
– she tutored kids as young as 3 on Saturdays for hours… and they were not allowed to sleep or eat until they had completed hours of study.
– in addition to teaching Spanish and English during the day, she tutored students in the evening until late at night… extra curricular activities are very rare
– the head of the school was quoted as saying that it was acceptable to cheat on up to 20% of the exam… more than that was unacceptable
– a nearby school brought in police to stop the cheating on tests and students, parents rioted saying it was not fair to prevent them from cheating as everyone else does. (Googled this today and found an article in the Telegraph here: http://bit.ly/1dZU0Bg )
– students are stressed and just study all day long. Many of them dream of leaving this system but don’t know any other way of doing it.
These are all comments made by someone that briefly lived the system… I am sure there must be some stories through a more positive lens but felt this would add to the conversation.
Love your final thoughts in this post.
Thanks Chris for adding this story – it has been interesting to see several similar stories being told. While I agree with the concerns (and from what I read so do many in Shanghai) this should not stop us from asking what we can learn from their success? Just as we have looked to Finland, we should look to other strong jurisdictions, and yes, be skeptical, but also enter with an open mind about what we might be able to learn – it is one of the best parts of PISA – it encourages us to see our system as part of a global network.
Thank you Chris, for your reflection of the PISA results, however I am more interested in the specifics of the status of Numeracy and Mathematics based programs in Canada. My general observation as someone who has been an Administrator in a Grade 1-12 school is that our students are definitely decreasing in basic Mathematics ability. I am well aware of the greatness of the Canadian Education system and we can be proud of BC, however I think we need to be truly reflective about whether our philosophical ideologies are really making a difference for our students. While the new provincial curriculum is exciting, and project based learning can create more passion for learning, we might also learn from our past and listen to opposing view points. Here is one example of a view point that I read in response to the CBC article below.
piRskwared
Every time a board or region swings the pendulum from inquiry to rote or back again they set the progress of math education back and loose a lot of kids through the cracks. The truth is there should be a balance.
Elementary kids need a lot of repetition to develop foundational numeracy, but the algorithms taught should not be devoid of sense making and reason. Often the standard algorithms for low-level math functions like rounding or 2-digit by 2-digit multiplication are taught in unfathomably abstract, rule-based ways when they could be taught using processes and algorithms that connect to some sense-making or visualization of the concept. Some of the “competent” math students coming into my grade 10 class might be able to invert and multiply to get the answer to a paper/ pencil fraction division problem but can’t tell me how many eighths are in one and a half; or they can calculate 15% of 250 but stare blankly when I ask them what three quarters of eight is.
Inquiry approaches often go too far and loose the demand for mental math and fluency but we have to allow room for kids to figure some things out for themselves — that’s what makes math fun! I think the traditional approach (algorithm, rules-based) is a decent place to start, but sense-making and reasoning definitely belong in math classes.
Thanks Balan for the comment. I would agree that the main conversations coming out of PISA 2012 do seem to be around math. I think one of the strengths of our system in BC is that we do always passionately debate the best approaches – the answers are often in the middle . . . .pendulums swinging too far in any direction can be a problem.
Hi Chris
Very much appreciate your post. Particularly interesting given the timing and where Linda and i are at the moment. We were at an OECD meeting in Barcelona last week with about 25 other country reps when the results came out and then went on to Manchester where we were working with school heads. The interest in Canada / BC was not about the rankings per se but what generalizations we could make from the results given the strong showing as a multicultural, multi lingual country with relatively high equity and high quality. Your post is very helpful in creating a context for further reflection. It would be foolish to blow off the PISA results without taking some time to understand what they can mean for us as a system in BC moving forward. Thanks!
Judy
Thanks Judy – the results are definitely moving some good conversations that we have as educators more publicly – what are characteristics of high performing systems? How do jurisdictions achieve excellence and equity? Questions we all want to pursue.
I am convinced that more than ever BC is going to be a destination for jurisdictions around the world looking for support in their learning journey.
[…] https://cultureofyes.ca/2013/12/08/some-pisa-thinking/ […]
Bring notice to the indicators in which BC performed very well in my opinion is as crucial as it is to identify the areas our systems have room to improve.
Equity and the high levels of performance in such diverse society are in my opinion the greater indicators of achievement we should celebrate and make sure we preserve in BC.
I believe the students could improve in math and science not by going back to a memorization model but rather continuing ascending in the spectrum of critical thinking and problem solving together with mastering mathematical operations.
I find very encouraging to see the conversations generated by the data from PISA and the fact our District which is my immediate experience with the education system in Canada is looking constantly ways to move forward benchmarking with other leaders and adopting what could work for us. As parent I don’t agree with excessive test taking training or long after school hours devoted to raising math scores. I rather have my children reading for mere pleasure, pursuing their passions and developing skills in arts and sports while taking the most advantage possible of the school hours and being challenge with stimulating homework that develops critical thinking and work habits.
It was sad for me to see how the students from Latin America and particularly Colombia, my native country, scored. The ranking doesn’t represent the highly educated Latin-American students performing successfully in foreign Universities or studying languages abroad nor the very innovative schools or the world class private schools we also have, but unfortunately reflects an even bigger problem: the profound inequality of our societies.
Thanks Vicky for the comment. You hit on a number of important issue. There is no pride in the concerning results in many countries. As you rightly point out – the difference between the highest and lowest achieving is alarming and a reflection on access to education in many countries.
The fact we are talking about what we can learn from others is a sign of success for PISA – whatever one thinks of the assessments, they lead to great conversations.
[…] For a very good perspective on the results of British Columbia students (some of the highest in PISA), Chris Kennedy, Superintendent of West Vancouver shares some observations in his ‘Culture of Yes’ blog (click HERE.) […]
[…] « Some PISA Thinking […]
It seems to be implied in you blog that the decrease in math performance is due to new countries participating in the pisa ranking. However, the numbers show a decrease in points obtained by canadian students through the years from 2003 on. Is this something to be considered, given the emphasis in the new way that math is taught at the elementary school level? The outcome does not support the idea that the new way to teach math is better than the traditional one.
Yes, the math results are lower than 2003 and I think it is worth exploring why. One possibility is the teaching methodology, but there may be other factors at play. I have seen a fair bit of coverage of Quebec’s math system (since they were highest performing in Canada) and that is useful to better understand what we might learn from their experiences.
Coming from Switzerland, another quite well performing country with a highly diverse student body I can see one big, but important difference in the two school systems: Canada, or at least the school our kids go to, is focussing a lot more on social skills than we do in Switzerland. Maybe you don’t recognize it as something special, but believe me, it is! How kids are talking to each other, how they care for each other, how they are interested in the other’s well being. It’s just amazing. Here you can find the roots for Canadas success in those testings – and I totally agree with all the comments about testing being very disputable. It’s the climate in your schools, that lets children grow, that supports them in developping self-esteem. And at the end, it’s the best basis for learning.
Please, keep an eye on that, it’s so important.
Thanks Bettina for the comment. I think social responsibility as for so long been part of what we do in schools, you are right, that we don’t realize that it is special and unique to our system. So glad your kids have had such good experience in Canadian schools.
Many Asian countries did better than Canada. I wonder what percent of the students in each Canadian province are Asian. That may help us understand why some provinces were higher than others (?)
[…] 1. British Columbia — high achievement, high diversity, high equity – lots to interest people […]
Actually, Singapore is the highest performing English-speaking country in the world. It’s also multicultural/lingual. The pisa tests were administered in English in Singapore
I find it interesting that some people think that international students go to Commonwealth countries because they have the best educational system. Do these people make that decision based on the marketing that tells that story or is it because parents feel that having a (more) bilingual kid is crucial? I think it is the latter, but I am sure the former does not hurt! Of course, we know how messed up is English orthography which makes learning to read English, learning to spell, and learning in general, a much more complicated (slow) process, but not many seem interested in improving the most critical element that underlies learning! Isn’t like wanting to renovate the roof without renovating the moldy, cracked basement? Or vice versa! 🙂 But, who am I? Ergo, I am not that bright! 🙂
[…] have written about PISA two times before (when both the 2009 and 2012 results were released – and I still hold to these commentaries). Beyond the high-level […]