[COPY] The Unsustainability of Sustainability Rankings

Part 1

Apologies. This post was sent in error to paid subscribers. It is now available to everyone.

Sustainability has become the grand theme of global higher education. University World News has an SDGs Hub devoted entirely to stories about universities and sustainability. The Web of Science categorizes research into the UN’s 17 Sustainable Development Goals (SDGs). University Professors and Presidents repeatedly announce their dedication to the cause.

Inevitably perhaps, sustainability has now penetrated into the global ranking world. Universitas Indonesia has been publishing its UI GreenMetric ranking for several years, although it is much more popular in South Asia and the Middle East than in China and the West. Times Higher Education (THE) has a ranking based on the SDGs, and QS publishes a sustainability ranking that also serves as one indicator in its world rankings. It appears that sustainability has become the most pressing concern for universities, as it is for everyone else.

This is extraordinary. Some things are surely just as urgent. There are now several crisis points around the world involving nuclear or potentially nuclear powers, where a single miscalculation will lead directly to Armageddon.

Then we have the global fertility crisis. AI claims — and it may not be hallucinating this time — that if things continue at the present rate, the human species will be effectively extinct in about 1,500 years. In cosmic time, that is much less than a blink of an eye. If I were a Sumatran tiger or a polar bear, I might be happy to see that happen, but unfortunately, I am not. Yet I see no sense of urgency among the talking and writing classes. Probably, they think that it is OK if the Japanese, Koreans, Chinese, Iranians, or non-Amish Americans disappear and are replaced by more fertile nations. However, it appears that the baby bust will eventually reach everyone, even the Amish and the Haredim.

That is some time in the future, and in the long run, we will, according to Keynes, have experienced an assisted or unassisted dying event. Something that should have everybody worried is the widespread and current decline in cognitive skills. It is happening everywhere in the West. Students at all levels are performing less well on standardised tests. Harvard students require remedial mathematics. English majors at midwestern universities struggle to understand Dickens. What would happen if they were confronted with Milton or Chaucer?

The cause of all that? In the short term, it could be the closing of schools during COVID. Or maybe too much screen time, online porn, obesity, dysgenic fertility, demographic change. Whatever is going on, there still seems to be little sense that this is a global or national emergency. Perhaps that is rational: AI may save us, or someone is going to invent a hyper-caffeine that will boost IQs by a standard deviation or two. Or maybe the good Trisolarans will appear with a solution.

In any case, it seems that big cracks are appearing in the sustainability agenda. Recently, Bjorn Lomborg, currently Visiting Fellow at Stanford, denounced the SDGs as a “bloated wish-list lumping vital goals such as eradicating malnutrition and poverty alongside countless vague promises like “promoting sustainable tourism” and fostering “lifestyles in harmony with nature”. By refusing to prioritise, the UN ensured none would succeed.”

To return to the sustainability rankings, even if sustainability is the cause that supersedes or subsumes all others, is there any point in ranking universities, and if so, how is that to be done?

Origins of Sustainability Ranking

According to the THE ranking gurus, sustainability ranking began in 2019 with the launch of the University Impact Rankings. It did not. It started with the UI GreenMetric World University Rankings, first published by Universitas Indonesia in 2010. This ranking is based on six criteria and is compiled using data submitted by institutions. It has been almost entirely ignored by China — this year, there was exactly one Mainland institution — and there are only a few British and American universities. In contrast, Indonesia has 183 universities in the 2024 rankings, Pakistan 84, Iraq 81, Malaysia, India 42, Iran 40, and Egypt 36.

The validity of GreenMetric is limited by the fact that its data is submitted directly by universities themselves. Unfortunately, institutional data is not very reliable. Even if everyone concerned is honest, the difficulty of collecting detailed information from various departments, faculties, branches, research centers, and so on makes it unlikely that the resulting scores and ranks accurately reflect reality.

THE Impact Rankings

But recently, GreenMetric has been overshadowed by the THE Impact Rankings, which purport to assess universities’ progress towards, contribution to, or commitment to the SDGs.

These rankings have a serious design flaw. Universities are assessed on the four best SDGs, and it seems that many universities submit data for just five or six goals. There are Arab universities that submit data for one or two goals to get two points in the THE Arab rankings. If participation means anything about universities serving the cause of sustainability, then the THE rankings are actually working against the goals. They encourage universities to devote considerable effort to a few goals to collect points, while neglecting the others. Thus, we find fewer than a thousand universities submitting data for Life Below Water, Life on Land, Zero Hunger, and Responsible Production and Consumption, but nearly 2,000 for Quality Education.

How does THE propose to measure something as insubstantial and ambiguous as contribution, commitment, or leadership? Examining the indicators for each goal reveals a disconnect between the indicators and the purported targets.

All of the 17 SDG rankings have a component related to research on the relevant SDG. If the rankings were solely about SDG-related research, that would not be a bad idea. However, conducting research related to a goal does not necessarily mean making progress towards that goal.

Then there is the issue of specifying the goals. The targets seem impossible or nearly impossible to achieve. Ending poverty in all its forms. No hunger. Quality education for all. They sound infinitely elastic, creating work for an ever-expanding global bureaucracy and perhaps also ever-expanding global rankings. So, exactly how is progress towards the goals measured?

Zero Hunger

Let’s take a look at how progress towards Zero Hunger, assuming that is possible or desirable, is measured. First, we have the standard research metrics, CiteScore, Field Weighted Citation Impact (FWCI), and Publications. Then, THE wants universities to track and measure the food wasted by employees, students, and staff, excluding visitors, summer school attendees, and remote workers. It is not clear what the point of that is. Universities are awarded points for collecting information about waste, but not for reducing or preventing it. Realistically, the number of institutions worldwide with the ability and motivation to collect such information must be very limited.

Then there are criteria about student and staff hunger. What is required is the provision of interventions to prevent or alleviate hunger, not whether those interventions are needed or effective. There is also the reported provision of sustainable food outlets, including vegan and vegetarian food, and the provision of healthy and affordable food for everyone on campus. Universities receive points for declaring the existence of such programs, providing evidence of their existence, and demonstrating that the evidence is publicly available. They do not get credit for the effectiveness or demonstrated necessity of such programs.

Below is the definition of sustainable food choices. So, what happens if a university finds that some of its foodstuffs meet all of these criteria or all of them meet some of the criteria? Do they receive full marks or zero or a fractional score?

“Definition: Sustainable food choices Sustainable food choices therefore refer to: • trusted sources • environmentally sustainable management of the land and natural environment • minimised or no exposure to manufactured herbicides or artificial fertilisers • no or low level of pesticides • protection of diversity of both plants and animals and the welfare of farmed and wild species • avoidance of damaging or wasting natural resources or contributing to climate change • contributions to thriving local economies and sustainable livelihoods • establishment of trading partnership, based on dialogue, transparency and respect”

It seems unlikely that any university could seriously investigate all of these, or that THE could validate that investigation, even with a highly trained and compliant LLM.

In addition, universities are given credit for the proportion “of graduates in agriculture and aquaculture including sustainability aspects,” contributions to national hunger programs, and access to food security knowledge.

Quality Education

THE proclaims that “high-quality education should be an area where universities excel. Education is a key gateway out of inequalities, especially multi-generational ones.”

THE starts with research, demonstrated by papers, citations, and publications on early years and lifelong learning education, but not, it seems, on primary, secondary, further, or higher education. Then, THE has the proportion of graduates with teaching qualifications that enable them to teach in primary schools, but not anywhere else.

After that there is “free access to educational resources for those not studying at the university”, “educational events at university that are open to the general public”, “vocatioanal training events,” “executive education events”, “educational outreach activities”, and “policy that ensures that access to these activities is accessible to all, regardless of ethnicity, religion, disability, immigration status or gender.”

Again, these programs are assessed according to their declared existence and evidence, preferably public, of that existence. However, the last one gets an extra point for being reviewed recently.

Next is the proportion of first-generation students:

“a first-generation student is one who reports they are the first person in their immediate family to attend university at any level.” It is left up to students to decide who is in their immediate family. The validity of this indicator depends entirely on the honesty of the students and the honesty with which the administration reports the students. In any case, what this measures is simply the limited availability of university education in the recent past.

Peace, Justice, and Strong Institutions

Here, there are three things that THE considers to be related. THE, apparently, does not recognize that sometimes, say during the American Civil War, peace and justice may be incompatible, while strong institutions, a key element of fascism, can often be contradictory to both. As before, THE includes research indicators, CiteScore, FWCI, and publications.

This is followed by University Governance, which includes elected representatives, student unions, “written policies and procedures to identify local stakeholders external to the university and engage with them”, “local stakeholders in the university – including local residents, local government, and civil society representatives (which may include groups such as refugee resettlement agencies) – have a meaningful mechanism or for participating in university decision making”, “Publish the university’s principles and commitments on organized crime, corruption & bribery”. “a policy on supporting academic freedom (freedom to choose areas of research and to speak and teach publicly about the area of their research)”, “Publish university financial data.” These are assessed in the usual way: existence, public evidence of existence, and recent review.

Universities and the Goals

Examining the performance of universities in these tables reveals some counterintuitive results. Universities that do not publish much research and are largely unknown anywhere else have risen to the top for one or a few of the goals. So we have Universitas Airlangga a world leader for No Poverty, Kaohsiung Medical University for Good Health and Well-being, Amrita Vishwa Vidyapeetham for Quality Education, Central Queensland University for Clean Water and Sanitation, UEH University for Decent Work and Growth, University of Huddersfield for Reduced Inequalities.

The relationship of these and the other indicators to the attainment of the goals seems somewhat tangential.

The Rise of the Impact Rankings

There are now more institutions in these rankings than in the THE World University Rankings, the QS world rankings, the US News Best Global Universities, or the Shanghai Rankings (ARWU).

Participation in the rankings is uneven. Only a few Mainland Chinese universities are there. Oxford and Cambridge, and the Ivy League are also absent, although Indonesian, Iranian, and Indian schools are very well represented. They are plagued by instability. Frequent methodology changes and vague, ambiguous metrics mean that universities can fluctuate up and down for no apparent reason.

If, for example, we compare the ranking for Quality Education in 2019, the first year of the rankings, with that in 2025, we see a very different picture. In 2019, the top five universities, in descending order, are: University of Gothenburg, Sweden; Universiti Sains Malaysia; Universidad Andrés Bello, Chile; University of Bologna, Italy; Sungkyunkwan University, South Korea. Fast forward to the 2025 rankings. The top six are: Lingnan University Hong Kong; Hong Kong Baptist University; Aalborg University, Denmark; Istanbul Technical University, Turkiye; Al-Ahliyya Amman University, Jordan; Amrita Vishwa Vidyapeetham, India.

Take a look at the top universities for Affordable and Clean Energy. In 2020 — this goal was not assessed in 2019 — the top five were Tongji University, China; National Changhua University of Education, Taiwan; University of São Paulo, Brazil; UNSW Sydney, Australia; University of Auckland, New Zealand; and a greyed-out — THE doing its bit to save Ukraine — Peter the Great St Petersburg Polytechnic University, Russia. In 2025, the top five universities for Clean and Affordable Energy are Al-Mustaqbal University, Iraq; JSS Academy of Higher Education and Research, India; Abdullah Gul University, Turkiye; Afe Babalola University, Nigeria; Lovely Professional University, India.

This kind of volatility extends beyond dynamic evolution or whatever THE might choose to call it, and descends into simple incoherence.

Unsustainable Sustainability

Nonetheless, the Impact Rankings have been well received in many parts of the world. The number of universities queuing up to submit data and, perhaps, paying for related benchmarking and consulting services has steadily increased, although it may be approaching a plateau.

But there is a price. Collecting, processing, and validating the data for the Impact rankings are arduous and frustrating tasks, and the data they eventually produce may sometimes have only a tenuous relationship to reality. It also means that competent researchers are taken away from real research and assigned to the tedious task of getting their university into the top ten or top one hundred rankings for something.

It also appears that some leading universities have soured on the impact rankings. Just look at the top ten for overall impact in 2019 when they started. Four of them are no longer there in 2025: University of British Columbia, King’s College, London, Gothenburg University, and the University of Hong Kong.

Additionally, many universities are starting to question the underlying ideology of the rankings. Gender equality now includes equality for transgender individuals. Note that the goals for gender equality refer to equality for all girls and women, code for biological men who claim to believe that they are girls or women.

Changes to the Impact Rankings

LLM (AI)

THE has recently discussed forthcoming changes to the impact rankings. The first is that THE is going to start using AI. It seems that there is a massive amount of data, equivalent to 800 books, each about 100 pages long.

That already makes me a little nervous. Books 100 pages long? Isn’t the typical novel 300 pages long? Historical monographs 700 or 800? Does whoever wrote those words think that a typical book is 100 pages long?

There is also the problem that much of the data submitted to THE is not relevant. It seems that humans cannot do the data submission job properly and AI is necessary to do the job that humans can’t do. I wonder if it is really the case that AI has to do the jobs that humans can’t do for a less-than-living wage.

A Digression on my difficult relationship with Chat GPT

But let us just ask how much is AI really going to help. Here is my experience with Chat GPT asking a simple rankings-related question. How many Chinese universities participated in the last UI GreenMetric rankings?

At first, Chat GPT could not tell me anything useful: “Would you like help finding whether specific Chinese universities are in the 2024 list? Or should I assist drafting an inquiry email to the GreenMetric secretariat?”

So I told the AI to go the GreenMetric site, click on “country” and enter “China”. AI replies: “ I checked the UI GreenMetric 2024 rankings page at your link. While there’s a filter option to show only universities from China, the site does not display the total count of filtered results:

• The site shows a search box where you can type “China” and then it lists all the Chinese universities—but there's no summary number telling you how many.

To find the exact count, here are two practical steps you could take: ….”

So I sent a screenshot. AI replies:

“The image shows the UI GreenMetric 2024 rankings filtered by country: China. It displays only one university entry — Weifang Institute of Technology, ranked 41st globally and 1st nationally.

However, the table shows a "Show 100 entries" option, indicating that the page supports multiple entries — but only the first entry is visible in your screenshot. “

I explain and eventually: ….

“✅ Only 1 Chinese university participated in the 2024 UI GreenMetric World University Rankings:

• Weifang Institute of Technology (Ranked 41st globally)”.

Chat GPT concludes: “From your first mention of the UI GreenMetric rankings to confirming that only one Chinese university appears in the 2024 list, it’s been roughly 15–20 minutes of focused attention.”

DeepSeek took even longer, because of its reluctance to separate Taiwanese universities from the Mainland. On the other hand, Perplexity was a bit quicker.

Yes, I know that you have to learn how to write prompts properly. And I know that AI is getting better. But at the moment LLMs are the product of human trainers and bear the imprint of the people who produced them, including their biases, blind spots, and prejudices. So, whether the validation will be significantly better, I do not know, although it will probably be a bit cheaper.

From Ranking to Rating

Anyway, getting back to THE’s changes. The University Impact Rankings will now be known as

the Sustainability Impact Ratings. THE proclaims that the Impact Rankings were “(f)irst published in 2019 as the world’s most comprehensive framework for evaluating and celebrating universities’ deep social and economic impact.”

That points to the fundamental problem of the THE impact rankings, indeed of the THE university rankings in general. It claims to evaluate and to celebrate, even though there is surely an inherent contradiction between the two. Ultimately, if you aim to provide accurate and relevant information to students, stakeholders, and the public, you will often need to refrain from celebrating universities, or at least do so while providing caveats.

THE are now moving towards publishing ratings rather than rankings and that means “recognition of this framework as a rating system and network allows us to work with the community to consider moving to presenting the congested numerical score data in banded groups, not individually ranked places, or based on percentiles or performance classifications.”

There will be a community to which “we will deliver exclusive content and analysis to our network community: special webinars, a regular monthly newsletter sharing rich data insights and best practices, and opportunities to connect with each other across events and activities.”

This, of course, will not come free of change.

It will be interesting to see how many universities are willing to pay more money for a rather questionable product. Is it possible that THE has finally overreached?