Fishing in the Bay

The ARC spend around m$300 per year, receive 4000 applications and fund around 1000 of them for an average k$300 per year each. The success rate is around 23%. On Nov 3 this year, they posted a “consultation document” (HERE) outlining what appear to be some pretty major changes to the Discovery scheme. If my understanding of this document is correct, the proposed changes are ill-conceived. They divert money to poorer projects, create perverse incentives and manifestly fail to solve the main problem that the ARC claim to be worried about.

Let’s get to the nuts and bolts then. It is all about grant success rates and some statistical patterns that appear when this measure is broken down by various factors.

Throughout the discussion below, an early career researcher (ECR) is someone less than 5 years post Ph.D. You might not like this definition, but it is the current one (the ARC are actually considering loosening it).

Issue 1: Experience

Success rates are lower for ECR only proposals - around 18.4% for ECR only proposals and around 25.5% for others, in 2009. Remember that the overall success rate is about 23%. This difference has persisted for many years, see HERE.  I do not think that any academic in Australia would be surprised that ECRs have a lower success rate. Is it unfair? Is it a problem that needs fixing?

First, success rates are not the right measure. They do not tell you how much money is going to ECRs. This is pertinent because many of the successful grants of senior researchers will be to support ECR post-docs! The glory may go to the CI, but the money is going to the ECR.

Second, accepting success rate as the key performance measure, imagine an ARC grant that is written by an ECR with no other experienced team members. Such proposals have a lower succss rate. The ARC seem to think that the success rate should be the same as that of a senior researcher.

In what parallel universe do ECR’s emerge from their PhD with the same skill base, research links, and maturity as a 45 year old Professor? In reality, many ECR proposals will be flawed. Moreover, lack of experience makes successful execution of the research program more risky than for an experienced researcher. So even with two equally good proposals, it would be perfectly sensible to fund the more experienced team.

A non-academic might reasonable ask “How do young researchers ever get started then?” We all know the answer. Mentoring. It was made clear to me, early in my career, that the best way to get an ARC grant was to team up with a more senior researcher. This is a good thing. The role of mentor and ECR is a mutually beneficial one, and has been a central pillar of academic culture for generations. Incentivising ECR-only projects is a direct attack on this healthy tradition.

Issue 2: Gender

Success rates have been consistently lower for females (by a modest amount). What could explain this? Well, it turns out that females are over-represented in ECRs and under-represented at higher experience levels. Could the gender gap then just be driven by their having lower experience?

The answer is no. The gender gap is real. If you restrict attention to ECR only proposals then the gap is 16.5% for females compared to 19.9% for males averaged over 2001-2009, see HERE for the graph.

Below is an even more revealing graph. It shows the success rates for males and females broken into 7 groups of experience (the average of the team). As you can see, the gender gap is consistent across 0-15 years - it is not just ECRs. The graph also shows the earlier stated fact that females are over-represented amongst the less experienced applicants - not that this explains the differing success rates because we have controlled for experience in this graph.

But it set the ARC onto quite the wrong track. You see, they got the idea that they could fix the gender problem by discriminating in favour of ECRs. 

The ARC proposal

The ARC consultation document says that

The ARC has hosted a number of discussions about ECR and gender to understand their relationship and thereby propose a means to address disadvantage in success rates….By addressing this issue … the aim is to remove gender difference across ARC schemes as a totality.

Laudible intent. Pity about the execution. What are they planning to do?

The proposal is to have a separate scheme* for ECRs (perhaps around 200 projects) and a new scheme for mid-to-late career researchers(LCR). So we will have three experience groups in mainly different pools - ECRs, LCRs and the rest.

*ECRs and LCRs can still apply in the general pool as well.

The problems

1. How will they control the total allocation to ECRs?

Why are they choosing 200 ECR projects? (This figure was mentioned in the ARC road-show and is not in the document) A fixed number of 200 awarded grants does not guarantee any particular success rate. So perhaps they will target equal success rates for the three groups, otherwise why bother? This raises the really important question:

What will they do if they get 2500 ECR proposals? Fund 500 of them? Which leaves 500 for the rest of the sector? What if only 200 apply?!

2. Gender bias is not actually addressed. 

Have a look at the Figure again, reproduced below. If the ARC carried their idea to its logical conclusion they could create separate pools for each of the 7 experience categories and impose an equal success rate on each. This would get rid of the “experience bias” completely but the gender bias remains completely unaddressed. Females ECRs still do worse than Male ECRs, even if they might do better than Male LCRs.

They could even impose a higher success rate on ECR’s and a decreasing success rate with more experience. This is of course perverse but it could be tuned in such a way that the overall success rates for females could be made to equal that of males. But the bias within experience groups would remain!!! Amongst ECR’s the 4% success rate gap would not change.

This is dangerously ignorant and thoughtless policy. Positive discrimination in favour of ECRs does not remove gender discrimination. It actually tries to hide it.

As a tax payer I expect the ARC to address the real issue. The real issue is that they have identified gender differences in success rates for all programs, but have not addressed the reason at all.

Looking at gender bias

It is possible that the quality of the applications for females was poorer and that this explains the lower success rate. This is not as fanciful as it may sound. Any kind of affirmative action at earlier career levels would result in the pool of female graduates being less capable than male graduates. And low success rates would not then imply bias. On the other hand, it might be that female applications are even better than male applications and the aggregate figures under-estimate the true level of gender bias. We just do not know yet. And we should know.

I believe that the ARC are ethically required to identify the extent of gender bias in their processes. A company with a poor safety record cannot ignore it. They are required to identify the sources of excess risk and eliminate them. In the same way, the ARC must properly assess the level and nature of gender bias, so as to remove it.

This is not impossible to do. For instance, one might take all 2009 applications, strip them of gender identifying information, and then rate the quality of the proposal, and the experience of the applicants. One can then measure the extent of different success rates for genders after accounting for the quality of the proposal and researchers.

The bottom line

So the net effect of the proposed changes will be to

• divert funds from higher quality projects to lower quality and riskier projects of ECRs
• incentivize fresh PhD’s to apply for their own grants rather than apply for post-docs under the supervision of a mentor
• totally ignore the issue of gender bias, in terms of checking whether it is really there, identifying the source or trying to correct it.

What can be done about this?

Perhaps not much. Margaret Shiels writes in her foreward

We are confident in the broad directions of change indicated in this paper, but would especially welcome commentary about details and implementation.

Translation. Save you ink. She has made up her mind. Nevertheless, being the idealist that I am, I will write my (suitably sanitised) comments down and send them to Mr Jonathan Rogers at the address DiscoveryConsultation@arc.gov.au. I really encourage all academics to do the same- whether or not you agree with my analysis. The deadline for submissions is December 1!

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.