The Papermill Alarm draws a line between papers which have the characteristics of papermill-products and those that don’t. Given data like that shown below, where red dots represent papermill-products and green dots represent normal papers, where should we draw that line?
Limit the spread
There’s well-known (and fun!) thought experiment in statistics that goes something like this:
There is a rare disease affecting one in ten thousand people. Someone creates a diagnostic test for this disease with 99% accuracy. You take the test and it comes up positive. What is the probability that you have the disease?
The obvious answer seems to be 99%. The test is 99% accurate after all. But let’s think about it a different way.
If we have 1,000,000 people (and the disease affects 1 in 10,000 of those people), then 100 of those people have the disease and 999,900 don’t have the disease. Let’s consider those groups separately.
The test is 99% accurate. This means:
- when we run the test on the 100 people who do have the disease, then 99 of them will be correctly diagnosed and 1 person will be misdiagnosed as not having the disease: a false-negative.
- However, when we run the test on the other 999,900 patients, who do not have the disease, 99% of them are correctly diagnosed as not having the disease, but 1% of them (9,999 people) will be ‘false-positives’. I.e. they will be incorrectly diagnosed as having the disease.
So, of the 10,098 people who test positive, only 99 actually have the disease. That means that if you test positive, even with 99% accuracy, the chance that you have this disease turns out to be less than 1%!
Isn’t that strange?
Precision and recall
The above is a common example in Bayesian statistics. What it means for Clear Skies is that, if we want to build detection algorithms for papermill-products, we have to control for imbalance in the data in order to be able to make useful predictions. We also have to be a bit more careful about what we mean when we say ‘accuracy’. There are 2 very handy metrics here.
What % of the detections that you make are accurate?
- High precision means that we can have a lot of confidence in predictions, but that we should expect to have a lot of false negatives.
- In the above example, precision is 0.98%
What % of the things you want to detect are you detecting?
- High recall means that we can expect to find a high percentage of what we are looking for, but the downside is that we will also see a lot of false positives.
- In the above example, recall is 99%
So, our imaginary diagnostic test has high recall and low precision. I know that medical researchers sometimes read this blog. Your words are different to ours, but hopefully the concepts are familiar!
(Also, I’m not a doctor, but I assume that high recall is a good choice for medical diagnostics. If you get misdiagnosed as having a disease, that’s not good. But you might be able to run other tests for confirmation. It seems far worse if you were misdiagnosed as not having a disease that you need to be treated for.)
When it comes to papermill detection, high recall would mean a situation like the one described above where we flag or ‘diagnose’ a very high percentage of papers, including a lot of non-papermill products, in order to ensure catching all of the papermill-products. The next stage of the process is to run further tests (one reason why we have multiple pipelines!) or for Research Integrity specialists to sift through those flagged papers and apply their own checks. In principle, this is quite a nice 2-stage process which can maximise the number of papermill-products we detect correctly.
However, there’s a downside here. We don’t want to waste the Research Integrity specialist’s time by flagging lots of papers which are likely to be ok.
But here’s the cool thing: we actually get to choose whether we want high precision or high recall. That’s nice, isn’t it? Which should we pick?
High precision would mean accepting that we miss a few papermill-products, but that a very high percentage of detections are what we are looking for.
We could try to find a balance in between the 2. But we can do better than that.
We know that papermills will test-the-waters with journals all the time. When they find one that has a high acceptance rate for their products, they target it.
So the effect of using a high-precision tool would be to reduce the chances of being targeted with a minimal amount of work (compared with a high-recall tool). When you think about it, we don’t need 100% recall. We just need to catch enough that it isn’t worth the mill’s time to keep submitting.
So there’s value in both. We want to have a high precision option and a high recall option. That’s why the Papermill Alarm issues red alerts and orange alerts. Red alerts have high precision and orange alerts have high recall.
So you get the best of both worlds.