![]() | This article is rated B-class on Wikipedia's content assessment scale. It is of interest to the following WikiProjects: | |||||||||||||||||||||||
|
[[[Introduction under-justifying and over-claiming]]]
I don’t like the final two sentences of the intro:
1. “ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution.”
Guessing at random provides a way of selecting “possibly” optimal models. Why is any ROC-based method preferred? What evidence is there that the ROC-based process leads to superior outcomes?
2. “ROC analysis is related in a direct and natural way to cost/benefit analysis of diagnostic decision making.”
It’s not clear what form this relationship takes, and the subsequent article doesn’t explain it. So it seems like a vague form of over-claiming to me. — Preceding unsigned comment added by Willbown (talk • contribs) 06:38, 16 July 2023 (UTC)
The minus sign in the derivation of the AUC right before the derivative of the FPR seems to be wrong. Let be the True Positive Rate as function of the False Positive Rate. The AUC is then given by:
Let be a parametrization of with and . According to Integration by substitution:
In terms of the True Negative Rate and its density:
Can anyone add information about the use of this method in clinical scenarios (eg. examination of risk factors for disease outcomes)?—Preceding unsigned comment added by 131.104.10.194 (talk • contribs) 23:08, 28 September 2006 (UTC)
I suggest that the articles "Receiver operating characteristic" and "Detection theory" (Signal detection theory) should be merged. The merged article should seek a middle path in terms of technical formality and jargon (as simple as possible, but not simpler). Merged or not, the two articles should be more clearly compatible, if not entirely consistent, since they are basically trying to explain the same underlying thing. Rbfuld (talk) 23:02, 27 January 2008 (UTC)rbfuld
I agree with the suggestion that the articles ROC and "Reciever operator characteristic" should be merged. Wicked Maven 20:00, 28 January 2007 (UTC)
I disagree with the suggestion that "Receiver operating characteristic" and "Detection theory" (Signal detection theory) should be merged. Signal detection theory appears to be a larger topic containing many concepts and methods. The combined page would have to be huge in order to treat all of them in sufficient detail. It is easier to maintain a collection of small pages that treat each component of the topic. Anecdote: I arrived at this page seeking to clarify my understanding of ROC curves, and in particular, the use of the area under the ROC curve. I already know about most of the stuff on the current Detection Theory page, and was only seeking information relating to ROC curves. This page is the correct level of granularity for what I wanted to learn. Note: Maven is suggesting that "ROC" be merged with this article? But it seems there is no other ROC article on this topic; the ROC disambiguation page points here. Bayle Shanks (talk) 22:01, 6 November 2009 (UTC)
For whom was this written? The author displays considerable erudition, but no desire to make his subject palatable to beginners. The second paragraph was enough to choke and die on. I'm not here to be stymied by a pedant - I need to understand ROC curves. I swear - if I ever learn enough about the subject to do so, I will join Wikipedia and rewrite this #$&* entry.64.59.144.85 02:55, 8 February 2007 (UTC)
I think that the example plot w/ points A, B, C, C' is misleading or wrong. C' is intended (I think) to be an example of the effects of inverting the output of the worse-than-random classifier C. If this is actually what it's meant to represent, the plot is wrong: inverting the output of the classifier doesn't correspond to a mirror reflection across the diagonal, but to a mirroring through the point (0.5,0.5). (Inverting the test set labels correponds to a mirror reflection across the diagonal.) I don't have the file used to create the diagram, or I would fix it myself, so I leave this to whoever posted the diagram. 128.62.104.34 22:27, 23 April 2007 (UTC)
No matter whether we have to mirror with the point (0.5, 0.5) or at the diagonal line, if we agree that any point under the diagonal line can be mirrored onto the other side then the lower right corner also represents a perfect classification. Therefore IMHO the two arrows labeled "better" and "worse" are misleading too. The closer towards the diagonal line, the worse; the closer towards either the left top corner or the right bottom corner, the better. One solution might be making both arrows two-headed. The heads pointing towards the edges should be labeled "better", the heads pointing towards the diagonal line should be labeled "worse". I am just afraid that people new to the topic will still be confused. Different approach: points under the diagonal line must be mirrored first before they can be compared. Put only one two-headed arrow in the upper triangle. Stevemiller 03:58, 9 October 2007 (UTC)
"The lower right triangle corresponds to worse-than-random classifiers"
Isn't a random classifier (the diagonal line) the worst case? Suppose our classifier is 100% of the time wrong, IOW it is at the right bottom point, then we actually have a perfect classifier, which has just the wrong labels. We would always flip a classifier that is below the diagonal line (AUC<0.5). I understand that with "behaving" normal distributions the ROC won't cross the diagonal line, switching sides. But other distributions might cause the ROC to cross the diagonal line. Stevemiller (talk) 17:36, 26 February 2022 (UTC)
The notation used in the figure ("How a ROC curve can be interpreted") in the fourth section ("Further interpretations") is inconsistent with the notation introduced in the first two sections ("Basic concept" and "ROC Space"). The figure uses TP, FP, TN and FN instead of TPR, FPR, TNR and FNR. Aside from being confusing, it is actually misleading since TP, FP, etc. were already introduced in the earlier sections as having different meanings than TPR, FPR, etc. What is more, the figure's notation isn't even internally consistent. The axis labels on the ROC graph should be "TP" and "FP", not "P(TP)" and "P(FP)". Alternatively, to show explicit dependence of the true positive rate and false positive rate on the threshold value, the axis labels could be "TP(θ)" and "FP(θ)", where the threshold value θ needs then to be introduced in the graph of the probability density curves for the detection statistic. And while I'm picking nits, why aren't the axes of the probability density graph labelled, and for that matter, why don't any of the three subfigures in this image have titles?
Don't get me wrong, I don't want to get rid of this image. For me it is the one illustration that allowed me to "get" what the ROC curve actually quantifies. Which is why I think it is important that it be brought into conformance with the rest of the article. I can see from the article's history that the image itself predates the discussion in the "Basic concept" and "ROC space" sections, so I imagine that the original creator of the image (Kku?) might be resistant to having it replaced with an updated version. However, our collective goal is for the overall article to be as clear as possible, and the best way that I can see to do that is to maintain the notation of the "Basic concept" section and to update the figure accordingly.
Here are the changes that I would propose to the figure:
- Titles for each of the three subfigures (these could be placed in the figure caption as long as the subfigures are labelled with a), b) and c)) - Axis labels where appropriate - Replace TP, FP, etc. with TPR, FPR, etc. - Introduce θ as the threshold value and replace P(TP) and P(FP) with TPR(θ) and FPR(θ)
One other thing worth mentioning on the topic of consistency, is that the confusion matrix in the figure is of a different form than that introduced in the "Basic concept" section, having its columns sum to 1 rather than to the respective probabilities of the underlying event occurring or not. I don't think that this should be changed in the name of consistency, however, because as it is, it provides a direct link between the two other subfigures in the image. If the confusion matrix were altered so that the columns sum respectively to P and N, then this link would be lost and the subfigure would only serve to introduce (dare I say it?) confusion.
Personally, if effort will be taken to update this figure, I think it might be wortwhile to introduce one more subfigure at the top showing the information flow (underlying two-state process --> observable data --> detection statistic --> decision), but this may not be the best choice of language if my stated goal is to enforce consistency with the rest of the article.
JanRu 20:26, 26 April 2007 (UTC)
In the section, "ROC Space", the info-box is referenced as containing evaluation metrics. Perhaps inadvertently, the word metric is hyperlinked to the wiki page on metrics, as in metric space distances. The reader may be inclined to believe from this that the info-box contains metrics. This is not the case. None of the "evaluation" metrics listed are true metrics in the mathematical sense. I would recommend deleting the hyperlink.
In a similar discussion, the notion of a ROC space is incorrect. What is meant by space? It is neither a vector space nor a topological space and so the verbiage is abused, even though it appears in some of the cited literature. A ROC graph is what is presented and its limitations are made clear, but the notion of a space is ill advised. I recommend titling the section as ROC graphs.
Snthor 14:21, 9 August 2007 (UTC)
The article says about d' "... under the assumption that both these distributions are normal with the SAME standard deviation" (my emphasis). But the article about d' uses "the standard deviation of the noise distribution". Stevemiller 04:46, 10 October 2007 (UTC)
Does the number of observations affect the ROC curve at all? With only one observation isn't it possible to have 100% sensitivity (no false negatives) and 100% specificity (no false positives)? Presumably I'm missing something because if that's the case having a good point on a ROC curve doesn't guarantee a good classifier. pgr94 (talk) 18:55, 24 January 2009 (UTC)
You'd have a good ROC curve, but no statistical reason to believe that this curve is representative of actual behaviour. --60.234.219.72 (talk) 01:27, 25 August 2009 (UTC)
This is an excellent addition to this article - very helpful for people wanting to dive deeper. Thanks so much. 128.220.160.6 (talk) 00:48, 9 March 2009 (UTC)
what does eqv. stand for????
From Revision #314192656 by 151.148.122.100 "Failed to parse (unknown function\MCC): \MCC = (TPTN - FPFN)/ \sqrt{P N P' N'}" ... reverted to working formula, however, the formula is rendered as a PNG - anyone who knows how to enforce text rendering, please be my guest. - Dlefree-loc-work (talk) 08:55, 21 September 2009 (UTC)
Perhaps something should be added to note that the further you move towards the upper-right in the ROC graph, the more often that the classifier gives you a positive answer. So, I think that movement in a diagonal direction towards the upper right corresponds to biasing the classifier to return a positive answer more often, without improving its accuracy. I'd add this myself, but I'd like someone more knowledgable to double-check it first. Bayle Shanks (talk) 22:07, 6 November 2009 (UTC)
ROC curve is also called a "lift curve" according to the book "Mastering Data Mining" by Berry and Linoff. —Preceding unsigned comment added by AndrewHZ (talk • contribs) 03:48, 6 December 2009 (UTC)
I thought that a lift chart had a different, but similar, X-axis than an ROC curve. The x-axis is the false positive rate in an ROC curve but it is the subset size (% of data tested) in a lift chart (see Data Mining: Practical Machine Learning Tools and Techniques by Witten and Frank). Mickeyg13 (talk) 22:27, 9 June 2010 (UTC)
Just gone looking for a source for the following summary statistic:
If you've got a reference for this one, it'd be much appreciated. —Preceding unsigned comment added by Noogz (talk • contribs) 06:23, 8 March 2010 (UTC)
As I understand, the ROC Curve is created by plotting quotients of integers against each other. But this means, that on both dimensions of the ROC space the irrational numbers do not have a ROC point. But if this is the case, the ROC "Curve" is defined for pairs of rational numbers only. But if this is the case the ROC curve is majorised by the Dirichlet function, which has a Lebesgue measure of 0. Thus the area under curve for a ROC "curve" is at most 0, if it ever exists. So in consequence, this means, that the AUC increases with every additional data point, however the convergence of the appropriate "measure" for the area under the curve is by no means guarantueed. It follows, that the AUC is a meaningless number, because it will rise with any additional observation and might achieve any number that wiill be given, supposed that enough observations for the classifier are available. Please point out my error, and I would happily accept that I am wrong. 80.153.50.105 (talk) 14:33, 4 June 2010 (UTC)
Under the Further Interpretations section, a statistic is defined as: "the intercept of the ROC curve with the line at 90 degrees to the no-discrimination line"
But there are an infinite number of lines perpendicular to any line. So this intercept can arrive at any point. Is this meant to be more specifically the intercept of the ROC curve with a line at 90 degrees to the no-discrimination line intersecting at its midpoint? —Preceding unsigned comment added by 32.166.60.40 (talk) 16:01, 4 November 2010 (UTC)
How about discussing the meaning and use of cut offs as illustrated here? (The cut offs are the labels 0.1, 0.2, etc on the curve.) AndrewHZ (talk) 16:55, 6 December 2010 (UTC)
The false positive rate is the conditional probability that a person truly has a disease given that they test positive. In other words, P(D|+). In signal detection theory there is something called the false alarm rate, which is being incorrectly used in this article as the false positive rate. This is all made clear in Statistical Methods For Rates And Proportions by David L. Fleiss.
This misunderstanding of the false positive rate is unfortunately fairly widespread, and having it misstated in this article is no help. —Preceding unsigned comment added by 12.190.115.98 (talk) 14:06, 24 January 2011 (UTC)
Is there a ref for this [3] thanks. Zulu Papa 5 * (talk) 02:10, 7 March 2011 (UTC)
Just noted that the Confusion matrix page gives the columns as predicted values and rows as actual values, whereas the confusing matrix used on this page has swapped the meaning of columns and rows. —Preceding unsigned comment added by 80.216.132.209 (talk) 20:49, 10 April 2011 (UTC)
"If a z-transformation is applied to the ROC curve, the curve will be transformed into a straight line." - How so? I don't see how that can follow, in the general case. (I assume we're talking about z-score and not the Z transform.) Converting the data to z-scores is a linear transform. --mcld (talk) 09:37, 23 November 2011 (UTC)
Mistakes in:
This should be a biased coin. And more is not good. — Preceding unsigned comment added by 129.125.21.37 (talk) 10:34, 16 April 2012 (UTC)
I came to the discussion page to look for an explanation of this, and I agree that the wording is unclear at best---it seems to suggest that for a small data set you would end up somewhere along the diagonal, and as the number of data points increases, you converge to (0.5,0.5). This is not what happens, you end up on the diagonal for large data sets, and where you end up is determined by the bias of the coin. --passerby — Preceding unsigned comment added by 24.6.143.43 (talk) 22:08, 9 July 2014 (UTC)
In the section "Other measures": The graph shows that if one detects at least 60% of the epitopes in a virus protein, at least 30% of the output is falsely marked as epitopes. I wonder if the second half of the sentence is wrong -- we should say "at least 30% of non-epitopes are falsely detected". MaigoAkisame (talk) 19:41, 8 August 2012 (UTC)
Can someone add some comments/references on the choice of the threshold? Indeed, after measuring the ROC curve of a system, one may want to optimize the system by choosing a specific point on the curve, i.e. a specific threshold, either by:
Lagaffe (talk) 10:42, 12 November 2012 (UTC)
The AUC section contains a formula with undefined variables, viz., X, Y and k. This is not acceptable in an article. I presume it was just copied from some text somewhere in which the definitions were given. If the author of that section is monitoring this talk page, I implore him/her to please define the variables. This is a bad (and annoying) practice and we should try to force clear definitions in every instance on wikipedia. Formulas are nearly useless without the definitions of their elements. Chafe66 (talk) 22:21, 30 January 2013 (UTC)
In the paragraph headed "Z-transformation", I think the blue link on the first occurrence of the phrase "Z-transformation" is in error. The linked page seems to have nothing to do with the transformation being referred to here. The reference should be to some transformation associated with the Normal distribution. Stephen Robertson (talk) 09:42, 24 April 2013 (UTC)
The Basic concept section (which begins "A classification model (classifier or diagnosis) is...") contains a piece of text identical to text I've found in page 3 of this document. That document has a publishing date of 2003, while the text here seems to have been added in 2007 by this edit. As a side note, while I initially thought the paragraphs following it were also a copyvio, they have in fact apparently been reproduced in this book published in 2010. AdventurousSquirrel (talk) 13:09, 13 January 2014 (UTC)
This article has been tagged for external link cleanup since March 2010. In looking over the existing external links, most of them are very borderline; they're mostly to personal pages of mathematicians working in the field or mathematicians' collections of academic papers on the subject. I can see an argument for each one to be included, but taken as a whole, they do feel kind of crufty and excessive.
I fully expect for at least part of this particular removal to be (justifiably) overridden by the regular editors -- I'm working through cleanup backlog of the external-link-template categories; I'm not a regular here -- because unlike the more egregious instances of "holy 73 links to personal webpages, Batman!" that the external-link-template tends to collect, I'm really on the fence about this one. I decided, however, to remove the external links section entirely, toss them here onto the talk page, and ask that instead of y'all reverting my edit wholesale that you instead take the chance to consider each link carefully and manually add back only the most justified and useful links to the article.
For what it's worth, and as an external opinion, the ones I really was this close to retaining were Kelly Zou's bibliography and the web-based calculator by John Eng.
--rahaeli (talk) 15:46, 18 May 2014 (UTC)
(PS: as someone with a nasty case of undiagnosed dyscalculia, leading to a strongly conditioned math-anxiety response, I have to commend the regular editors here: this is a wonderful example of specialist subject matter being written in such a way as to be crystal clear to the layperson, ie, me. I think this is one of the articles where Wikipedia really shines. --rahaeli (talk) 15:51, 18 May 2014 (UTC))
Just curious how the hell the name came about, as it seems a bit flaunty given that it's just true positives vs false positives. — Preceding unsigned comment added by Mmkstarr (talk • contribs) 00:56, 18 March 2016 (UTC)
http://www.math.utah.edu/~gamez/files/ROC-Curves.pdf See last paragraph. Hous21 (talk) 00:59, 18 March 2016 (UTC)
The article here holds a relatively critical view of AUC for estimating classifier performance based on the paper by Hand. This research seems to be mostly superseded by the more recent paper by Flach cited later. "Small-sample precision of ROC-related estimates" explicitly talks about small sample estimates, which is not relevant for machine learning applications. Therefore paragraph seems inconsistent to me. There seems to be no current literature suggesting that AUC actually is not a good measure. Still the article goes on saying "One recent explanation of the problem with ROC AUC " which implies that there is a problem. Andreas Mueller (talk) 20:47, 29 April 2016 (UTC)
Hi,
I am new to Wikipedia, so please pardon me if this is not the correct space for this.
I would like to add information about the Total Operating Characteristic (TOC) curve. This is an alternative to the ROC curve that still provides all of the information the ROC curve presents such as area under the curve. However, it is designed slightly different so that you would be able to recreate a confusion matrix showing the hits, misses, correct rejections, and false alarms for each threshold. Dr. Robert Gil Pontius came up with the TOC curve as an improvement to the ROC curve, and he has a paper published in the International Journal of Geographical Information Science called, “The total operating characteristic to measure diagnostic ability for multiple thresholds” which explains how to construct a TOC curve and the advantages of using it over the ROC.
Should the TOC be added to the ROC article, or should it have its own article? Because we have a lot of content on the TOC curve, I think we should try to make a separate page for TOC. However, based on the little information I know about Wiki, creating a page and getting it approved is a more difficult task?
Thank you!
Crobbins5 (talk) 06:55, 19 March 2018 (UTC)
Doesn't seem like there are a lot of sources for this TOC. So maybe just a mention here. Without sources, it's difficult to justify a separate article. Zulu Papa 5 * (talk) 21:00, 19 March 2018 (UTC)
This is because the journal article was published fairly recently, and scientists are used to using the ROC curve. However, 26 have already cited Pontius and Si’s article. The TOC curve provides all of the information that the ROC curve does but is more intuitive, and Wikipedia seems like the perfect platform for spreading awareness to the public so that more can benefit from it.
Crobbins5 (talk) 03:47, 20 March 2018 (UTC)
There is a year-old Wikipedia article at Total operating characteristic. --Rumping (talk) 17:13, 17 April 2018 (UTC)
I just created an ROC image which might be interesting for this article:
--MartinThoma (talk) 19:53, 24 June 2018 (UTC)
Most ROC examples show a curve in a step-wise form, i.e., at a given cutoff value either TPR or FPR change, leading to a curve that is comprised of horizontal and vertical lines which are then only "smoothed" by the number of points. If at a given cutoff value, though, both, a TRUE and and FALSE data point are classifed as positive, because for some reason they have the same predicted value, the curve should provide a diagonal line piece as I understand (and as it is implemented, e.g., in the R package ROCR). I have seen versions, though, where in this case the curve is rather created by still using horizontal and vertical line pieces, usually in a manner that a fictive cutoff point is created as if by an infinite small amount the FALSE observation is classified as positive before the TRUE observation is classified. Obviously, this will reduce the AUC in comparison to the straight (diagonal) line.
Is there a "correct" approach to this? Jenskauf (talk) 14:19, 24 May 2020 (UTC)
@MartinThoma: and @Jahibadkaret: Is there a reason to have the ROC graph in a hand-drawn style? I can redraw the graph in a more conventional style if it fits the tone of the article better. Cheers, cmɢʟee⎆τaʟκ 16:04, 13 August 2021 (UTC)
Would anyone object if I simply booted this entire paragraph out of the lead section?
The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The true-positive rate is also known as sensitivity, recall or probability of detection in machine learning. The false-positive rate is also known as probability of false alarm and can be calculated as (1 − specificity). It can also be thought of as a plot of the power as a function of the Type I Error of the decision rule (when the performance is calculated from just a sample of the population, it can be thought of as estimators of these quantities). The ROC curve is thus the sensitivity or recall as a function of fall-out. In general, if the probability distributions for both detection and false alarm are known, the ROC curve can be generated by plotting the cumulative distribution function (area under the probability distribution from to the discrimination threshold) of the detection probability in the y-axis versus the cumulative distribution function of the false-alarm probability on the x-axis.
As the second paragraph in the entire article, it's really a bit much, a perfect example of the complaints about readability that have been expressed here numerous times over the years. Why are we being given detailed instructions on how to create and interpret a ROC curve, when we've barely been told what that even is or why we'd want to?
With the nightmare above excised, the entire lead then becomes this:
A receiver operating characteristic curve, or ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The method was originally developed for operators of military radar receivers starting in 1941, which led to its name.
ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. ROC analysis is related in a direct and natural way to cost/benefit analysis of diagnostic decision making.
The ROC curve was first developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battlefields and was soon introduced to psychology to account for perceptual detection of stimuli. ROC analysis since then has been used in medicine, radiology, biometrics, forecasting of natural hazards, meteorology, model performance assessment, and other areas for many decades and is increasingly used in machine learning and data mining research.
The ROC is also known as a relative operating characteristic curve, because it is a comparison of two operating characteristics (TPR and FPR) as the criterion changes.
To summarize: If used correctly, ROC curves are a very powerful tool as a statistical performance measure in detection/classification theory and hypothesis testing, since they allow having all relevant quantities in one plot.
Which, you know, is actually reasonable. It needs some minor cleanup to address the redundancy in the first and third paragraphs, now glaringly apparent without all that faff between them. And that "to summarize" simply has to go. (It's a lead section, duh! Its entire job is to summarize.) But, some rough edges notwithstanding, it's at least a passable attempt at introducing the concept of a ROC curve to the reader.
If we don't boot the Paragraph From Hell, then I'd like to at least trim it way, waaaay down. Like, literally a single sentence. Perhaps:
The ROC curve for a given decision model plots the rate of true positive results vs. false positive results at various threshold settings, providing a means of evaluating the model's performance.
Oversimplified? Hell yes. But that's the point of a lead section, and the reason there's an entire article right below it that fills in the details. (As long as it's merely oversimplified, and not wrong. I don't claim to be a topic expert; heck I'm not even particularly familiar with the topic.) -- (please use ((reply to|FeRDNYC))
on reply) FeRDNYC (talk) 16:42, 20 October 2021 (UTC)
that "to summarize" simply has to go. (It's a lead section, duh! Its entire job is to summarize.)There should be a list of Wikipedia red-flags (probably is, somewhere), and that should definitely be on it. If you have to summarize your lead section, something's not right up there. -- FeRDNYC (talk) 16:52, 20 October 2021 (UTC)
Fellow Wikipedians: I've proposed some changes to the formula infobox transcluded into this article, with the goal of trimming down its overpowering (if not excessive) width. My original message with some explanatory notes is at Template talk:Confusion matrix terms#Template_width, and you can see the revised template layout I've proposed by viewing its sandbox version.
There have been no responses over there in well over two months, and since the changes I'm proposing are significant enough to possibly be contentious, I wanted to invite any interested Wikipedians to discuss them over at the template's talk page. Thanks! FeRDNYC (talk) 00:12, 5 January 2022 (UTC)