ACM KDD CUP

The contestants were evaluated based on their performance on the validation data set. The following performance metrics were considered:

a) Gains chart, i.e., lift table listing the cumulative percent of responders recovered in the top quantiles of the file;
b) Receiver operating characteristics (ROC) curve analysis and the area under the ROC curve;
c) Statistical tests, i.e., analysis of variance and various correlational measures between the actual dependent variable and the predicted probability estimate/score.

The results were almost always indicative of the 'photo finish' situation between the BNB software and the Gain software. MineSet software was the consistent runner-up following the top two constants with very close scores.

Because the results were too close to call, we pursued additional analyses by repeatedly sampling at random from the validation data sets and compared the results. In terms of the performance metric, we settled on the gains charts as the ROC curve analysis results were closely mirroring these results. Final calls were made based on the combination of the performance in the top 10 and 40 percent of the file. The performance in the top 10 percent is looked at as a measure of precision while the performance in the top 40 percent of the file is related to the stability and marketing coverage criteria.

An overall performance metric based on the average cumulative percent of responders recovered up to the 40th percentile of the validation data set as a whole is listed in Table 1. Table 2 and 3 list the average performance in the top 10 and 40 percent of the files repeatedly sampled at random from the validation data set.

Table 1: Average Overall Performance
Score (rounded to the nearest digit) gain     99 BNB      99 MineSet  97

Table 2: Average Performance (in TOP 10% of File)
Score (rounded to the nearest digit) BNB      100 gain     97 MineSet  95

Table 3: Average Performance (in TOP 40% of File)
Score (rounded to the nearest digit) gain     100 BNB      98 MineSet  98