Tuesday, March 25, 2014

Parameter Tuning

Update 5/13 B: Looks about the same with heaven rankings

Closeness definition: (ranges from 0: bad to 1: good)


Graphs; Beware: they alternate Heaven+Hell, pD+pF





















Update 5/13: Looking good against default parameters

The learners:
Learner# of Parameters# of Param Permutations
SKL Gaussian Bayes01
SKL Multinomial Bayes214
SKL Bernoulli Bayes398
Total113


"Unoptimized" learners are the following default parameters:

  • SKL Gaussian Bayes: {}
  • SKL Multinomial Bayes {'alpha': 1.0, 'fit_prior': True}
  • SKL Bernoulli Bayes {'binarize': 0.5, 'alpha': 1.0, 'fit_prior': True}

Getting away from the whole "top ranked" reporting, I'm now reporting top 10 mean g.
  • This still isn't the best, considering it's a summary statistic
  • How do you feel about non-dominated sort on pD, 1-pF?




















 Update 4/08: Interfaced with JMOO, nothing too fancy yet.


Update: More Evals, New Sets, Corrected Stats

These results generated with the following Xval/permutation params: (this took about 1.5hrs)
default_cross_val_folds=4
default_cross_val_repeats=4
default_param_permutations=5
default_percent_test=1.0/default_cross_val_folds

Stats generated with:
self.metrics={
'accuracy':metrics.accuracy_score(self.actual,self.predicted),
'precision':metrics.precision_score(self.actual,self.predicted),
'recall':metrics.recall_score(self.actual,self.predicted),
'confusion matrix':metrics.confusion_matrix(self.actual,self.predicted),
'Jaccard Score':metrics.jaccard_similarity_score(self.actual,self.predicted),
'ROC-AUC':metrics.accuracy_score(self.actual,self.predicted),
'class precision':metrics.precision_score(self.actual,self.predicted, average=None)
}
cm=self.metrics['confusion matrix']
d=cm[1][1]                                               #######from here out, it's basically your ABCD code, but I had to flip the CM
b=cm[1][0]
c=cm[0][1]
a=cm[0][0]
pd = 1.0*d / (b+d)
pf = 1.0*c / (a+c)
pn = 1.0*(b+d) / (a+c)
prec = 1.0*d / (c+d)
g = 2.0*(1-pf)*pd / (1-pf+pd)
f = 2.0*prec*pd/(prec+pd)
assert(sum(cm[1])==sum(self.actual))
self.metrics.update({
'pD':pd,
'pF':pf,
'g':g,
'f':f,

'pN':pn
})

Tabular results:
http://unbox.org/things/var/ben/consolidated/pres/results_summary_3-25-14.txt
http://unbox.org/things/var/ben/consolidated/pres/full_tuning_results_3-25-14.txt

Note about the charts: I switched to 1-pF so that heaven, hell are always (1,1) and (0,0).
















Results from train on N, test on N+1






http://unbox.org/things/var/ben/consolidated/