#AI competitions don't produce useful models.
— Luke Oakden-Rayner (@DrLukeOR) September 19, 2019
As promised, here is a new blogpost explaining why AI competitions never seem to lead to products, how you can overfit on a hold out test set, and why imagenet results since the mid-2010s are suspect.https://t.co/F9H5iicgVq pic.twitter.com/fDUzEYRPDb
The test set (stage 1) for this competition is 400k cases.
— Andres Torrubia (@antor) September 19, 2019
In the recent Molecular competition I do believe (+ want to) we contributed to science, truth that different top models a few decimals across may be luck... But it's different than your assertion of "not useful".
It would be fair to say in many cases the difference between 1st place and 10th place (for instance) isn't very important - but the insights from most competition results I've studied are fantastic
— Jeremy Howard (@jeremyphoward) September 19, 2019
I should clarify, there are a bunch of things that are obviously correct in the post. But, the headline/takeaway is not.
— Xavier Amatriain (@xamat) September 19, 2019
In my experience most #ML competition results are far more reliable and reproducible then, oh, I don’t know, most of scientific research.https://t.co/pWBO6clJLP
— Bojan Tunguz (@tunguz) September 20, 2019
