One odd thing on Twitter (& elsewhere, but it's so common here) is people making insightful points with such hostility that they greatly damage their chances of reaching the people they think they're trying to reach.
— michael_nielsen (@michael_nielsen) September 21, 2019
I think this is an important observation by @karpathy. Somehow the areas of active learning class imbalance in ML and experimental design in stats haven’t quite addressed this (please correct me if I’m wrong). We focus too much on standard datasets and iidiness. https://t.co/ktajEKTZjp
— Nando de Freitas (@NandoDF) September 21, 2019
(likely an artifact of most of academia focused on finding models conditioned on standard datasets)
— Andrej Karpathy (@karpathy) September 20, 2019
We see more significant improvements from training data distribution search (data splits + oversampling factor ratios) than neural architecture search. The latter is so overrated :)
— Andrej Karpathy (@karpathy) September 20, 2019
In my experience most #ML competition results are far more reliable and reproducible then, oh, I don’t know, most of scientific research.https://t.co/pWBO6clJLP
— Bojan Tunguz (@tunguz) September 20, 2019
I do the opposite and also tell my students to. I find by keeping the randomness as I experiment I get a better intuitive feel for the modeling problem.
— Jeremy Howard (@jeremyphoward) September 20, 2019
Very interesting to hear your very different approach! :)
- Your model runs will have the exact same perplexity spikes (hits confusing data at the same time)
— Smerity (@Smerity) September 19, 2019
- You can compare timestamp / batch results in early training as a pseudo-estimate of convergence
- Improved gradient flow visibly helps the same init do better
i.e. If there's almost any possibility that your model can cheat (i.e. information leakage in the inputs), don't worry, it'll tell you*.
— Smerity (@Smerity) September 19, 2019
* Assuming your model isn't _really_ broken. If it's _really_ broken then getting it to learn to cheat is an important first step 🥳
I should clarify, there are a bunch of things that are obviously correct in the post. But, the headline/takeaway is not.
— Xavier Amatriain (@xamat) September 19, 2019
It would be fair to say in many cases the difference between 1st place and 10th place (for instance) isn't very important - but the insights from most competition results I've studied are fantastic
— Jeremy Howard (@jeremyphoward) September 19, 2019
The CS community has responded to ethical concerns by promoting a distinct field, algorithmic fairness. There are many questionable things about this approach. "Fairness" is a narrow conception of ethical responsibility, and it tends to get further reduced to a technical problem.
— Arvind Narayanan (@random_walker) September 19, 2019
Many of today's prominent computer scientists entered the field decades ago, inspired by the potential of digital technology to hold powerful institutions to account. But gradually tech has become an instrument of power. The tech industry itself badly needs more accountability.
— Arvind Narayanan (@random_walker) September 19, 2019
