Even my continued fiddling with the SHA-RNN model shows there's a _lot_ to be studied and explored. I haven't published new incremental progress but you can tie the RNN across the 4 layers to substantially decrease total params yet get nearly equivalent perplexity results.
— Smerity (@Smerity) January 28, 2020
I feel like good-old LSTM (or QRNN) are usually better for text classification indeed.
— Thomas Wolf (@Thom_Wolf) January 28, 2020
Note that for those who want to give a try at text classification with pretrained Bert models, you can give a look at the experimental section of this paper https://t.co/w4MWPTB79u
The other dimension here is monolingual vs multilingual models. I think monolingual models in low-resource languages currently have an edge e.g. as seen in our MultiFiT paper.
— Sebastian Ruder (@seb_ruder) January 28, 2020
I have thought about these two paragraphs so many times over the years.
— Chris Albon (@chrisalbon) January 24, 2020
Huge thanks to @ericpfreeman for finding the source. pic.twitter.com/3oJ29ib1WF
Soros on Science vs Alchemy:
— hardmaru (@hardmaru) January 24, 2020
“Scientific method seeks to understand things as they are, while alchemy seeks to bring about a desired state of affairs. To put it another way, the primary objective of science is truth—that of alchemy, operational success.”https://t.co/KvfMYYvM0Q
A good way to be unsure about something is to ask for one more opinion. More opinions often lead to indecision, not clarity. If necessary, ask for a few, add your own, make a call, and move on. Nearly all decisions are temporary, but stalling is permanent time lost.
— Jason Fried (@jasonfried) January 22, 2020
Less Software:
— Jason Fried (@jasonfried) January 22, 2020
You’d think that twice as much code would make your software only twice as complex. But actually, each time you increase the amount of code, your software grows exponentially more complicated.https://t.co/fOZn3S6gAZ
There are unfair advantages in the game of life, like being born white, male, or wealthy. It’s not right or OK, but for now it’s still the case.
— Brandon Rohrer (@_brohrer_) January 19, 2020
What you do with your advantages says a lot about you. If you attribute them to your own cleverness, that says a lot about you too. https://t.co/IVvhxBDpNn
4 ways computational research can help address social problems
— Rachel Thomas (@math_rachel) January 18, 2020
diagnostic: understand & measure
formalizer: shape how problems are defined
rebuttal: illuminates what's possible
synecdoche: make long-standing problem salient to publichttps://t.co/UFifZN9ksm @red_abebe @s010n pic.twitter.com/2GUqvJfal8
As someone with a nascent interest in causal inference, I agree with this sentiment.
— Andrew Beam (@AndrewLBeam) January 17, 2020
I think when most ML folks hear causal inference, they think "learning the causal structure of the world from data".
This is actually causal discovery, and is much harder. https://t.co/PswILRpSyD
In my experience very few disciplines care enough about their data.
— Neil Lawrence (@lawrennd) January 16, 2020
Very weird to pick on statisticians ... they care much more than most.
The only discipline I know that has understood the full importance of their data is accountancy, where data≡money. https://t.co/AOQ16KD713
5. Spend 4 days to do it well. Open-sourcing a good code base takes some time but you should consider it as important as your paper
— Thomas Wolf (@Thom_Wolf) January 14, 2020
6. Consider merging with a larger repo: are you working on language models? 🤗Transformers is probably happy to help you ➡️ https://t.co/iNdvPcRFPa
