Deep learning training tip that I realized I do but never learned from anyone - when tweaking your model for improving gradient flow / speed to converge, keep the exact same random seed (hyperparameters and weight initializations) and only modify the model interactions.
— Smerity (@Smerity) September 19, 2019
- Your model runs will have the exact same perplexity spikes (hits confusing data at the same time)
— Smerity (@Smerity) September 19, 2019
- You can compare timestamp / batch results in early training as a pseudo-estimate of convergence
- Improved gradient flow visibly helps the same init do better
This may or may not be a soft pseudo-science variant of the Lottery Ticket Hypothesis / @hardmaru et al's Weight Agnostic Neural Networks. Either way it has worked multiple times over multiple datasets for me and the results seem to generalize.
— Smerity (@Smerity) September 19, 2019
I do the opposite and also tell my students to. I find by keeping the randomness as I experiment I get a better intuitive feel for the modeling problem.
— Jeremy Howard (@jeremyphoward) September 20, 2019
Very interesting to hear your very different approach! :)
