Tweeted By @karpathy
Excellent and unintuitive read on GPUs. The chip doing the compute has tiny amount of memory & is connected to the main memory literally through a straw. Most of the energy goes to data movement too. Many repercussions. E.g. latency better predicted by # activations than # flops https://t.co/67PBOfEcNK
— Andrej Karpathy (@karpathy) March 15, 2022