Lazy is reasonable in C++, where application-logic gets compiled and is in order of nano-seconds, it quickly breaks down in Python, where application logic is interpreted. Doing N for-loops queuing compute for a batch-size N will effectively introduce N * 1us overhead. per op
— Soumith Chintala (@soumithchintala) October 6, 2018
No, it does matter. CNNs have ops so costly that the eager overhead on CPU is entirely hidden behind GPU execution, so you don’t pay anything for it. But eager never postpones running the kernels, so it will have lower latency than lazy (and it’s still somewhat cheaper).
— Adam Paszke (@apaszke) October 6, 2018
Basically lazy pays graph construction + GPU cost (because you only start computing once you see all of it), while eager pays only the GPU cost (because GPU runs async while you queue the kernels).
— Adam Paszke (@apaszke) October 6, 2018
As someone who hasn't played with lazy graph frameworks, I learned a lot from this discussion of the pros and cons https://t.co/f5DA1F4xiK
— Jeremy Howard (@jeremyphoward) October 6, 2018
