Homepage
Close
Menu

Site Navigation

  • Home
  • Archive(TODO)
    • By Day
    • By Month
  • About(TODO)
  • Stats
Close
by Tim_Dettmers on 2021-07-06 (UTC).

In a few equations how it relates to grad clipping. This is the most important part of LAMB/LARS to understand.

Grad clip:

if norm(grad) > max_norm:
grad *= norm(grad)/max_norm

Update clip:

if norm(update) > max_unorm:
update *= norm(update)/max_unorm

— Tim Dettmers (@Tim_Dettmers) July 6, 2021
tutoriallearning
by bartoldson on 2021-07-06 (UTC).

Should this be "/=" instead of "*="? Thanks for sharing this!

— Brian Bartoldson (@bartoldson) July 6, 2021
misc

Tags

learning tutorial misc nlp rstats gan ethics research dataviz survey python tool security kaggle video thought bayesian humour tensorflow w_code bias dataset pytorch cv tip application javascript forecast swift golang rl jax julia gnn causal surey diffusion
© Copyright Philosophy 2018 Site Template by Colorlib