Homepage
Close
Menu

Site Navigation

  • Home
  • Archive(TODO)
    • By Day
    • By Month
  • About(TODO)
  • Stats
Close
by HoloMarkeD on 2018-10-19 (UTC).

I avoid cron if I can and use event based functions, e.g. trigger when a file arrives. If its simple enough then that can run for free on cloud functions, otherwise Airflow for DAGs that can check in to Git, but if lots of data/streaming or row by row then Apache Beam

— Mark Edmondson (@HoloMarkeD) October 19, 2018
tip
by HoloMarkeD on 2018-10-19 (UTC).

Also good to have data validation checks, that trigger reloads if data not looking right. Airflow good for the email alerts and SLAs in that regard

— Mark Edmondson (@HoloMarkeD) October 19, 2018
tip

Tags

learning tutorial misc nlp rstats gan ethics research dataviz survey python tool security kaggle video thought bayesian humour tensorflow w_code bias dataset pytorch cv tip application javascript forecast swift golang rl jax julia gnn causal surey diffusion
© Copyright Philosophy 2018 Site Template by Colorlib