Lauri Koobas: Data Engineering - from early startup to scaling | Ep. 4

Lauri Koobas, ex-Microsoft and currently Head of Data Platform at Bondora, shed insights on data engineering - from early startup to scaling.We mostly focused on analytics and building data warehouse - real-world challenges from both data engineering and software engineering sides. We also discussed GDPR and PII challenges when dealing with data.You can find video version on MapForEngineers YouTube channel: https://www.youtube.com/@mapforengineersAnnotated chapters in timeline:00:00:00 Sneak peek of episode00:01:21 Episode overview00:02:44 Introduction, Lauri's background00:20:48 Starship robots: huge amount of data there00:23:37 Data lake, data warehouse, data lakehouse00:26:44 Devil is in the details: timestamps, texts, character sets...00:49:44 Moving data from prod to data warehouse00:53:09 Analytics tools: PostHog, Amplitude, Redash, Databricks01:00:15 Analytics tools vs real-time monitoring like Prometheus/Grafana01:04:15 Usability matters: each tool for its job01:06:38 Startup grows: needs in data analytics01:11:09 Multiple data sources: when data warehouse really begins01:19:55 Data and (de-)coupling: software engineers should not be blocked by analytics01:22:51 Data ETL01:24:59 Changes in data model: multi-phase migrations01:29:38 Change data capture, incremental imports01:34:21 Should analytics have new data in real time? Maybe not?01:39:02 Importing data into DWH through business events01:43:37 When DWH subscribes to business events, data model can evolve freely01:47:16 Quick recap what we discussed so far01:52:25 GDPR and Data Compliance: start early01:56:05 PII data: know exactly where you store it, control it well02:03:37 Lauri's books recommendations on data engineering - Kimball02:07:18 Lauri's podcast on data engineering, in Estonian02:08:28 Wrap up This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit log.mapforengineers.com