Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Ingest streaming data into Delta Lake with Auto Loader and Databricks

Microsoft DevRadio
12.4K subscribers
<__slot-el>
<__slot-el>
4.5K views 3 years ago
We’ll walk through how to simplify the process of bringing streaming data into Delta Lake as a starting point for live decision-making. https://github.com/jodb/DatabricksAnd... ...more
...more
These chapters are auto-generated

Intro

0:00

Welcome

0:25

Databricks

1:35

Autoloader

4:50

Demo Overview

11:50

Case Setup

14:35

First Data Set

15:05

Schema Loc

16:20

Two sides of the coin

19:45

Triggers

21:15

Incremental ETL

25:20

Viewing the data after ingestion

26:25

Delta table as a source of a stream

27:35

Silver table as source of a stream

29:10

Adding data to Bronze table again

31:10

Looking at Bronze table again

32:25

Event Hub Connector

36:40

Installing Libraries

37:55

Sending Small Data to Event Hub

39:25

Defining the schema

43:25

Writing the data to Delta

45:25

Testing the data

49:00

Viewing the data

50:10

Conclusion

51:05

Wrapup

52:45
Ingest streaming data into Delta Lake with Auto Loader and Databricks
36Likes
4,504Views
2021Nov 11
We’ll walk through how to simplify the process of bringing streaming data into Delta Lake as a starting point for live decision-making. https://github.com/jodb/DatabricksAnd...

Follow along using the transcript.

Microsoft DevRadio

12.4K subscribers