Skip to main content
Back to top
Ctrl
+
K
DE4E: Data Engineering for Everybody
๐ Check List
1. Introduction to Data Engineering
์ฐ๋ฆฌ๋ ๋น ๋ฐ์ดํฐ ์๋์ ์ด๊ณ ์์ต๋๋ค
1.1. ๋ฐ์ดํฐ ์์ง๋์ด๋ง์ด๋ ๋ฌด์์ผ๊น?
1.2. ์ฌ๋ก๋ฅผ ํตํด ์์๋ณด๋ ๋ฐ์ดํฐ ์์ง๋์ด๋ง
1.3. ๋ฐ์ดํฐ ์์ง๋์ด๋ ์ด๋ป๊ฒ ์งํํ๊ณ ์์๊น?
1.4. ๋ฐ์ดํฐ ์์ง๋์ด๋ ์ด๋ค ๊ณ ๋ฏผ์ ํ ๊น?
Recap
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
2. Data Source and Data Collection
2.1 ๋ฐ์ดํฐ ์์ง๋์ด๊ฐ ๋ค๋ฃจ๋ ๋ฐ์ดํฐ
2.2 ๋ฐ์ดํฐ๋ฅผ ์์งํ๋ ๋ฐฉ๋ฒ
2.3 ๋ฐ์ดํฐ ์์ง์์์ ๊ณ ๋ คํ ์
2.4 ๋ฐ์ดํฐ ์ฒ๋ฆฌ๋ฐฉ์ ์์๋ณด๊ธฐ
2.5 ๐ป Index ์ค์ต
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
3. Data Transformation and Cleaning
(์์ ๋ชฉ)
4. Data Storage
4.1 data warehouse / data lake / data lakehouse
4.2 Data Warehouse
4.3 Data Mart
4.4 Data Lake
4.5 Data Lakehouse
4.6 ๐ป Data Pipeline ์ค์ต
4.7 Databases
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
5. Data Processing Frameworks I
5.1 Data Processing์ ์ํ ๊ธฐ์ด ์ง์
5.2 ๋๊ท๋ชจ ๋ฐ์ดํฐ ์ฒ๋ฆฌ ํ์์ฑ๊ณผ ํ๋ ์์ํฌ
5.3 ๋ถ์ฐ์ฒ๋ฆฌ์ ๋ณ๋ ฌ์ฒ๋ฆฌ
5.4 ์บ์ฑ(Cache + ing)
5.5 ์ธ๋ฑ์ฑ
5.6 ๋ฉ๋ชจ๋ฆฌ ์ต์ ํ
5.7 ์ธ๋ฉ๋ชจ๋ฆฌ ๋๋น๋?
5.8 ๋ถ์ฐ ์์คํ
5.9 Hadoop
5.10 Hadoop Ecosystem
5.11 HDFS
5.12 MapReduce
๐ป MapReduce ์ค์ต
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
6. Data Processing Frameworks II
6.1 Stream Processing Framework
6.2 ์ํ์น ์คํํฌ(Apache Spark)
๐ป ์ค์ต: ์ํ์น ์คํํฌ
6.3 ์ํ์น ์นดํ์นด (Apache Kafka)
6.4 ์ํ์น ์คํฐ (Apache Storm)
6.5 ์ํ์น ํ๋งํฌ (Apache Flink)
6.6 ๋ฐ์ดํฐ ์ฒ๋ฆฌ๋ฐฉ๋ฒ ๋น๊ต
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
7. Introduction to Apache Airflow
7.1 Introduction to Apache Airflow
7.2 Airflow Executor
7.3 Airflow Use Cases
7.4 Airflow Hands-On
๐ป ์ค์ต: Airflow
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
8. Cloud Computing and Data Enginnering
8.1 Cloud Computing
8.2 Data Engineering
8.3 ํด๋ผ์ฐ๋ ์ปดํจํ ์์์ ๋ฐ์ดํฐ ์์ง๋์ด๋ง๊ณผ ํ๋ก์ธ์ค๋ค
8.4.
์ฌ๋ก ํ์ต - ํด๋ผ์ฐ๋ ์ปดํจํ ์์์ ๋ฐ์ดํฐ ์์ง๋์ด๋ง
8.5.
๋ฉํฐ ํด๋ผ์ฐ๋ ํ๊ฒฝ์์์ ๋ฐ์ดํฐ ์์ง๋์ด๋ง
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
9. Kafka Zero-To-Hero
9.1. Kafka Intro
์นดํ์นด ๊ธฐ๋ณธ ๊ตฌ์กฐ
ํ ํฝ๊ณผ ํํฐ์
ํํฐ์ ๊ณผ ํ๋ก๋์ & ์ปจ์๋จธ
์นดํ์นด ์ฑ๋ฅ
์นดํ์นด ์ฅ์ ๋์
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
๐ Project - MovieFlix
1. ์ํคํ ์ฒ ์ค๋ช
2. Infra
3. ๋ฐ์ดํฐ ์์ฑ
4. Data Flow
5. Monitoring
๐๏ธย ์ฐธ๊ณ ์๋ฃ(Reference)
Repository
Open issue
.md
.pdf
4.1 data warehouse / data lake / data lakehouse
4.1 data warehouse / data lake / data lakehouse
#