Data Engineering

Kubernetes - ํ”„๋กœ๋•์…˜ ๋ฐ ์Šค์ผ€์ผ๋ง

Helm charts, HPA/VPA, StatefulSets, PersistentVolumes, RBAC, GKE, resource limits, Prometheus/Grafana, health probes

20 ๋ฉด์ ‘ ์งˆ๋ฌธยท
Senior
1

Kubernetes ์ƒํƒœ๊ณ„์—์„œ Helm์˜ ์ฃผ์š” ์—ญํ• ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ?

๋‹ต๋ณ€

Helm์€ Kubernetes์šฉ ํŒจํ‚ค์ง€ ๋งค๋‹ˆ์ €์ž…๋‹ˆ๋‹ค. ํ…œํ”Œ๋ฆฟํ™”๋œ YAML ํŒŒ์ผ์˜ ๋ชจ์Œ์ธ chart๋ฅผ ํ†ตํ•ด ๋ณต์žกํ•œ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์„ ์ •์˜, ์„ค์น˜ ๋ฐ ์—…๊ทธ๋ ˆ์ด๋“œํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Helm์€ ์ข…์†์„ฑ, ๋ฒ„์ „ ๋ฐ ์„ค์ •์„ ์žฌํ˜„ ๊ฐ€๋Šฅํ•œ ๋ฐฉ์‹์œผ๋กœ ๊ด€๋ฆฌํ•˜์—ฌ ๋ฐฐํฌ๋ฅผ ๋‹จ์ˆœํ™”ํ•ฉ๋‹ˆ๋‹ค.

2

Deployment์™€ StatefulSet์˜ ๊ทผ๋ณธ์ ์ธ ์ฐจ์ด์ ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ?

๋‹ต๋ณ€

StatefulSet์€ ๊ฐ pod์— ๋Œ€ํ•ด ์•ˆ์ •์ ์ด๊ณ  ์˜๊ตฌ์ ์ธ ID (๋„คํŠธ์›Œํฌ ์ด๋ฆ„, ์Šคํ† ๋ฆฌ์ง€)๋ฅผ ๋ณด์žฅํ•˜๋Š” ๋ฐ˜๋ฉด, Deployment๋Š” pod๋ฅผ ๊ตํ™˜ ๊ฐ€๋Šฅํ•œ ๊ฒƒ์œผ๋กœ ์ทจ๊ธ‰ํ•ฉ๋‹ˆ๋‹ค. StatefulSet์€ ๊ฐ ์ธ์Šคํ„ด์Šค๊ฐ€ ์žฌ์‹œ์ž‘ ์‹œ์—๋„ ID์™€ ๋ฐ์ดํ„ฐ๋ฅผ ์œ ์ง€ํ•ด์•ผ ํ•˜๋Š” ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค์™€ ๊ฐ™์€ stateful ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์— ํ•„์ˆ˜์ ์ž…๋‹ˆ๋‹ค.

3

Horizontal Pod Autoscaler (HPA)๋Š” ๋ ˆํ”Œ๋ฆฌ์นด ์ˆ˜๋ฅผ ์กฐ์ •ํ•˜๊ธฐ ์œ„ํ•ด ์–ด๋–ป๊ฒŒ ์ž‘๋™ํ•ฉ๋‹ˆ๊นŒ?

๋‹ต๋ณ€

HPA๋Š” Metrics Server๋ฅผ ํ†ตํ•ด pod ๋ฉ”ํŠธ๋ฆญ (CPU, ๋ฉ”๋ชจ๋ฆฌ ๋˜๋Š” ์‚ฌ์šฉ์ž ์ •์˜ ๋ฉ”ํŠธ๋ฆญ)์„ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๊ณ  ๋ชฉํ‘œ ์‚ฌ์šฉ๋ฅ ์„ ์œ ์ง€ํ•˜๊ธฐ ์œ„ํ•ด ๋ ˆํ”Œ๋ฆฌ์นด ์ˆ˜๋ฅผ ์ž๋™์œผ๋กœ ์กฐ์ •ํ•ฉ๋‹ˆ๋‹ค. ํ˜„์žฌ ์‚ฌ์šฉ๋Ÿ‰๊ณผ ๋ชฉํ‘œ ์‚ฌ์ด์˜ ๋น„์œจ์„ ๊ณ„์‚ฐํ•œ ๋‹ค์Œ ๊ทธ์— ๋”ฐ๋ผ ์Šค์ผ€์ผ ์—… ๋˜๋Š” ๋‹ค์šดํ•˜๋ฉฐ, thrashing์„ ๋ฐฉ์ง€ํ•˜๊ธฐ ์œ„ํ•ด cooldown ๊ธฐ๊ฐ„์„ ๋‘ก๋‹ˆ๋‹ค.

4

HPA (Horizontal Pod Autoscaler)์™€ VPA (Vertical Pod Autoscaler)์˜ ์ฐจ์ด์ ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ?

5

Kubernetes์—์„œ PersistentVolume (PV)์™€ PersistentVolumeClaim (PVC)์˜ ์—ญํ• ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ?

+17 ๋ฉด์ ‘ ์งˆ๋ฌธ

๊ธฐํƒ€ Data Engineering ๋ฉด์ ‘ ์ฃผ์ œ

Linux & Shell - ๊ธฐ์ดˆ

Junior
20๊ฐœ ์งˆ๋ฌธ

Git & GitHub - ๊ธฐ์ดˆ

Junior
20๊ฐœ ์งˆ๋ฌธ

๋ฐ์ดํ„ฐ ์—”์ง€๋‹ˆ์–ด๋ง์„ ์œ„ํ•œ ๊ณ ๊ธ‰ Python

Junior
25๊ฐœ ์งˆ๋ฌธ

Docker - ๊ธฐ์ดˆ

Junior
25๊ฐœ ์งˆ๋ฌธ

Google Cloud Platform - ๊ธฐ์ดˆ

Junior
20๊ฐœ ์งˆ๋ฌธ

CI/CD ๋ฐ ์ฝ”๋“œ ํ’ˆ์งˆ

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Docker Compose

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

FastAPI - ๋ฐ์ดํ„ฐ API

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Data Engineering์„ ์œ„ํ•œ ๊ณ ๊ธ‰ SQL

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Data Lake - ์•„ํ‚คํ…์ฒ˜ ๋ฐ ์ˆ˜์ง‘

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

๋ฐ์ดํ„ฐ ์—”์ง€๋‹ˆ์–ด๋ง์„ ์œ„ํ•œ BigQuery

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

PostgreSQL - ๊ด€๋ฆฌ

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Data Engineering์„ ์œ„ํ•œ Data Modeling

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Fivetran & Airbyte - ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

dbt - ๊ธฐ์ดˆ

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Apache Airflow - ๊ธฐ์ดˆ

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

Kubernetes - ๊ธฐ์ดˆ

Mid-Level
20๊ฐœ ์งˆ๋ฌธ

dbt - ๊ณ ๊ธ‰ ๊ธฐ๋Šฅ

Senior
20๊ฐœ ์งˆ๋ฌธ

ETL / ELT / ETLT ํŒจํ„ด

Senior
20๊ฐœ ์งˆ๋ฌธ

Apache Airflow - ๊ณ ๊ธ‰

Senior
20๊ฐœ ์งˆ๋ฌธ

Airflow + dbt - ํŒŒ์ดํ”„๋ผ์ธ ์˜ค์ผ€์ŠคํŠธ๋ ˆ์ด์…˜

Senior
20๊ฐœ ์งˆ๋ฌธ

PySpark - ๋Œ€๊ทœ๋ชจ ์ฒ˜๋ฆฌ

Senior
20๊ฐœ ์งˆ๋ฌธ

Google Pub/Sub - ๋ฐ์ดํ„ฐ ์ŠคํŠธ๋ฆฌ๋ฐ

Senior
20๊ฐœ ์งˆ๋ฌธ

Apache Beam & Dataflow

Senior
20๊ฐœ ์งˆ๋ฌธ

Terraform - Infrastructure as Code

Senior
20๊ฐœ ์งˆ๋ฌธ

NoSQL ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค

Senior
20๊ฐœ ์งˆ๋ฌธ

๋ชจ๋˜ Data Architecture

Senior
20๊ฐœ ์งˆ๋ฌธ

๋ชจ๋‹ˆํ„ฐ๋ง ๋ฐ ๊ด€์ฐฐ ๊ฐ€๋Šฅ์„ฑ

Senior
20๊ฐœ ์งˆ๋ฌธ

IAM ๋ฐ ๋ฐ์ดํ„ฐ ๋ณด์•ˆ

Senior
20๊ฐœ ์งˆ๋ฌธ

๋‹ค์Œ ๋ฉด์ ‘์„ ์œ„ํ•ด Data Engineering์„ ๋งˆ์Šคํ„ฐํ•˜์„ธ์š”

๋ชจ๋“  ์งˆ๋ฌธ, flashcards, ๊ธฐ์ˆ  ํ…Œ์ŠคํŠธ, ์ฝ”๋“œ ๋ฆฌ๋ทฐ ์—ฐ์Šต, ๋ฉด์ ‘ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ์— ์ ‘๊ทผํ•˜์„ธ์š”.

๋ฌด๋ฃŒ๋กœ ์‹œ์ž‘ํ•˜๊ธฐ