> For the complete documentation index, see [llms.txt](https://tech-de.gitbook.io/muse-data-engineer/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://tech-de.gitbook.io/muse-data-engineer/readme.md).

# muse-data-engineer

* Technical Skills to becoming a data engineer

## Data Ingestion

* Kafka
  * [이진로그 Binary log(Binlog)](https://tech-de.gitbook.io/muse-data-engineer/data-ingestion/data-ingestion/binlog.md)
  * [변경 데이터 캡처 CDC](https://tech-de.gitbook.io/muse-data-engineer/data-ingestion/data-ingestion/cdc.md)
* Embulk
  * [Embulk & Digdag](https://tech-de.gitbook.io/muse-data-engineer/data-ingestion/data-ingestion-1/embulk.md)
  * [Embulk 코드](https://tech-de.gitbook.io/muse-data-engineer/data-ingestion/data-ingestion-1/embulk_code.md)

## Batch Processing

* Hadoop
  * [Hadoop과 HDFS](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing/hadoop_hdfs.md)
  * [Mapreduce와 YARN](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing/hadoop_map_yarn.md)
  * [Hadoop ECO System](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing/hadoop_eco.md)
* Spark
  * [Spark 개요](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing-1/spark_base.md)
  * [Spark 튜닝](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing-1/spark_tuning.md)
  * [Spark 최적화](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing-1/spark_optimization.md)
  * [Spark Yarn](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing-1/spark_yarn.md)
  * [Spark 클러스터 매니저](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing-1/spark_cluster_manager.md)
  * [Spark 조인과 셔플](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-processing-1/spark_join.md)
* Batch SQL
  * Presto
    * [Presto 개요](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_base.md)
    * [Presto 튜닝](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_tuning.md)
    * [Presto 쿼리 Processing](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_query_processing.md)
    * [Trino 개요](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing-1/trino_base.md)
    * [Trino 최적화](https://github.com/mjs1995/muse-data-engineer/blob/main/doc/batch-sql/batch-processing-1/trino_tuning.md)
  * Hive
    * [Hive 개요](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing/hive_base.md)
    * [Hive 아키텍처](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing/hive_architecture.md)
    * [Hive 포맷](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing/hive_format.md)
    * [HiveQL](https://tech-de.gitbook.io/muse-data-engineer/doc/batch-sql/batch-processing/hive_hiveql.md)

## workflow

* Airflow
  * [Airflow 개요](https://tech-de.gitbook.io/muse-data-engineer/workflow/airflow/airflow_base.md)
  * [Airflow 아키텍처](https://tech-de.gitbook.io/muse-data-engineer/workflow/airflow/airflow_architecture.md)
* Dbt
  * [dbt 개요](https://tech-de.gitbook.io/muse-data-engineer/workflow/dbt_base.md)
* Prefect
  * [Prefect 개요](https://tech-de.gitbook.io/muse-data-engineer/workflow/prefect_base.md)

## BI

* [OLAP vs OLTP](https://tech-de.gitbook.io/muse-data-engineer/bi/olap.md)
* [데이터 모델링과 DW/DM](https://tech-de.gitbook.io/muse-data-engineer/bi/data_modeling_dw_dm.md)
* [데이터 레이크와 클라우드 DW](https://tech-de.gitbook.io/muse-data-engineer/bi/data_lake.md)
* [데이터 메시와 데이터 패브릭](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/data_mesh_fabric.md)
* Snowflake
  * [Snowflake 개요](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_base.md)
  * [Snowflake\_아키텍처](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_arc.md)
  * [Snowflake 코드](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_handson.md)

## Back-End Development

* [인프라 기초](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/infra_based.md)
* [클라우드와 온프레미스](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/onpremises_cloud.md)
* [모놀리틱 아키텍처와 마이크로서비스 아키텍처](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/msa.md)
* Kubernetes
  * [Kubernetes 개요](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_base.md)
  * [Kubernetes 오브젝트 모델](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_object.md)
  * [Kubernetes 파드](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_pod.md)
  * [Kubernetes 레플리케이션](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_replica.md)
  * [Kubernetes 서비스](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_service.md)
  * [Kubernetes 볼륨](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_volume.md)
  * [Kubernetes 컨피그맵과 시크릿](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_config_secret.md)
  * [Kubernetes 디플로이먼트](https://tech-de.gitbook.io/muse-data-engineer/back-end-development/kubernetes/kubernetes_deployment.md)
* Docker
  * [Docker 개요](https://github.com/mjs1995/muse-data-engineer/blob/main/back-end-development/docker/docker_base.md)
* IaC
  * [IaC와 Terraform](https://github.com/mjs1995/muse-data-engineer/blob/main/back-end-development/terraform.md)

## Programming Language

* Python
  * [Python과 프로파일링](https://tech-de.gitbook.io/muse-data-engineer/programming-language/programming-language/python_profiling.md)
  * [Python 컴파일](https://tech-de.gitbook.io/muse-data-engineer/programming-language/programming-language/python_comfile.md)
  * [Python 비동기](https://tech-de.gitbook.io/muse-data-engineer/programming-language/programming-language/python_async.md)
  * [Python multiprocessing](https://tech-de.gitbook.io/muse-data-engineer/programming-language/programming-language/python_multiprocessing.md)
