> For the complete documentation index, see [llms.txt](https://tech-de.gitbook.io/muse-data-engineer/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://tech-de.gitbook.io/muse-data-engineer/readme.md).

# muse-data-engineer

* Technical Skills to becoming a data engineer

## Data Ingestion

* Kafka
  * [이진로그 Binary log(Binlog)](/muse-data-engineer/data-ingestion/data-ingestion/binlog.md)
  * [변경 데이터 캡처 CDC](/muse-data-engineer/data-ingestion/data-ingestion/cdc.md)
* Embulk
  * [Embulk & Digdag](/muse-data-engineer/data-ingestion/data-ingestion-1/embulk.md)
  * [Embulk 코드](/muse-data-engineer/data-ingestion/data-ingestion-1/embulk_code.md)

## Batch Processing

* Hadoop
  * [Hadoop과 HDFS](/muse-data-engineer/doc/batch-processing/hadoop_hdfs.md)
  * [Mapreduce와 YARN](/muse-data-engineer/doc/batch-processing/hadoop_map_yarn.md)
  * [Hadoop ECO System](/muse-data-engineer/doc/batch-processing/hadoop_eco.md)
* Spark
  * [Spark 개요](/muse-data-engineer/doc/batch-processing-1/spark_base.md)
  * [Spark 튜닝](/muse-data-engineer/doc/batch-processing-1/spark_tuning.md)
  * [Spark 최적화](/muse-data-engineer/doc/batch-processing-1/spark_optimization.md)
  * [Spark Yarn](/muse-data-engineer/doc/batch-processing-1/spark_yarn.md)
  * [Spark 클러스터 매니저](/muse-data-engineer/doc/batch-processing-1/spark_cluster_manager.md)
  * [Spark 조인과 셔플](/muse-data-engineer/doc/batch-processing-1/spark_join.md)
* Batch SQL
  * Presto
    * [Presto 개요](/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_base.md)
    * [Presto 튜닝](/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_tuning.md)
    * [Presto 쿼리 Processing](/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_query_processing.md)
    * [Trino 개요](/muse-data-engineer/doc/batch-sql/batch-processing-1/trino_base.md)
    * [Trino 최적화](https://github.com/mjs1995/muse-data-engineer/blob/main/doc/batch-sql/batch-processing-1/trino_tuning.md)
  * Hive
    * [Hive 개요](/muse-data-engineer/doc/batch-sql/batch-processing/hive_base.md)
    * [Hive 아키텍처](/muse-data-engineer/doc/batch-sql/batch-processing/hive_architecture.md)
    * [Hive 포맷](/muse-data-engineer/doc/batch-sql/batch-processing/hive_format.md)
    * [HiveQL](/muse-data-engineer/doc/batch-sql/batch-processing/hive_hiveql.md)

## workflow

* Airflow
  * [Airflow 개요](/muse-data-engineer/workflow/airflow/airflow_base.md)
  * [Airflow 아키텍처](/muse-data-engineer/workflow/airflow/airflow_architecture.md)
* Dbt
  * [dbt 개요](/muse-data-engineer/workflow/dbt_base.md)
* Prefect
  * [Prefect 개요](/muse-data-engineer/workflow/prefect_base.md)

## BI

* [OLAP vs OLTP](/muse-data-engineer/bi/olap.md)
* [데이터 모델링과 DW/DM](/muse-data-engineer/bi/data_modeling_dw_dm.md)
* [데이터 레이크와 클라우드 DW](/muse-data-engineer/bi/data_lake.md)
* [데이터 메시와 데이터 패브릭](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/data_mesh_fabric.md)
* Snowflake
  * [Snowflake 개요](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_base.md)
  * [Snowflake\_아키텍처](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_arc.md)
  * [Snowflake 코드](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_handson.md)

## Back-End Development

* [인프라 기초](/muse-data-engineer/back-end-development/infra_based.md)
* [클라우드와 온프레미스](/muse-data-engineer/back-end-development/onpremises_cloud.md)
* [모놀리틱 아키텍처와 마이크로서비스 아키텍처](/muse-data-engineer/back-end-development/msa.md)
* Kubernetes
  * [Kubernetes 개요](/muse-data-engineer/back-end-development/kubernetes/kubernetes_base.md)
  * [Kubernetes 오브젝트 모델](/muse-data-engineer/back-end-development/kubernetes/kubernetes_object.md)
  * [Kubernetes 파드](/muse-data-engineer/back-end-development/kubernetes/kubernetes_pod.md)
  * [Kubernetes 레플리케이션](/muse-data-engineer/back-end-development/kubernetes/kubernetes_replica.md)
  * [Kubernetes 서비스](/muse-data-engineer/back-end-development/kubernetes/kubernetes_service.md)
  * [Kubernetes 볼륨](/muse-data-engineer/back-end-development/kubernetes/kubernetes_volume.md)
  * [Kubernetes 컨피그맵과 시크릿](/muse-data-engineer/back-end-development/kubernetes/kubernetes_config_secret.md)
  * [Kubernetes 디플로이먼트](/muse-data-engineer/back-end-development/kubernetes/kubernetes_deployment.md)
* Docker
  * [Docker 개요](https://github.com/mjs1995/muse-data-engineer/blob/main/back-end-development/docker/docker_base.md)
* IaC
  * [IaC와 Terraform](https://github.com/mjs1995/muse-data-engineer/blob/main/back-end-development/terraform.md)

## Programming Language

* Python
  * [Python과 프로파일링](/muse-data-engineer/programming-language/programming-language/python_profiling.md)
  * [Python 컴파일](/muse-data-engineer/programming-language/programming-language/python_comfile.md)
  * [Python 비동기](/muse-data-engineer/programming-language/programming-language/python_async.md)
  * [Python multiprocessing](/muse-data-engineer/programming-language/programming-language/python_multiprocessing.md)