# muse-data-engineer

* Technical Skills to becoming a data engineer

## Data Ingestion

* Kafka
  * [이진로그 Binary log(Binlog)](/muse-data-engineer/data-ingestion/data-ingestion/binlog.md)
  * [변경 데이터 캡처 CDC](/muse-data-engineer/data-ingestion/data-ingestion/cdc.md)
* Embulk
  * [Embulk & Digdag](/muse-data-engineer/data-ingestion/data-ingestion-1/embulk.md)
  * [Embulk 코드](/muse-data-engineer/data-ingestion/data-ingestion-1/embulk_code.md)

## Batch Processing

* Hadoop
  * [Hadoop과 HDFS](/muse-data-engineer/doc/batch-processing/hadoop_hdfs.md)
  * [Mapreduce와 YARN](/muse-data-engineer/doc/batch-processing/hadoop_map_yarn.md)
  * [Hadoop ECO System](/muse-data-engineer/doc/batch-processing/hadoop_eco.md)
* Spark
  * [Spark 개요](/muse-data-engineer/doc/batch-processing-1/spark_base.md)
  * [Spark 튜닝](/muse-data-engineer/doc/batch-processing-1/spark_tuning.md)
  * [Spark 최적화](/muse-data-engineer/doc/batch-processing-1/spark_optimization.md)
  * [Spark Yarn](/muse-data-engineer/doc/batch-processing-1/spark_yarn.md)
  * [Spark 클러스터 매니저](/muse-data-engineer/doc/batch-processing-1/spark_cluster_manager.md)
  * [Spark 조인과 셔플](/muse-data-engineer/doc/batch-processing-1/spark_join.md)
* Batch SQL
  * Presto
    * [Presto 개요](/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_base.md)
    * [Presto 튜닝](/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_tuning.md)
    * [Presto 쿼리 Processing](/muse-data-engineer/doc/batch-sql/batch-processing-1/presto_query_processing.md)
    * [Trino 개요](/muse-data-engineer/doc/batch-sql/batch-processing-1/trino_base.md)
    * [Trino 최적화](https://github.com/mjs1995/muse-data-engineer/blob/main/doc/batch-sql/batch-processing-1/trino_tuning.md)
  * Hive
    * [Hive 개요](/muse-data-engineer/doc/batch-sql/batch-processing/hive_base.md)
    * [Hive 아키텍처](/muse-data-engineer/doc/batch-sql/batch-processing/hive_architecture.md)
    * [Hive 포맷](/muse-data-engineer/doc/batch-sql/batch-processing/hive_format.md)
    * [HiveQL](/muse-data-engineer/doc/batch-sql/batch-processing/hive_hiveql.md)

## workflow

* Airflow
  * [Airflow 개요](/muse-data-engineer/workflow/airflow/airflow_base.md)
  * [Airflow 아키텍처](/muse-data-engineer/workflow/airflow/airflow_architecture.md)
* Dbt
  * [dbt 개요](/muse-data-engineer/workflow/dbt_base.md)
* Prefect
  * [Prefect 개요](/muse-data-engineer/workflow/prefect_base.md)

## BI

* [OLAP vs OLTP](/muse-data-engineer/bi/olap.md)
* [데이터 모델링과 DW/DM](/muse-data-engineer/bi/data_modeling_dw_dm.md)
* [데이터 레이크와 클라우드 DW](/muse-data-engineer/bi/data_lake.md)
* [데이터 메시와 데이터 패브릭](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/data_mesh_fabric.md)
* Snowflake
  * [Snowflake 개요](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_base.md)
  * [Snowflake\_아키텍처](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_arc.md)
  * [Snowflake 코드](https://github.com/mjs1995/muse-data-engineer/blob/main/bi/snowflake/snowflake_handson.md)

## Back-End Development

* [인프라 기초](/muse-data-engineer/back-end-development/infra_based.md)
* [클라우드와 온프레미스](/muse-data-engineer/back-end-development/onpremises_cloud.md)
* [모놀리틱 아키텍처와 마이크로서비스 아키텍처](/muse-data-engineer/back-end-development/msa.md)
* Kubernetes
  * [Kubernetes 개요](/muse-data-engineer/back-end-development/kubernetes/kubernetes_base.md)
  * [Kubernetes 오브젝트 모델](/muse-data-engineer/back-end-development/kubernetes/kubernetes_object.md)
  * [Kubernetes 파드](/muse-data-engineer/back-end-development/kubernetes/kubernetes_pod.md)
  * [Kubernetes 레플리케이션](/muse-data-engineer/back-end-development/kubernetes/kubernetes_replica.md)
  * [Kubernetes 서비스](/muse-data-engineer/back-end-development/kubernetes/kubernetes_service.md)
  * [Kubernetes 볼륨](/muse-data-engineer/back-end-development/kubernetes/kubernetes_volume.md)
  * [Kubernetes 컨피그맵과 시크릿](/muse-data-engineer/back-end-development/kubernetes/kubernetes_config_secret.md)
  * [Kubernetes 디플로이먼트](/muse-data-engineer/back-end-development/kubernetes/kubernetes_deployment.md)
* Docker
  * [Docker 개요](https://github.com/mjs1995/muse-data-engineer/blob/main/back-end-development/docker/docker_base.md)
* IaC
  * [IaC와 Terraform](https://github.com/mjs1995/muse-data-engineer/blob/main/back-end-development/terraform.md)

## Programming Language

* Python
  * [Python과 프로파일링](/muse-data-engineer/programming-language/programming-language/python_profiling.md)
  * [Python 컴파일](/muse-data-engineer/programming-language/programming-language/python_comfile.md)
  * [Python 비동기](/muse-data-engineer/programming-language/programming-language/python_async.md)
  * [Python multiprocessing](/muse-data-engineer/programming-language/programming-language/python_multiprocessing.md)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://tech-de.gitbook.io/muse-data-engineer/readme.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.