BigData / Data pipeline interview questions
1. What is Data pipeline?
A data pipeline is a series of data processing steps. If the data is not currently loaded into the data platform, then it is ingested at the beginning of the pipeline....
2. Why is Data pipeline required?
Data need to be streamlined for Data Science, Machine learning, Business Analytics and Reporting.
3. Difference Between a Data Scientist and a Data Engineer.
The main responsibility of a data scientist is to analyze data and produce suggestions for actions to take to improve a business metric, and then monitor the results of implementing those actions. In contrast, a data engineer is responsible for...
4. Difference between Data pipeline and ETL pipeline.
Data Pipelines and ETL Pipelines, both signify processes for moving data from one system to the other; they are not entirely the same thing. Below are three key differences: Data Pipeline Is an Umbrella Term of Which ETL Pipelines Are...
5. Types of Data processing Systems.
There are t2 types of data processing systems: online analytical processing (OLAP) and online transaction processing (OLTP). The main difference is that OLAP uses data to gain valuable insights, while the other is purely operational....
6. What are the differences between a database and a data warehouse?
A database is any collection of data organized for storage, accessibility, and retrieval. A data warehouse is a type of database the integrates copies of transaction data from disparate source systems and provisions them for analytical use.
7. What is Master Data Management (MDM)?
Master Data Management helps to create one single master reference source for all critical business data, leading to fewer errors and less redundancy in business processes.
8. Mention a few popular cloud data warehouses.
Snowflake, Google BigQuery, Amazon Redshift, Azure Synapse Analytics, and IBM Db2 warehouse.
9. What is a data lake?
A data lake is a centralized repository designed to store, process, and secure large amounts of structured, semistructured, and unstructured data. It can store data in its native format and process any variety of it, ignoring size limits.
10. Types of Data Pipelines.
The most common types of data pipelines include: Batch : When companies need to move a large amount of data regularly, they often choose a batch processing system. With a batch data pipeline, this data is not transferred in real-time...
11. Mention a few Open-source Data pipeline tools.
Talend Apache Kafka Apache Airflow
12. Explain the Key Components of a Data Pipeline.
The Data Sources can be a variety of systems, such as databases, APIs, and flat files. The data pipelines must extract the data from these sources and bring it into the pipeline....
