BigData / Hadoop basics
1. What is Apache Hadoop?
Hadoop is a opensource framework that facilitates the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop provides the reliable, scalable way of distributed computing....
2. Hadoop Core modules.
Hadoop Common. The common utilities framework that is leveraged by other Hadoop modules....
3. Applications of Hadoop.
Business Intelligence. Business Analytics.
4. What is Big Data?
Big data refers to the large amount of data, a collection of large datasets that is infeasible to handle/process using existing computer technology/programmming model. Thus Big Data includes huge volume, high velocity and extensible variety of data, categorized into three...
5. Apache Hadoop Deployment modes.
Standalone Mode. Pseudo-Distributed Mode.
6. Advantages of using Apache Hadoop.
Fault Tolerant. Complex Data Analytics.
7. HDFS.
HDFS (Hadoop Distributed File System) is the file system used by Hadoop Framework for data Storage on Distributed Computing Environment. HDFS Is exceptionally scalable and faults tolerant.
8. What are the advantages of using HDFS?
The features of Hadoop Distributed File Systems are, Portable. Scalable.
9. Describe HDFS Architecture.
Below are some of the major components of Hadoop Distributed File System (HDFS). NameNode.
10. What is NameNode?
NameNode is a daemon process in HDFS which is responsible for all processes related to storage in Hadoop. It Manages the Hadoop File system namespace....
11. What is DataNode?
DataNode is a commodity hardware which actually stores the data and performs file-level operations as suggested by the NameNode. DataNode sends the signals to NameNode for its status....
12. What is Secondary NameNode?
As name described Secondary NameNode is not a backup of the NameNode. Whereas from time to time it stores the NameNode Metadata files and in the case of NameNode failure It helps NameNode by providing Metadata file....
13. What is Backup Node?
Backup Node is corresponding to Checkpoint NameNode, but it keeps the updated copy of FsImage in RAM memory and is always synchronized with NameNode.
14. What are the 5 V's of Big Data?
The below are the identified characteristics of any big data. Volume, Variety, Velocity, Value, and Veracity....
15. How Data is stored in HDFS?
First the files are divided into blocks and then those Blocks are stored on different DataNodes. NameNode stores the metadata....
16. Explain the data flow in Hadoop system.
Capture Big Data. Process and Analyse.
17. What are the most commonly used Input Formats in Hadoop ?
The below are the most commonly input formats used in Hadoop system. TextInputFormat.
18. What is Functional Programming?
Functional programming is a way of implementing parallel programming in which one function gets exclusive access to the data that it processes and release the lock on the data when it passes to the other function.
19. What are the different Distributed Programming available on Hadoop system?
Distributed Programming is the core feature of the Big Data system that supports the Parallel Programming Paradigm and leverages the power of Distributed Storage System. Below are some of the Distributed Programming available on Hadoop System....
20. What are the available output formats in Hadoop system?
The available output format in Hadoop system are, TextOutputFormat. MapFileOutputFormat.
21. Difference between fsImage and editLog file.
fsImage file contains all the information about modifications/changes made across the cluster ever since the nameNode was started. It is stored in HardDisk memory....
22. What is checkpointing in HDFS?
Checkpointing refers to the process of combining edit log with FSImage. Secondary Namenode gets the copy of fsImage and editLog from NameNode and it consolidates the files to prepare the final FSImage file.
23. The Controls that trigger the checkpoint reconciliation process.
fs.checkpoint.period controls the number of seconds between two periodic checkpoints. Default value is 3600 Sec (1 hour)....
24. What is Data block size in HDFS?
The block size is the smallest unit of data that a file system can store. The default Block Size on Hadoop 1 is 64MB, Hadoop 2 is 128MB.
25. What happens to the existing data if the block size is changed in HDFS?
Changing the block size in hdfs-site.XML configuration file won't affect the existing data that are already stored in HDFS.
26. What is the single point of failure in a Hadoop cluster?
The single point of failure in a Hadoop cluster is the NameNode . While the loss of any other machine does not result in data loss, NameNode loss results in cluster unavailability....
27. What is Replication factor in HDFS?
Replication factor facilitates fault tolerance in Hadoop cluster. HDFS stores files as data blocks and distributes these blocks across the entire cluster....
28. How does Tweaking block size affect the system?
While tweaking the block size, a very small block size will increase seek time to read the file that affects performance and making large block size we will lose the parallelism while reading a file.
29. Explain HDFS Data Write Pipeline Workflow.
The HDFS client sends a WRITE request on DistributedFileSystem API. DistributedFileSystem issue a RPC call to the name node to create a new file in FS namespace....
30. Explain HDFS Read a file workflow.
Client opens the file it wishes to read by calling open() on the Distributed FileSystem (HDFS). DistributedFileSystem makes an RPC call to the name node to determine the locations of the blocks for the first few blocks in the file....
31. Explain Hadoop Common module.
This library provide filesystem and OS level abstractions and includes the necessary Java files and scripts required to start Hadoop. This module is required by other Hadoop modules.
