BigData / Hadoop basics
1. What is Apache Hadoop?
Hadoop is a opensource framework that facilitates the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop provides the reliable, scalable way of distributed computing. It is designed to scale up from single machine(server) to thousands of...
2. Hadoop Core modules.
Hadoop Common. The common utilities framework that is leveraged by other Hadoop modules. Hadoop Distributed File System (HDFS). A distributed file system that provides high-throughput access to the application data. Hadoop YARN. A job scheduling and cluster resource management framework. Hadoop M...
3. Applications of Hadoop.
Business Intelligence. Business Analytics. Data Warehousing. Log Processing. Semantic Analysis. Video and Image Analysis. Searching and Data Mining. Pattern recognition. Graph Analysis. Text Mining. Digital signal processing.
4. What is Big Data?
Big data refers to the large amount of data, a collection of large datasets that is infeasible to handle/process using existing computer technology/programmming model. Thus Big Data includes huge volume, high velocity and extensible variety of data, categorized into three types. Structured data :...
5. Apache Hadoop Deployment modes.
Standalone Mode. Pseudo-Distributed Mode. Distributed or Clustered Mode.
6. Advantages of using Apache Hadoop.
Fault Tolerant. Complex Data Analytics. Scalable and Storage Flexibility. cost effective (Commodity Hardware). Open Source. Fast and flexible.
7. HDFS.
HDFS (Hadoop Distributed File System) is the file system used by Hadoop Framework for data Storage on Distributed Computing Environment. HDFS Is exceptionally scalable and faults tolerant.
8. What are the advantages of using HDFS?
The features of Hadoop Distributed File Systems are, Portable. Scalable. Reliable. Fault tolerant. Recovery from Hardware failures. Data Coherency. Enabling MapReduce Process.
9. Describe HDFS Architecture.
Below are some of the major components of Hadoop Distributed File System (HDFS). NameNode. DataNode. Secondary NameNode. Backup Node.
10. What is NameNode?
NameNode is a daemon process in HDFS which is responsible for all processes related to storage in Hadoop. It Manages the Hadoop File system namespace. NameNode Itself doesnot store the actual data, whereas it stores the meta data in RAM for supporting faster operations. NameNode is a Single point...
11. What is DataNode?
DataNode is a commodity hardware which actually stores the data and performs file-level operations as suggested by the NameNode. DataNode sends the signals to NameNode for its status. If DataNode is not able to send heartbeats to the NameNode than Name node marked particular DataNode as dead. Sla...
12. What is Secondary NameNode?
As name described Secondary NameNode is not a backup of the NameNode. Whereas from time to time it stores the NameNode Metadata files and in the case of NameNode failure It helps NameNode by providing Metadata file. So basically, it only support and provide required file to NameNode in the case o...
13. What is Backup Node?
Backup Node is corresponding to Checkpoint NameNode, but it keeps the updated copy of FsImage in RAM memory and is always synchronized with NameNode.
14. What are the 5 V's of Big Data?
The below are the identified characteristics of any big data. Volume, Variety, Velocity, Value, and Veracity. Volume refers to the scale of data, the main characteristic that makes data big or huge. The amount of data/information over internet from social networks for e.g. are growing exponential...
15. How Data is stored in HDFS?
First the files are divided into blocks and then those Blocks are stored on different DataNodes. NameNode stores the metadata. (file information, location etc.) There are 2 terms used in HDFS Data Storage. Block. Replication. Blocks are the smallest unit of storage in HDFS and ranges from 64MB to...
16. Explain the data flow in Hadoop system.
Capture Big Data. Process and Analyse. Distribute Results. Feedback (Audit).
17. What are the most commonly used Input Formats in Hadoop ?
The below are the most commonly input formats used in Hadoop system. TextInputFormat. KeyValueInputFormat. DBInputFormat. and SequenceFileInputFormat. TextInputFormat is the default input format. The other input formats include, CombineFileInputFormat. CustomInputFormat.
18. What is Functional Programming?
Functional programming is a way of implementing parallel programming in which one function gets exclusive access to the data that it processes and release the lock on the data when it passes to the other function.
19. What are the different Distributed Programming available on Hadoop system?
Distributed Programming is the core feature of the Big Data system that supports the Parallel Programming Paradigm and leverages the power of Distributed Storage System. Below are some of the Distributed Programming available on Hadoop System. Hadoop MapReduce. Apache Hive. Apache Pig. Apache Spark.
20. What are the available output formats in Hadoop system?
The available output format in Hadoop system are, TextOutputFormat. MapFileOutputFormat. SequenceFileOutputFormat. SequenceFileAsBinaryOutputFormat. MultipleOutputFormat. MultipalTextOutputFormat. MultipalSequenceFileOutputFormat. TextOutputFormat is the default output format provided by the Hado...
21. Difference between fsImage and editLog file.
fsImage file contains all the information about modifications/changes made across the cluster ever since the nameNode was started. It is stored in HardDisk memory. editLog also has metadata about modifications however it has only recent changes (usually past 1 hour). It is maitained in RAM.
22. What is checkpointing in HDFS?
Checkpointing refers to the process of combining edit log with FSImage. Secondary Namenode gets the copy of fsImage and editLog from NameNode and it consolidates the files to prepare the final FSImage file. FsImage (final) = fsImage + editLog
23. The Controls that trigger the checkpoint reconciliation process.
fs.checkpoint.period controls the number of seconds between two periodic checkpoints. Default value is 3600 Sec (1 hour). fs.checkpoint.size refers to the size of the current edit log (in bytes) that triggers a periodic checkpoint even if the fs.checkpoint.period hasn't expired. The default value...
24. What is Data block size in HDFS?
The block size is the smallest unit of data that a file system can store. The default Block Size on Hadoop 1 is 64MB, Hadoop 2 is 128MB.
25. What happens to the existing data if the block size is changed in HDFS?
Changing the block size in hdfs-site.XML configuration file won't affect the existing data that are already stored in HDFS.
26. What is the single point of failure in a Hadoop cluster?
The single point of failure in a Hadoop cluster is the NameNode . While the loss of any other machine does not result in data loss, NameNode loss results in cluster unavailability. The permanent loss of NameNode data would render the cluster's HDFS inoperable. Therefore, the step should be taken ...
27. What is Replication factor in HDFS?
Replication factor facilitates fault tolerance in Hadoop cluster. HDFS stores files as data blocks and distributes these blocks across the entire cluster. As HDFS was designed to be fault-tolerant and to run on commodity hardware, blocks are replicated a number of times to ensure high data availa...
28. How does Tweaking block size affect the system?
While tweaking the block size, a very small block size will increase seek time to read the file that affects performance and making large block size we will lose the parallelism while reading a file.
29. Explain HDFS Data Write Pipeline Workflow.
The HDFS client sends a WRITE request on DistributedFileSystem API. DistributedFileSystem issue a RPC call to the name node to create a new file in FS namespace. After various checks, client gets the permission or an IOException. The DistributedFileSystem return FSDataOutputStream to the client f...
30. Explain HDFS Read a file workflow.
Client opens the file it wishes to read by calling open() on the Distributed FileSystem (HDFS). DistributedFileSystem makes an RPC call to the name node to determine the locations of the blocks for the first few blocks in the file. For each block, the name node returns the addresses of the data n...
31. Explain Hadoop Common module.
This library provide filesystem and OS level abstractions and includes the necessary Java files and scripts required to start Hadoop. This module is required by other Hadoop modules.
