Big data Interview Questions

Interviewers use this question to test calm incident handling as much as Spark knowledge. Structure your reply as a short incident story: contain, diagnose, recover, prevent.

1. Contain and communicate first. Acknowledge the alert, tell the dashboard owners that data is late and give an honest ETA. Stakeholders forgive a delay far more easily than silence.
2. Find the real error. Open the Spark UI or history server, locate the failed stage and read the executor logs, not just the driver stack trace. Typical causes are executor OutOfMemoryError, a FetchFailedException during a shuffle, a missing upstream partition or a schema change in the source.
3. Check what changed. Compare input volume, recent deployments, config changes and cluster capacity with the last successful run. A sudden spike in one key usually points to skew.
4. Recover safely. Rerun only the failed date or partition with a targeted fix, such as more shuffle partitions or AQE skew handling. Make sure the write is idempotent (overwrite the partition or use MERGE) so the rerun cannot create duplicates.
5. Validate before announcing. Reconcile row counts and key totals against the source, then confirm the dashboards have refreshed.
6. Prevent a repeat. Write a blameless post-mortem, add freshness and volume alerts, and add a test or data contract for the source that changed.

Example point: “The job died at 2 a.m. with executor OOM in a join stage. One merchant ID had 40 times its usual rows after a festive-sale campaign. I salted that key, reran the partition by 7:30 a.m. and later added an alert on key distribution.”

Note: Quantify the outcome, such as recovery time or reports restored, and name one lasting improvement you made.

Cost questions check whether you think like an owner. Use STAR, and make sure the result has a number in it: a percentage saved, rupees per month, or cost per pipeline run.

• Situation: for example, “Our monthly cloud bill for the analytics platform had doubled in a year while data volume grew only 30 percent.”
• Task: find the biggest cost drivers without breaking SLAs.
• Action: explain how you measured first. Tag clusters and jobs by team, pull billing exports and rank the top ten jobs by spend. Then describe concrete levers:
  • Right-size executors and turn on autoscaling or dynamic allocation instead of fixed, oversized clusters.
  • Auto-terminate idle interactive clusters and move dev notebooks to smaller pools.
  • Use spot or preemptible instances for executors while keeping the driver on on-demand capacity.
  • Convert CSV or JSON to Parquet with compression, compact small files and partition by the columns people actually filter on, so queries scan far less data.
  • Apply storage lifecycle rules that move old raw data to cheaper tiers, and expire unused snapshots.
  • Kill duplicate pipelines that produce the same table for different teams.
• Result: “We cut spend by 38 percent in three months, and the nightly batch finished 40 minutes earlier because it scanned less data.”

Close by mentioning how you kept it from creeping back, for example a weekly cost dashboard per team or budget alerts, because interviewers value sustained savings over a one-time clean-up.

Note: Never claim savings that hurt reliability; say explicitly that SLAs and data quality stayed the same or improved.

This question tests planning and risk management. Show that you treated the migration as a programme with phases, not a single copy job.

1. Inventory everything. List Hive databases and tables, Oozie or Airflow workflows, Spark and MapReduce jobs, upstream sources, downstream consumers and SLAs. Tools such as Hive metastore queries and job history logs help build this map.
2. Choose the target and patterns. For example, object storage with Spark and an open table format such as Delta Lake or Apache Iceberg, plus a managed catalogue. Call out differences that bite: object stores do not have atomic directory renames like HDFS, so jobs that rely on rename-based commits need a table format or proper committers.
3. Move in waves. Start with low-risk, high-visibility pipelines, then the critical ones. Copy history with DistCp or a cloud transfer service and set up incremental sync for the cutover window.
4. Run in parallel and reconcile. Run old and new pipelines side by side for a few cycles and compare row counts, checksums and key business metrics.
5. Rebuild security. Map Kerberos and Ranger policies to cloud IAM roles and catalogue-level permissions, and keep audit logging on from day one.
6. Cut over and decommission. Switch consumers, freeze the old jobs, and retire hardware only after a sign-off period.

Example result: “We moved 300 Hive tables and 120 jobs in four waves over five months with zero missed SLAs, and cut infrastructure cost by about a quarter.”

Note: Mention one thing that went wrong, such as a timezone or small-files issue, and how you caught it during the parallel run; it makes the story credible.

Structure the answer as measure, locate the bottleneck, relieve pressure, fix properly. Interviewers want to see that you do not simply “add more consumers” without checking the limits.

1. Measure the lag per partition. Use kafka-consumer-groups.sh --describe --group <group> or your monitoring (for example a Prometheus lag exporter). If one partition lags badly and others are fine, the problem is a hot key, not total throughput.
2. Check whether the consumer or the sink is slow. Often the consumer spends most of its time writing to a database or API. Look at processing time per batch and at errors or throttling on the sink.
3. Watch for rebalance storms. If processing a batch takes longer than max.poll.interval.ms, the consumer is kicked out of the group, partitions are reassigned and work repeats. Reduce max.poll.records or speed up processing.
4. Scale sensibly. Add consumer instances up to the number of partitions; beyond that, extra consumers sit idle. Adding partitions helps but changes key-to-partition mapping, which can affect per-key ordering.
5. Tune the pipeline. Batch writes to the sink, increase fetch sizes, and in Spark Structured Streaming set maxOffsetsPerTrigger so each micro-batch is predictable.
6. Communicate. Tell report users the delay and expected catch-up time.

Example: “Lag hit 20 lakh messages during a Diwali sale. The Postgres sink was the bottleneck, so we switched to batched upserts and added four consumers. Lag cleared in 35 minutes, and we now load-test before every sale.”

Note: End with prevention: capacity tests before known peaks and lag-based alerts with clear thresholds.

This is a data trust question. Show a methodical reconciliation process and good communication, not defensiveness.

1. Acknowledge and scope. Ask for the exact report, date range and expected figure. Agree who owns the “source of truth” for the comparison.
2. Reconcile layer by layer. Compare counts and totals at each stage: source extract, raw zone, cleaned tables, and the final aggregate. The first layer where numbers diverge tells you where to dig.
3. Check the usual suspects:
   • Duplicates from at-least-once delivery or a rerun that appended instead of overwrote.
   • Late-arriving or back-dated transactions missed by a date filter.
   • Timezone boundaries, for example UTC in the pipeline versus IST in finance.
   • Join fan-out, where a one-to-many join silently multiplies rows.
   • Definition gaps: gross versus net, refunds, cancellations, GST included or excluded.
4. Fix and backfill. Correct the logic, reprocess the affected partitions idempotently and share a before-and-after reconciliation.
5. Prevent a repeat. Add automated checks with tools such as Great Expectations, Deequ or dbt tests (uniqueness, not-null, totals within tolerance), plus a daily reconciliation job against finance, and document the metric definition in the data catalogue.

Example: “The gap was 2.3 percent. Refunds were posted in finance on the refund date but our pipeline netted them against the original order date. We aligned the definition with finance and added a daily reconciliation that alerts if the difference exceeds 0.1 percent.”

Note: Emphasise that you kept stakeholders updated during the investigation; rebuilding trust in data matters as much as the fix.

Governance questions are increasingly common in India since the Digital Personal Data Protection Act, 2023. Show that you understand both the technical controls and the process around them.

• Situation and task: for example, “An internal audit found customer phone numbers and PAN details readable by every analyst in our data lake. We had six weeks to fix it.”
• Discover and classify. Scan tables and columns for personal data, tag them in the data catalogue (for example Unity Catalog, AWS Glue with Lake Formation, or Apache Atlas) and agree on sensitivity levels with legal and security teams.
• Minimise and protect.
  • Stop ingesting fields nobody needs.
  • Mask or tokenise sensitive columns in shared layers; use salted hashing when analysts only need to join on the value.
  • Apply column-level and row-level access policies (Apache Ranger, Lake Formation or catalogue grants) so only approved roles see raw values.
  • Keep encryption at rest and in transit, and turn on access audit logs.
• Retention and erasure. Define retention periods and a process for deletion requests. In Delta Lake or Iceberg, a DELETE must be followed by VACUUM or snapshot expiry, otherwise old files still hold the data.
• Result: “Raw PII access dropped from about 400 users to 12, the audit point was closed, and analytics use cases kept working with masked data.”

Mention how you brought people along: short training for analysts and a simple request process for exceptions, so that the new controls were not bypassed.

Note: Avoid claiming legal expertise; say you worked with the legal or privacy team on what the law requires and owned the technical implementation.

This question tests judgement. Strong candidates show they clarified the real need before choosing technology, and that they compared options on latency, cost and operational effort.

1. Clarify “real time”. Ask what decision the data drives and how fast it must happen. “Real time” often turns out to mean every 15 minutes, which a scheduled micro-batch can meet.
2. Lay out the options in a short comparison for stakeholders:

1. Prototype and measure. Build a small proof of concept and show actual latency and cost rather than arguing in theory.
2. Decide together. Present the trade-off, recommend one option and document the reasoning so the choice can be revisited if needs change.

Example: “The marketing team asked for a real-time campaign dashboard. After discussion, they needed updates every 10 minutes to shift ad budget. A micro-batch job on the existing lakehouse met that at roughly a fifth of the cost of a streaming stack. Fraud detection, where seconds matter, stayed on true streaming.”

Note: Show that you are not biased towards the newest tool; interviewers like candidates who pick the simplest design that meets the requirement.

Interviewers want to see curiosity, caution and a plan. The key message: understand before you change, and reduce risk step by step.

1. Map what exists. Read the Oozie workflow and coordinator definitions, list the Hive scripts, and use the metastore and query logs to trace inputs and outputs. Draw a simple lineage diagram of sources, tables and downstream consumers.
2. Talk to people. Find former owners, the teams consuming the outputs and anyone who gets paged when it breaks. Ask which outputs are business-critical and what “correct” looks like.
3. Observe it running. Track runtimes, failures, retries and data volumes for a few cycles. Add basic monitoring and alerts if none exist.
4. Protect behaviour with tests. Snapshot outputs for a few known dates and write characterisation tests, so any later change can be compared against current results.
5. Document as you go. A short runbook covering how to rerun, common failures and contacts is often the most valuable deliverable in the first month.
6. Improve incrementally. Fix the riskiest items first, such as hard-coded paths or non-idempotent inserts, and avoid a big-bang rewrite. If migrating to Spark or Airflow later, move one workflow at a time and compare outputs.

Example: “I took over a 40-step Oozie flow with no documentation. In three weeks I built lineage and a runbook, found that two steps wrote to a table nobody read, and removed them. Failures dropped from weekly to about once a quarter.”

Note: Stress that you communicated progress to your manager and stakeholders; ownership includes making the system understandable to others, not just to yourself.

Answer in two halves: the immediate recovery, then the process fix. Keep the tone collaborative; interviewers listen for blame.

• Immediate response:
  • Identify the change quickly by comparing the incoming schema with the expected one, for example a renamed column, a type change from integer to string, or a new nested field.
  • Stop bad data spreading. Route failing records to a quarantine or dead-letter location instead of letting nulls flow into reports.
  • Patch the mapping, reprocess the affected window idempotently and confirm downstream tables are correct.
  • Inform consumers of the delay and what was affected.
• Root cause and prevention:
  • Agree a data contract with the producing team: schema, meaning of fields, freshness and a notice period for breaking changes.
  • For Kafka, use a schema registry with Avro or Protobuf and set a compatibility mode such as BACKWARD, so incompatible changes are rejected when the producer tries to register them.
  • Add schema drift detection to the pipeline and an automated check in the producer's CI.
  • Design ingestion to tolerate additive changes, such as new optional columns, using schema evolution features in Delta Lake or Iceberg.

Example: “The payments team renamed amount to txn_amount, and our revenue table filled with nulls. We quarantined the records, fixed the mapping and backfilled within four hours. We then set up a schema registry with backward compatibility and a shared Slack channel for changes. We had no schema-related outages in the next year.”

Note: Frame the upstream team as a partner; saying “we agreed a contract together” sounds far stronger than “they broke our pipeline”.

This is a performance-tuning story. Show that you diagnosed with evidence from the Spark UI before changing settings, and quantify the improvement.

1. Read the Spark UI. Look at which stage takes the time, then compare median and maximum task duration. A few tasks running far longer than the rest means skew. Check shuffle read and write sizes, spill to disk and GC time.
2. Reduce the data early. Select only needed columns, push filters before joins and make sure partition pruning actually happens (check the plan with explain()).
3. Fix the joins. Broadcast small dimension tables, handle skewed keys with AQE skew-join handling or salting, and avoid joining on nulls.
4. Right-size partitions. The default spark.sql.shuffle.partitions of 200 is too few for terabytes and too many for small data. Aim for tasks processing roughly 100 to 200 MB each, or let AQE coalesce partitions.
5. Replace slow code. Row-by-row Python UDFs are expensive; use built-in functions or vectorised pandas UDFs. Cache only DataFrames that are reused, and unpersist them afterwards.
6. Fix the files. Thousands of tiny input files slow planning and scans; compact them.

Example: “A customer 360 job took 3 hours in production. The UI showed one join stage where two tasks ran for 90 minutes because of a default customer ID used for guest checkouts. We filtered those rows into a separate path, broadcast a lookup table and replaced a Python UDF with built-in functions. Runtime dropped to 25 minutes and the cluster size halved.”

Note: Mention how you made the test data more realistic afterwards, for example by sampling production key distributions, so the problem is caught earlier next time.

Big Data refers to complicated and massive datasets. Big data operations require specialized tools and techniques because a relational database cannot handle such a large amount of data.

Big data enables businesses to gain a deeper understanding of their industry and assists them in extracting valuable information from the unstructured and raw data that is routinely gathered. Big data enables businesses to make more informed business decisions.

The terms "big data" and "Hadoop" are essentially interchangeable. Hadoop, a framework that focuses on big data operations, gained popularity along with the growth of big data. Professionals can use the framework to evaluate big data and assist organizations in decision-making.

There are three steps to be followed:

• Data Ingestion
• Data Storage
• Data Processing.

Data ingestion, or the extraction of data from diverse sources, is the initial step in the deployment of a big data solution. A CRM like Salesforce, an ERP like SAP, an RDBMS like MySQL, or any other log files, documents, social media feeds, etc., could be the data source. Either batch jobs or real-time streams can be used to ingest the data. The obtained information is then kept in HDFS.

Data Storage: The Second Step in Deploying a Big Data Solution

The next step after data input is to store the extracted data. Either a NoSQL database or HDFS will be used to store the data (i.e. HBase). HBase is better for random read/write access, while HDFS storage is better for sequential access.

Data processing is the last stage of deploying a big data solution. One of the processing frameworks, such as Spark, MapReduce, Pig, etc.

Enterprises are dealing with a huge volume of structured, unstructured, and semi-structured data because data analysis has become one of the primary determinants of a company.

It might be challenging to analyze unstructured data, which is where Hadoop's skills come into play.

Hadoop is also open source and uses standard hardware. Consequently, it is a cost-benefit strategy for companies.

Fsck is an acronym for file system check. It is a command that HDFS employs. This command is used to check for discrepancies and determine whether the file has any errors. For instance, HDFS is alerted by this command if any file blocks are missing.

"$ hdfs namenode -format"

How to Proceed: The question is subjective, thus there isn't a right or wrong response because it relies on your prior experiences. Inquiring about your prior experience during a big data interview will help the interviewer determine whether you are a good fit for the project's requirements.

So, how will you respond to the query? If you have prior experience, begin by discussing your responsibilities in a previous role before gradually introducing more information. Inform them of the ways in which you helped to make the project a success.

How to Proceed:

Although it is a difficult topic, it is frequently asked in big data interviews. You are prompted to select between good data or good models. You should attempt to respond to it from your experience as a candidate. Many companies want to follow a strict process of evaluating data, which means they have already selected data models. Good data can change the game in this situation. The other way around also works as a model is chosen based on good data.

How to Proceed:

This question should always be answered in the affirmative. Performance in the real world is important and is independent of the data or model you are utilizing in your project.

How to Proceed:

One of the most important phases in big data projects is data preparation. At least one question based on data preparation might be asked during a big data interview. The interviewer wants to know what processes or safety measures you use when preparing your data, which is why he asks you this question.

As you are already aware, data preparation is vital to obtain the appropriate data, which can then be utilized for modeling. This is the message you should provide to the interviewer. Additionally, be sure to highlight the kind of model you'll be using and the factors that went into your decision. In addition, you should go through key terms for data preparation such as converting variables, outlier values, and unstructured data.

How to Proceed:

In large data, unstructured data is particularly prevalent. To achieve accurate data analysis, the unstructured data should be converted into structured data. By quickly contrasting the two, you might begin by responding to the question. 

Once finished, you can now talk about the techniques you employ to change one form into another. You could also describe the circumstance in which you actually did it. You can post information about your academic projects if you recently graduated.

For conducting Hadoop operations, dual processors or core machines with a configuration of 4 or 8 GB RAM and ECC memory are appropriate. However, the hardware needs to be customized in accordance with the project-specific workflow and process flow because it fluctuates.

HDFS NameNode supports exclusive write only. As a result, the second user will be rejected and only the first user will be granted access to the file.

For processing, the HDFS physically separates the input data into units known as HDFS Blocks.

Input Split is a logical division of data by mapper for mapping operation.

Big Data refers to substantial and frequently complex data collections that are so large that they are impossible to manage with traditional software solutions. Big Data consists of organized and unstructured data sets, including websites, multimedia, audio, video, and photo information.

Businesses can gather the data they require in a variety of methods, including:

• cookies on websites
• email monitoring
• Smartphones
• Smartwatches
• Forms of online transactions
• interactions with websites
• Transactional records
• posts on social media
• Companies that collect and market customer and valuable data are known as third-party trackers.
• 
  

Three groups of tasks are involved in working with large data:

Integration is the process of combining data, frequently from multiple sources, and transforming it into a format that can be analyzed to offer.

Management: Big data needs to be kept in a repository that can be easily accessed and gathered. Big Data is primarily unstructured, making it unsuitable for traditional relational databases, which require data in a formatted table and row format.

Analysis: The investment return from big data includes a variety of valuable market insights, such as information on consumer preferences and purchasing trends. These are demonstrated by using tools powered by AI and machine learning to analyze huge data sets. transforming it into a format that can be examined to offer.

Volume: The volume is reflected in the size of the data that is kept in data warehouses. Since there is a chance that the data will reach arbitrary heights, it must be reviewed and processed. which could be between terabytes and petabytes or more.

Velocity: Velocity basically describes how quickly real-time data is generated. Imagine the number of Facebook, Instagram, or Twitter postings that are created per second, for an hour or longer, to provide a straightforward illustration of recognition.

Variety: Big Data consists of data that has been acquired from various sources and is structured, unstructured, and semi-structured. This diverse range of data necessitates the use of distinct and suitable algorithms, as well as extremely varied and specific analyzing and processing procedures.

Veracity: In a simple sense, we can describe data veracity as the caliber of the data examined. It pertains to how trustworthy the data is in general.



Value: Unprocessed data has no purpose or meaning but can be transformed into something worthwhile. We can glean knowledge that is useful.

Regardless of the company's divisions or breadth, data is now a crucial instrument that organizations must use. Big data is widely used by businesses to get an advantage over rivals in their industry.

One step in the big data process is auditing the datasets a business acquires. Big data experts also need to understand what the business expects from the application and how they intend to use the data.

Decision-making with confidence: Analytics strives to foster decision-making, and big data endures to support this. Due to the abundance of data available, big data can assist businesses in making decisions more quickly while maintaining the same level of certainty. Moving quickly and responding to larger trends and operational changes today is a big commercial advantage in a fast-paced environment.

Asset optimization: Big data indicates that companies can have individual-level control over their assets. This suggests that they can effectively optimize assets based on the data source, increase production, increase the lifespan of equipment, and minimize any necessary downtime for specific assets. Guaranteeing the company is making the most of its resources and ties with falling costs, gives it a competitive advantage.

Cost reduction: Big data can assist companies in cutting costs. Data gathered by businesses can assist them in identifying areas where cost savings can be made without having a negative impact on business operations, from analyzing energy usage to evaluating the effectiveness of staff operating patterns.

Improve Customer Engagement: In order to establish and adapt consumer dialogue, which might subsequently be translated into higher sales, consumers make confident choices while responding to online surveys about their decisions, habits, and tendencies. Understanding each customer's preferences through the data gathered on them allows you to target them with certain products while also providing the personalized experience that many modern consumers have grown to expect.

Identify fresh sources of income: Analytics can help businesses find new sources of income and diversify their operations. For instance, understanding consumer patterns and choices enables businesses to choose their course of action. The data that firms gather may also be sold, adding new revenue sources and the opportunity to form commercial partnerships.

Hadoop is also discussed while talking about big data. Therefore, from the standpoint of an interview, this is one of the most important questions. you may undoubtedly encounter. An

open-source framework called Hadoop is used to store, process, and interpret large, unorganized data collections in order to gain information and insights. Thus, the relationship between Hadoop and Big Data is as follows.

The amount of organized, semi-structured, and unstructured data that make up big data makes it a challenging undertaking to analyze and handle. A device or piece of technology was required to aid in the quick processing of the data. Hadoop is therefore utilized as a result of its processing and storage capabilities. Hadoop is also an open-source piece of software.

Cost considerations are advantageous for company solutions.



Its widespread use in recent years is mostly due to the framework's ability to disperse the processing of large data sets via cross-computer clusters employing straightforward programming paradigms.

Hadoop helps with both data storage and big data processing. It is the most dependable method for overcoming significant data obstacles. Some key characteristics of Hadoop include:

Distributed Processing: Hadoop facilitates distributed data processing, which results in faster processing. MapReduce is responsible for the parallel processing of the distributed data that is collected in Hadoop HDFS.

Open Source - Because Hadoop is an open-source framework, it is free to use. The source code may be modified to suit the needs of the user.

Fault Tolerance: Hadoop has a high level of fault tolerance. It makes three replicas at different nodes by default for each block. This replica count might be changed based on the situation. Therefore, if one of the nodes fails, we can retrieve the data from another node. Data restoration and node failure detection are done automatically.

Scalability: We can quickly reach the new device even though it has different hardware.



Reliability: Hadoop stores data in a cluster in a secure manner that is independent of the machine. So, any machine failures have no impact on the data saved in the Hadoop environment.

A protocol that enables users to access files over a network is called NFS (Network File System). Even if the files are stored on a networked device's disc, NFS clients would provide access to them as if they were local.

A distributed file system, such as HDFS (Hadoop Distributed File System), is one that is shared by numerous networked computers or nodes. Because it stores several copies of files on the file system—the default replication level is three—HDFS is fault-tolerant.

Replication/Fault Tolerance is the key distinction between the two. Failures were designed to be tolerated by HDFS. There is no built-in fault tolerance

The IT industry has been using data modeling as a business model for many years. The data model is a method for arriving at the diagram by thoroughly understanding the data in question. The process of visualizing the data allows business and technology experts to comprehend the data and comprehend how it will be used.

The benefits of data modeling

• Companies can gain a variety of advantages from data modeling as part of their data management, including:

• You've cleansed, arranged, and modeled your data to project what your next step should entail before you ever establish a database. Databases are constrained, prone to errors, and poorly designed due to data modeling, which also improves data quality.

• Data modeling creates a visual representation of the flow of your planned data organization. Employee understanding of data trends and their role in the data management puzzle is aided by this. Additionally, it fosters data-related communication between organizational departments.

• Data modeling makes it possible to create databases with greater depth, which leads to the development of future applications and data-based business insights.

The three basic steps involved in deploying a model onto a big data platform are,

• Data Ingestion
• Data Storage
• Data Processing

Data ingestion: This process entails gathering data from various platforms, including social media, corporate applications, log files, etc.

Data Storage: After data extraction is finished, storing the vast volume of data in the database presents a challenge. Here, the Hadoop Distributed File System (HDFS) is crucial.

Data processing: After storing the data in HDFS or HBase, the next step is to use specific algorithms to analyze and visualize this massive amount of data for better data processing. Again, using Hadoop, Apache Spark, Pig, etc. will make this work simpler

After carrying out these crucial actions, one can successfully install a big data model.

Hadoop's typical input formats are:

Text Input Format: Hadoop uses this as its default input format.

Key Value Input Format: Hadoop uses the Key-Value Input Format to read plain text files.

Sequence File Input Format: Hadoop reads files in a succession using the Sequence File Input format.

Big Data processing methods analyze big data sets at a massive scale. Offline batch data processing is typically full power and full scale, tackling arbitrary BI scenarios. In contrast, real-time stream processing is conducted on the most recent slice of data for data profiling to pick outliers, impostor transaction exposures, safety monitoring, etc. However, the most challenging task is to do fast or real-time ad-hoc analytics on a big comprehensive data set. It substantially means you need to scan tons of data within seconds. This is only probable when data is processed with high parallelism.

Different techniques of Big Data Processing are:

• Batch Processing of Big Data
• Big Data Stream Processing 
• Real-Time Big Data Processing
• Map Reduce

Hadoop A software architecture called MapReduce is used to process huge data sets. It is the primary part of the Hadoop system for processing data. It separates the incoming data into different components and executes a program on each piece of data in parallel. The terms "MapReduce" refer to two distinct jobs. The first is the map operation, which turns a set of data into a varied collection of data in which individual components are separated into tuples. The key-based data tuples are combined via the reduced operation, which also changes the key's value.

A Reducer's primary methods are:

Setup(): this is a method that is only used to set up the reducer's various arguments

Reduce: The primary function of the reducer is reduce(). This method's specific purpose is to specify the task that needs to be completed for each unique group of values that share a key.

Cleanup: After completing the reduce() task, cleaning() is used to clean up or destroy any temporary files or data.

Hadoop can provide an option wherein a particular set of lousy input records could be skipped while processing map inputs. SkipBadRecords class in Hadoop offers an optional mode of execution in which the bad records can be detected and neglected in multiple attempts. This may happen due to the presence of some bugs in the map function. The user has to manually fix it, which may sometimes be possible because the bug may be in third-party libraries. With the help of this feature, only a small amount of data is lost, which may be acceptable because we are dealing with a large amount of data.

Outliers are data points that are extremely dispersed from the group and do not belong to any clusters or groups. This could have an impact on how the model behaves, cause it to forecast the wrong outcomes, or make them exceedingly inaccurate. As a result, outliers must be handled cautiously because they may reveal useful information. These outliers may cause a Big Data model or a machine learning model to be inaccurate. These outcomes could be,



• poor outcomes
• reduced precision
• Extended Training

The practice of cleaning and changing raw data before processing and analysis is known as data preparation. Prior to processing, this critical stage frequently entails reformatting data, making enhancements, and combining data sets to enrich data.

For data specialists or business users, data preparation is a never-ending effort. However, it is crucial to put data into context in order to gain insights and then be able to remove the biased results discovered as a result of bad data quality.

For instance, standardizing data formats, improving source data, and/or removing outliers are all common steps in the data construction process.

It is a tool used for concurrent data copying to and from Hadoop file systems with very huge amounts of data. Its distribution, error handling, recovery, and reporting are all impacted by MapReduce. A list of files and directories is expanded into a series of inputs to map jobs, each of which copies a specific subset of the files listed in the source list.

An open-source framework called Hadoop is designed to store and handle large amounts of data in a distributed fashion.

The Key Elements of Hadoop

Hadoop's main storage system is HDFS (Hadoop Distributed File System): The vast amount of data is kept on HDFS. It was primarily designed for storing enormous datasets on inexpensive technology.

Hadoop Map Reduce: Hadoop's MapReduce layer is in charge of handling data processing. It submits a request for the processing of already-stored structured and unstructured data in HDFS. By dividing up the data into separate jobs, it is responsible for the parallel processing of a large amount of data. Map and Reduce are the two phases of processing. A map is a stage where data blocks are read and created, to put it simply.

YARN: YARN is the processing framework used by Hadoop. YARN manages resources and offers a variety of data processing engines, including real-time streaming, data science, and batch processing.

The different Output formats in Hadoop are -

• Textoutputformat
• Mapfileoutputformat
• DBoutputformat
• Sequencefileoutputformat
• SequencefileAsBinaryoutputformat

Spark uses lazy evaluation. Transformations such as filter, select or groupBy only build a plan; nothing runs until an action such as count(), collect() or write is called.

1. Planning. For DataFrames and SQL, the Catalyst optimiser turns the code into a logical plan, optimises it (predicate pushdown, column pruning, constant folding) and chooses a physical plan. Tungsten then generates efficient code through whole-stage code generation.
2. Job. Each action submits one job (sometimes more, for example when a query needs to sample data first).
3. Stages. The DAGScheduler splits the job into stages at shuffle boundaries. Narrow dependencies, where each output partition depends on one input partition (map, filter, union), are pipelined inside one stage. Wide dependencies, where an output partition needs data from many input partitions (groupBy, reduceByKey, distinct, repartition, most joins), force a shuffle and start a new stage.
4. Tasks. Each stage runs one task per partition. The TaskScheduler sends tasks to executor cores, preferring nodes where the data already lives.

df = spark.read.parquet('s3://bucket/orders')
out = (df.filter(df.status == 'PAID')   # narrow: same stage
         .groupBy('city')                # wide: shuffle, new stage
         .sum('amount'))
out.write.parquet('s3://bucket/city_sales')  # action: job starts

This job has two stages: stage one reads, filters and writes shuffle files; stage two reads the shuffled data, aggregates and writes the output. In explain() output, each Exchange node marks a shuffle and therefore a stage boundary. If a shuffle file is lost, Spark reruns only the parent stage that produced it, using lineage.

Note: Counting the Exchange nodes in a plan is a quick way to estimate how expensive a query will be.

Skew means a few keys hold far more rows than the rest, so a few tasks run much longer than others and may run out of memory. First confirm it: in the Spark UI, the maximum task duration and shuffle read are many times the median, and a groupBy(key).count() shows the hot keys.

• Broadcast the small side. If one table is small enough (below spark.sql.autoBroadcastJoinThreshold, 10 MB by default, or with an explicit broadcast() hint), a broadcast hash join avoids shuffling the big table and removes the skew problem.
• AQE skew join (Spark 3.x). With spark.sql.adaptive.enabled and spark.sql.adaptive.skewJoin.enabled on, Spark checks shuffle partition sizes at runtime. A partition larger than both skewedPartitionFactor (default 5) times the median and skewedPartitionThresholdInBytes (default 256 MB) is split into smaller pieces, and the matching partition on the other side is duplicated for each piece.
• Salting. Add a random salt to the skewed side and replicate the other side once per salt value, then join on key plus salt, so a hot key is spread across N tasks.
• Handle nulls and default keys separately. Null or placeholder keys (such as “guest” or 0) are a common cause; filter them out or process them apart.
• Isolate hot keys. Process the top few keys with a broadcast join and union the result with a normal join for the rest.

from pyspark.sql import functions as F
N = 16
big = big.withColumn('salt', (F.rand() * N).cast('int'))
small = small.withColumn('salt', F.explode(F.array([F.lit(i) for i in range(N)])))
joined = big.join(small, ['merchant_id', 'salt'])

For skewed aggregations, use two-stage aggregation: aggregate by key plus salt first, then aggregate again by key.

Note: Salting multiplies the small side N times, so pick the smallest N that evens out the tasks.

All three are binary, compressed, schema-aware formats, but they are optimised for different access patterns.

• Choose Parquet as the default for analytics and lakehouse tables. Columnar storage means a query reading 5 of 200 columns reads only those columns, and statistics enable predicate pushdown. Delta Lake stores data as Parquet, and it is the most common choice for Apache Iceberg.
• Choose ORC when you are on a Hive-centric stack, especially Hive ACID transactional tables, which require ORC. It compresses very well and has rich built-in indexes.
• Choose Avro for streaming and ingestion, where whole records are written and read one at a time. With a schema registry, each Kafka message carries only a small schema ID, and compatibility rules make evolution safe. A common pattern is Avro in Kafka and the landing zone, then Parquet in curated tables.

Whatever the format, aim for files of roughly 128 MB to 1 GB and use a splittable codec such as Snappy or ZSTD. CSV and JSON are fine for exchange but poor for large-scale analytics.

Note: Interviewers often follow up with “row versus columnar”; explain it with the example of reading a few columns from a wide table.

• Topic and partitions. A topic is split into partitions, each an ordered, append-only log. Partitions are replicated across brokers: one leader handles reads and writes, and followers in the in-sync replica set (ISR) copy it. With acks=all and min.insync.replicas=2, a write is acknowledged only when enough replicas have it.
• Keys and ordering. The producer hashes the record key to choose a partition, so all events for one key land in the same partition and stay in order. There is no ordering guarantee across partitions.
• Consumer groups. Within one group, each partition is assigned to exactly one consumer, so the partition count caps parallelism. Different groups read the same topic independently.
• Offsets. Each consumer group commits the position it has processed to the internal __consumer_offsets topic. On restart or rebalance, consumption resumes from the last committed offset.

Delivery semantics depend on when offsets are committed:

• At-most-once: commit before processing; a crash loses records.
• At-least-once: commit after processing; a crash causes reprocessing, so duplicates are possible. This is the common default.
• Exactly-once: requires three things together: an idempotent producer (enable.idempotence=true, the default since Kafka 3.0) that removes duplicates from retries; transactions (transactional.id) that write output records and the consumed offsets atomically; and downstream consumers reading with isolation.level=read_committed. Kafka Streams wraps this as processing.guarantee=exactly_once_v2.

These guarantees apply within Kafka. When writing to an external database or file system, you also need an idempotent or transactional sink, for example upserts keyed on a business ID, or storing offsets in the same transaction as the data.

Note: Choose the partition count with future consumer parallelism in mind; increasing it later changes which partition a key maps to.

A lakehouse combines the cheap, open storage of a data lake (files such as Parquet on S3, ADLS or GCS) with warehouse features: ACID transactions, schema enforcement, time travel, fine-grained governance and good SQL performance. Several engines, such as Spark, Trino, Flink and Snowflake, can work on the same tables.

Object stores offer no multi-file transactions, so the table format adds a metadata layer that defines exactly which files make up each version of a table.

• Delta Lake keeps a transaction log in a _delta_log folder. Each commit is a numbered JSON file listing files added and removed, and a Parquet checkpoint is written periodically (every 10 commits by default) so readers need not replay the whole log. Writers write new data files first, then try to create the next log entry atomically. If another writer committed first, the writer checks for conflicts and retries (optimistic concurrency). Readers always see a complete committed version.
• Apache Iceberg uses a tree: a catalogue (Hive Metastore, AWS Glue, Nessie or a REST catalogue) points to the current metadata.json, which lists snapshots; each snapshot has a manifest list, and manifests track data files with column statistics. A commit is an atomic swap of the catalogue pointer. Iceberg also offers hidden partitioning and partition evolution without rewriting data.

Because every version is recorded, both formats support:

• MERGE, UPDATE and DELETE on files that are otherwise immutable.
• Time travel, for example SELECT * FROM orders VERSION AS OF 42.
• Schema evolution with enforcement.
• Maintenance: compaction (OPTIMIZE in Delta, rewrite_data_files in Iceberg) and clean-up of old files (VACUUM, expire_snapshots).

Apache Hudi is a third format built on similar ideas, with a strong focus on upserts and incremental processing.

Note: Old snapshots cost storage and keep deleted data around; set retention deliberately, especially for personal data.

Both organise data to reduce work at query time, but in different ways.

• Partitioning splits a table into directories by the value of one or more columns, for example dt=2026-09-12/country=IN/. Queries that filter on the partition column read only the matching directories (partition pruning). Use it for low-cardinality columns that appear in most filters, such as date, region or event type.
• Bucketing hashes a column into a fixed number of files, for example 64 buckets on user_id. Rows with the same key always land in the same bucket, so two tables bucketed the same way on the join key can be joined bucket by bucket without a full shuffle (bucket map join or sort-merge bucket join in Hive, bucketed joins in Spark). Use it for high-cardinality join or aggregation keys.

CREATE TABLE sales (
  order_id BIGINT,
  user_id  BIGINT,
  amount   DECIMAL(12,2)
)
PARTITIONED BY (dt STRING)
CLUSTERED BY (user_id) INTO 64 BUCKETS
STORED AS ORC;

Common pitfalls:

• Partitioning on a high-cardinality column such as user_id creates millions of tiny directories and files, overloads the metastore and slows every query.
• Bucketed joins only help when both tables use the same bucket column and compatible bucket counts.
• Spark and Hive use different hash functions, so a table bucketed by one engine is not treated as bucketed by the other.
• Changing the bucket count later means rewriting the table.

A practical rule is to partition by date, keep each partition at least a few hundred megabytes, and consider bucketing only for large tables that are frequently joined on the same key. Modern table formats offer alternatives: Iceberg hidden partitioning (for example days(event_ts)) and Delta Lake Z-ordering or liquid clustering, which co-locate related data without a fixed directory layout.

Note: If asked which to use, answer with the query pattern first: filters suggest partitioning, repeated joins suggest bucketing.