Question 1

How does boolean indexing work in pandas, and what are the common pitfalls?

Accepted Answer

Boolean indexing filters a DataFrame by passing a boolean Series or array of the same length as the index. Common pitfalls include using Python's and/or instead of &/| and forgetting to wrap compound conditions in parentheses, both of which raise errors or produce wrong results.

Question 2

How do you apply multiple aggregation functions to different columns in a single GroupBy call?

Accepted Answer

Pass a dictionary to agg() mapping each column to one or more functions, or use named aggregations with the keyword-argument form (pandas 0.25+) to control output column names directly. Both approaches avoid chained GroupBy calls and produce a clean, single-pass result.

Question 3

How do you detect and remove duplicate rows in pandas, and how do you control which duplicate to keep?

Accepted Answer

duplicated() returns a boolean mask of rows that are duplicates of an earlier row; drop_duplicates() removes them. Both accept a subset parameter to restrict comparison to specific columns and a keep parameter ('first', 'last', or False) to control which instance is retained or whether all copies are dropped.

Question 4

What is the difference between loc and iloc in pandas, and when should you use each?

Accepted Answer

loc selects rows and columns by label (index value or column name), while iloc selects by integer position. Use loc when your index carries meaningful labels like dates or IDs; use iloc for positional slicing regardless of what the index contains.

Question 5

How do common SQL operations map to pandas, and when should you use SQL instead of pandas?

Accepted Answer

Every core SQL clause — SELECT, WHERE, GROUP BY, HAVING, JOIN, ORDER BY, LIMIT — has a direct pandas equivalent, but SQL executes inside a database engine with optimized query planning and disk-backed storage, while pandas requires all data to fit in RAM. Use SQL for large persistent datasets and pandas for in-memory transformation, feature engineering, and integration with the Python ML ecosystem.

Question 6

What is the split-apply-combine model in pandas GroupBy?

Accepted Answer

GroupBy splits a DataFrame into subgroups by key, applies a function independently to each group, then combines the results back into a single object. Understanding which phase each method targets — agg collapses, transform preserves shape, filter removes entire groups — determines which API to reach for.

Question 7

How do you work with string data in pandas using the .str accessor, and how does it compare to applying Python string methods manually?

Accepted Answer

The .str accessor vectorizes Python string methods across a Series without a Python-level loop, propagates NaN automatically, and integrates cleanly into method chains. Calling apply(lambda x: x.upper()) does the same work slower and breaks on NaN unless you add a null check.

Question 8

What is the difference between wide and long (tidy) data formats, and why does it matter for analysis?

Accepted Answer

Wide format stores multiple measurements as separate columns per subject; long (tidy) format stores one measurement per row with a variable-name column and a value column. Long format is required by most statistical and visualization libraries, makes adding new variables trivial, and is the standard expected by groupby and merge operations.

Question 9

When should you use apply, map, or applymap versus vectorized pandas operations, and what are the performance implications?

Accepted Answer

Vectorized pandas and NumPy operations operate on entire arrays in compiled C/Fortran code and should always be your first choice. apply runs a Python function row- or column-wise in a Python loop, map transforms a single Series element-by-element, and applymap (DataFrame.map in pandas 2.1+) applies a function to every scalar — all three are orders of magnitude slower than vectorized equivalents.

Question 10

How does the categorical dtype reduce memory and speed up operations in pandas?

Accepted Answer

Categorical dtype stores a column's unique values once in a lookup table and represents each row as a small integer code, replacing repeated Python string objects. This cuts memory by an order of magnitude for low-cardinality string columns and accelerates GroupBy, sorting, and equality comparisons because pandas operates on integer codes rather than string comparisons.

Question 11

How does the category dtype work in pandas and when should you use it?

Accepted Answer

CategoricalDtype stores a column as integer codes plus a small lookup table of unique values, dramatically reducing memory for low-cardinality string columns. It also enforces a fixed set of valid values, enables natural ordering, and speeds up groupby and sort operations.

Question 12

How do you parse, manipulate, and extract features from datetime columns in pandas?

Accepted Answer

Convert string columns to datetime with pd.to_datetime(), then use the .dt accessor to extract components like year, month, day, and day of week, compute time deltas, and perform resampling. Setting a DatetimeIndex unlocks time-series-specific operations like resample, rolling, and time-aware interpolation.

Question 13

When should you use pandas eval() and query(), and what are their limitations?

Accepted Answer

eval() and query() parse expression strings and delegate evaluation to numexpr, which uses multi-threaded SIMD operations and avoids allocating intermediate arrays — giving 2-10x speedups on large DataFrames. They are most beneficial on DataFrames larger than a few hundred thousand rows where intermediate array allocation dominates; for small frames, the expression parsing overhead makes them slower than standard indexing.

Question 14

What are the different strategies for handling missing data in pandas — isna, fillna, dropna, and interpolate?

Accepted Answer

isna/notna detect missing values; dropna removes rows or columns containing them; fillna replaces them with a scalar, dict, or forward/backward fill; interpolate estimates values from neighboring points using a chosen method. The right strategy depends on whether missingness is random, structural, or time-ordered.

Question 15

What merge types does pandas support, and what does the validate parameter do?

Accepted Answer

pandas merge supports inner, left, right, and outer joins that mirror SQL semantics. The validate parameter enforces key cardinality ('one-to-one', 'one-to-many', 'many-to-one', 'many-to-many') and raises MergeError immediately when the data violates the expectation, preventing silent row multiplication.

Question 16

What is the difference between merge, join, and concat in pandas?

Accepted Answer

concat stacks DataFrames along an axis without matching keys; join aligns on the index (or a single key column) using a convenient shorthand; merge is the most general, joining on any column(s) with full SQL-style control over the join type, key names, and suffix handling.

Question 17

What is method chaining in pandas and how do you use pipe() to extend it?

Accepted Answer

Method chaining applies a sequence of transformations in a single expression without intermediate variables, improving readability and reducing accidental mutation. pipe() inserts any callable — including custom functions and sklearn transformers — into the chain, keeping data flow linear even when a function takes the DataFrame as a non-first argument.

Question 18

How do GroupBy and multi-index interact in pandas, and how do you flatten a multi-index result?

Accepted Answer

Grouping on multiple keys produces a MultiIndex on the result by default. You can suppress it with as_index=False or groupby(..., as_index=False), or reset it afterward with reset_index(). Stacking and unstacking let you pivot between long and wide forms once a MultiIndex exists.

Question 19

How does Polars differ from pandas, and when should you choose one over the other?

Accepted Answer

Polars is a Rust-native DataFrame library built on Apache Arrow that executes a lazy query plan with parallel, multi-threaded evaluation — making it 5-50x faster than pandas on large datasets. pandas has a broader ecosystem and is the right choice for exploratory work, small datasets, and libraries that expect a pandas DataFrame; Polars wins on throughput, memory efficiency, and correctness (no implicit index, no silent copies).

Question 20

What is the difference between pivot, pivot_table, and melt in pandas, and when do you use each?

Accepted Answer

pivot reshapes long-format data to wide by spreading a column's values into new column headers — it requires unique index/column combinations and has no aggregation. pivot_table is the aggregating version that handles duplicates via a specified aggfunc. melt is the inverse: it takes wide-format data and collapses multiple columns into key-value rows (long format).

Question 21

How do you reduce memory usage in a pandas DataFrame using dtypes, category encoding, and downcasting?

Accepted Answer

The biggest wins come from converting low-cardinality string columns to category dtype (often 10x smaller), downcasting int64 and float64 to the smallest type that fits the data range, and using sparse arrays or chunked reads for data that doesn't need to live fully in memory.

Question 22

How do rolling and expanding windows work in pandas, and when do you use each?

Accepted Answer

rolling() computes statistics over a fixed-size sliding window, discarding data outside the window; expanding() grows the window from the first row to the current row, equivalent to an ever-increasing cumulative calculation. Both return objects you chain .mean(), .sum(), .std(), or a custom .apply() onto.

Question 23

What causes SettingWithCopyWarning in pandas and how do you fix it?

Accepted Answer

SettingWithCopyWarning fires when you try to set a value on what pandas suspects is a copy of a slice rather than the original DataFrame, so the write may silently fail. The fix is to always use .loc on the original DataFrame for assignments, or call .copy() explicitly when you intend to work on a detached copy.

Question 24

What is the difference between GroupBy transform and agg in pandas?

Accepted Answer

agg collapses each group into a single scalar, returning a result with one row per group. transform returns a Series or DataFrame with the same index as the original, broadcasting the group-level result back to every row — making it ideal for adding derived columns without a merge.

Question 25

Why is pandas slow, and what are the main strategies to speed it up?

Accepted Answer

pandas is slow primarily because Python loops bypass NumPy's vectorized C kernels, object-dtype columns prevent SIMD optimizations, and keeping entire datasets in memory limits scalability. The fixes are vectorization, categorical encoding, eval/query for large frames, chunking for out-of-core data, and switching to Polars or DuckDB for compute-heavy pipelines.