Data Transformation: Normalization, Discretization, Sampling, Compression

Last lesson left us with messy data pulled in from many sources and a promise: before any of it can be analysed together, it has to be cleaned and reshaped. So picture the result of that pull.

One column is “age” in years, running 18 to 90. The next is “income” in rupees, running ₹10,000 to ₹20,00,000. Feed both into a model raw, and income will swamp the result — not because income matters more, but because its numbers are simply bigger and nobody put the two columns on the same ruler.

Putting columns on the same ruler is the first of four small prep moves. Together, they are what turns raw records into analysis-ready ones:

• We rescale a column so its size stops lying.
• We bucket a number into a category when that is what we actually want.
• We sample the rows when there are too many to use whole.
• We shrink the bytes when storage bites.

Four moves, and GATE expects you to perform the arithmetic of each by hand. Let us walk them one at a time.

Rescaling: min-max vs z-score

Two rescalings cover almost every GATE question, and the whole trick is reading which one a problem wants.

Min-max normalization squashes every value into the range [0, 1]:

x' = (x − min) / (max − min)

The smallest value becomes 0, the largest becomes 1, and everything else lands in between. Bounded, and easy to read at a glance.

Z-score standardization instead centers the column at mean 0 and rescales so its standard deviation is 1:

z = (x − μ) / σ

Now most values sit in [−3, +3] — but a z-score can be negative, and it is not capped. A wildly extreme value can land at z = 5 or beyond, which min-max could never do.

That last sentence reads like a point in min-max’s favour. It is worth pausing on, because it usually is not. “Bounded” sounds like the safe, tidy choice; the boundedness is bought at a price. Min-max lets the single most extreme value in the column define the entire scale.

Watch what one outlier does to the column 10, 20, 30, 40, 900. The min is 10 and the max is 900, so the divisor is 890:

10  →  0/890    = 0.000
20  →  10/890   = 0.011
30  →  20/890   = 0.022
40  →  30/890   = 0.034
900 →  890/890  = 1.000

Four of the five values are now squeezed into the bottom three percent of the range, effectively indistinguishable from each other. The outlier sits at exactly 1.000 — looking for all the world like a perfectly ordinary maximum.

Min-max has hidden the anomaly by definition, because its top value always maps to 1 no matter how absurd it is. A z-score, by contrast, would hand that same value back as a conspicuously large positive number — the anomaly stays visible and you can decide what to do about it. Bounded output and trustworthy output are not the same thing.

Discretization: turning numbers into bins

Sometimes you do not want a continuous number at all — you want a bucket, “young / middle / old” instead of an exact age. Turning a number into a bucket is discretization, or binning, and there are two honest ways to cut.

• Equal-width bins — split the range into bins of the same width. Ages [18, 90] into three bins gives [18, 42), [42, 66), [66, 90]. Simple, but one bin may end up nearly empty.
• Equal-frequency bins — make each bin hold the same count of rows. Sort the column, then cut at every n/k-th value. The bins differ in width, but the row counts come out balanced. With nine sorted ages and three bins, you simply cut after the 3rd and the 6th value: three rows per bin, whatever ages they happen to be.

Sampling: picking rows instead of using them all

When the dataset is too large to analyse whole, sample it — work with a representative handful instead of the lot.

• Random sampling — every row has the same chance of being picked.
• Stratified sampling — split the population into groups (strata) and sample from each in proportion. Reach for this when a small subgroup would otherwise be missed entirely.
• With replacement — a picked row can be picked again. This is what the bootstrap does: draw many same-sized resampled copies of the data, which is how confidence intervals and bagged ensembles (many models each trained on a different resample) are built.
• Without replacement — once picked, a row is out. This is the default for surveys.

Compression: shrinking the bytes

Two flavours, and the whole decision is which one you can afford.

• Lossless — every original bit is recoverable. This is for text, numeric tables, and code (zip, gzip).
• Lossy — it throws away detail to shrink harder. This is for images, audio, and video (JPEG, MP3), where tiny reconstruction errors are invisible to a human.

The rule of thumb writes itself: tabular and textual data must be lossless; only perceptual media can be lossy.

How GATE asks this

Almost always a NAT — a column value, a min/max or a mean/SD, and “compute the normalized value to 3 decimals.” Occasionally an MCQ on which rescaling preserves which property, or an MSQ listing sampling methods or compression facts.

Scan the question for μ, σ, min, max: those four symbols tell you instantly which formula the problem wants.

Worked example — GATE DA 2024, Q17

A person’s salary is ₹106000. The population has mean μ = ₹96000 and standard deviation σ = ₹21000. Find the z-score of this salary.

Plug straight into the standardization formula and reduce one step at a time:

z = (x − μ) / σ
  = (106000 − 96000) / 21000
  =      10000       / 21000
  ≈ 0.476

So z ≈ 0.476. This is the real GATE DA 2024, Q17. The salary is about half a standard deviation above the mean — comfortably above average, yet well inside the typical range, just as the prediction suggested.

In one breath

Four prep moves ready raw data for analysis:

• Rescaling: min-max rescales a column onto a bounded [0, 1]; z-score centers it at mean 0 with SD 1 but leaves it unbounded and possibly negative.
• Discretization buckets a number by equal width or equal frequency.
• Sampling picks a representative subset (random or stratified, with or without replacement).
• Compression shrinks the bytes losslessly for tables and text but only lossily for perceptual media.

Practice

A question to carry forward

So the data is clean now — rescaled, binned where it helps, sampled to a workable size. It is finally fit to analyse. But fit to analyse where?

Running heavy “total sales by category, by state, by month, across five years” queries against the live operational database would crawl and would fight with the checkout traffic for the same rows. Analytics needs its own home: a store built for big read queries rather than tiny writes.

Inside it, the tables are deliberately shaped, not in the tidy normalized form you just spent two lessons perfecting, but in a layout that minimises joins on read. Here is the thread onward: what does that analytics-first store look like, why does it sometimes choose redundancy on purpose, and what are the two standard table shapes it picks between?