Sets

A list answers “what is the ordered collection, duplicates and all?” A set answers a different and often more useful question: “what distinct things are in here?” The moment you stop caring about order and position, and start caring only about membership and uniqueness, a set replaces a surprising amount of bookkeeping — the little seen lists, the if x not in result guards, the manual de-duplication loops. Let us see why.

Creating a set

You write a set with curly braces, like a dict but without the colons. Two small traps hide in the syntax, so meet them up front:

unique_tags = {"python", "data", "python", "ml", "data"}
print(len(unique_tags))     # duplicates collapsed
print(sorted(unique_tags))  # sorted only so the output is stable to show

empty = set()               # {} would be an empty DICT, not a set
print(type(empty))

ids = set([1, 2, 2, 3, 3, 3, 4])
print(ids)

3
['data', 'ml', 'python']
<class 'set'>
{1, 2, 3, 4}

Two things to carry forward. First, {} is an empty dictionary, not an empty set — for an empty set you must write set(). Second, and this is why we printed sorted(unique_tags) rather than the set directly: a set has no order at all. Unlike a dict, which keeps insertion order, a set may print its elements in a different sequence from one run to the next. Never index a set, and never rely on its print order — when you need a stable view, sort it.

Deduplication — the everyday job

Removing duplicates is what sets are reached for most of the time, and it collapses to a single call:

user_ids_in_log = [42, 17, 42, 99, 17, 42, 8, 8, 17]

# Order not preserved — but every duplicate is gone.
unique = set(user_ids_in_log)
print(sorted(unique))

# Need to KEEP first-seen order? dict.fromkeys does it cleanly.
unique_ordered = list(dict.fromkeys(user_ids_in_log))
print(unique_ordered)

[8, 17, 42, 99]
[42, 17, 99, 8]

If order does not matter, set(iterable) is the answer. If it does, dict.fromkeys is the tidiest order-preserving dedup in the language — it leans on the fact that dictionaries keep insertion order and cannot hold a duplicate key.

Membership tests — O(1) versus O(n)

Here is the property that makes a set more than a convenience. A set is backed by a hash table: to check x in s, Python computes a numeric fingerprint of x, jumps straight to the one bucket that fingerprint points to, and looks only there. The size of the set barely matters — that is what “O(1), constant time” means. A list has no such shortcut: x in lst walks the elements one by one until it finds a match or reaches the end, which is O(n) and gets slower as the list grows.

big_list = list(range(1_000_000))
big_set = set(big_list)

# Both find the element. The difference is how much work each does.
print(999_999 in big_list)   # True — after scanning up to 1,000,000 elements
print(999_999 in big_set)    # True — after ONE hash and ONE bucket check

True
True

Both lines print True, but count the work behind them. The element 999_999 is the very last in the list, so in big_list has to examine all million entries to reach it. The set check does a single hash and inspects a single bucket — the same handful of operations whether the set holds ten items or ten million. For one lookup you would never notice; for thousands of lookups against a large collection, it is the gap between “instant” and “the script appears to hang.”

Set algebra — union, intersection, difference

This is where sets pay back over hand-written loops. The operators read like the mathematics they come from, and they mean exactly what they say:

signed_up = {"alice", "bob", "chen", "dani", "eli"}
made_purchase = {"bob", "chen", "fiona", "gabe"}

# (sorted only so the printed output is stable)
print("union:        ", sorted(signed_up | made_purchase))  # either action
print("intersection: ", sorted(signed_up & made_purchase))  # did BOTH
print("difference:   ", sorted(signed_up - made_purchase))  # signed up, didn't buy
print("sym diff:     ", sorted(signed_up ^ made_purchase))  # in exactly one

union:         ['alice', 'bob', 'chen', 'dani', 'eli', 'fiona', 'gabe']
intersection:  ['bob', 'chen']
difference:    ['alice', 'dani', 'eli']
sym diff:      ['alice', 'dani', 'eli', 'fiona', 'gabe']

Read those four lines as business questions and the power is obvious. The intersection is your activated users — they signed up and bought. The difference signed_up - made_purchase is your conversion target — signed up but never purchased. Once you think in set algebra, “find the users who did X but not Y” is a single operator instead of a ten-line loop. Each operator also has a named method (signed_up.union(...), signed_up.intersection(...)) if you prefer words to symbols.

What is allowed in a set?

Set elements must be hashable — Python has to compute a stable fingerprint for each, and that is only possible if the value cannot change after creation. Numbers, strings, tuples of hashables, and frozensets all qualify. Lists, sets, and dicts do not, because they are mutable and their fingerprint could shift out from under the table:

ok = {1, "two", (3, 4), frozenset([5, 6])}
print(len(ok))            # all four are hashable, all stored

try:
    bad = {[1, 2], [3, 4]}     # lists are mutable — unhashable
except TypeError as e:
    print("TypeError:", e)

4
TypeError: unhashable type: 'list'

Deduping a list of dicts — the messy real case

Dictionaries are not hashable, so you cannot pour a list of dicts straight into set() — and yet messy records shaped exactly like that are everywhere: API responses, CSV rows, database results. The idiom is to keep a seen set of the key that defines “same”, and skip anything whose key you have already recorded:

records = [
    {"email": "alice@x.com", "name": "Alice", "source": "web"},
    {"email": "BOB@x.com",   "name": "Bob",   "source": "api"},
    {"email": "alice@x.com", "name": "Alice", "source": "csv"},
    {"email": "bob@x.com",   "name": "Bob",   "source": "web"},
    {"email": "chen@x.com",  "name": "Chen",  "source": "web"},
]

# Dedupe by normalised email, keeping the first occurrence.
seen = set()
deduped = []
for r in records:
    key = r["email"].lower()
    if key not in seen:
        seen.add(key)
        deduped.append(r)

for r in deduped:
    print(r)

{'email': 'alice@x.com', 'name': 'Alice', 'source': 'web'}
{'email': 'BOB@x.com', 'name': 'Bob', 'source': 'api'}
{'email': 'chen@x.com', 'name': 'Chen', 'source': 'web'}

Look at what survived. The second Alice (from csv) and the lower-case bob@x.com both vanished, because normalising the email to lower case made them duplicates of rows already seen — the BOB@x.com API row won by arriving first. This seen-set pattern is the most reached-for dedup idiom in working Python: whenever the task is “process each item but skip ones already handled,” a set is the tool.

frozenset — an immutable set

frozenset is to set what a tuple is to a list: the same operations, but nothing can be added or removed after it is built. And because it cannot change, it is itself hashable — so a frozenset can be a dictionary key or an element of another set:

# A cache keyed by an unordered combination of tags.
cache = {}
cache[frozenset({"python", "data"})] = "matched A"
cache[frozenset({"data", "python"})] = "matched A again"   # SAME key — order ignored

Because the two frozensets contain the same elements, they hash equal and address the same slot — which is exactly why you would use one when the combination matters but the order does not.

In one breath

• A set holds unique, hashable elements and has no order — sort it when you need a stable view.
• set(iterable) dedupes; dict.fromkeys dedupes while keeping first-seen order.
• Membership in a set is O(1) (one hash, one bucket); in a list it is O(n) — convert before looping.
• |, &, -, ^ are union, intersection, difference, symmetric difference.
• frozenset is the immutable, hashable set — usable as a dict key or inside another set.

Practice

What’s next

Sets store unique values. To map a key to a value — the most common data shape in any real application — you need the dictionary, and that is next.