AmanOG

Bloom Filters

aman Jaiswal's photo
aman Jaiswal
·

4 min read

Bloom filters are approximate data structures (also known as probabilistic data structures) that allow us to efficiently determine whether an element is not part of a set with 100% certainty. However, they can only give probabilistic guarantees about an element's presence in the set, meaning there is a small chance of false positives but no false negatives.

Why Use a Bloom Filter?

Imagine a scenario where a platform like Instagram wants to recommend reels to its users but does not want to show something the user has already watched. One way to achieve this is by keeping track of all the IDs of reels watched by a particular user and ensuring they aren’t shown again. For example:

user = {p1, p2, p3, p1024}

In this case, every time a new reel is to be recommended, we can check if its ID exists in the set of watched reels. However, users watch hundreds of reels every single day, and over time, the set of watched posts will become enormous. To determine if a user has watched a particular reel, the entire set would need to be loaded into memory, making the operation increasingly resource-intensive and time-consuming.

Key Insight: Approximation is Often Enough

In many cases, such as recommending new content, an approximate solution is sufficient. Instead of maintaining a large, exact set of watched IDs, we can use a Bloom filter to:

  • Efficiently check if a reel has already been watched.

  • Avoid the memory overhead of storing the entire set of watched IDs.

How a Bloom Filter Works

A Bloom filter is essentially a bit array of fixed size, initialized with all bits set to 0. It uses multiple hash functions to map elements to positions in the array. Here’s how it works:

  1. Adding an Element:

    • When an element is added to the Bloom filter, it is hashed by each of the hash functions.

    • Each hash function produces an index, and the corresponding bits in the bit array are set to 1.

  2. Checking for Membership:

    • To check if an element is in the Bloom filter, it is hashed by the same hash functions.

    • If all the bits corresponding to the hash indices are set to 1, the element is probably in the set.

    • If any of the bits is 0, the element is definitely not in the set.

Advantages of Bloom Filters

  1. Space Efficiency:

    • A Bloom filter requires significantly less memory than storing the actual set of elements.

  2. Fast Operations:

    • Both insertion and membership checks are O(k), where k is the number of hash functions.

  3. No False Negatives:

    • If the filter says an element is not in the set, it is guaranteed to be correct.

Disadvantages of Bloom Filters

  1. False Positives:

    • A Bloom filter can incorrectly report that an element is in the set when it isn’t.

  2. Non-removable Elements:

    • Once an element is added, it cannot be removed (though variations like Counting Bloom Filters address this).

Use Cases for Bloom Filters

  1. Web Caches:

    • Preventing redundant requests by checking if a URL has been accessed.

  2. Database Systems:

    • Quickly determining if a key might exist in a database before performing a costly lookup.

  3. Distributed Systems:

    • Efficiently synchronizing data across systems by identifying elements that might need updating.

Example: Instagram Reel Recommendations

Using a Bloom filter, Instagram can keep track of watched reel IDs without storing the entire set of IDs. Here’s how the process works:

  1. When a user watches a reel, its ID is added to the Bloom filter.

  2. Before recommending a new reel, its ID is checked against the Bloom filter.

    • If the ID is not in the filter, the reel is recommended.

    • If the ID is possibly in the filter, it is skipped to avoid repetition.

This approach ensures fast and memory-efficient operations, even as the number of watched reels grows over time.

Tried to implement Bloom Filter checkout - https://github.com/JAmanOG/Tried-to-Implement-Bloom-Filter

bloom filterSystem Design