Removing Duplicates

Redundant data is a waste of processing power. Distinct and DistinctBy are your tools for ensuring every record in your stream is unique.

1. Distinct (Primitive and Reference)

For primitives (int, string), Distinct() works out of the box. For complex objects, it uses Reference Equality. **Architect Note:** If you want Distinct() to work on objects by value, you MUST implement IEquatable or provide a custom IEqualityComparer.

2. DistinctBy (Added in .NET 6+)

The modern way to filter unique objects based on a specific property. No more custom comparers needed!

// Get unique users based ONLY on their Email
var uniqueUsers = users.DistinctBy(u => u.Email);
    

3. Architect Insight

Q: "Is Distinct expensive?"

Architect Answer: "Yes. Under the hood, Distinct builds a internal **HashSet**. This has an O(N) memory cost. For massive streams, consider using a database-level DISTINCT or GROUP BY to ensure the uniqueness is handled at the storage layer before it flows into your application."