“You can’t compare apples to oranges” is a fine and meaningful expression within certain contexts: for example, if someone says that the movie version of a book had better special effects than the book, it’d be fair to take that person to task for comparing apples to oranges. They’re comparing two things based on a quality that belongs to only one of those things.
The problem with the expression is that it gets over-applied too often. If someone says they prefer movies to books, for example, it wouldn’t be right to reply, “You can’t compare apples to oranges!” That would be as meaningful as protesting when someone says that they think oranges are tastier than apples, or that apples are on special this week at the grocery store but oranges aren’t.
