People play competitive games for an emotional reason: to try; to grow so that their trying means more; to conquer something that once conquered them. Many games also are bundled with stories, but if the story is the only good thing about the game it shouldn't be formatted as a game.
Games with no challenge - like Lego games - and no story are degenerate because the only reason left to play them is empowerment: the low feeling of "I can make stuff blow up on a screen by pressing a button! Cool!" In other words, if you like such games, you value power in a completely imaginary world.
Competitive games are not so because the point of them is not the in-universe power but the out-of-universe skill: nobody plays Prismata to blow stuff up or Dark Souls to kill monsters. People play those games to use and grow their skill at the game. And that skill itself is not imaginary: it's a mental adeptness possessed by the real you, not by your avatar in a game world. The challenge you overcome and that is the source of your enjoyment is not the one faced by a sprite or an imaginary character, but the one you, a real person, face against the game, a real object. Casual games are degenerate because their appeal is purely the power over something imaginary; competitive games are not degenerate because their appeal is the struggle against a real obstacle - toward a goal with itself token value - that exercises real mental abilities, and in a way that reflects the growth arc of many real-world endeavors.
Lego Star Wars and similar games may serve legitimately as a "gateway into gaming": a lot of older adults that grew up without video games are kind of phobic of them and it's possible that playing a game like this could help them to see that the medium isn't "some newfangled thing for youngsters" or "for hardcore competitors only".
A legitimate "casual game" is one like Codenames or Dominion; one that has skill and allows trying, but also allows enjoying it without being into it.