The answer is simple. Privilege is a baseless concept. It is the phrenology of the 21st century, purporting to explain reality and failing. And all too often, rather than dealing with the failings of the concept, proponents simply double down and insist it must be true or (more shamefully) lash out against those that raise issue with it.
And if you believe that it is worthwhile to contemplate privilege, I also invite you to contemplate original sin. Both concepts are equally valid.
I'm supremely irritated when the word "privilege" is thrown around as a means of silencing people, and doubly so when it comes from people are in the majority of said privilege. But lets not pretend its not a tangible thing. Study after study, after study show that certain minority groups, women, LGBT persons systematically face greater discrimination, greater rates of violence, and have fewer economic opportunities for lesser pay. That's hard data, peer reviewed, and a position held by the overwhelming majority of sociologists who've studied it. Frankly, its plainly visible to anyone who hasn't preconceived that it cannot be so.
Denying outright that privilege exists is tantamount to denying climate change -- Its easy enough to do from an air-conditioned apartment, meanwhile those along the coast are left to drowning. We can argue about symptoms vs. causes, root causes, who to blame, and what precisely to do about it until we're blue in the face but that doesn't achieve anything and is energy wasted to inaction.
We all have a set of privileges we were born into -- men have privileges that women don't have, whites have privileges that blacks don't have, women have privileges that men don't have, blacks have privileges that whites don't have. Its not all or nothing, its shades and degrees along as many axes as you can name. But generally, if you're a white male who had access to a quality education and quality healthcare, the playing field is tipped greatly in your favor for no reason other than the circumstances of your birth -- and that you can identify distinct demographics that by-and-large aren't born into similarly smooth-sailing by virtue of their skin, or gender, or economic class -- well, being a white dude is a privilege to be damn sure.
And having privilege doesn't make you a villain, and it doesn't mean you've never faced hardship or overcome a hand you were dealt. Having or lacking privilege is not pre-destiny, it just means the course of different parts of your life are swimming with or against currents you don't control.
So none of us should feel bad for having the privileges we do -- the place we want to get to is a place where greater equality is achieved extending privileges to those who go without today, not by taking them from those who enjoy them today. Lets all become richer, not poorer -- basic dignities are not a commodity in a zero-sum game.
While that work is going on, the least we can all do is stop dismissing one another's trials, and try to understand them instead.