Tl;DR. I'm white. I worked my ass off to get to where I am now, I make good money, and no magic "CIS White Privelege fairy" ever came down and sprinkled money or opportunity on me. Simply put, The whole "privelege" concept is ridiculous and insulting.
Taking your story at face value, just because you did not feel privileged does not mean that privilege does not exist, or even that you didn't actually benefit from it. As I said, what we call "privilege" is merely a status quo that those in the majority, largely unaware, experience as a lack of resistance, whereas someone who suffers the other end of the same status quo experiences as a penalty. You can't say it doesn't exist or wasn't in effect because you did not feel it, the experiences of those who suffer under it prove its effect. Your experience is your experience, you cannot use it to disprove the experience of others.
I was never dirt-poor (well, perhaps briefly) growing up, but I also come from meager, rural means and now make a good living at everyone's favorite tech company out in beautiful Seattle. I worked hard too to get where I am, and never had much active benefit bestowed upon me. Still, I recognize that I certainly have gained from all kinds of socially-ingrained, passive benefits. Privilege is not usually defined by you getting to take the express elevator but by those on the other end of things having to take the stairs nearly everywhere they go. Had every opportunity in your story been made available to you, but with only a change in your skin tone, its entirely possible that all you have today might never have come to be. I mean that literally -- all of your extant success could have been derailed merely by someone thinking to themselves "Well, he seems entusiastic, and he's well-qualified, but he's black."
You can be offended if someone suggests that you didn't work hard, or don't deserve what you've achieved, as I would be if someone suggested the same of me. As an immigrant to this country you likely have felt discrimination, and had to work harder than a born-citizen would to overcome it. But even still, you cannot rightfully be offended by the mere notion that privilege exists, just because you were one of the lucky few who's hard work actually paid off -- and again, congrats, sounds like you've earned it, but there are people who work just as hard, or harder, and are not able to climb as high as you.