Under priveleged , over privelged and what the hell is privelge anyway

I was talking to a friend before about how much this concept of being "Under" privileged just really chaps my hide.
Like most Americans, I've earned every single thing I own, have a mortgage or car payment on those things I do not yet own, and yet some how there's this concept that I'm privileged.
I've always thought personally that this is just a justification that criminals use to get something for nothing and yet the idea has caught on to such a degree that we find the word in use in everyday life.
And I'm sorry but I just don't get it. I've been poor, still pretty much am or I would not be living in the West Ward (lets call that spade a spade), I've been among the working homeless, I've been abused in pretty much every way a child can be , and yet somehow, I don't deserve what little I've managed to work for and hold onto?
For no greater reason that A) I was born White, or B) someone just wants someone else to blame for their own bad choices...or wants what I have without doing the work, I get the joy of being called privileged.
So I'll ask anyone one out there who cares to do so to explain how this word applies to me or anyone else like me?
I'll add that as a veteran, I've earned my freedoms such as they are as generations did before me, so let's get into this, after millions of dead, trillions of dollars and rivers of blood to make this nation what it is how is it that Americans and myself as one of them are privileged?
Just wondering really.
Peace,
David