Yes, it does have a sigma attached to it. I used to work in a health care facility for years and the stigma wasn't a historic one. Most of the individuals that used it , used it heavily. Accurately or not those individuals were thought of as possible users. Any one coming to work with very strong fragrance use was eyed as drinking or smoking before coming into work. It just our culture to think that way.
Consider me stigmatized (and not caring). I use a handmade patchouli soap bar every day, and I think it smells great. I haven't inhaled for decades, but for me, the scent is a reminder of better times, not pot smoking. If those around me think otherwise, so be it.
I like it but enough is enough. Same goes for Vetiver, I like a hint of it but nothing really strong. I just finished a tub of Salters French Vetiver shaving cream, I enjoy it but it's knock you over strong. Both are great blended with other scents.
Patchouli was refered to as dead head deodorant (this coming from a dead-head), there is still a stigma attached to it. I have worked in places where if they smelled it on you, you would get an alcohold and 10 panel drug test instantly. No other reason was necessary. The smell doesn't work for me, but the people that I have met that use it, use lots of it. YMMV I guess.
I can only take patchouli in very small doses. I love Mystic Water shaving soap, but the patchouli in the Marrakesh flavor is way too much for me.
Stigma?, depends on to whom. Memories and associations?, definitely. I've been to the 60s (a recovering Dead Head). For me, if I smell it, I run the other way. But if there's one thing I learned from the 60s, it's wear what pleases you and do what you like if it doesn't hurt others. For us old guys patchouli = not shaving, not...shaving!
I really used to like Barbershop brand's Almond cologne, just enough patchouli to accent the almond. Strangely, the first time I smelled it, at all of 16, was that it was a very "genteel" scent.