I too, am a Stupidist. Unfortunately, society has kept me in the closet too long. It's sad, you see, but it's not politically correct to hate the many who cling to idiocy.
I had a similar encounter once. I was randomly accosted by a middle-aged black man when I was in Hyde Park (Chicago) going to the University of Chicago. He asked how long I was going to live there, to which I said 5 more years (that was '96 and I ended up not leaving until 2000 -- who'da thunk I'd be right?). He didn't believe that and vented on me his frustration with the university (being a white, non-Chicagoan student made me a representative, I suppose). At that time, the university was doubling its undergraduate population. To house them all, they had to build more dorms, and to do that, they had to tear down some houses on the neighborhood land they owned -- houses that had been lived in by the black families of the neighborhood for the last several decades. The guy told me that he was born in that house and had lived there some 40 odd years and he resented that he was getting displaced so that more students who were not and never would be a part of the community could move in. He was wrong to vent all his frustrations on an unsuspecting and very nice young man such as myself. But, I could understand why he was so upset. I wonder if a similar dynamic was working with the people you encountered -- that the changes in Harlem you mentioned were displacing people or otherwise cutting up the fabric of the community and that there was some resultant hostility directed at the perceived outsiders who were causing or benefiting from this development. Not that it's ok to throw stuff at you, I was just wondering -- in writing and at length.
