Disclaimer: I am not trolling, I am an autistic person who doesn’t understand so many social nuances. Also I am from New Hampshire (97% white), so I just don’t have any close African-American friends that I am willing to risk asking such a loaded question.
It became associated with black people as the food was relatively cheap and therefore commonly eaten by black people. Then it became integrated to comedic shows of people doing blackface as a way to deride black people.
Everyone loves fried chicken and watermelon. The problem isn’t the food, but cultural stigma attached to it in certain cases.