Every small-to-medium sized community trying to build will have a handful of users like this. I managed to discover the serial downvoters on my old lemm.ee comm and when I banned them (about 4 of them?) it had a huge impact. They didn’t all downvote /everything/ but they downvoted a lot of things, and no contribution. And if they got in early, they could sink new threads.

Made a huge difference.