Probably useful to look at https://hexbear.net/api/v3/site slur_filter_regex

"(f(a|4)gg(o|0)t)|((\\W|_|^)(f(a|4)g)g?)|((f(a|4)g)g?s?(\\W|_|$))|*removed*?|(n(i|1)gg((a|er)?(s|z)?)){1,}|(nig){2,}|*removed*(s?){1,}|*removed*?s?|(k(i|y|1|l)kes?){1,}|(*removed*?){1,}|(towel\\s*heads?){1,}|\\bspi(c|k)s?\\b|(spi(c|k)s){2,}|\\b*removed*?|(ch(i|1)nks?){1,}|(n(i|1|l)glets?){1,}|be(a|@|4)ners?|\\bjaps?\\b|(*removed*){2,}|(jungle\\s*bunn(y|ies?)){1,}|(jigg?aboo?s?){1,}|\\bpakis?\\b|(*removed*?){2,}|(rag\\s*heads?){1,}|(*removed*?){1,}|(*removed*?){1,}|(*removed*?){1,}|\\btr(a|@)nn?(y|(i|1|l)es?|ers?)|(tr(a|@)nn?(y|(i|1|l)es?|ers?)){1,}|(l(a|@|4)dyboy(s?)){1,}|ret(a|4)rd(s|ed)?|(*removed*){1,}|(porch\\s?monkey){1,}|(zh(y|i)d(ovka)?){1,}|\\bching\\s?chong\\b|(ching\\s?chong\\s?){1,}|(chong\\s?ching\\s?){1,}|(hefem(a|@|4)le){1,}|(*removed*){1,}|(*removed*){1,}|(\\babb?o\\b)|(abb?o){2,}|(*removed*){1,}|(*removed*?){1,}|(shem(en|an|ales?))|(chinam(an|en))|(redsk(i|l)ns?)|(*removed*?)|(degen(s|era(tes?|cy)))|(curry\\s?munchers?)|(bugm(e|a)n?)|(*removed*^yei])|(glow(ie|y|-y))|((\\W|gang)rap(e|ed|ing|ist))|(\\Wb(i|1)tch(es)?\\W)"

I am not sure how that’ll render (because slur filter doesn’t work on post preview). Some of the slurs in the slur filter code are replaced with *removed* but those which are broken up with other characters are not. To see every slur unmangled, go to the URL above and type “slur_filter” into the searchbox at top labeled “Filter JSON”, then use the left-hand arrows to uncollapse 3 levels site_view > local_site > slur_filter_regex. That’s how it works on Firefox on desktop, if instruction is not sufficient for anyone I can try to improve it.

That regex is too long and complicated for me to fix personally. But assuming somebody is able to work with it, it shouldn’t be too difficult to do something like add a list of common false positives as exceptions, or narrow the scope of the matching a bit more.

One issue with the way the regex works is that it seems to remove a leading space then not put it back in. I started to write a post about issues of the slur filter but decided against sharing it, in this case because it was too nitpicky. I will include here in case anybody decides to fix any aspect, perhaps they would be able to do this at the same time. Maybe it has to do with word boundary in the wrong place or something, I’m not sure. This is from an article that was posted:

separate complaint on behalf of their clients over allegations ofremoved, torture and humiliation.

There should be a space between “of” and “*removed” of *removed* not of*removed*.

I also think it should be prohibited to intentionally put in *removed* to make a joke. Example, there is a tagline you can find in the above URL to the api:

“all cops are removed”

When I read jokes of this format the effect in my brain is to instantly try to think of every slur I have ever heard to find the one that fits. I resent people making me think of all those slurs. I decided against posting this when I originally wrote it because it might be too dictatorial and coddling even in theory, plus likely impossible to enforce.

That’s kind of the whole difficulty of balance with these sorts of interventions, you want to discourage, not encourage the behaviour, but at some point it works at counter purposes like “parental advisory” stickers.

It would be very useful to the idea if some information was available about how often the filter is actually activated in what kinds of scenarios. Because then it could be known if the “wrong” filtering of slurs is actually as common as perceived. I can’t imagine that is logged anywhere?

Now that I am editing this text after posting it, I can see how the source I originally had, with the regex as above, has been modified from the original, for example, {1,}|*removed*?s? is part of the source, instead of whatever the term was originally. I couldn’t recreate my original text from what I have here. So it is done by the server at the time of posting, and cannot be undone. The original content is (I guess) erased, unless there is some log retained that would be visible only to admins/mods. I think what is being proposed in OP is more like if instead of replacing someslur with *removed*, it was replaced with <span class="slurword">someslur</span>, and a secondary regex filter that was identical to the main one, but starts and ends with linebreaks instead of word boundaries would be added, which would insert at the beginning of a paragraph containing a slur: <div class="slurparagraph">, and then at the end of the paragraph, </div> to close it. Then it would be for the client to render however it wanted according to stylesheet. You could make it styled as a collapsing section like spoilers, you could even make just the word itself collapsible/toggled. The word becomes visible if you hover over it, click it etc. Some forums have word-level spoiler features.

This is definitely a suggestion for lemmy main, not hexbear: having a separate implementation of the slur filter when previewing (or better yet: writing) that would highlight slurs, or even add warning text below the composition box like “Your text contains 3 terms which are consider slurs on this site, they will be removed from your post” + link to some info about that would be helpful to good-faith posters. I think it would require changes to both server and client/interface of lemmy to implement. It would certainly require every client to be updated to conform and I have doubts about that happening. I suppose it should be possible for a client to query the API for the slurs and run the filter while composing or previewing locally without any change to the servers, if a dev wanted to do that.

I am not really convinced any of the changes discussed in this comment solve more problems than they introduce. Except the one about fixing the leading space.

I would like this, because sometimes (very rarely) it messes up and censors something that shouldn’t be.

You’d probably need it implemented upstream instead of being Hexbear specific functionality. I doubt Hexbear wants to maintain a custom implementation.

If Nutomic and Dessalines are overall not against that approach, it’s unlikely IMO that it’s something they’ll prioritize doing, but it’d be good for them to at least be theoretically on board before anyone actually tries to make the change and opens a PR.

Counterargument: it’s really funny

It would also prevent breaking of links which have things it detects as a slur which is really annoying. I’ve seen a lot of false positives. Truthfully these kinds of simplistic filters with large word lists are very problematic without allowlist exception lists about 5x the size of them also included. A few slurs like the racial n-word have very few words that contain them but a lot of the more exotic ones here on HB are ones I and many people have never heard of and are found in somewhat frequently used words that are themselves entirely benign. Hard filters like these work when you have only a few words you want censored and have exception-lists that are actually maintained and updated to unbreak things. Set and forget hard filters are a lot more problematic when you dump tons of words in them including huge trigger lists of words that are not slurs. The word for sexual assault that starts with r for example is used in a ton of headlines and therefore links mentioning it break.

At the very least an instance option to exclude links from the slur filter or replace the link if it’s not a hyperlink with text (e.g. “Click here” where that hides the actual http address) that says “potential slur link” rather than the https://www/. somesite .soviet /innocuouswordcontainingslurcharacters)

If above is not possible, can “removed” be turned into “=removed=” or something like it with symbols on the sides? It can be difficult to read when you have something like sremovedhorpe, where the edges of “removed” blend into the word itself, e.g. s=removed=horpe. This could just be a me problem.

Not very pretty version of OP using only local edits, inline with current tech:

s

horpe