We’ve had some trouble recently with posts from aggregator links like Google Amp, MSN, and Yahoo.
We’re now requiring links go to the OG source, and not a conduit.
In an example like this, it can give the wrong attribution to the MBFC bot, and can give a more or less reliable rating than the original source, but it also makes it harder to run down duplicates.
So anything not linked to the original source, but is stuck on Google Amp, MSN, Yahoo, etc. will be removed.
I already sent it. It’s here:
https://ponder.cat/wp/wp-sources.zip
Edit: You don’t need to do the import initially, since there’s already a sources file with some small modifications. The import is the only complicated part. Use categorize.py to categorize a source, or lookup.py to run a quick command-line test.
Ok i will look into it, thanks i thought it was just the sources not the code.
Ok i implemented it into the bot and it took about 1 hour and 6 minutes to fetch all links and i am now implementing the part where it is inserted into the new text.
Sounds good. If you redid the import, I think you’ll want to make some manual fixes to the .json. Off the top of my head, I think you just need to add bbc.co.uk and aljazeera.com to the URL lists for those sources.