Next automation task?

[fishcharlie]

@blocklistproject What would you like to see next for automation on this project? You obviously run a lot of the day to day of this project and have an extremely good sense of what people want, and what takes up most of your time. So just looking for some of that insight in terms of what to prioritize on the automation front.

[blocklistproject]

What about deduplication in each list? Most of the lists I have deduplicated but as we add more URLs and grow the lists there is bound to be duplicates.

[fishcharlie]

@blocklistproject #342. Originally I was going down the path of removing duplicates if they existed in multiple lists, but decided that was a bad idea. So this PR only removes duplicates if they exist in the same list. Duplicates in different lists won't be removed.

Let me know if you think we should change that.

[fishcharlie]

@blocklistproject I think a lot of the automation ideas I originally had are now complete. Do you have any ideas or suggestions for future automation tasks?

Here are a few ideas I have. However I'm not sure if they are worth it or if they'd even be a good idea.

• There are some entries like 0.0.0.0 220.181.12.120/29 (in abuse.txt). It might be a good idea to separate this out into individual IP addresses instead of using net masks. My concern with this is that I'm not sure how many entries this will actually add. I also think this might be causing issues on the dnsmasq lists.
• Sort entries within each list
• Add linter to ensure domain names and lists are valid and in a good format
• For each domain that starts with www. also include same domain without the www. prefix
• Automate creating PRs for add and remove requests from issues

[blocklistproject]

• I need to go through and remove all IP. The devices that use these lists for the most part can't have IPs included.

• An automation to alpha sort each list would be nice.

• can you explain what the linter would do

• adding a www for each domain would be great.

• what would be the benefit for creating a PR for each add/remove

[fishcharlie]

@blocklistproject

I need to go through and remove all IP. The devices that use these lists for the most part can't have IPs included.

Although my system isn't using IP addresses currently, I have a feature in my systems backlog to add support for blocking IPs. Not gonna dictate how things should go here, especially since I'm only one user. But maybe put those in a different or special list for the time being? Or have a different list category for IPs?

An automation to alpha sort each list would be nice.

👍

can you explain what the linter would do

It would basically ensure different rules that we set. This could include ensuring that all IP addresses are 0.0.0.0. Ensuring that only spaces are used between 0.0.0.0 and the domain name. Ensuring that the domain name doesn't have any spaces in it. After you remove all IP addresses it could ensure that the domain name isn't an IP address. List could go on and on. But it'd basically just validate that the format is valid. I've run into a few instances where these things weren't actually met and had to be corrected. So at least having an alert about errors here might be good.

adding a www for each domain would be great

👍. How about removing www if there is no non www version of the domain? ie. should it be bidirectional adding and removing www?

what would be the benefit for creating a PR for each add/remove

Instead of having to do the work of adding it to a text field, you'd be able to write a comment on an issue that says @blocklistproject-bot Create PR or something along those lines, and it would read the original comment, and create a PR for you automatically. This would save you time from having to manually open your text editor, add the domains, create a commit, open a PR, etc.

I'm not sure exactly how much time this would save you. But creating a PR to resolve an issue would just be 1 comment, instead of a multi step process.

[blocklistproject]

Also would like to work on creating a version of the lists for adblock

Syntax can be seen here: https://adblockplus.org/en/filter-cheatsheet

I am going to see if I can get the script to work for it and will post it here for you to check if you don't mind

[fishcharlie]

@blocklistproject Happy to review this. Not quite sure I understand the syntax of that link you sent. But I'd be happy to review the PR when you post it! Just tag me in it or something.