Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ufdbGuard is unable to optimize easylist because of too many lines #9

Open
f-galland opened this issue Apr 3, 2022 · 1 comment
Open

ufdbGuard is unable to optimize easylist because of too many lines #9

f-galland opened this issue Apr 3, 2022 · 1 comment

Comments

@f-galland
Copy link

f-galland commented Apr 3, 2022 •
edited
Loading

Hi there!

I've successfully set up ufdbguard with easylists using your script. Nice.

Once thing, though, is that ufdb complaints that it's too long a list to optimize, and that it will process urls one by one:

2022-04-03 17:58:39 [18335] ufdbCategory adblock
2022-04-03 17:58:39 [18335] ufdbCategoryExpressionList adblock/easylist
2022-04-03 17:58:39 [18335] loading regular expressions from "/usr/local/ufdbguard/blacklists/adblock/easylist"
2022-04-03 17:58:45 [18335] WARNING: the expressionlist has 25101 expressions and may use many resources  *****
2022-04-03 17:58:45 [18335]       Note that large numbers of expressions may impact performance considerably
2022-04-03 17:58:50 [18335] ERROR: UFDBoptimizeExprList: unable to optimise 25101 expressions of /usr/local/ufdbguard/blacklists/adblock/easylist  (error 12)  *****
2022-04-03 17:58:50 [18335] Since the 25101 expressions could not be optimised into one expression, they will be evaluated one by one which impacts performance  *****
2022-04-03 17:58:50 [18335] ufdbCategoryExpressionList adblock/easyprivacy
2022-04-03 17:58:50 [18335] loading regular expressions from "/usr/local/ufdbguard/blacklists/adblock/easyprivacy"
2022-04-03 17:59:06 [18335] WARNING: the expressionlist has 45652 expressions and may use many resources  *****
2022-04-03 17:59:06 [18335]       Note that large numbers of expressions may impact performance considerably
2022-04-03 17:59:11 [18335] ERROR: UFDBoptimizeExprList: unable to optimise 45652 expressions of /usr/local/ufdbguard/blacklists/adblock/easyprivacy  (error 12)  *****
2022-04-03 17:59:11 [18335] Since the 45652 expressions could not be optimised into one expression, they will be evaluated one by one which impacts performance  *****

there is some heavy RAM usage (2GB+) on the server as well.

I was wondering if splitting the output file into separate smaller files could be a simple solution for this?

Thanks a lot for the script!

EDIT:

I something like the following

cd /usr/local/ufdbguard/blacklists/adblock
split -a5 -l400 -d easyprivacy easyprivacy_
split -a5 -l400 -d easylist easylist_
echo -ne "\n\n\n\n\n\n\n"
for i in $(ls /usr/local/ufdbguard/blacklists/adblock); do echo "expressionlist adblock/$i"; done

Which splits the easylist and easyprivacy files in smaller 400-lines files (the maximum recommended by the docs is 500) and then outputs the needed expressionlists for the ufdbGuard.conf file.

Then I modified my ufdbGuard.conf like so:

category adblock {
    expressionlist adblock/easylist_00000
    expressionlist adblock/easylist_00001
    expressionlist adblock/easylist_00002
    expressionlist adblock/easylist_00003
    expressionlist adblock/easylist_00004
    expressionlist adblock/easylist_00005
    expressionlist adblock/easylist_00006
    expressionlist adblock/easylist_00007
    expressionlist adblock/easylist_00008
    expressionlist adblock/easylist_00009
    expressionlist adblock/easylist_00010
    expressionlist adblock/easylist_00011
    expressionlist adblock/easylist_00012
    expressionlist adblock/easylist_00013
    expressionlist adblock/easylist_00014
    expressionlist adblock/easylist_00015
    expressionlist adblock/easylist_00016
    expressionlist adblock/easylist_00017
    expressionlist adblock/easylist_00018
    expressionlist adblock/easylist_00019
    expressionlist adblock/easylist_00020
    expressionlist adblock/easylist_00021
    expressionlist adblock/easylist_00022
    expressionlist adblock/easylist_00023
    expressionlist adblock/easylist_00024
    expressionlist adblock/easylist_00025
    expressionlist adblock/easylist_00026
    expressionlist adblock/easylist_00027
    expressionlist adblock/easylist_00028
    expressionlist adblock/easylist_00029
    expressionlist adblock/easylist_00030
    expressionlist adblock/easylist_00031
    expressionlist adblock/easylist_00032
    expressionlist adblock/easylist_00033
    expressionlist adblock/easylist_00034
    expressionlist adblock/easylist_00035
    expressionlist adblock/easylist_00036
    expressionlist adblock/easylist_00037
    expressionlist adblock/easylist_00038
    expressionlist adblock/easylist_00039
    expressionlist adblock/easylist_00040
    expressionlist adblock/easylist_00041
    expressionlist adblock/easylist_00042
    expressionlist adblock/easylist_00043
    expressionlist adblock/easylist_00044
    expressionlist adblock/easylist_00045
    expressionlist adblock/easylist_00046
    expressionlist adblock/easylist_00047
    expressionlist adblock/easylist_00048
    expressionlist adblock/easylist_00049
    expressionlist adblock/easylist_00050
    expressionlist adblock/easylist_00051
    expressionlist adblock/easylist_00052
    expressionlist adblock/easylist_00053
    expressionlist adblock/easylist_00054
    expressionlist adblock/easylist_00055
    expressionlist adblock/easylist_00056
    expressionlist adblock/easylist_00057
    expressionlist adblock/easylist_00058
    expressionlist adblock/easylist_00059
    expressionlist adblock/easylist_00060
    expressionlist adblock/easylist_00061
    expressionlist adblock/easylist_00062
    expressionlist adblock/easyprivacy_00000
    expressionlist adblock/easyprivacy_00001
    expressionlist adblock/easyprivacy_00002
    expressionlist adblock/easyprivacy_00003
    expressionlist adblock/easyprivacy_00004
    expressionlist adblock/easyprivacy_00005
    expressionlist adblock/easyprivacy_00006
    expressionlist adblock/easyprivacy_00007
    expressionlist adblock/easyprivacy_00008
    expressionlist adblock/easyprivacy_00009
    expressionlist adblock/easyprivacy_00010
    expressionlist adblock/easyprivacy_00011
    expressionlist adblock/easyprivacy_00012
    expressionlist adblock/easyprivacy_00013
    expressionlist adblock/easyprivacy_00014
    expressionlist adblock/easyprivacy_00015
    expressionlist adblock/easyprivacy_00016
    expressionlist adblock/easyprivacy_00017
    expressionlist adblock/easyprivacy_00018
    expressionlist adblock/easyprivacy_00019
    expressionlist adblock/easyprivacy_00020
    expressionlist adblock/easyprivacy_00021
    expressionlist adblock/easyprivacy_00022
    expressionlist adblock/easyprivacy_00023
    expressionlist adblock/easyprivacy_00024
    expressionlist adblock/easyprivacy_00025
    expressionlist adblock/easyprivacy_00026
    expressionlist adblock/easyprivacy_00027
    expressionlist adblock/easyprivacy_00028
    expressionlist adblock/easyprivacy_00029
    expressionlist adblock/easyprivacy_00030
    expressionlist adblock/easyprivacy_00031
    expressionlist adblock/easyprivacy_00032
    expressionlist adblock/easyprivacy_00033
    expressionlist adblock/easyprivacy_00034
    expressionlist adblock/easyprivacy_00035
    expressionlist adblock/easyprivacy_00036
    expressionlist adblock/easyprivacy_00037
    expressionlist adblock/easyprivacy_00038
    expressionlist adblock/easyprivacy_00039
    expressionlist adblock/easyprivacy_00040
    expressionlist adblock/easyprivacy_00041
    expressionlist adblock/easyprivacy_00042
    expressionlist adblock/easyprivacy_00043
    expressionlist adblock/easyprivacy_00044
    expressionlist adblock/easyprivacy_00045
    expressionlist adblock/easyprivacy_00046
    expressionlist adblock/easyprivacy_00047
    expressionlist adblock/easyprivacy_00048
    expressionlist adblock/easyprivacy_00049
    expressionlist adblock/easyprivacy_00050
    expressionlist adblock/easyprivacy_00051
    redirect http://MY_IP:8081/cgi-bin/URLblocked.cgi?admin=%A&mode=default&color=red&size=normal&clientaddr=%a&clientname=%n&clientuser=%i&clientgroup=%s&targetgroup=%t&url=%u
}

And now I'm not getting any warning whatsoever and the memory usage dropped down to 500MB, which is still a lot, but much better than 2GB.

Curious how such a dirty solution worked.

Anyway, I hope it helps someone

Cheers
@f-galland
Copy link
Author

Another suggestion:

If you are using ssl bump to transparently proxy HTTPS with squid, your category adblock block, should include the following:

option block-bumped-connect on

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@f-galland