Skip to content

Extreme memory usage when using large ipsets #738

@principis

Description

@principis

What happened:
I'm using this guide to create a large country blocklist using ipsets.

Adding the ipsets takes a long time (around 5 minutes) causes high cpu usage and steadily increases the memory usage.
When reloading, the memory usage increased to 35%+ (of 8 GB).

After the reload (or restart) firewalld uses 1.5 GB memory (without the ipset it uses 26.7 MB). This seems way too much.

I tried to remove the ipset, and run firewall-cmd --reload after which the ipset was removed, but the memory usage stayed the same, 1.5 GB. This also seems like a bug. Restarting firewalld (without the blacklist) fixes the problem, after which the memory usage dropped to around 26 MB.

What you expected to happen:
Not use this much of memory, and reload faster.

How to reproduce it (as minimally and precisely as possible):

countries="af al am ar bd ca cl cn ee fr gt hk id in kp kr la pl ro ru sg ua um us uy uz ve vg vi vn vu wf ye zm zw"

# Create a temporary working directory
ipdeny_tmp_dir=$(mktemp -d -t blacklist-XXXXXXXXXX)
pushd $ipdeny_tmp_dir

# Download the latest network addresses by country file
curl -LO http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
tar xf all-zones.tar.gz

firewall-cmd -q --permanent --new-ipset=blacklist --type=hash:net \
    --option=family=inet --option=hashsize=4096 --option=maxelem=200000

# Add the address ranges by country per ipdeny.com to the blacklist
for country in $countries; do
    firewall-cmd -q --permanent --ipset=blacklist \
        --add-entries-from-file=./$country.zone && \
        echo "Added $country to blacklist ipset."
done

firewall-cmd --permanent --zone=drop --add-source=ipset:blacklist
firewall-cmd -q --reload

popd
rm -rf $ipdeny_tmp_dir

Anything else we need to know?:

Environment:

  • Firewalld Version: 0.8.4-1.fc32
  • Firewalld Backend: nftables
  • OS: Fedora 32 (Server Edition)
  • Others:
    • nftables: 0.9.3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions