小内存 Debian 服务器防护改造：从 CrowdSec 迁移到 Caddy GeoIP 与 Fail2ban

On a Debian server with limited memory, I originally used CrowdSec to analyze SSH and web logs, then used Firewall Bouncer to synchronize ban decisions to nftables.

This setup could dynamically ban IP addresses based on attack behavior. However, for a server with only a few hundred MiB of RAM running several personal services, its resource overhead needed to be reconsidered. I ultimately adopted the following combination:

• Website: Caddy GeoIP and a manual IP allowlist, with an option to exempt CDN-to-origin traffic on a per-site basis.
• SSH: Fail2ban reads the systemd journal and bans anomalous sources through nftables.

This article documents the complete migration procedure, as well as the log-source and startup-ordering issues encountered during deployment. All domains, addresses, and host identifiers have been replaced with examples. Commands are run as the root user.

1. First, clarify the changes in protection capabilities

Country allowlists cannot serve as an equivalent substitute for behavior detection. Malicious requests from allowed countries may still reach the application, and Caddy’s GeoIP rules do not cover other public ports that do not pass through it.

2. When checking memory usage, inspect RSS and Swap together

First, get an overall picture:

free -h

systemctl show crowdsec crowdsec-firewall-bouncer \
    -p Id -p MemoryCurrent

ps -eo pid,comm,rss,args --sort=-rss | head -n 15

Looking at RSS alone may underestimate a process’s memory usage because some anonymous memory has already been swapped out. You can sort by the combined total of RSS and Swap:

for status_file in /proc/[0-9]*/status; do
    awk '
        /^Name:/   { name=$2 }
        /^VmRSS:/  { rss=$2 }
        /^VmSwap:/ { swap=$2 }
        END {
            if (rss+swap > 0)
                printf "%12d %12d %12d %s\n", rss+swap, rss, swap, name
        }
    ' "$status_file" 2>/dev/null
done | sort -nr | head -n 5

The four columns are: combined KiB, RSS KiB, Swap KiB, and process name. This is a process ranking; if the same service has multiple processes, they need to be aggregated further.

During this investigation, CrowdSec used approximately 42 MiB of RSS and 225 MiB of VmSwap, for a rough combined total of 267 MiB. This shows that it was indeed a component worth slimming down, but all 267 MiB cannot be treated as physical memory.

You also need to distinguish between “Swap is already in use” and “frequent paging is currently occurring”:

vmstat 1 10

Observe the sampling data after the first row. si and so represent the amount of data swapped in and out per second, respectively. A large amount of Swap in use does not, by itself, mean that sustained paging is currently taking place.

For other high-usage processes, you can use the PID to identify the service:

systemctl status 实际PID --no-pager --full

After confirming its purpose, decide whether to disable it:

systemctl disable --now 实际服务名.service

3. Back up and disable CrowdSec

It is recommended to save the configurations before beginning; if possible, validate the new setup before disabling the old one to minimize the protection gap.

The backup directory below assumes this is the first migration. If repeating the operation, use a different directory to avoid overwriting the initial state.

install -d -m 0700 /root/security-migration-backup

cp -a /etc/crowdsec /root/security-migration-backup/
cp -a /etc/caddy /root/security-migration-backup/
cp -a /usr/bin/caddy /root/security-migration-backup/caddy.original

nft list ruleset > /root/security-migration-backup/nftables.before.nft

systemctl list-unit-files --no-pager 'crowdsec*' \
    > /root/security-migration-backup/crowdsec-units.before.txt

Here, the original Caddy binary is assumed to be located at /usr/bin/caddy. If it is actually elsewhere, inspect the service definition first:

systemctl cat caddy

3.1 Disable scheduled jobs and resident services

First, inspect the CrowdSec-related units:

systemctl list-unit-files --no-pager 'crowdsec*'

The following loop uses Bash to disable every CrowdSec timer it finds:

while read -r crowdsec_timer _; do
    if [ -n "$crowdsec_timer" ]; then
        systemctl disable --now "$crowdsec_timer"
    fi
done < <(
    systemctl list-unit-files \
        --type=timer --no-legend --no-pager 'crowdsec*'
)

Stop the Bouncer first and then the engine, and mask both units to prevent them from being started again:

systemctl disable --now crowdsec-firewall-bouncer.service
systemctl disable --now crowdsec.service

systemctl mask crowdsec.service crowdsec-firewall-bouncer.service

Check the results:

systemctl is-active crowdsec crowdsec-firewall-bouncer
systemctl is-enabled crowdsec crowdsec-firewall-bouncer

The expected results are inactive and masked, respectively. The SSH, Caddy, and Nginx Collections are all executed by the CrowdSec engine and do not need to be disabled individually.

3.2 Check for residual firewall rules

nft list tables

The default configuration generally uses table ip crowdsec and table ip6 crowdsec6. If they still exist after CrowdSec has been disabled, compare them with the actual configuration under /etc/crowdsec/bouncers/, confirm that they are tables used exclusively by the Bouncer, and then delete them:

nft delete table ip crowdsec
nft delete table ip6 crowdsec6

Only run the commands for tables that actually exist and have been confirmed to belong to CrowdSec. Do not run nft flush ruleset, and do not disable the nftables service.

If CrowdSec rules were previously added manually to /etc/nftables.conf or a file referenced by it, remove the corresponding persistent definitions as well.

Temporarily retain the packages and configurations to make rollback possible; disabling the services already releases the resources they were using at runtime.

4. Add the GeoIP module to Caddy

This article uses the third-party module porech/caddy-maxmind-geolocation. It matches the client’s source using a local, MaxMind-compatible MMDB database.

4.1 Record the existing version and plugins

/usr/bin/caddy version
/usr/bin/caddy list-modules --packages --versions
systemctl cat caddy

Existing plugins for DNS, rate limiting, and other features must also be retained. Do not directly replace the original file with a new binary that contains only the GeoIP module.

4.2 Add the module to a copy

On a server with limited memory, you can use Caddy’s online build-and-download feature to avoid running the Go compiler locally:

install -d -m 0755 /usr/local/lib/caddy

install -m 0755 \
    /usr/bin/caddy \
    /usr/local/lib/caddy/caddy

/usr/local/lib/caddy/caddy add-package \
    github.com/porech/caddy-maxmind-geolocation

add-package updates to the latest Caddy release and rebuilds the binary with both the existing registered plugins and the new plugin; it does not guarantee that the original core version will be preserved. At this point, the copy is being modified; the running service and the original /usr/bin/caddy remain unchanged.

If the core and plugin versions must be pinned, build the binary with xcaddy on another machine with sufficient resources, specifying the required versions, and upload a binary that matches the target operating system and architecture.

Inspect the new version, plugin list, and compatibility with the existing configuration:

/usr/local/lib/caddy/caddy version
/usr/local/lib/caddy/caddy list-modules --packages --versions

/usr/local/lib/caddy/caddy validate \
    --config /etc/caddy/Caddyfile \
    --adapter caddyfile

If the existing plugins cannot be rebuilt or validation of the original configuration fails, resolve the compatibility issue before switching the service.

4.3 Install the Country database

Obtain the GeoLite2 Country MMDB database from the MaxMind official website. After downloading and extracting it, assume the file has been placed at /tmp/GeoLite2-Country.mmdb:

install -d -o root -g caddy -m 0750 /var/lib/caddy/geoip

install -o root -g caddy -m 0640 \
    /tmp/GeoLite2-Country.mmdb \
    /var/lib/caddy/geoip/GeoLite2-Country.mmdb

runuser -u caddy -- test -r \
    /var/lib/caddy/geoip/GeoLite2-Country.mmdb

The final command exits successfully if the caddy user can read the database. This assumes that both the service’s runtime user and group are caddy; they should match the actual unit definition.

The Country database is sufficient when only the country or region needs to be determined. Xray’s geoip.dat is not an MMDB file that this plugin can use directly.

5. Maintain the country, manual IP, and CDN-to-origin allowlists separately

5.1 Prepare three matcher configurations

Place the three files under /etc/caddy/. Each file defines a site-level matcher.

geo_whitelist.conf defines the allowed countries. A source IP that does not belong to CN, HK, or JP matches @geo_denied.

# 在站点块中引用此文件。
@geo_denied {
    not {
        maxmind_geolocation {
            db_path /var/lib/caddy/geoip/GeoLite2-Country.mmdb
            allow_countries CN HK JP
        }
    }
}

custom_allowlist.conf evaluates requests according to the client address recognized by Caddy. An empty allowlist is represented by expression true, in which case every request matches @not_custom_allowed. To add manually allowed addresses, replace expression true with the commented not client_ip line below and enter the actual addresses.

# 在站点块中引用此文件。
# 默认手动白名单为空。启用时，用下方的 not client_ip 行替换 expression true。
# 在同一行填写实际 IPv4、IPv6 或 CIDR，以空格分隔。
@not_custom_allowed expression true
# @not_custom_allowed not client_ip 203.0.113.10 198.51.100.0/24 2001:db8::10

cdn_allowlist.conf uses remote_ip to match the source of CDN-to-origin connections. Replace the example ranges below with the actual origin CIDRs supplied by the CDN provider; IPv4 and IPv6 can both be placed on the same line.

@not_cdn_origin not remote_ip 192.0.2.0/24 2001:db8:100::/48

5.2 Define two reusable snippets

Place the following two snippets after the global options block in the Caddyfile and outside the site blocks.

(access_whitelist) {
    import /etc/caddy/custom_allowlist.conf
    import /etc/caddy/geo_whitelist.conf

    route @not_custom_allowed {
        route @geo_denied {
            respond "Forbidden" 403
        }
    }
}

(access_whitelist_with_cdn) {
    import /etc/caddy/cdn_allowlist.conf
    import /etc/caddy/custom_allowlist.conf
    import /etc/caddy/geo_whitelist.conf

    route @not_cdn_origin {
        route @not_custom_allowed {
            route @geo_denied {
                respond "Forbidden" 403
            }
        }
    }
}

Each route layer checks whether the corresponding allowlist was not matched. A 403 is returned only when every layer matches. Once the checks pass, the request continues through the site’s application routing.

5.3 Reference them in site blocks

Choose one snippet for each site. Both snippets define @not_custom_allowed and @geo_denied, so avoid importing them more than once in the same site.

direct.example.com {
    import access_whitelist

    reverse_proxy 127.0.0.1:8080
}

cdn.example.com {
    import access_whitelist_with_cdn

    reverse_proxy 127.0.0.1:9090
}

For these two sites, Caddy’s default ordering executes route before reverse_proxy. If a site uses handle, handle_path, or redir, inspect the adapted route order to ensure that the allowlist checks occur before any application handling that could respond early.

/usr/local/lib/caddy/caddy adapt \
    --config /etc/caddy/Caddyfile \
    --adapter caddyfile \
    --pretty

5.4 CDN-to-origin requests and client IPs

access_whitelist_with_cdn treats CDN-to-origin traffic as an independent allow condition. When a request reaches the origin through an allowlisted CDN node, the origin skips the manual IP and GeoIP checks, so this origin-side rule will not restrict the countries of visitors behind the CDN. If you want to allow only CDN-to-origin traffic, or also want to check the visitor’s country behind the CDN, design different rules for the corresponding objective.

The manual allowlist and GeoIP lookup use the client IP recognized by Caddy. To check by the visitor’s address behind a CDN, configure trusted_proxies and the client IP header according to the CDN provider’s documentation. After the allowlist checks pass, existing mTLS or application authentication continues to run as usual.

6. Make systemd use a separate Caddy binary

Using a separate path and a drop-in can prevent a later APT update to /usr/bin/caddy from overwriting the custom binary currently in use. You will also need to maintain updates to the custom binary yourself.

install -d -m 0755 /etc/systemd/system/caddy.service.d

vim /etc/systemd/system/caddy.service.d/90-geoip-binary.conf

Write the following:

[Service]
ExecStart=
ExecStart=/usr/local/lib/caddy/caddy run --environ --config /etc/caddy/Caddyfile

ExecReload=
ExecReload=/usr/local/lib/caddy/caddy reload --config /etc/caddy/Caddyfile --force

If the original service uses different arguments, a configuration path, or an adapter, adjust them according to the actual startup command.

Validate the configuration and then switch:

/usr/local/lib/caddy/caddy validate \
    --config /etc/caddy/Caddyfile \
    --adapter caddyfile

After validation succeeds, run:

systemctl daemon-reload
systemctl restart caddy

systemctl status caddy --no-pager
journalctl -u caddy -n 50 --no-pager

The initial switch to the new binary requires a restart and briefly interrupts connections. Afterward, when only modifying the configuration, use:

systemctl reload caddy

Test using egress addresses in different countries, or use a test server that is not included in the manual allowlist:

curl -I https://app.example.com/

Allowed sources should proceed to backend processing, while other sources should receive a 403. The local machine and private-network addresses generally have no country attribution; if they need to be monitored locally, explicitly add the necessary IP addresses to the allowlist.

7. Protect SSH with Fail2ban

The Fail2ban version actually tested in this article is version 1.0.2 from the Debian 12 repository.

7.1 Installation and inspection

apt update
apt install fail2ban python3-systemd nftables

fail2ban-client -V

/usr/sbin/sshd -T | grep '^port '
ss -lntp | grep sshd
journalctl -u ssh.service -n 20 --no-pager

The following configuration assumes that the SSH port is 22; it must be changed if the actual port is different.

7.2 Create an SSH jail

vim /etc/fail2ban/jail.d/90-sshd.local

Write the following:

[sshd]
enabled = true

backend = systemd
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd
filter = sshd[mode=normal]

port = 22
protocol = tcp
banaction = nftables[type=multiport]

usedns = no
ignoreip = 127.0.0.1/8 ::1

findtime = 10m
maxretry = 5
bantime = 1h

If the same source accumulates 5 matching failure events within 10 minutes, access to the SSH port will be banned for one hour. The number of matching events does not necessarily equal the number of connections.

The + in journalmatch represents OR between two groups of conditions. When using the systemd backend, do not set a file-based logpath. If there is already a fixed, trusted management egress, you may add ignoreip as needed.

Use .local files for configuration overrides to avoid modifying package-provided .conf files.

7.3 Validate and Start

fail2ban-client -t

After the check passes:

systemctl enable fail2ban
systemctl restart fail2ban

Wait a few seconds, then check:

fail2ban-client ping
fail2ban-client status
fail2ban-client status sshd

The expected result is that ping returns pong, and the Jail list contains only the items that need to be enabled. If other jails are enabled by an old configuration, explicitly disable them in the corresponding .local file.

Keep the existing SSH session open and open another terminal to verify normal key-based login.

8. Two Common Startup Issues

8.1 SSH Log File Not Found

ERROR Failed during configuration: Have not found any log file for sshd jail

This usually means that the current jail is looking for a file-based log while the system is actually using the journal. First, use journalctl -u ssh.service to confirm that the logs exist, then configure backend = systemd, install python3-systemd, and restart.

8.2 Client Cannot Find the Socket

ERROR Failed to access socket path: /var/run/fail2ban/fail2ban.sock.
Is fail2ban running?

This indicates that the client failed to connect; this line alone does not mean that the service failed to start. In this deployment, the client ran its check in the same second that the service started, while the service reported Server ready in the next second, so checking again later worked normally.

Troubleshooting commands:

fail2ban-client -t
systemctl status fail2ban --no-pager --full
journalctl -u fail2ban -b -n 80 --no-pager
tail -n 80 /var/log/fail2ban.log

There is no need to create the socket manually; the service process should create it.

If you see:

WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'

This is a warning caused by using the default value auto; it did not prevent startup in this deployment.

9. Verify Fail2ban’s Actual Blocking Effectiveness

9.1 First Verify IPv4 and IPv6 Rule Generation

Use documentation-reserved addresses to test rule synchronization:

fail2ban-client set sshd banip 203.0.113.10
fail2ban-client set sshd banip 2001:db8::10

nft list table inet f2b-table

Check whether the addresses appear in the sets and whether the associated rules match the actual SSH port. The default nftables action may create rules only when the first ban occurs, so if there are no bans yet, the table may not be visible; this is not necessarily a fault.

After completing the test, remove the bans:

fail2ban-client set sshd unbanip 203.0.113.10
fail2ban-client set sshd unbanip 2001:db8::10

This test validates the execution path, but it does not prove that both the log-detection conditions and actual network blocking have been verified. Ban records may survive a service restart, so they should be explicitly removed when the test is finished.

9.2 Verify Connections from Another Controlled Server

On the target server, ban the actual public IP of another controlled server:

fail2ban-client set sshd banip 测试服务器实际公网IPv4

Then, on the banned server, open a new connection:

nc -4 -vz -w 5 目标服务器实际公网IPv4 22

You can also observe the SSH handshake:

ssh -4 -vvv \
    -o ConnectTimeout=5 \
    -o ConnectionAttempts=1 \
    -p 22 testuser@目标服务器实际公网IPv4

Replace the Chinese placeholder values and the port in the examples with the actual values. The default nftables action is reject, so the connection may immediately report a rejection; if it is changed to drop, it will typically time out. If the connection reaches the SSH host-key prompt or authentication stage, this new connection was not blocked by the ban.

When necessary, capture packets on the target server to confirm whether the request arrives:

tcpdump -ni any -nn \
    'host 测试服务器实际公网IPv4 and tcp port 22'

Seeing the SYN in tcpdump does not mean that the firewall allowed it.

After removing the ban, repeat the connection from the test server:

fail2ban-client set sshd unbanip 测试服务器实际公网IPv4

A connection failure while the ban is active and a successful connection after it is removed are the intuitive comparative results. Use an explicit IPv4 address and -4 to prevent the connection from actually using IPv6, since only the IPv4 address is banned. Do not use the only current management access source for the ban test.

10. Plugin Overhead and Database Maintenance

According to the project source reviewed, the GeoIP plugin opens the database reader on the first match and reuses it thereafter. When the matcher runs, it performs a local query; there are no remote GeoIP API requests and no additional caching of IP query results.

The maxminddb-golang library it uses supports mmap on Linux. The mapped file still consumes physical memory, but the database file’s size cannot be directly treated as additional Go heap memory. A snippet is configuration reuse and does not guarantee that all sites share a single plugin instance; mapped pages from the same file can be shared by the operating system.

For a personal service, the additional overhead is expected to be small, but no definitive QPS, query latency, or net increase in plugin memory was measured on this machine. Data frames in established long-lived connections, such as WebSocket connections, will not rerun this GeoIP check for every packet.

Using only the Country database and disabling unnecessary global debug during normal operation can reduce unnecessary data volumes and per-request debug logging.

To view the actual memory usage of the entire Caddy process:

systemctl show caddy -p MemoryCurrent

caddy_pid=$(systemctl show caddy -p MainPID --value)

awk '/^(Rss|Pss|Swap|SwapPss):/' \
    "/proc/$caddy_pid/smaps_rollup"

The database requires ongoing maintenance. When updating it, first write a new file in the same directory and then replace the original by renaming it, avoiding directly truncating the original file while it is being used via mmap:

install -o root -g caddy -m 0640 \
    /tmp/GeoLite2-Country.mmdb \
    /var/lib/caddy/geoip/GeoLite2-Country.mmdb.new

mv -f \
    /var/lib/caddy/geoip/GeoLite2-Country.mmdb.new \
    /var/lib/caddy/geoip/GeoLite2-Country.mmdb

systemctl reload caddy

The reload here relies on the --force in the earlier drop-in, ensuring that it reloads even when the configuration text has not changed. The plugin itself does not automatically download the database.

11. Acceptance and Rollback

Confirm the following during acceptance:

• CrowdSec and Firewall Bouncer have been stopped, and their related timers have been disabled.
• The new binary actually running under Caddy includes GeoIP and the pre-existing plugins.
• Directly connected sites allow requests from manually whitelisted sources and allowed regions, while other sources return 403. Sites with CDN exemptions also allow listed origin nodes.
• Fail2ban’s SSH jail has begun processing new journal logs.
• IPv4 and IPv6 rule generation, as well as the ban/unban comparison test using a real connection, have passed.
• Normal SSH key-based login and existing application access have not been unintentionally affected.

To view service status and overall memory usage:

systemctl is-active caddy fail2ban
systemctl is-active crowdsec crowdsec-firewall-bouncer

free -h

To roll back to the original setup:

systemctl disable --now fail2ban

cp -a /root/security-migration-backup/caddy/. /etc/caddy/

rm /etc/systemd/system/caddy.service.d/90-geoip-binary.conf

systemctl daemon-reload
systemctl restart caddy

systemctl unmask crowdsec.service crowdsec-firewall-bouncer.service
systemctl enable --now crowdsec.service
systemctl enable --now crowdsec-firewall-bouncer.service

This procedure preserves the original /usr/bin/caddy binary and CrowdSec configuration. If timers had previously been enabled, restore the corresponding items according to the backed-up unit status. After the Bouncer starts, it will resynchronize its decisions; there is no need to blindly overwrite the entire old firewall snapshot.

References

• Caddy MaxMind Geolocation Module
• Caddy Command Line and add-package
• Caddy systemd Service and drop-in
• Caddy import and snippet
• Caddy Request Matchers: client_ip and not
• Caddy Directive Execution Order
• Caddy route
• MaxMind GeoLite Database
• maxminddb-golang v1.13.1 Documentation
• Debian 12 Fail2ban Package
• Debian 12 Fail2ban jail.conf Manual
• Debian 12 Fail2ban nftables Action Source
• CrowdSec Firewall Bouncer
• Linux /proc Filesystem