jogamp.org server ...

Just want to acknowledge the long timeouts of our jogamp.org server :|

A while ago I moved to mpm_event to solve the issues and other details,
now I upped certain metrics (by simply using defaults, installation had a bad config).

Hope it helps, may fiddle w/ details if re-occurring.
If anybody has more experience (we just have a 4 core CPU, 32GB)
how to finetune .. you are welcome.

Let's see how it works now.

jogamp-scripting commit c04b48166645e81bc402bfd59bd7e2d531c8fea5

    apache2: Bump up mpm_event: ServerLimit 24, ThreadsPerChild 25, MaxRequestWorkers 24*25=600

- updated bugzilla to 5.2
- fine tuned MariaDB + ZFS
(see jogamp-scripting repo)

After monitoring the server for a while
the load on bugzilla (perl) was visible.
Since the update wasn't good enough, I realized that MariaDB
and the bugzilla/perl processes consumed each 25% CPU.
MariaDB seems to have  limited overall performance considerably?

I found this nice page for MariaDB + ZFS:
<https://shatteredsilicon.net/mysql-mariadb-innodb-on-zfs/>

After creating a ZFS dataset for /var/lib/mysql w/ recordsize 16K
and adding the following to /etc/mysql/my.cnf

[mysqld]
            #
            # * InnoDB on ZFS
            #   https://shatteredsilicon.net/mysql-mariadb-innodb-on-zfs/
            #
            # innodb_log_write_ahead_size = 16384 (adjusted)
            innodb_log_write_ahead_size = 4096
            innodb_doublewrite = 0
            # innodb_checksum_algorithm = none (no more supported)
            innodb_flush_neighbors = 0
            innodb_use_native_aio = 0

MariaDB is reduced to < 10%, while bugzilla/perl is at ~100%,
i.e. bugzilla is no more DB limited :)

The DoS/SYN or whatever (attack) is sadly persistent over last days :(
Perhaps we just became famous again :)

Can't even SSH to it sporadically

OK, will spend more time analyzing the root cause and what to do.

Earlier I have adjusted
- linux tcp backlog to 5000, also apache2 backlog to 5000
- apache2 mpm metrics won't make a big difference in range 600-1200 max connections
- visible bugzilla/perl resource hog
  - optimized mariadb, good but not enough
  - disabled via release/rc builds helped a lot
...
bear with me ..

https://jogamp.org/cgit/jogamp-scripting.git/log/

commit d776cf17498fa57b9cbb79a79721ea7fd688bf17

    server/setup/05-service-settings: Update MediaWiki Upgrade Info for 1.44
   
    This includes a patch for the Bugzilla extension,
    see server/setup/05-service-settings/patches/commit-c99dc10-wiki1_41-01.patch
    applied on top of https://github.com/mozilla/mediawiki-bugzilla
    commit ef950c90922fd07aaa3e6b2f85cf597bebf1b6aa

commit 4e9f399a7328735c0ccfdf090f9f187f4ad0ecc9

    apache2 mpm_event: We settle w/ defaults to avoid blocking all tcp access for now
   
    Previous enhancements ..
    - mariadb-zfs
    - bugzilla upgrade
    - mediawiki upgrade to 1.44 w/ caching working hopefully
      - including mediawiki Bugzilla extension patch
      - be aware that our mediawiki causes bugzilla requests
    - robots.txt adding bugzilla and wiki
   
    Let's hope this is working well so far.

Had to add the following two /16 networks to the firewall block list :-(
Not obeying robots.txt and I doubt its genuine interest :)
Sure, probably 'just' a handful each .. but can't isolate 'em,
so I blocked the network .. poor mans solution :(

# 8.210.30.40     # AlibabaCloud_HK .. but too many ip
8.210.0.0/16    # AlibabaCloud_HK
# 47.243.251.69   # ALIBABA-CLOUD---HK .. but too many ip
47.243.0.0/16   # ALIBABA-CLOUD---HK

jogamp-scripting commit 6d59cb4e51ad37eb31176656a7f32f26dbd0784e

    Bump: Server/apache settings (back to mpm_event, finetuning for SYNREQ attack)
   
    Bugzilla running in mpm_prefork and mod_perl doesn't solve the server situation.
    Hence reverted.
   
    We have to disable bugzilla for now or from time to time
    or perhaps run it in a separate apache2 instance w/ mod_perl + mpm_prefork?
   
    Or 'just' moving to an alternative w/ better launch performance :/

testing the 2nd apache server instance w/ mod_perl + mpm_prefork,
reached via a proxy config in the main server :)

Seems to work.

scripts netstat-list-abuse.sh and net-count-subs.sh in jogamp-scripting/server/setup/attack/scripts
as well as netstat-openconnections.sh
helps analyzing SYN attacks.

The last one shows SYN, and a manual run w/o watch shows the IP source.

The first one then counts the /16 net and also gives the proper netmask,
to be added to iptables to block w/o active reject.

Turns out tons of questionable bots within the tencent cloud caused
quite some additional troubles sadly.

If anybody has a much more NICE approach than blocking /16 networks,
please share. Thank you.

I have patched cgit (our git web server face)
- cgit_add_repo_with_default_branch <https://jausoft.com/cgit/cgit.git/commit/?h=cgit_add_repo_with_default_branch>
- cgit_sendfile_timeout <https://jausoft.com/cgit/cgit.git/commit/?h=cgit_sendfile_timeout>

Esp cgit_sendfile_timeout finally removes slow-attack instances, which have piled up
and caused resource starvation.

Hence I have re-enabled the apache bugzilla instance, to see how it fares now.

Sven Gothel wrote

I have patched cgit (our git web server face)

Forgot: cgit origin <https://git.zx2c4.com/cgit/about/>
and yes, I have posted both patches on their mailinglist.

Sven Gothel wrote

Machines

- Server stability (against DoS, ...)
  - cgit patches
  - various fine tuning kernel/apache/..
  - we may need a more powerful machine due to incr load(?)

Last week we had a harddisk failure.
Since we use ZFS, nothing was lost, disk has been replaced
and I could resilver it.

I took the chance to cleanup the mem-swap setup
- disabled zfs vdev swap (buggy)
- added native swap partition
See <https://jausoft.com/cgit/jaunotes.git/about/config/misc/linux-boot-uefi_dual-or-single.md>

Further I updated the ZFS backup,
pulling from remote to my local server via
<https://jausoft.com/cgit/openzfs/openzfs_build_env.git/tree/scenarios/backup/zsync-poolFromRemote.sh>.
This has also been done and I will do it regularly again w/ incremental snapshots.

Then I hacked cgit and added a cgit-reaper for reasons described
<https://jausoft.com/cgit/cgit-reaper.git/about/#goals>

Last but not least, I updated our iptables setup
as a final attempt to get rid of the DoS attacks
<https://jausoft.com/cgit/jaunotes.git/tree/config/server/02-firewall/etc/iptables/ip4tables_bad_fwdmz_good-secure#n264>
It features for ipv4 and ipv6
- better syn-flood protection, now limited to 50/s
- same IP http(s) rate-limit to 10 (15) per second
- same ip max-concurrent http(s) connections to 10 (30) per second

So far it shows we got hit a lot by
- syn-flood (wasn't handled properly by kernel)
- http(s) rate-limit

Hope it makes the server more accessible again.
At least I can lower the connection rate per IP
in case this is not enough ...

Cheers

~Sven

EDIT: If interested, I use this script to monitor the filtering results via 'watch -n 2 print-network-stats.sh`
<https://jausoft.com/cgit/jaunotes.git/tree/config/server/xx-attack/scripts/print-network-stats.sh>

Sven Gothel wrote

It features for ipv4 and ipv6
- better syn-flood protection, now limited to 50/s
- same IP http(s) rate-limit to 10 (15) per second
- same ip max-concurrent http(s) connections to 10 (30) per second

So far it shows we got hit a lot by
- syn-flood (wasn't handled properly by kernel)
- http(s) rate-limit

Hope it makes the server more accessible again.
At least I can lower the connection rate per IP
in case this is not enough ...

I have tightened the limits:

    Per Client IP
        New Connections (syn-flooding)
            5/s http   *drop*
            8/s https  *drop*
            10/s total *drop*
        Concurrent Connections
            5 http
            10 https

Further the following new-connection limits per IP are counted
    2/s total
    6/s total
    10/s total *drop*

See <http://jordan.goethel.localnet/cgit/jaunotes.git/about/config/server/xx-attack/DoS-Mitigation.md>

Quite a few offenders are dropped
- bursting w/ multiple new connections >8
- attempting to maintain more than 10 concurrent connections

Will see how it fares ..

If you experience issues with service, like our maven (https) please say so.

The firewall for SYN, DoS and DDos attacks has elaborated - as nothing simple just worked :|

Spoiler: New AI bots starting around July/August 2026 ignoring robots.txt are taking down
self hosting small server like ours. See the Haskell/Lightpanda incident.

Besides JogAmp, Haskell also had a very notable incident.

- 1.4 Million distinct IPv4 addresses have been used with this bot user agent
- the IP addresses stem from 225 different countries. I think we have ~ 30 more distinct ISO country codes.
- we have some known faces! like spacex which got flagged 100% time on their 21000 distinct IP addresses they’re using to DDoS us
- we also have quite a lot of domestic use ASNs, which indicates that botnets are involved, too!
- Xe noted, that we can proudly say this counts as a DDoS

On JogAmp (and jausoft), I have also collected 550k unique IPs (so far) - insane.

The only mitigation here is
- use an updated 'badbots' user-agent query
- blocking @ webserver directly when it happens
- blocking w/ fail2ban (actual nftable) preemptively afterwards

These IPs are banned for a long time (2d and will be kept if still offending)!

Knowing this, we may can relax the even tighter new-connection (syn) and concurrent-connection
limits - but IMO they should be reasonable:

    Per Client IP
        New Connections (syn-flooding)
            5/s http / https
            6/s total
        Concurrent Connections
            5 http / https

These are only banned initially for 5min, but may be increased if continued attempts are made
or removed if not.

My whole document here: See the DoS-Mitigation