/*-----------------------------------------------------------------------------
Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, Alan Kennington.
You may distribute this software under the terms of Alan Kennington's
modified Artistic Licence, as specified in the accompanying LICENCE file.
-----------------------------------------------------------------------------*/

The author of this software disclaims any express or implied guarantee of the
fitness of this software for any purpose. In no event shall the author of this
software be held liable for any direct, indirect, incidental, special,
exemplary, or consequential damages (including, but not limited to, procurement
of substitute services; loss of use, data, or profits; or business interruption)
however caused and on any theory of liability, whether in contract, strict
liability, or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.

Automatic rampant downloaders will download large numbers of useless warning messages.
The original motivation for mod_bwshare was to prevent automatic download of all 900 MBytes of material on one of my web sites which was connected to a 33k modem, which would have taken several days if it had succeeded.
Here is my currently running bwshare admin interface on www.topology.org.

2006-5-16: Note that the mod_bwshare Apache module probably does not work for Apache 1.x.y any more. I haven't tried to build it for Apache 1 for many years. I'll back-port it to Apache 1.x again when I get some spare time.
2006-8-10: The bwshare module does not compile for the Win32 API. I don't intend to port it to Win32 because unix is much better and Win32 is too flaky.

Some other design objectives were CPU efficiency and memory compactness.
Perhaps I was a bit too cautious/thrifty/frugal/miserly because I first learned computer software in 1972 with a 1 MHz CPU and 100 kBytes RAM.
Therefore there is plenty of capacity for additional functionality in future enhancements.
However, I am resisting the temptation to turn mod_bwshare into sluggish bloatware through feature creep.

moose /root# /usr/local/apache2/bin/apxs -c mod_bwshare.c
moose /root# /usr/local/apache2/bin/apxs -i mod_bwshare.la
moose /root# /usr/local/apache2/bin/apachectl restart

I had to add the following line to the httpd.conf file first.

LoadModule bwshare_module modules/mod_bwshare.so

I also added this sort of thing in a file bwshare.conf which I included into the httpd.conf file before doing the apachectl restart command:

<IfModule mod_bwshare.c>
    <Location /bwshare-info>
        SetHandler bwshare-info
    </Location>

    <Location /bwshare-trace>
        SetHandler bwshare-trace
    </Location>

    # Some bandwidth control parameters.
    <Directory />
    BW_tx1debt_max          25
    BW_tx1cred_rate         0.095
    BW_tx2debt_max          3000000
    BW_tx2cred_rate         2500
    </Directory>
</IfModule>

moose /root# /usr/local/apache2/bin/apxs -c mod_bwshare.c
/usr/local/apache2/build/libtool --silent --mode=compile gcc -prefer-pic
 -DAP_HAVE_DESIGNATED_INITIALIZER -DLINUX=2 -D_REENTRANT -D_XOPEN_SOURCE=500
 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE -g -O2 -pthread
 -I/usr/local/apache2/include  -I/usr/local/apache2/include
 -I/usr/local/apache2/include   -c -o mod_bwshare.lo mod_bwshare.c && touch
 mod_bwshare.slo
/usr/local/apache2/build/libtool --silent --mode=link gcc -o mod_bwshare.la
 -rpath /usr/local/apache2/modules -module -avoid-version    mod_bwshare.lo

moose /root# ls -l mod* .libs
-rw-r--r--  1 root root 152562 Nov 15 22:39 mod_bwshare.c
-rw-r--r--  1 root root    822 Nov 22 19:27 mod_bwshare.la
-rw-r--r--  1 root root    324 Nov 22 19:27 mod_bwshare.lo
-rw-r--r--  1 root root 123752 Nov 22 19:27 mod_bwshare.o
-rw-r--r--  1 root root      0 Nov 22 19:27 mod_bwshare.slo

.libs:
total 337
drwxr-xr-x   2 root root    208 Nov 22 19:27 .
drwx------  20 root root    944 Nov 22 19:27 ..
-rw-r--r--   1 root root 123904 Nov 22 19:27 mod_bwshare.a
lrwxrwxrwx   1 root root     17 Nov 22 19:27 mod_bwshare.la -> ../mod_bwshare.la
-rw-r--r--   1 root root    823 Nov 22 19:27 mod_bwshare.lai
-rw-r--r--   1 root root 123752 Nov 22 19:27 mod_bwshare.o
-rwxr-xr-x   1 root root  85438 Nov 22 19:27 mod_bwshare.so

moose /root# /usr/local/apache2/bin/apxs -i mod_bwshare.la
/usr/local/apache2/build/instdso.sh
 SH_LIBTOOL='/usr/local/apache2/build/libtool' mod_bwshare.la
 /usr/local/apache2/modules
/usr/local/apache2/build/libtool --mode=install cp mod_bwshare.la
 /usr/local/apache2/modules/
cp .libs/mod_bwshare.so /usr/local/apache2/modules/mod_bwshare.so
cp .libs/mod_bwshare.lai /usr/local/apache2/modules/mod_bwshare.la
cp .libs/mod_bwshare.a /usr/local/apache2/modules/mod_bwshare.a
ranlib /usr/local/apache2/modules/mod_bwshare.a
chmod 644 /usr/local/apache2/modules/mod_bwshare.a
PATH="$PATH:/sbin" ldconfig -n /usr/local/apache2/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/local/apache2/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /usr/local/apache2/modules/mod_bwshare.so

moose /root# /usr/local/apache2/bin/apachectl restart

apxs2 -c mod_bwshare.c
su -c "apxs2 -i mod_bwshare.la"

I do the above as an ordinary user and then give the super-user password for the second line when it prompts me. Then I run apache2ctl stop and apache2ctl start. In fact, if you want to avoid process exceptions (i.e. program crash), it's best to stop Apache first, then install the new module, and then restart Apache.

I actually use the following as root user:

    apache2ctl stop ; apxs2 -i mod_bwshare.la ; apache2ctl start

I do that after running apxs2 -c mod_bwshare.c as an ordinary user.

On my SuSE 9 systems, I don't need to modify the httpd.conf file, but when I run Apache 2.0.50 compiled from source, I have to add this line to the httpd.conf file:

LoadModule bwshare_module modules/mod_bwshare.so

<IfModule mod_bwshare.c>
# Some bandwidth control parameters.
BW_tx1cred_rate         0.067
BW_tx1debt_max          30
BW_tx2cred_rate         1000
BW_tx2debt_max          1000000

<Location /bwshare-info>
    SetHandler bwshare-info
</Location>

<Location /bwshare-trace>
    SetHandler bwshare-trace
</Location>
</IfModule>

The following global floating-point parameters are defined for control of the mod_bwshare module.

• BW_tx1cred_rate: sets the maximum rate of serving files (files/second).
• BW_tx1debt_max: sets the maximum files to serve in excess of BW_tx1cred_rate (files).
• BW_tx2cred_rate: sets the maximum rate of serving bytes (bytes/second).
• BW_tx2debt_max: sets the maximum bytes to serve in excess of BW_tx2cred_rate (bytes).

Be careful when setting these parameters. The module makes very little attempt to check that your choice of parameters is sane.

It seems that you can specify there parameters pretty much anywhere in the configurations files, even inside virtual host and per-directory contexts. But the effect is always global to the whole bwshare module.

<VirtualHost *:80>
# Other stuff.
# .....

# Set mod_bwshare parameters for this virtual host.
<IfModule mod_bwshare.c>

# Turn off throttling for just this virtual host.
BW_throttle_off         1

</IfModule>

</VirtualHost>

<IfModule mod_bwshare.c>
    # Local network.
    BW_subnet_limit net = 192.168.1.0/24 \
        tx1rate =   80.00 files/min  tx1max =      200 files \
        tx2rate = 1000000 bits/sec   tx2max = 10000000 bytes

    # Googlebot.
    BW_subnet_limit net = 66.249.66.65 / 0xffffff00 \
        tx1rate =   24.00 files/min  tx1max = 80.0 files \
        tx2rate =  120000 bits/sec   tx2max = 4000000 bytes
    BW_subnet_limit net = 66.249.65.205/24 \
        tx1rate =  24.00 files/min  tx1max = 80.0 files \
        tx2rate = 120000 bits/sec   tx2max = 4000000 bytes
</IfModule>

The arguments of the keyword BW_subnet_limit must all be on the same line. A line may be extended by the back-slash character. On the same argument line after BW_subnet_limit, any number of the following parameter settings may be given.

<subnet-limit> := BW_subnet_limit <args>
<args> := [<arg>]...
<arg> := <net> | <tx1rate> | <tx1max> | <tx2rate> | <tx2max>

<net> := (net | network | host | hosts) [ = ] <host> [ / <netmask> ]
<host> := <quad-dotted-decimal>
<netmask> := <int0..32> | <hex8digits>

<tx1rate> := tx1rate [ = ] <value> (file | files | request | requests) / <time>
<tx1max>  := tx1max  [ = ] <value> (file | files | request | requests)
<tx2rate> := tx2rate [ = ] <value> (bit | bits | byte | bytes) / <time>
<tx2max>  := tx2max  [ = ] <value> (bit | bits | byte | bytes)

<value> := <non-negative-real-number>
<time> := <sec> | <min> | <hr> | <day>
<sec> := sec | secs | second | seconds
<min> := min | mins | minute | minutes
<hr>  :=  hr |  hrs |   hour |   hours
<day> := day | days

If you understand the above kludge-BNF syntax, you know more about computers than I do! If any errors are found, the particular BW_subnet_limit configuration is silently ignored. Any parameters which do not appear in the list are replaced with defaults.

The defaults value for the network host/mask is 0.0.0.0/0. Therefore if the subnet is not defined, the bandwidth limitation settings will apply to all HTTP clients. If only the host part of the subnet is specified, the network mask is then 0xffffffff. (Note that the netmask is completely arbitrary. It does not need to be a left-justified sequence of ones. The host part does not need to have zeroes where the netmask has zeroes. The host bits where the netmask is zero are ignored.) In the case of a hexadecimal netmask, precisely 8 digits must be given. Hex digits may be upper or lower case.

If any of the bandwidth limitation parameters are not defined, they are replaced by the corresponding global parameters BW_tx1cred_rate, BW_tx1debt_max, BW_tx2cred_rate and BW_tx2debt_max. All rates and maxima must be non-negative floating point numbers. All time units are converted internally to seconds. The data quantity units are converted internally to bytes. Files and requests mean the same thing.

If subnets overlap, the first matching subnet in the list is used. Therefore if one of the subnets is 0.0.0.0/0, this will prevent all following subnets from taking effect. Today (2006-5-15), the maximum number of subnets is 500. This may be changed in the mod_bwshare.c source file.

To see whether your subnet bandwidth limitations are working, there is a new option in the bwshare-info screen called show bw limits. If you set this, you can see the limits applied to every HTTP client IP address.

To install this software, create a directory apache/modules/bwshare in your apache source tree.
Then copy Makefile.in, config.m4 and mod_bwshare.c to the apache/modules/bwshare directory.
It seems like you might have to run autoconf in the Apache top directory to remake configure from configure.in.
But I found that this created an erroneous line as follows:

LTFLAGS="$LTFLAGS -export-dynamic"

If autoconf causes this line to appear in your configure file, you will have to remove it or comment it out.
I have no idea how this bug gets in there.

Linux reef 2.4.18-586tsc #1 Sun Apr 14 10:57:57 EST 2002 i586 GNU/Linux

The only trouble I had is that I created the source files in the wrong
places.  I wound up with
/.../.../httpd-2.0.50/apache/modules/bwshare/<bwshare sources>

which is wrong. It's:
/.../.../httpd-2.0.50/modules/bwshare/<bwshare sources>

In this case, the <bwshare sources> are the three files you specified
should be copied into a certain directory inside the apache source
directory.

The parameter

--enable-bwshare

must be used with the configure command in the top-level Apache source directory (in addition to any other command line arguments you normally use) if you choose the no option in the config.m4 file.

If there are any problems with the Makefile.in or config.m4 files, compare them with the corresponding files in other modules, e.g. in apache/modules/experimental.
The file apache/modules/experimental/README may also contain useful hints for installing this module.

The parameter

--activate-module=src/modules/bwshare/mod_bwshare.o

should be used with the configure command in the top-level Apache source directory (in addition to any other command line arguments which you normally use).

If there are any problems with the Makefile.tmpl file, compare it with the Makefile.tmpl file in other modules, e.g. in apache/src/modules/example.
The file apache/src/modules/example/README may also contain useful hints for installing this module.
Please let me know of any changes you made to build the bwshare module on your system so that I can incorporate them.

The following line may be suitable in the httpd.conf file.
This should go after the other AddModule commands.

AddModule mod_bwshare.c

See also my notes on installing Apache 1.3.23 with PHP 4.1.2 and bwshare 0.1.2.

1. I have heard that the mod_bwshare module forces the Apache logs to use reverse DNS even when the configuration files specify no host lookup. There should be an option in mod_bwshare which does not activate the reverse DNS at all.
2. Bwshare versions up to 0.1.8 are reported not to work for Apache 2 on solaris if IPv6 is not supported in the Apache build. The fix for this should be in version 0.1.9.
3. The reverse DNS lookup is ver slow because the getnameinfo() function is used instead of ap_get_remote_host(). This is fixed in bwshare version 0.2.0.
4. No others as far as I know... Please find more for me.

The purpose of this module is to give the web site operator some control over bandwidth utilization by individual client hosts.
The bwshare module temporarily blocks access by excessive users.
This is aimed especially at users who download whole websites at great speed.
Excessive speed is considered bad etiquette for search engine robots.

The mod_bwshare module uses a form of statistical shaping.
This means that the software measures the statistical behaviour of the subscriber in the past, and uses this as a basis for controlling the current access to resources by the user.
The principal algorithm used in mod_bwshare is the token bucket.

The guiding principle in this module is that two categories of clients should be allowed unhindered access: human users and well-behaved search engines.
Non-human impolite clients should be throttled as soon as possible.
On an Internet where almost everyone is desperately trying to increase their hit rate, it might seem crazy to try to dampen the enthusiasm of visitors to a site.
But in Australia, we pay by the megabyte for our traffic, and anywhere in the world, you don't make much profit out of visitors who download your whole website if it's really big.
(E.g. if you're a mirror for a few huge mega-sites.)

There are many actions which a throttling module can take when an excessive user is identified.
You can slow them down or you can discard their requests.
You can do this with or without a visual indication to the user.
And the throttling can be permanent or temporary.
I have chosen to implement temporary request discard with an indication to the user.
Discard is much easier to implement than delay (slowing them down), and I wanted a real human user to be able to continue access if they were just temporarily over-using the server.
So that's why the discard is temporary and there is an explicit indication to the user.

Currently, the module bwshare has the following features.

• Records the number of files requested by each client host according to IP address.
• Records the number of bytes downloaded by each client host according to IP address.
• Permits the operator to view the recorded data in an HTML table through a handler called bwshare_trace.
  The table shows the number of bytes downloaded by each host, the number of requests made, and various leaky bucket based measures of mean bandwidth.
  In particular, clients who exceed their bandwidth bounds will set off a red light on the management screen, and they get the 503 status code for every download after the first 30 requests or so.

  503a.html a typical 503 message for the too demanding client screenshot39.html screen shot 39, 2003-9-26 screenshot40.html screen shot 40, 2003-10-30 [running 8 months, no problems!] screenshot43.html screen shot 43, 2005-5-2 [bwshare version 0.1.4 with Apache 2.0.49] screenshot47.html screen shot 47, 2005-7-17 [bwshare version 0.1.4 with Apache 2.0.49] screenshot55.html screen shot 55: bwshare-trace, 2006-5-16 [bwshare version 0.1.9 with Apache 2.0.55] screenshot56.html screen shot 56: bwshare-info, 2006-5-16 [bwshare version 0.1.9 with Apache 2.0.55]

• Calculates the current file TX debt tx1debt of each client host according to the following rules.
  • If the client requests a file, then tx1debt is incremented by 1.
  • If the client is idle for T seconds, then tx1debt is reduced by T * tx1cred_rate, where tx1cred_rate is a specified permitted long term average file request rate for each host.
    (Example: tx1cred_rate = 0.05 files/second means that one file can be requested every 20 seconds.)
  • If the tx1debt falls to zero, it does not go negative. I.e. credit cannot be accumulated.
  • If the tx1debt exceeds a specified maximum debt tx1debt_max when a new request is made, the client is sent a 503 status code (HTTP_SERVICE_UNAVAILABLE).
    (A typical value of tx1debt_max for a 33.6k modem might be 30 files.)
  • This means that the limit for file download is tx1debt_max + T * tx1cred_rate files within any time interval of T seconds.
  • As an example, if tx1debt_max = 30 files and tx1cred_rate = 50/1000 files/second, this means that the client can
    • download 30 files as quickly as they like, and
    • download a further file every 20 seconds (because 1000/50 = 20).
• Calculates the current byte TX debt tx2debt of each client host according to the following rules.
  • If the client downloads a file, then tx2debt is incremented by the number of bytes in the file.
  • If the client is idle for T seconds, then tx2debt is reduced by T * tx2cred_rate, where tx2cred_rate is a specified permitted long term average file download rate for each host.
    (Example: tx2cred_rate = 1000 bytes/second means that 1000 bytes can be downloaded every second (obviously).)
  • If the tx2debt falls to zero, it does not go negative.
    I.e. credit cannot be accumulated.
  • If the tx2debt exceeds a specified maximum debt tx2debt_max when a new request is made, then the client is sent a 503 status code (HTTP_SERVICE_UNAVAILABLE).
    (A typical value of tx2debt_max for a 33.6k modem might be 300,000 bytes.)
  • This means that the limit for file download is tx2debt_max + T * tx2cred_rate bytes within any time interval of T seconds.
  • As an example, if tx2debt_max = 500,000 bytes and tx2cred_rate = 8000/8 bytes/second, this means that the client can
    • download 500,000 bytes as quickly as they like, and
    • download a further 1000 bytes every second.