Difference between revisions of "IT/Benchfile"

From CA Greens wiki
Jump to: navigation, search
(Slow delivery agent for AOL)
m (Slow delivery agent for AOL)
Line 66: Line 66:
 
Roughly one in ten subscribers to our Mailman lists are at AOL.com addresses.  AOL blocks incoming email from any sender whose non-deliverable rate goes over about 1% on a given day.  That's close to the noise level due to addresses aging out.  AOL users are allowed to create multiple, anonymous "screen name" identities.  An unused, forgotten, or abandoned screen name contributes noise.  A few of them at any one time are enough to get AOL to block incoming email from our server.  We need to send to AOL addresses slowly, one by one, so we can identify bad screen names from the bounces and log entries, and remove them.  It's also helpful when political adversaries sign up for Mailman lists and then report the list traffic as spam.
 
Roughly one in ten subscribers to our Mailman lists are at AOL.com addresses.  AOL blocks incoming email from any sender whose non-deliverable rate goes over about 1% on a given day.  That's close to the noise level due to addresses aging out.  AOL users are allowed to create multiple, anonymous "screen name" identities.  An unused, forgotten, or abandoned screen name contributes noise.  A few of them at any one time are enough to get AOL to block incoming email from our server.  We need to send to AOL addresses slowly, one by one, so we can identify bad screen names from the bounces and log entries, and remove them.  It's also helpful when political adversaries sign up for Mailman lists and then report the list traffic as spam.
  
Postfix has a mechanism for transmitting some messages through a transport besides its own built-in smtp sender.  A transport named "slow" is described at the end of /etc/postfix/master.cf and mentioned ("<code>slow_destination_recipient_limit = 1</code>") in main.cf.  It's a shell script that queues messages for later delivery.  The list of addresses to be diverted is the hash:/etc/postfix/transport mapfile mentioned ("transport_maps =") in main.cf.  You can edit transport and type make.  The syntax is obvious.
+
Postfix has a mechanism for transmitting some messages through a transport besides its own built-in smtp sender.  A transport named "slow" is described at the end of /etc/postfix/master.cf and mentioned ("<code>slow_destination_recipient_limit = 1</code>") in main.cf.  It's a shell script that queues messages for later delivery.  The list of addresses to be diverted is the hash:/etc/postfix/transport mapfile mentioned ("<code>transport_maps =</code>") in main.cf.  You can edit transport and type make.  The syntax is obvious.
  
 
A job in Cameron's crontab examines this queue every few minutes, and relays a few messages out through Cameron's "rachel" server.  A post to a large Mailman list will be broadcast everywhere within a few seconds, except for the AOL destinations which will take half an hour to two hours to trickle out.  Messages the relay won't accept (high spam score) pile up in ~cls/stuck where they must be examined and discarded or forwarded manually.
 
A job in Cameron's crontab examines this queue every few minutes, and relays a few messages out through Cameron's "rachel" server.  A post to a large Mailman list will be broadcast everywhere within a few seconds, except for the AOL destinations which will take half an hour to two hours to trickle out.  Messages the relay won't accept (high spam score) pile up in ~cls/stuck where they must be examined and discarded or forwarded manually.

Revision as of 16:08, 24 December 2011

Benchfile

A "benchfile" is the book you leave in your desk drawer for your replacement in case you suddenly get a better job.

Provisioning

The Green Party of California rents a virtual private server (VPS) from Godmama's Forge. It's Ubuntu-9.0 ("Karmic Koala") based on Debian-5.0 ("Squeeze") plus some work-in-progress from Debian Unstable ("Sid"). The virtual host is named wangari.cagreens.org and runs under Vmware Enterprise. The init scripts are in /etc/init.d/.

Secrets and Authentication

There's no root password. Members of the "admin" user group can get a root shell by

sudo su -

or run other single commands under sudo. That's how Ubuntu does things. Passwords such as the MySQL root user's are shown in a text file "notes" in root's home directory.

Several users not involved with server administration maintain files in /w/cal appearing on the "old web site". They connect via SSH2. There is no FTP nor Telnet server. Almost everyone uses interactive keyboard authentication, plain old passwords. A few users have installed RSA or DSA public keys.

Domain Name Service

The cagreens.org domain is served by two DNS vendors with five public name servers. The master is amybiehl.greens.org running on Cameron's "bract" VPS (bract.truffula.us, bract.cagreens.org...) at Chunkhost. The four slaves are on Dyn.com ("The Dynamic DNS") where Cameron has a "VIP" account.

About fifty other domains depend on a name server cesarchavez.cagreens.org, which is running right now on Cameron's rachel dedicated server. These domains include addictedtowar.com, interfaithpeacecoalition.org, sjvgreens.org, migreens.org, etc. The Green Party of California lent its domain name to this effort in about 1998, to serve the larger Green movement and commemorate Cesar Chavez. I don't know whether there was a formal decision to do it or it seemed a "no brainer" at the time.

The cesarchavez.cagreens.org name server is not authoritative for the cagreens.org domain. That's because it's considered poor practice (in some circles, anyway) to put a name server in its own domain. If you don't do that, you don't need your registrar to handle a "glue record" for you.

LAMP Stack

apt-get install apache2-mpm-prefork php5-pear php5-gd php5-cli libapache2-mod-php5 mysql-server mysql-client

Look in the /etc/apache2/sites-enabled, /etc/php5, and /var/log/apache2 directories.

Backups

apt-get install rsync

Backup is a nightly rsync via ssh to a host in Cameron's garage. Authentication is by a key in ~root/.ssh/authorized_keys.

Local name service

apt-get install maradns

A lightweight caching name server, itself authoritative for no domains, listens on address 127.0.0.53. It queries customer name servers belonging to got.net and Godmama's Forge. It's mentioned first in our resolver configuration file /etc/resolv.conf which is obliquely mentioned (as "files") in /etc/nsswitch.conf. To see its configuration,

egrep -v '^#|^$' /etc/maradns/mararc

A local caching name server, also known as a "DNS forwarder", means you don't have to use the network for most name queries. For an email server that's looking up sender's PTR records, it's a real speed-up.

DNS block list

apt-get install rbldnsd

The blocklist file is /var/lib/rbldns/iparanges. It's maintained on another server, and pushed here by rsync via ssh. Configuration is in /etc/default/rbldnsd. This is one of two servers of the "dnsbl.cagreens.org" blocklist. At least four hosts refer to this DNSBL.

Email server: Postfix and Mailman

Postfix

Postfix is set up the "Debian Way". Postfix' configuration files are in /etc/postfix and the mapfiles they use are in /etc/postfix/maps. Tradition has the email aliases file in /etc/, so we have a symlink there pointing at the real one.

apt-get install postfix postfix-pcre mailman

There's a Makefile in /etc/postfix. Most of the time you can edit a file and type

make

and it will do whatever's needed. If you're not sure, run

make -n

first and decide. The Makefile also knows how to pull in updates to the map files used for spam reduction, currently from a public rsync server on Cameron's garage PC. We use maps in Postfix mapfile formats "hash", PCRE, and CIDR.

In a wide terminal window,

tail -f /var/log/mail.info

Mailman

Mailman runs as the "list" user. GNU Mailman installs into its data directories and runs there. Debian installs the software in /usr/lib/cgi-bin/mailman and /usr/lib/mailman/, with data files in /var/lib/mailman/. They've done a nice job of separating the files their package maintainer updates from those Mailman updates as it runs, so you can do

apt-get install mailman

when there's an update, without losing/breaking anything. The /var/lib/mailman run directory has a bunch of symlinks under it, pointing into the software location. There's a qrunner daemon with eight processes, started and stopped by /var/lib/mailman/bin/mailmanctl.

Mailman generates two files /var/lib/mailman/data/aliases and /var/lib/mailman/data/aliases.db which are mentioned in /etc/postfix/main.cf. That's where the posting and admin email addresses are defined.

There's a popular script for further integration between Postfix and Mailman. Debian distributes it with Mailman but we're not using it. It makes Postfix depend on Python. The main advantage would be pre-queue rejection of rejectable messages, reducing backscatter and administrative noise.

Slow delivery agent for AOL

Roughly one in ten subscribers to our Mailman lists are at AOL.com addresses. AOL blocks incoming email from any sender whose non-deliverable rate goes over about 1% on a given day. That's close to the noise level due to addresses aging out. AOL users are allowed to create multiple, anonymous "screen name" identities. An unused, forgotten, or abandoned screen name contributes noise. A few of them at any one time are enough to get AOL to block incoming email from our server. We need to send to AOL addresses slowly, one by one, so we can identify bad screen names from the bounces and log entries, and remove them. It's also helpful when political adversaries sign up for Mailman lists and then report the list traffic as spam.

Postfix has a mechanism for transmitting some messages through a transport besides its own built-in smtp sender. A transport named "slow" is described at the end of /etc/postfix/master.cf and mentioned ("slow_destination_recipient_limit = 1") in main.cf. It's a shell script that queues messages for later delivery. The list of addresses to be diverted is the hash:/etc/postfix/transport mapfile mentioned ("transport_maps =") in main.cf. You can edit transport and type make. The syntax is obvious.

A job in Cameron's crontab examines this queue every few minutes, and relays a few messages out through Cameron's "rachel" server. A post to a large Mailman list will be broadcast everywhere within a few seconds, except for the AOL destinations which will take half an hour to two hours to trickle out. Messages the relay won't accept (high spam score) pile up in ~cls/stuck where they must be examined and discarded or forwarded manually.

You are invited to write an improved version of our slow AOL transport that doesn't require an external relay and a cron job.

Drupal and carve-outs

Most of the www.cagreens.org site is on Drupal, installed by the mbrennan user in /w/d7 Sept. 2011. Margot Brennan is a principal of the Radical Designs web design shop. Many directories and individual files are "carved out" of the www.cagreens.org virtual web server. These "carve-outs" are listed as Alias directives in the file /etc/apache2/sites-available/d7.cagreens.org. They refer to files on the pre-Drupal web site, in /w/cal/. That site appears intact at http://files.cagreens.org.

CiviCRM

A demonstration instance of CiviCRM was installed in March 2011. This instance was abandoned because no first-tier technical support was available for its lone user.