Post Feed
Comment Feed

About Me

My MP3 Player

Moby - Play: The B Sides Dream Theater - Systematic Chaos I am Legend - (audio book) Yanni - Live at the Acropolis

New Comments

2017 Archives

July All Archives...

Spam and Disk Space

Sunday, April 22nd, 2012

On a weekend (of course) a while ago there was a tiny issue with my email server at work. I got a call from someone saying they hadn't gotten any email for a while on their smartphone and they couldn't login using webmail. When I got a second to check it, sure enough it wasn't working. Over the years I've learned that one of the first things to check when a server goes crazy is how much available disk space there is. In this case the operating system disk was 100% utilized. I log everything on that server and those logs add up, especially with a few iPhones and iPads connected to it, so I deleted about 7GB of log files. Everything seemed to be back to normal, email started flowing again and I could get into webmail without issue. I should have known that was too easy, but I went back to my weekend.

Sunday night I received a text message from someone else saying email isn't working. I didn't notice the message until I was going to bed so I decided to wait until the morning to look at it again. On Monday morning when I checked it out, another set of disks was full in the same server, this one for the transaction logs. These logs are supposed to be purged every day when an online backup is performed, so these disks should never fill up. Since transaction logs hold data that hasn't yet been committed to the database, you can't just delete them to free up space, if you do you'll lose data. The only safe way to remove the logs is to perform an online backup. The problem is when there is no more room for transaction logs the information store dismounts, going offline and making an online backup impossible. Luckily, I actually planned for just such an eventuality and have a multi-gigabyte dummy file sitting on every disk on this server. I just deleted the dummy file which left enough space to mount the information store and perform the backup to purge the logs. After an hour or so email was flowing again.

Then I noticed the disk space on the OS drive was getting low again! I cleared 7GB of logs two days before and it only had 1GB free. I continued looking around and found all sorts of spam in my queues waiting to be sent, all 'from' the same teacher's account. This teacher's password had obviously been compromised and someone was using his credentials to send out spam from my server. The nerve. I changed his password, cleared out the queues and everything returned to normal.

It didn't make sense at the time, but this spammer was trying to send so much mail that the connection logs filled up the OS disk. Then the volume of messages caused so many transactions that the transaction log disk filled up before the nightly backup could purge the logs.

This was super annoying because it didn't matter that my system was up to date and correctly configured to prevent relaying spam. The bad guy had a valid username and password which gave him access. Originally, allowing relaying based on authentication was enabled for because people had smartphones that required IMAP/SMTP access. Now almost all smartphones (Android, iPhone, Blackberry) can talk directly to Exchange server using ActiveSync, so I might be able to disable it. It was a great way to start the week.

Make a comment!

Numbers from below


Email Address

Website URL

Remember me