Barracuda Luncheon

Today I went to a lunch seminar put on by Barracuda Networks and Layer 8 at Mario's, a very nice italian restaurant. I love these things; you get a free lunch, maybe learning something (assuming it just isn't a sales pitch) and you get out of work for a couple hours.

I have a Barracuda web filter at school ruining all the kid's fun, which is why I got the invite. They probably want me to buy the spam filter, the IM filter and who could live without their email archiver? Not only was it at a very nice local italian place, it was about 1 mile from work. When I got there the car that pulled in behind me contained Brad, an old friend from RIT, and a co-worker of his. This was awesome because I got to see him again and I had someone to talk to. Usually when I go to these things alone I don't talk to anyone (I'm shy).

The food was fantastic (as expected) and the presentation was very good. I felt bad for Tim, Barracuda's regional sales manager who was doing the presentation. He's a sales guy, and all these ornery nerds were asking questions and getting into debates (sometimes with each other) over stupid things like HIPAA compliance and something he might have misspoken. When he was talking about their email archiver he said something about being able to remove quota's from users' mailboxes and these guys jumped all over him for that. I knew what Tim meant even though he said something contradictory. I'd like to think these guys could just chill while they enjoy their free lunch, but no.

It wouldn't have been an IT seminar without some freebies and sure enough at the end of the presentation Tim was giving away hats and shirts to people who could answer questions correctly. Brad got a hat and I got the "email retentive" shirt!

First Week Dell Problems

The first week of school is always crazy. Teachers show up after being MIA the entire summer and suddenly find all sorts of problems that they need fixed that second. One and half weeks ago was the first day of school for Harley and was no exception.

There was the usual "help, my printer doesn't work" and "my password doesn't work!!!", but I got quite a few messages about computers not turning on. Usually this is just them being unplugged or powerstrip switches being off, but sure enough some weren't working. Lights would turn on when I hit the power button, but no drives spun or fans blew. This was actually a well known problem with certain Dell models that relates back to bad capacitors used on the mainboards. This was an easy diagnosis and fix; you just needed a new mainboard. The annoying part was that I had 7 of these systems (Optiplex SX280) all fail at the same time with the same problem and all about 1 month after the warranty ran out.

You can see the top of the left capacitor is bulging compared to the two on its right. Sometimes this is enough to cause the board to fail. Sometimes they'll run for a while with a bulging cap, but eventually they'll explode:

This one went one step past bulging and started leaking.

I called Dell to see if I could complain my way into some free mainboard replacements. After all, this is a manufacturing defect on their part and I personally feel they should have fixed the problem at their cost. My complaining didn't go as well as I had hoped. First I spoke with a lovely Indian in their out-of-warranty service department. She wasn't all that sympathetic and really didn't understand what I wanted. She said those boards are $280 each! So I pressed on and asked about getting a limited warranty extension. Again she didn't get it and quoted me $195 (each) for a 1 year warranty extension. Finally I just told her flat out, "I want these boards for free, it's your companies fault they failed." She told me I'd have to speak to general support for that. She transferred me there and I went through the same thing finally have to just spell it out for the guy. He put me on hold and talked to his supervisor who said no. I asked to talk to this supervisor who told me that Dell "took a stand" on this problem and decided to not cover any more claims after January 2008. I informed him that over years Harley has spent hundreds of thousands of dollars with Dell and I thought they should cover $1800 in parts; he disagreed.

In the end I had to buy the boards from Partstock who gave me a more reasonable price. After I replaced all the boards I ended up stripping one of the dead boards of all its capacitors and used those to replace the single bulging or leaking cap on the rest of the boards. Assuming that worked, I now have 6 spare mainboards for when more of them fail (which they will eventually).

I also had another problem with one of the same models: it would halt on boot complaining about not being able to detect a heat sink. I had no idea these things were smart enough to know if the heatsink was present or not. After opening it up and taking out all the parts (these are very small machines with a lot of parts crammed in) I saw this:

This is the one side that was fine. Apparently that anchor is not just structural, but allows for a continuity check across the heatsink.

This was the other end. Either the machine got smacked around or just had some bad soldering, the anchor ripped right out of the board. I soldered it back down which ended up being surprisingly difficult. You can't remove the z-clip in between the two anchors, it had to be soldered down with the tension of the clip on it.

Doing the Dance

It's that time of year again, where I have to order replacement workstations and servers for the school. Nothing is more annoying than playing the quote game with manufacturers. You email/call some guy from some company and say "Hey I need 40 computers with these specs, send me a quote", and get an email later (hopefully that same day) with some outrageous price. Then you repeat the process with a different manufacturer.

Once you have quotes from a few different places, you start setting them against each other. You take the lowest price and tell others that companyX is beating them and see if they can do any better. They'll usually come back with a slightly better price. Then you repeat the entire process over and over again until you can't squeezing any more deals out of them and you finally go with whoever gave you the best price.

One of the companies I'm looking at for servers this year is HP, and I'm kind of astounded how difficult it's been to get an accurate quote from them. I went on their website and chatted with some support rep, gave him the specs I want and he said someone would be in contact with me about it. Later that day (at 10pm) I get an email from some rep with a quote attached. The quote was for $4k and the specs were no where near what I asked for. She quoted me for 1 dual-core processor instead of 2 quad-core processors and 1GB of memory instead of the 24GB of memory I asked for, among other incorrect specs. I replied reiterating the needed specs and asked her to update the quote. When she finally got back to me, the new quote is for $20k!! Obviously something was messed up since that was more than twice what everyone else (Dell and IBM) gave me, but I couldn't tell since the quote isn't itemized. Finally a few days later I got an accurate (and competitive) quote from that HP rep, I can't wait to see how hard it'll be to get the price down.

Random IT Surveys

I hate phone surveys. Not a week goes by that I don't get at least one random phone call from some ESL call-center slave. These surveys take like 20 minutes and are made up of these ridiculous 15 part questions with things like having to rate on a scale from 1-9 how specific phrases apply to our infrastructure or how likely we are to purchase technology X in the next 3-6, 6-12 and 12-24 months.

Today I got 3 of these survey calls. During the last one I finally said that something came up and hung up on the girl because I couldn't stand it anymore. Sometimes it's so tough to understand what these people are talking about since they're speaking a mile-a-minute and with terrible accents; it was giving me a headache.

Sometimes they offer money to take these surveys, which is really the only reason I even consider wasting my time doing them. Sometimes they don't tell you until after if that is the case, but it's usually between $10 and $30. At least that's what they say, I can only honestly remember getting a couple of checks and a couple of Amazon Gift cards.

Failures at Work

This has been a bad week for computers at work. First the librarian brings me a laptop with all the keys in the wrong order. One of these spoiled rich kids felt it necessary to rip every key off the keyboard and put them back on in the wrong spots. Not a huge problem, just annoying.

Then on Tuesday I'm sitting in my office eating lunch watching a youtube video when the video suddenly stops playing. So I wait a little bit and let it buffer, but it doesn't start up again. Then I start to notice a very distinctive odor, that of electronics on fire. Ok, not really on fire, but when capacitors get too hot and explode, they have a very specific scent. So I start sniffing all the servers in my office and find the culprit: the one that routes our internet connections. I thought it was just the power supply, so I changed that out, but it still didn't work. Finally I had to find a spare similar piece of hardware, install Linux and set it up as the router again. All together it was about 2 hours of down time.

The next day I find more fallout from the dead server. Apparently years ago a policy was set that changed everyone's IE homepage to a page on that server. This was for displaying custom alerts and messages to faculty and staff. Since then almost everyone has changed their homepage to something else. The few who did not however got the "This page cannot be displayed" message when opening IE. So they naturally thought the internet was down. I told them to change their homepage and all would be fine. Then I disabled that policy since it really wasn't in use anymore. My expectation in disabling it was that all homepages would stay the same and it would simply stop forcing the local page for new computers. I was wrong. When it was disabled it reset EVERY domain computer's homepage to whatever the default was (MSN or Dell's site). Any time anything changes, users freak out; imagine opening IE and not seeing what you're used to seeing. So another email to everyone went out telling them to just change their homepage and all was well.

Then I noticed a blinking amber light on my Oracle database server where there is usually a solid blue light. I yanked off the bezel and saw one of the six disks announcing its impending death. Luckily this disk was part of a RAID1 array (mirrored, so all the data on this disk is on another disk) and this particular array only holds the operating system, the Oracle data is on a different array of disks. I didn't have any spare disks for this machine so I ordered a few; they should be arriving this afternoon.

I can't wait to see what breaks later today or tomorrow.

November

Not a whole lot has been going on for the past month, just the usual everyday stuff. Now that I think about it, the only things coming to mind are the 2007 NYSCATE conference and Thanksgiving.

Nyscate was fun; they had a like 9 timeslots for little talks and like 20 difference choices for each timeslot. They were more geared toward educators than technology people, but I found a few I liked. One of them was on implementing an enterprise class wireless network. It was rather refreshing to see a session on how and why you should do things a certain way without being told "you'll need to buy our product in order to do this." It just so happens that I'm planning a project to improve the wireless at Harley soon, so this was a good one. I also saw a bunch of sessions on specific products, which is good to know what's out there, but I like the generic ones better. Then when you weren't in a session there was a huge room with booths from over 100 vendors you could go chat with. Rachelle (who also went to the conference) liked going around with me to the vendors, because they seemed to give me better free stuff than they were giving out to the teachers. I got stuff like coffee mugs, travel mugs, a leather and nylon portfolio, a rocket-ship clock and all sorts of other stuff. I finally got to meet my CDWG rep, John, as he was one of the guys manning their booth.

The day after the conference we left for NJ for Thanksgiving. We got to hang out with my parents and a couple old friends. I finally got to see JMW; I think the last time we saw him was at his wedding a couple years go. Then we did Thanksgiving over my aunt and uncle's house. I ate wayyyyy too much.

Virtually Worthless

Over the past few years Harley has been transitioning to a new HVAC system. As areas in the building are renovated, they're converted to work with the new system.

Recently (this week) I realized that the old system is controlled by DOS software running on some ancient Windows 95 machine in the maintenance shop. After futzing around for a while trying to run this software on newer machines (2k or xp), it just won't work on anything other than the machine it's on. Of course the company that wrote the software won't even speak to us about it since it hasn't been supported in years.

The only option I can think of for replacing this decrepit computer is to host the Windows 95 operating system as a virtual machine inside XP. Brilliant right? Now the question is how to get the contents of the ancient machine into a virtual hard disk file (.vhd). Microsoft Virtual PC as the ability to link to a physical drive and convert it to a virtual hard drive. So yesterday I opened up this ancient desktop getting coated in dust that was probably older than I am and yanked out the hard drive. I set the drive up in the replacement machine as a secondary, fired up Virtual PC and started the conversion process. I tried this about 3 times each failing miserably before I gave up.

My next thought was to use Ghost to copy the drive. Ghost will copy the entire contents of the drive to a server on the network and then I can use it on the virtual machine to copy the contents down from the server to the virtual hard drive. This worked very well, but it was annoying to have to jump through all the extra hoops (finding DOS drivers for the old machine, building a CAB files with those drivers, and then ghosting both machines). MS has the functionality built into Virtual PC to convert from a physical drive to a virtual drive, but it didn’t work for me at all. In fact I searched around online and found more than a few people complaining that it just doesn't work.

SELECT brain FROM ass;

To quote a wise man (my dad, quoting another wise man (his dad)), "There's never enough time to do it right, but always enough time to do it over."

We have a new website at school where content is role-based. This means when you login you are presented with content based on what roles are applied to your account: student, parent, staff, alumni, trustee, etc. Upper school parents probably don't care about nursery newsletters and students probably don't care about board meeting minutes.

In order for all this to work, someone (me) has to export all of our constituents from one of our databases and get them into a form the website company can work with. This was a major pain in the ass. The database that had all the information I needed didn't have a very useful export feature. I basically had to export all the fields I needed and save it as an MS Access database, and then had to write a bunch of queries and modules to do various things to the data like combine fields to make usernames and unique temporary passwords. Then I had to export the output from the Access queries and save it as an Excel spreadsheet like the company needed.

This was all done months ago to input the data into the website database, but not until recently did they need to generate and mail letters to parents informing them of their usernames and passwords. I sent the mail-merge people the same spreadsheet I sent the website people, but heard that there were more than just current parents in that file; there were some former parents and some grand parents. So in a mad rush they had me re-generate this file for their mailing with only current parents in it. In my haste to get this done I fumbled one of the steps and used the wrong username formula. This means the letters were generated and mailed with the wrong usernames.

UGH.

On the bright side, of the almost 500 letters that went out, only 17 people so far have inquired about their nonworking accounts.

Recent WTFs

Rachelle got me a watch for our 1 year anniversary, but it needed to have a couple of links taken out of the band to fit me. No problem. Under normal circumstances I would have done it myself, but since we moved, I can't find anything, including tools I would use to take the pins out. We brought it back to the place she bought it from to have them do it. When we got to the counter the woman said that the only person who does that is on a lunch break (at 7:00pm) and would be back in an hour. So my first thought is wondering why they only have 1 person that can do this? It's not rocket surgery.

So while we were waiting for the lone watch-band-link-remover to come back from lunch, we walked around to a few other stores that size watches. They generally all want $14 and like 20 minutes to take a link out of a watch band! It probably takes maybe 5 minutes to do this and at $14, that equates to something along the lines of $168/hr! Absolutely ridiculous.

I wanted to take some pictures of the new kitchen faucet I installed last week, but the batteries in my camera were dead. Rachelle then informed me that she bought an 8-pack of "Super Heavy Duty" Sunbeam brand AA batteries for $1 somewhere. After laughing for a while and wondering how useless these batteries were probably going to be, I joking said, "If I can get 3 pictures out these batteries, I'll be impressed." I only got 1 picture! At least they'll work well in remote controls.

This week at work our CFO was complaining that she wasn't getting an email from M&T Bank when she requested her password. I checked in to it, and their email wasn't getting caught by a spam filter, it was getting rejected due to a sanity check. Basically email connections to Harley are checked to make sure they conform to the standards. Many spammers use shitty custom software that doesn't conform to SMTP standards, so rejecting weird looking connections is almost always safe. Except when it comes to M&T. I was kind of astounded that a national (maybe even global) bank can't get their shit together and just follow published guidelines and standards. It makes me wonder about the settings of their other IT services. I'd like to think things like their databases of bank account info are securely set up, but who knows...

Power Lines

So we had some pretty bad storms yesterday, and they knocked power out to Harley around 1:30pm. This is always fun, and by fun I mean an enormous pain the ass. First of all, the power at Harley is incredibly stable. Even during the 2003 Northeast blackout harley still had power. I remember leaving work not knowing anything was wrong until my car radio wasn't picking anything up and all the street lights were out during my drive home. Last week I checked the uptime on one of my student file servers and it was up around 530 days! That is respectable for any server, but especially one not connected to any kind of battery backup. So when my office went pitch black and deadly silent I figured some major shit just hit the fan. Since day camp is going on right now, all the little kids were screaming too, that made it a little more creepy.

The most treacherous part of the blackout ended up being the exit strategy from my office. I'm in the bowels of the building with no windows and I had my door closed, so it was completely pitch black. My office also happens to be a minefield of computers, monitors, wires, parts, tools, tables, an extra chair and even a hand-truck. I made it out alive, but I did trip on some stuff on the way there. After that, the rest of the 26ers just hung out upstairs in the gallery for an hour or so until we all decided to go home. I had a softball practice at the school later that night so I figured I could turn everything back on when I got back for practice, assuming the electricity would be back on by then. It was back on when I returned and it took about an hour to get everything back in business. Everything should have come back up on its own, but when there is a chance for something to go wrong, why wouldn't it?

I'm wondering why there isn't more of an effort to move the big power lines underground. Our old house had power lines on poles weaving in and out of trees before it reached the house. Every time we had a storm or heavy snow I expected one of those trees to let go and take out the power line, but it never happened. Our new house has all the utilities (electric, cable, phone) coming in under ground, so I won't have to worry about losing anything when the next ice storm hits.

Oh yeah, the cause the blackout...a tree fell and took out the line feeding our transformer. At least that's better than the transformer getting hit by lightning like 2 years go and being down for 3 days while RG&E finds another one.

Master Nerd

This morning I received an email from Experts Exchange saying I've earned a free t-shirt! Apparently I've earned "Master" status in the Exchange Server, Linux Networking and Miscellaneous Networking categories. I don't think I'll look as cool as this guy wearing it, but free stuff is cool.

Experts Exchange is a site where people can ask mostly IT related questions and have EE's vast community of nerds offer suggestions to solve your problems. Back in 2003 when I signed up it was free and they gave you like 500 free points to ask questions. When you post questions you assign a point value; obviously the more points assigned the more urgent the problem. If you post the accepted solution you get the points multiplied by a grade they assign your post. An 'A' having a multiplier of 4, 'B' having a 3, you get the picture. So you can rack up some serious points if you know what you're doing. Since then I believe you actually have to pay to sign up and you can't even read the questions/answers unless you're logged in.

Most of the time I just lurk the site reading about problems and solutions other people have with software I also use. It's good to know what common problems people have and how to fix them before they happen to you. It makes you look like a rock star to your boss.

Corporately Challenged

Yesterday was the JPMorgan Chase Corporate Challenge in Rochester over on RIT's campus. I'm still not sure what possessed me to sign up, but aside from that 3.5 mile running thing, it was pretty fun. Harley had a tent in the "bowl" and was cooking burgers and dogs after the race.

Check out Al Simone's face as they blew the starting horns! HAHAHA, Priceless...

I can't believe how many people were at this event: 10,397. We (Rachelle and I (she signed up for her School's team)) were so far back at the start that it took us about 6 minutes just to pass the starting line! And even though we walked just about the entire thing there were still thousands of people behind us.

We parked over at Marketplace Mall and took a shuttle over to RIT. It was great getting there; we just walked up and got right on the bus. On the way back it was crazy. We waited in line for about an hour, not because it wasn't moving but because of the length of the line. There was this woman in front of us who just had to complain about everything; I hate those people. Maybe she was drunk, but she didn't look it, just sounded like it. Then after waiting all that time two buses showed up at the same time and people in the line behind us just broke out and ran to the other bus! We were all like WTF, great way to start a riot! I was fully prepared to throw some shit down if one of those line jumpers took our spots on a bus, luckily we found spots (seats even!) on the bus and got back to the car without incident.

My 1337 Email Server

Every day most of my servers at work will email statistics and log excerpts to me so I can make sure they're healthy and everything is running well. I thought this one was amusing since the number of email messages processed by this particular server yesterday was exactly 1337. If you aren't familiar with this number (and I'm talking about the meaning other than just one thousand three hundred thirty seven) check out the wikipedia entry for leet.

You could also have some fun with this leet-speak generator.

Educational Software

Educational software is the worst! I'm not talking about its content, the little kids at Harley learn a lot from it, I'm talking about its development. It seems like all these companies wrote their software back in the late 80s and early 90s when computers were showing up in classrooms and never got around to updating it. By update I mean design it to work well on a contemporary operating system; something a littler more recent than Windows 3.1. It seems to me that these companies just apply some dirty hack to get the program to function (if it doesn't already) on newer operating systems like XP.

My favorite behavior by some of the ed software Harley has is to have extremely outdated multimedia requirements, like Quicktime version 2. The current (as of this writing) version of QT is 7. So if I'm trying to install some POS educational software it will ignore the fact that QT 7 is installed on the computer and insist that it installs the bundled QT 2 on the CD. Some of them you can trick, some you can't.

Then there is the way they store data. Many of these software titles have the ability to track multiple users (think class) and track their progress, etc. That was great for the classroom with a single computer that everyone used individually. What about when 15 kids go into a computer lab and all use the software at the same time? And then the next day they each sit down at different computers and try to pick up where they left? That's what a network is for, but this software is so old it won't let you use the "\\servername\datafolder\kidsfiles" notation of specifying a remote location to work from. Even some of the "Network Versions" still require you to map a drive letter to work from. This is better than nothing, but completely useless when you have a bunch of titles that each requires their own mapped drive, it becomes a maintenance nightmare.

This week I found something else out about some of our educational software. One title in particular which worked decently before started turning the screen upside down! Apparently it messes with the video drivers or something that causes the screen to rotate 180°. Sometimes it's when you start the program and sometimes it's when you quit the program. Since it started doing acting stupid this week, I can only assume it was a Windows Update that screwed everything up. Go figure.

Daylight Sucking Time

Daylight savings this year has been a major pain in my ass. Not just because the dates changed from April 1 to March 11 but because Microsoft really didn't handle the change very well. I've known about the change for a pretty long time and I assumed that Microsoft would release patches via windows update way in advance since they did know about it back in 2005. Not only did they not release updates way in advance, they only released them for the current versions of windows (XP and Server 2003). What about windows 2000 you ask? Well you can purchase an update from MS for $4000 (!!) or follow an MS knowledge base article that tells you how to create a couple scripts to patch your system. So if they released an article that tells you exactly how to fix it (which didn't work exactly), why couldn't they just create an update for it? Because they didn't consider this a "critical update."

The workstation updates weren't that bad, but some of the server updates were kind of a pain. We use Exchange server at work for our groupware, and it was a bit more complicated to patch. There was an update for the Server OS, there was an update for Exchange itself, and there was a tool to fix everyone's calendar items that fall in the change period (between the old and new DST times). Then we also have Blackberries to add to the mix. So each Blackberry device needed an update, the Blackberry Enterprise Server (which I'm running on 2000 Server) needed as OS update, a patch for the send-as permissions, the exchange update and probably something else I'm forgetting about. Confused yet? I didn't mention that all this stuff has to happen in a certain order and that order is different depending on who you're asking. MS had one order and Research In Motion (the Blackberry company) had a different order and some steps had prerequisites such as having to run from some third computer (a workstation) with only Office 2003 or 2007 installed (we use Office 2002 of course).

I read as much as I could about this and waited as long as I could to see what problems people had, but since the time changed this morning I guess I've either done it or not. We'll see how many people complain Monday morning if anything is screwed up.

Ink Cartridge Hell

Small yield ink-slingers are the bane of my existence. By small yield I'm talking about the home and small office inkjet printers. At work they are primarily used in the lower school. Each (LS) classroom will typically have one printer shared among the 3 or 4 workstations while the other divisions better utilize the bigger centralized printers like the networked lasers and copiers. Not a week goes by where I don't get a few calls about these stupid printers not working, jamming, dried up ink, etc.

Typically speaking, the smaller capacity the printer, the more expensive it is to run. We go through a decent number of ink cartridges so this year I thought I'd try to save a little money by using recycled cartridges. There is a pretty big effort to make Harley "greener" so I figured I do my part. I also decided to go with a local company (which shall remain nameless) to support local business and to get things moving a little faster by cutting the shipping cost/time out the equation. That decision is starting to turn around and bite me on the ass.

These recycled cartridges are less than stellar in performance. Don't get me wrong, when they work they work great but a significant percentage of them don't work. No big deal; I can just call up the local company I bought them from and they'll drive out replacements for any duds. Recently I've gotten a lot of duds. Between Friday and today (Monday was a holiday) I probably went though 8 cartridges between 3 printers and I still haven't gotten 1 of them working at an acceptable level. With new ink I can just hand the person a cartridge when they ask for one and they'll put it in themselves and it'll just work. With these refills I can't do that because chances are the first or even second cartridge won't work. It's infuriating; it takes like 1000 times longer to perform the simplest of jobs.

I've already decided I'm never going to use recycled cartridges after this year, but I'm still debating on trying to return what cartridges we've already purchased.

Fight Spam with a Fake MX

I just can't seem to get ahead of the spammers. I'm constantly looking for new techniques and little tricks to catch up to them. I usually do for a little while, but as time goes on they keep pulling ahead.

For those of you who don't know how email actually works, here's the quick and dirty version. Say you want to email user@example.com, you compose you message and hit "send" in your email client. Your client will connect to your email server (most likely your ISP's server) and send the message there. Once that server has the message it looks up the MX record for example.com. An MX record simply tells you what server(s) handle mail for the domain. You can (and really should) have more than 1 MX record with different weights. The server will then connect to the server with the lowest weight and pass the message along. From there it might get passed along (relayed) a few more times internally before it lands in your mailbox, but that's just about it. If the lowest weight MX doesn't answer (is down or busy) the sending server should then try an MX with the next higher weight and so on until the message is delivered or returned.

This whole idea of using the lowest weighted MX first and then moving up is set by the RFC (standards) for the Simple Mail Transport Protocol. Spammers don't use standards; in fact they usually do the complete opposite.

Yesterday I was reading about a trick that I haven't yet heard of, the idea of a fake MX. Basically you just add an additional MX record for your domain with a really high weight. All legit email servers should use the lower weight servers to deliver mail, and never touch this fake MX. Spammers on the other hand will sometimes use the highest weighted MX record first on purpose because they are usually just backup mail servers with less spam filtering. Then you simply set the mail server on the fake MX to tempfail all incoming mail. This basically means the server will reply with a "I'm busy, try again later" message and reject the email. Again according to the standards, legit email server will try again later and continue to for a number of days before returning mail as undeliverable. Spammers don't retry, at least most don't. They are all about sending as much spam as fast as possible. If they get tempfailed they usually just move on and don't come back.

I thought that was a pretty good idea. I haven't set this up yet, I'm trying to figure out a way of testing it to get some numbers without actually rejecting email, at least until I'm sure it's actually working.

I'm a Little Cyan Today

The color laser printer in the Art lab has been proclaiming its lack of blue toner for a while now. It usually just flashes a warning that says "Order C Toner" when you're running low but not out, so you have a chance to get more before. An old trick to get more out of your toner cartridges is to take them out, shake them up and put them back in the printer. I guess one of the art teachers tried doing this, but in the process of taking the cartridge and drum out she flipped the lever to release the cartridge from the drum. Yeah that equals toner all over the place. She failed to mention this to me until later. So since all the remaining toner spilled on the floor, the cartridge was completely empty when I checked the printer this morning. I put a new cartridge on, and noticed a little loose toner on the outside of the drum. Thinking it was only a little bit, I tried blowing it off over the garbage can. It was significantly more than a little (because of the spillage) and my little blow produced a cyan colored cloud of death. I looked like the newest member of the blue man group when the dust settled. I'm exaggerating a little, but it was a good thing I had on jeans and a blue shirt today. Later in the day I blew my nose and found some cyan boogs in the tissue.

A Blackberry Turned Me Green

Over the summer when Harley's new head came on board he got a Nextel Blackberry. It's a neat little device. I had to setup a Blackberry Enterprise Server (BES) on the network so it could do mail/calendar/contacts/everything else over the air instead of having to sync it manually with your computer. After I played with it for a while I decided I really wanted one. After I got all working I handed it to him, haven't seen it since and learned to live without one.

Now our CFO just got one today. It's sitting on my desk while it "activates" with Nextel and my BES. Now that I'm playing with her's, I totally want one again!

Database Shitfest

The title says it all; it's the best way to describe what is going on at school lately. We have an enterprise-esque database system at the center of our school. This DB has its fingers in everything: scheduling, transcripts, grades, attendance, admissions, general ledger and all sorts of other stuff. It was (and still is) functioning fine, but is slightly out of date. We stopped receiving updates more than a year ago and nobody noticed. Then this summer we discover we are at version 803 when the current version is 816.

Normally this wouldn't be a big deal, but I'm having a hell of a time trying to get up to date. Not only are we trying to update the actual data and the programs accessing it, the server hardware itself will be replaced once the update is successful. One of the problems is the fact that this is a production database and is in use about 101% of the working day. Sure I can kick people out of it for a while, but a few people's jobs revolve around accessing this database so I can't keep them out for any length of time.

What I've been doing is kicking people out, copying the data to the new server and then running the updates on the new hardware. This way I can put the old server back in action if there are problems upgrading. The first time the update failed about 10 minutes after I started. It started barking about some missing column in some table. I called the support line and they had me send them a copy of the database so they can figure out the problem.

A couple days later they get back to me saying they fixed the problem. Today one of the support people successfully updated the old data I copied over the first time. So I try running the update on a new copy of the current database. Guess what happens? First I ran out of disk space on the drive that the data was held on. There were so many changes to the data to bring it up 13 versions it created thousands of log files (to keep track of transactions), each being around 1mb, and filled the disk up. I noticed this when I went back to work around 5:30pm today to check on the update progress. I made some more room and tried the update again (3rd time). Yup, failed again. At least this time it was in a different place and I was able to get the error messages. I forwarded them to the support people along with another copy of the database.

This update has been a real pain in the butt. I have to work around about 6 people's schedules, wait for the support people to figure out the problems and find time to work with them. And this is all on top of the 10000 other things that I have come up since the middle of August and need to be done by the start of school (8 days).

IT Security Strategies

I was reading this is one of the many trade-mags I get at work. Most people probably won't find this funny, but anyone in the IT business should. This is from page 82 of the July 20, 2006 issue of Network Computing:

My favorites are #7 and #3.

Screwed by 3GB

At work I have a server setup to store all the backup data from the other servers. Backing up to disk over the network is much faster than backing up all those servers to tape and is a lot cheaper. I have a RAID (Redundant Array of Independent Disks) system setup in this server to prevent any data loss if any of the disks should go bad. There are a few different flavors of RAID; the one I'm using is RAID 1, or mirroring. This involves two "identical" disks having the same data. When data is written to the array it gets put on both disks at the same time. In the event that one of the disks should fail, the other one will continue to function and no data will be lost. Obviously if both disks fail you're screwed.

When I was planning this server I ordered two 200GB disks for the backups. These disks were identical; they were the same manufacturer, make, model and size. So when I created the array I used the maximum amount of space on the disks for the partitions. It worked great for a year or so. Last week one of those disks failed. It was still under warranty, but I bought a new one instead because if I tried to get a replacement they would want the old disk back and it still has all sorts of private data on it that I can't remove.

I ordered a new 200GB disk last week (the day the old one failed) and it arrived yesterday morning. I transplanted the drives and run the "raidhotadd" command expecting to have the system recognize the new disk and add it the array. It didn't. After some hair pulling I found out that the new disk is actually smaller than the original. Both times I asked for 200GB disks and both times I received 200GB disks (according to the box and the labels on the disks), but the original pair were actually 203GB. You can't mirror two disks when one of them is 3GB larger than the other.

Luckily I was had enough room on another drive in that server copy all the data so I could recreate the partitions on the one old drive and the one new drive to around 185GB so they would both match and could mirror each other. I purposely left some room in case this happens again and I get a disk that is 199GB.

Stupid Google

I was looking through some of my server logs today at work and found a bunch of entries from one of Google's web crawlers. The weird part was what pages the bot was requesting.

66.249.65.51 [19/Jun/2006:00:55:33] "GET /calendar/?m=12&y=2008 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:34] "GET /calendar/?m=166&y=2006 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:34] "GET /calendar/?m=88&y=2006 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:35] "GET /calendar/?m=7&y=2004 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:35] "GET /calendar/?m=12&y=2003 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:36] "GET /calendar/?m=12&y=2008 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:37] "GET /calendar/?m=74&y=2006 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:37] "GET /calendar/?m=7&y=1992 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:38] "GET /calendar/?m=1&y=2020 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:38] "GET /calendar/?m=175&y=2006 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:39] "GET /calendar/?m=9&y=2011 HTTP/1.1"
66.249.65.51 [19/Jun/2006:00:55:40] "GET /calendar/?m=166&y=2006 HTTP/1.1"

Each of these requests is for my calendar page with different arguments. If you request the page with no arguments you'll be shown the calendar of the current month and year. The "m" argument is for the month and "y" is for the year you wish to view. Check out some of the months and years Google's crawler was requesting: 9/2011, 12/2008, 1/2020 and more interestingly 175/2006, 166/2006, 88/2006 and 74/2006. I've never tested my calendar application with dates such as those, but it actually handles them rather gracefully. Using 13 for the month is handled as January of the next year. So 1/2007 and 13/2006 are the same date to the program. Either way, WTF is the bot doing making requests like that? The bot should only be indexing pages that are linked from other pages; I don't think there is a link anywhere to Harley's calendar for October 2019.

Play Ball!

Recently at work Mike emailed a bunch of guys asking if anyone wanted to start a softball team. I thought it sounded fun so I agreed to be on the team. I was pretty nervous because I hadn't swung a bat or thrown a softball since high school gym class, and I vaguely remember sucking at both anyway. Then I heard that many of the other players were in the same position and that we are playing in a beginner's league, so it wasn't that bad.

We had a practice this past Friday to sort of scrape the rust off which was a great idea; I was really freaking about the first game on Sunday. Schifty gave me some tips when it was my turn to practice batting and they really helped. I didn't really warm up or stretch before the practice and ended up pulling just about every muscle in my body.

I got a call Sunday morning from Mike saying that the plan changed and that because of how the teams worked out we had to play two games back to back that night. Wonderful. I assumed that the games would be easier than the practice because there would be a lot of down time. This was true, there was down time but the non-down times were way more strenuous. The first game was against some team sponsored by Jeremiah's Tavern. They all had matching uniforms and looked like they meant business. They did. We got our asses pretty much handed to us. They have a rule in the league where if at any time a team is winning by 11 runs the game is over, even if it is only in the first inning. Yeah, the game ended pretty early. I had fun and I know everyone else did so it was ok. The second game was worse. This team, also sponsored by Jeremiah's, was not a bunch of beginners and had no business playing us. They were really fast, and could all whack the ball pretty far. One of them hit a foul ball that actually hit a girl on the foot who was standing by the next field down. They weren't very nice either. Oh well. By the end of the game the Ibuprofen I took was wearing off and my legs were killing. That game officially ended early too, but they decided to keep playing.

I'm glad I joined the team; I'm not very athletic, but it's fun and might get me into a little better shape.

Spammed by a University

While looking through my server logs this morning at work I noticed a bunch of email from a server at Loyola University. I looked a little closer and the email was coming from the a bunch of bogus addresses (random@gmail.com, random@hotmail.com and random@comcast.net) and all aimed at 1 user at Harley. Once I saw the from addresses I knew it was spam so I emailed Loyola letting them know they have a problem, including some entries from my logs as proof.

While I was waiting for a response I stopped up and talked to the teacher getting all of this spam and she told me she was a Loyola alumnus. I looked at some of the actual messages and found they were addressed to one of their alumni mailing lists. It looks like the university was running a mailing list that was completely open to the public. This means Joe Spammer can send 1 email to this list and Loyola will forward it to all their alumni, making the spammers job that much easier! Bah!

A little while later I get an email back from someone at the school saying that the person spamming was on the IP address 24.97.xx.xx. They said they forwarded my complaint to Road Runner's abuse department because that IP address is registered to RR and told me that they added the address to their blacklist. The problem was that the IP address above is MY mail server's address! This guy obviously had no clue what was going on and I couldn't email him back to tell him this because I was blacklisted. So I dug up a phone number online and gave this guy a call. By the time I got to talk to him he had figured out that he made a mistake and that someone on Bellsouth's network was doing all of this.

I added their server to Harley's blacklist until they stuck cork in it, which took an additional hour and a half after my phone call.

I assumed a University to be more on the ball about this stuff, but maybe that is too much to expect.

Worst Week Ever

It really hasn't been the worst week ever, it just felt like it. I totally blame this on that little hour we lost last weekend. I've felt like a zombie ever since sunday.

Saturday was Harley's big fund raiser, BLAST, and was also the drawing for the hybrid raffle. I didn't win. Not winning didn't really bother me; what did was the fact that one of the richest men in Rochester won. And he only bought 2 tickets!

Monday was kind of a shitfest because all sorts of stuff broke at work. One of my new domain controllers decided it needed to reboot randomly every few minutes. When I pulled it off the shelf and put it on my bench it wouldn't boot at all. An hour and a half on the horn with Dell later they end up sending someone out to replace like everything but the hard drives. I was impressed to see the parts for the server arrived via UPS about 1.5 hours after the call to Dell. It took longer for the guy to change the parts than for them to arrive in the mail.

The rest of the week was pretty slow, but I feel like I've been in a daze the whole time.

Sticking it to the State

Recently Jason, who works for the state of NY, and I have been trying to get him on Instant Messenger at work. They seem to have cut the network off completely to the internet where he works. Since only internet activity allowed is browsing web pages, the ONLY way out of his network is through their proxy server that will only play nice on port 80 (for web sites) and 443 (for secure web sites).

After a couple weeks of trying different things we finally found a combination that worked. The first barrier was getting traffic in and out of his lan. For that we used an SSH tunnel. This would allow his office computer to make an encrypted "tunnel" to his home laptop and forward the IM traffic there. The default port for ssh however is 22 which was blocked of course. The sneaky part was setting up the ssh server on his laptop at home to listen on port 443 (remember that port is allowed because it is used for secure web sites). Since SSH uses the same technology as secure web site traffic, SSL, the proxy server at work won't notice that it isn't web traffic and will let it through.

The second barrier was how to relay the IM traffic from his laptop at home to the internet. For that we used a SOCKS proxy server. I found a free one called Antinat that would work as a windows service.

All he needs to do to get on IM now is to open Putty (a free windows SSH client) and connect to his laptop at home. Then when he sets his IM client to use the proxy on his laptop (throught the tunnel) it works like a charm! This can work for ANY application...you just need to tunnel the right ports. He could even set his web browser to use that proxy to get to websites his corporate proxy will block.

The Disease-Bag is back!!

OMG I'm SIIIIIIIICCKKKK!!!

For those of you who don't know what a disease-bag is...it's a plastic Wegmans bag used to carry around a box of tissues and used snot rags. I believe the term was originally coined by Avino back when we lived at RIT.

Today I emailed in sick because I'm on my death bed. So guess what happens at work today? Harley's internet connection goes down. Since our email, website and everything else is hosted onsite it really couldn't wait until tomorrow to be fixed. So I had to get dressed, drag my sorry ass across the soccer field and reset the cable modem.

So later this evening I notice that Harley's connection is still flaky. I called Time Warner to see if there is a problem with their network in that area but of course the problem is on my end. They're stopping by tomorrow between 8am-12pm to do some tests and maybe replace some hardware. This might suck because if I'm feeling the same or worse I wasn't planning on going to work. Figures.

Laptop Surgery

So a co-worker of mine tells me the other day that her laptop was dropped (again) and the power jack was damaged. Basically the laptop fell on the right-angle power connecter breaking off the pin in the middle of the jack inside the laptop. Ouch. Since it was already off warrantee and she used this as an excuse to buy a new laptop I asked her to bring it in so I could try fixing it.

She brought it in this morning and let me check it out. After taking out all 35 screws and removing basically every internal part besides the processor I was able to get to the back of the power connector.

(Click to enlarge any of the pictures)

You can see in the first picture how the pin broke off from the part soldered to the motherboard (and destroyed the plastic that held it in the right position. In this pic I have the power plug inserted and the pin in the connector to hold it in place.

Here is the connector with the pin soldered back to the post. I was using a really shitty overpowered soldering iron for the job but it was all I had. I added a lot of solder to hopefully add more strength to hold up under the pressure of removing and inserting the power cord. It actually held up very well and worked fine without any strain-relief, but I did end up squirting some RTV in the gap between the plastic and the pin.

Here is the outside where the power cord plugs in. You can see how when it was dropped it cracked the outside of the case too. This is an after picture showing the pin back in its proper place.

After taking twice as long to put back together and having only 1 screw left over, the final test was turning it on (yes it still worked) and see the power cord charge the battery.

Finally!

I finally figured out what is killing one of my servers at work. Every once in a while the server that runs the website would go apeshit and its load average would jump to around 30 (12 is considered critical (right now it's idling around 0.02)). It would do this for about 20 minutes and kill the MySQL and Apache services. Well it looks like the problem is related to SquirrelMail (a free webmail client) accessing very large mailboxes on an Exchange 2003 server. My mailbox has around 5000 items in it and when I tried accessing it with SM last night the server went nuts. I originally thought the problem was with MySQL (since it was one of the programs crashing and burning, but it looks like that was just collateral damage. It might be a problem with Apache's setup or just a bug in SM's software. Either way I've found the problem and removed the software. Go me!

Work Calls at Home

Today everyone left to go back home. Since they had a 6 hour drive and Kelly had to catch a ferry back to Long Island they had to leave pretty early. 10am is early for me on a weekend. So just after they leave while I'm eating my waffles the house phone rings. I look at the tv (caller ID on TV) and see the last name of someone I work with. I didn't answer it...and they didn't leave a message. This kind of annoys me because it was 10am on a holiday and I'm sure it was some stupid question that really could have waited 1 day until we are back at work. I would have been really pissed off if that call woke me up.

I used to have my cell phone number on the phone list at work. Then everyone started calling for every stupid little problem on my cell phone before they would email or call my office. I actually got a call last summer on the beach in North Carolina about some printer problem even though my office voicemail said I would be on vacation and not to call me.

The day after I got back from North Carolina I asked to have my cell number removed from the list. It's a shame I had to do that because I would like people to call me if there is an emergency (fire, explosion or theft) but not for things like printers running out of paper. Sometimes I think people just need to take a few seconds and a few deep breaths and think about what they are doing. It would make everyone's life easier.

Lunch at an Art Museum

On Friday a bunch of coworkers and I were invited by my boss and the head of the Advancement Office for a catered lunch at this odd art museum in town called Artisan Works. Very strange place. It is like a warehouse that is stuffed to the gills with all sorts of art. The guy that owns the place actually owns all of the art and like 99% of the stuff there is for sale. There are a few "areas" where local artists can keep their stuff and create more.

Lunch was great. They had Chicken French, Veggie Lasagne, Caesar Salad, some kind of rice and this really good caramel cheesecake for dessert. I'm not a big wine (or any booze) drinker but they also had a lot of wine for everyone.

After lunch one guy that worked there took us on a tour of the place. There was all sorts of neat stuff there: sculptures, paintings, cars, motorcycles, stuff that just looked like junk to me (but I'm sure can be considered "art"), pictures of naked women everywhere and I even saw a near-life-sized statue of an Ent. The coolest thing I found was a Harley Davidson constructed entirely out of wood!

Definitely check this place out if you get the chance!