Tom Moertel's Weblog: Category hardware

Repairing my Kenwood A/V receiver's remote-control sensor

Tom Moertel — Fri, 25 May 2007 17:42:00 -0400

The Kenwood audio-video receiver that forms the core of my home theater system stopped responding to its remote control. As I discovered shortly thereafter, having to leave the couch to fiddle with knobs degrades the “home theater experience.” Clearly, something had to be done.

I knew the receiver was the culprit because the remote control worked fine with other components of my system. I figured the IR sensor had gone bad and did a little Googling for “Kenwood” and “IR sensor” and “problem”. The results revealed that many other Kenwood customers had the same problem.

The cause of the problem, I learned, was that the solder joints which connect the IR sensor’s leads to the display board eventually fail because of thermal expansion. That explanation seemed to account for what I was observing, so I cracked the case in search of visual confirmation.

First, I found the joints where the IR sensor was connected to the circuit board. The vertical red line shows where I found them:

Then I examined the joints closely. Sure enough, at least one had completely failed:

The problem confirmed, I moved to the solution phase of the project. With my soldering iron, I touched-up the joints:

It wasn’t my best work, but it did the job.

Now my receiver is back on speaking terms with its remote control, and I have returned to the modern world. Life is good

A bad way to start the evening

Tom Moertel — Mon, 29 Jan 2007 22:08:00 -0500

I just checked my inbox and noticed the following urgent message from the SMART daemon on my laptop:

The following warning/error was logged by the smartd daemon:

Device: /dev/hda, FAILED SMART self-check. BACK UP DATA NOW!

For details see host's SYSLOG (default: /var/log/messages).

Crap.

Luckily, I recently ordered a new laptop, and it should be arriving tomorrow. Good timing.

Once it arrives, I should be able to transfer my dying laptop’s personality to the new laptop from the back-up copy on my development RAID system. Still, getting things set up just right will probably eat half a day.

How to make sure your servers come back up after an extended power outage

Tom Moertel — Wed, 09 Aug 2006 00:35:00 -0400

If an extended power outage drains your UPS, and your servers are forced to shut down, will they automatically start up again when the power is eventually restored? It’s a good question, especially if your servers are in some distant, unattended server room. Unless you’ve tested your servers, don’t assume that the answer is Yes.

Many servers offer a BIOS configuration option that forces them to automatically power on when they receive line voltage. If your servers have this option, just set it and you’re done.

Unfortunately, some servers, including a Dell PowerEdge 1600SC that I’m using, lack this configuration option. When these servers turn themselves off as the final step of a UPS-controlled shutdown, they don’t start up again when the power is restored. Because they were shut down before the power was cut off, they think they are supposed to remain off when the power is restored. That is, they remember their on/off status across power outages.

Fortunately, there is a way to make sure these servers automatically power on: shut them down without powering them off; halt them instead. That way, when the UPS finally cuts off the supply voltage, the servers will still be in their “on” state, and they will remember this state across the outage. Later, when the power is restored, the servers will automatically restore their pre-outage state and power up.

With Fedora Core Linux and Network UPS Tools, it’s not difficult to make sure the servers are halted instead of powered off, but the implementation isn’t obvious. To spare you the digging, here are the important bits.

When the power fails and the UPS-monitoring software decides that the batteries are almost depleted, it will initiate a server shutdown using the command defined in the /etc/ups/upsmon.conf file. The default command is this:
```
SHUTDOWNCMD "/sbin/shutdown -h +0" 
```
The shutdown command will tell the init process to enter runlevel 0, which is the prepare-to-halt-the-system runlevel.
The init process will stop all of the running services in an orderly fashion, and then, as the last step, invoke the final script in the shutdown process: /etc/rc.d/rc0.d/S01halt.
The final lines of the S01halt script will power off the server. Unless, that is, the file /halt is present, in which case the script will halt the server instead.

Thus the trick is to make sure that the /halt file does exist. The trick turns out to be easy to pull off; just redefine the shutdown command in /etc/ups/upsmon.conf:

SHUTDOWNCMD "/bin/touch /halt; /sbin/shutdown -h +0"

And that’s all there is to it!

Dell-support follow-up survey

Tom Moertel — Tue, 27 Jun 2006 17:41:00 -0400

Recently I wrote about a bad support experience I had with Dell. Today in my inbox was an invitation to participate in a survey about my experience.

I accepted.

My survey responses, which were completely honest, were not glowing. Although Dell ultimately resolved my problem (the cabling shipment arrived the following day), the hoops I was forced to jump through were unacceptable. Dell’s support seemed fundamentally broken, and I had to fight to make it work.

When asked what Dell would need to do before I would feel comfortable recommending them to others, I wrote:

I would need confidence that Dell makes it easy for clued-in technical customers to speak with clued-in support personnel. As things stand, clued-in customers waste too much time on the phone with ineffective support personnel. In my case, I was handed off numerous times and ended up speaking with seven support persons, and only the final person had the knowledge and empowerment to make the situation right for this customer.

I also gave them a link to my article about the experience. It will be interesting to see if anybody reads it.

In any case, I am glad I received the survey invitation. At least it shows that Dell is trying to improve. Further, the survey asked the right questions: I was able to adequately express my dissatisfaction and point out where I thought their process had broken down.

I do hope somebody at Dell figures it out because support is the company’s Achilles heel. HP, in my experience, smokes Dell in this regard.

Update: It looks like my article got the attention of CMP Media’s CRN, a source of “vital information for VARs and technology integrators.” Edward F. Moltzen linked to my article in his article of 6 July 2006: Dell Works, Spends To Get Back Into Good Graces.

Perhaps not entirely coincidentally, on the very same day, a “customer advocate” with Dell’s headquarters in Round Rock, Texas, sent me an email offering help and asking for feedback. Owing to a mix-up, I did not receive his email until he re-sent it on 11 July, but I have since responded with a detailed summary of my experience.

Things are getting interesting.

Update 2006-07-21: It seems the Dell customer advocate was serious about fixing problems. He reviewed my case and was able to identify a user-interface problem with Dell’s web site that probably led to a good part of my difficulties. The problem is that a non-support phone number is offered in a portion of the support section of Dell’s web site. Customers, like me, who call the phone number are connected to people trained to handle pre-invoice issues, not support. Oops. As of this writing, the UI problem still exists, but I trust that it will be solved soon.

New server arrives: my adventure in Dell customer-support Hell

Tom Moertel — Fri, 23 Jun 2006 14:56:00 -0400

Yesterday I received a new server from Dell to replace the server I decommissioned earlier this week. My initial impression was favorable: the build quality looked good, the power supply seemed up to the task, and the heat sinks and blowers seemed almost absurdly beefy.

And then I noticed the drive bays.

Dell, doin’ it’s own thing, server style

You need to understand that Dell engineers like to do their own thing on occasion. Sometimes they will use power supplies that have weird mounting configurations. Or snap-in blowers that are hard to source. Or, in this case, drive bays that require goofy mounting hardware.

On this server there are two non-standard things about the drive bays:

They require proprietary, snap-in drive carriers (but for servers this is fairly common).
They are located so close to the edge of the server’s case that special low-profile, right-angle power and SATA data cables are required for drive mounting. This is basically a non-stop train to Goofy Town.

Now, here’s the head scratcher. Somebody at Dell was smart enough to fill each of the bays with a proprietary drive carrier. That person realized that if Dell didn’t provide the carriers, the bays were pretty much useless, and customers would likely be upset because they couldn’t actually use the drive bays they had just purchased as part of their shiny new servers. Likewise, somebody was smart enough to provide the special low-profile, right-angle power cable required for each bay. But nobody thought to provide the special low-profile, right-angle SATA cable required for each bay. Oops.

If it were a standard cable, I could understand the omission. In this case, however, the cable is effectively proprietary and thus should have been considered an essential part of the bay itself, just like the carrier and power connecter are, and provided out of the box.

Dell’s phone support: “You are in a maze of twisty little passages, all alike”

So I called Dell, using the phone number on my order screen, to get the required cabling. Once Dell picked up, the problem was solved with a simple, 17-step process, requiring only about two-and-a-half hours in phone-maze hell:

After phone-tree surfing, I ended up talking with Temi. She didn’t know the part number for the cable, so she said she would have somebody call me back.
An hour later, Scott – in sales – called me back. He said he really couldn’t sell me the cables because “there was essentially no resale value to them.” But he said maybe “parts” could help me out. So he transferred me to Manuel in the parts department.
Manuel was able to narrow the selection down to four potential cables. But he didn’t know which of the four I needed. So he said he would conference in a tech specialist, who would be able to pick the right cable.
I ended up back at the main menu of the phone tree. Manuel was not on the line. Oops.
Once again, I surfed the phone tree to business customer service.
This time I was connected to Cathy. I explained the situation. She said she couldn’t help me but would transfer me to somebody who could. (At this time, I had been on the phone for one solid hour.)
George picked up. He seemed clued in. After I explained the situation, I could sense that he got it: It is not cool to ship a customer a server with effectively unusable drive bays.
Unfortunately, George said he was not the right person to take care of the issue. (I got the feeling he was in the support group for big-money enterprise customers and that my small company didn’t quite make the cut.) He said he would give me the exact phone number and extension to call to speak with the people who could get the job done.
When he gave me the number, I noticed it was the same number that had been on my order screen and had started my mad quest through Dell’s customer-support, phone-tree hell. When I informed George of this, he seemed surprised. In that case, he said, he would personally transfer me to a “resolution specialist” who had the clout to get things done. Further, he assured me, he would make sure the specialist understood the situation before he handed off the call. Cool.
George conferenced in Erica and briefly explained the situation. OK, she said, she would help me out. George said goodbye, and I thanked him.
Erica, now in charge, asked me what I wanted her to do. I said, figure out what the right part is, and send a shipment to me. Erica said that she didn’t know what the right part was, but she could transfer me to parts, and they could probably help me out. I said, no way, I had already talked to parts – about an hour and a half ago – and now that I was speaking with a resolution specialist I didn’t want to be de-escalated.
At this point, everything fell apart. Erica said that she couldn’t get me the parts. All she could really do, in fact, was arrange for the server to be picked up for a refund. This blew my mind.
Staying calm, I pointed out the absurdity of the situation: “Wouldn’t it make more sense to conference in the right group and just have them send me the parts? Think about it, you just sold me the server. Now you’re telling me that the solution is to send the server back for a refund? If I do that, it will be as if I had not done business with Dell in the first place. Are you absolutely certain that the best solution Dell has to offer is effectively the same as not doing business with Dell? Doesn’t that strike you as absurd? Aren’t you empowered to do something that makes a little more sense, both for me and for Dell? George told me that you were a ‘resolution specialist’ who had the power to make things right for customers. Isn’t that the case?”
Apparently, the situation struck somebody as absurd because at that moment a gentleman by the name of Michael broke into the conversation. He thanked Erica for her help and said that he would be taking over the call.
After Erica left the conversation, Michael explained that he and George (from step 10) had been monitoring the conversation since George’s hand-off, just to make sure the situation was handled properly. Because the call seemed to be headed in the wrong direction, they felt it was time to take the call back and make things right themselves.
Michael – who seemed like a no-nonsense kind of guy – said that he was going to find out what I needed, make sure it was in stock, and get it to me. And that’s exactly what he did. In about three minutes, he had confirmed the part number of the correct cabling, verified that it was in stock, and then handed the call over to George, who (1) arranged for the shipment, (2) gave it some kind of insane priority that he said would get it fulfilled before the shift change in the next hour, and (3) got me a tracking number. I thanked George for his help, and he gave me his direct line, just in case I ever needed it.
Problem solved.

All in all, I am not happy with Dell’s support. Even though Michael and George kicked ass on behalf of this customer – note to Dell: you need more guys like them – it was too late to undo the damage caused by nearly two hours of ineffective prior support.

I have some more thoughts that I will share later, especially regarding the comparative merits of HP’s support.

Until then, does anybody have any other entertaining phone-support tales to tell?

Update: See Dell-support follow-up survey to read about how my problems with Dell support ended up getting the attention of a business magazine and Dell headquarters.

My new rig is up and running

Tom Moertel — Fri, 17 Feb 2006 21:58:00 -0500

I am happy to report that I am typing this post on my new homebrew workstation. It sure does feel snappy!

Vitals: AMD Opteron 165 (dual core), 4-GB ECC RAM, 500-GB RAID5 storage (hot-swap trays), Fedora Core 4 Linux (workstation install). I went with the AMD Opteron because of the on-chip memory controllers and better I/O architecture.

Here’s a snapshot taken halfway through the assembly process:

The heat-pipe system that AMD provided to remove heat from the Opteron 165 reminds me of the exhaust systems on top-fuel funny cars:

Dig that shiny copper!

How to flash your BIOS when you don't have a floppy drive

Tom Moertel — Fri, 17 Feb 2006 02:23:00 -0500

Tonight while building a new workstation, I needed to update the BIOS on the motherboard, a Tyan Tomcat K8E. Tyan, however, offers only floppy-based BIOS flashing software to do the job. Worse, the software requires me to boot into DOS first, using a DOS boot floppy that is neither provided nor lying around the office (I’m a Linux guy).

One more thing: it turns out that my new floppy drive is junk.

Thus we arrive at tonight’s problem: If you do not have a floppy drive, how can you flash a motherboard’s BIOS when its manufacturer provides only a DOS-floppy-based BIOS flasher?

Fortunately, the problem can be solved. In case you ever need the solution, here it is.

Disclaimer: This recipe worked fine for me, but might not for you. If you follow these instructions, you do so at your own risk and assume all responsibility for whatever happens, even if your computer catches on fire or your pants explode. You have been warned.

First, download a bootable floppy image from the FreeDOS Project. The one you want is the 2.88-MB ODIN image because it has about 1.5 MB of free space, enough to hold the contents of the BIOS flasher’s floppy.

Second, mount the floppy image so that you can edit it:

mkdir /tmp/image
mount -o loop /path/to/odin2880.img /tmp/image

Third, copy the BIOS flasher and associated files into the mounted floppy image. I just unziped Tyan’s archive directly into the image:

unzip /tmp/tyan_2865_301.zip -d /tmp/image

Fourth, unmount the image.

umount -d /tmp/image

Fifth, create a bootable CD-ROM from the floppy image.

cd /tmp
mkdir boot_cd
mv /path/to/odin2880.img boot_cd
mkisofs -o odin-cdrom.img -b odin2880.img -c boot.catalog boot_cd
cdrecord -v -eject odin-cdrom.img

Finally, reboot your PC using the CD-ROM and flash away! (Note: If FreeDOS asks, you don’t want to use extended memory or anything like that because BIOS flashers don’t like it. You want old 8086-style unprotected memory.)

Verizon FiOS fiber-optic Internet service: a first look

Tom Moertel — Tue, 15 Nov 2005 14:23:00 -0500

Recently I had Verizon’s fiber-optic service “FiOS” installed at my home. The installation process took about a half day and involved placing the following boxes around my house:

optical network terminal (ONT, installed outside of house)
battery backup unit (BBU, installed in basement)
power adapter (plugged into household electrical outlet)

The ONT was installed next to my old POTS junction box:

The ONT acts like a miniature central office. To my house it provides four POTS lines for voice service and one 10/100 Mbps Ethernet port for data service. The ONT accepts a single fiber-optic cable that connects all of these services back to Verizon’s central office.

As part of the installation process, Verizon moved my POTS lines from copper over to the ONT’s POTS interfaces. Verizon wanted to remove my copper-based service altogether, but I forbade them from doing so because I have non-Verizon business lines that I want to keep on copper, which competitive carriers can use to offer me service. (Verizon is not required to share its fiber cables with competitive carriers.)

If you look closely at the ONT, you’ll see that it also is capable of handling video service:

(At present Pennsylvania’s cable-franchise laws prevent Verizon from offering video service, but I’m sure Verizon’s lobbyists are working to change that situation.)

Unlike copper wires, fiber-optic cables do not carry power. The ONT, therefore, must be powered from my home’s electrical service. If the power goes out, the battery backup unit (BBU) will supply power for the ONT’s voice services for about four hours.

VoIP users beware: When the household power fails, the ONT’s data services will be dropped immediately in order to conserve the BBU’s battery. This seems pretty lame to me, but Verizon confirmed this behavior when I called them to ask about it. If you need data service during a power failure, make sure your ONT is powered via a UPS under your control.

To provide data service to my house, the installer ran a CAT-5 cable from the ONT’s 10/100 Ethernet port into my house, where it plugs into a D-Link 4-port “Ethernet Broadband Router,” provided by Verizon for free. Although the provided router has NAT and firewall features, I placed a Linux-based firewall between it and the rest of my home network as an added precaution.

I have been using the service for several days now, and here is my verdict:

It’s just broadband.

Practically speaking, I can’t tell any difference between FiOS and my Adelphia cable-modem service. I ordered 5-Mbps service from both providers, and both services provide about 5 Mbps down, which is faster than fast enough for me. The FiOS service has slightly lower latency – I can ping www.google.com in about 9 ms – and that’s a nice plus.

The big benefit of FiOS is competition: Verizon’s price is about $10/month less than Adelphia’s. When I called Adelphia to cancel my service, their representative attempted to change my mind by offering me a 3-month promotional discount and trying to sell me extra television channels.

I passed.

Replacing the fan array in my HP ProCurve 4000M switch

Tom Moertel — Fri, 11 Nov 2005 21:42:00 -0500

The main network switch in my home office is an HP ProCurve 4000m, which has been running non-stop for over half a decade. It is a great switch, and even though it is getting old, it is still dependable.

A while ago I noticed that the 4000m’s fault indicator was lit. So I logged into the switch and checked the log: fan 1 was dead. The switch has built-in redundancy (three fans), and so I didn’t worry about it, but I did call HP ProCurve tech support.

The woman I spoke with was friendly and helpful. I told her what was wrong, and she said a new fan array would be on my doorstep within 48 hours. No charge. (I guess the ProCurve warranty really is worth something.)

Today, I installed the array. This meant opening up the switch, which is a fun thing to do. If you are curious about what is inside of a 4000m, I took photos of the operation.

During the process, I recalled why I love old-style HP engineering:

The replacement parts came with clear instructions that showed me how to remove the old array and install the new one. They were easy to follow and didn’t leave anything to guess.
The 4000m is solid – inside and out.
The electrical components are top quality.
The industrial engineering is superb. For example, all of the user-removable screws have non-stripping torx heads and are designed not to fall out and get lost; instead they remain attached to the module or panel you are removing. (See this photo of removed modules to see how the screws stay in place.)

Everything about the process made me think, wow, this is really well engineered.

The thing is, I know, as I sit here and watch the blinking LEDs on my now-restored 4000m, that my next network switch will probably be a Dell.

As much as I love the ProCurve engineering, the Dell price is compelling. Even if I expect the Dells to fail twice as often (and the Dell warranties are comparatively lame), I can buy twice as many Dells and keep spares on the shelf – and still save money compared to the equivalent ProCurve equipment.

I find the situation somewhat sad. I am an engineering guy to the core. So when I go for the cheaper product because it is so darn cheap, I know that much of the market will do likewise. That bodes ill for HP. Like HP’s calculators, the ProCurves too may pass into history.