Pablo Averbuj

ReadyNAS round 3: Fatality!

Today went much better than yesterday. I turned the ReadyNAS on and at first the fan only spun slowly. After it started getting past the early boot stages it spun up and stayed at high RPM. The problem is not entirely fixed. The OS doesn’t seem to recognize the fan, so I can’t get metrics on fan speed, and if the fan stops, I won’t know until it overheats and shuts down. However, it seems to be working ok since 7am when I started it. I did seems to get warm around 6:45pm but seems to be alright since. Frankly, I’d be surprised if it stays up the entire time we are gone. I think that either the fan is flaky or the motherboard is having issues. My hope is to have Infrant send a new fan tomorrow and install it when I get home from the trip. If that fan doesn’t work then I’ll start thinking it’s more than just bad luck with fans.

Sun Jun 17 19:00:23 EDT 2007 Disk on channel 2 has reached warning level (55C/131F).
Sun Jun 17 18:58:23 EDT 2007 Disk on channel 2 has reached warning level (55C/131F).
Sun Jun 17 18:56:23 EDT 2007 Disk on channel 2 has reached warning level (56C/132F).
Sun Jun 17 18:54:23 EDT 2007 Disk on channel 2 has reached warning level (56C/132F).
Sun Jun 17 18:53:04 EDT 2007 Disk on channel 2 has reached warning level (57C/134F). Disk on channel 3 has reached warning level (55C/131F).
Sun Jun 17 18:48:24 EDT 2007 Disk on channel 2 has reached warning level (56C/132F).
Sun Jun 17 18:46:23 EDT 2007 Disk on channel 2 has reached warning level (56C/132F).
Sun Jun 17 18:44:38 EDT 2007 Disk on channel 2 has reached warning level (56C/132F). Disk on channel 3 has reached warning level (55C/131F).
Sun Jun 17 18:42:23 EDT 2007 Disk on channel 2 has reached warning level (56C/132F).
Sun Jun 17 08:55:37 EDT 2007 System is up.

ReadyNAS round 2: FIGHT!

Today I fought with the ReadyNAS on and off. At first I was not able to get the fan to spin up at all. I tried the new fan, the old fan, I tried spinning it by hand. It sensed the fan (ie, when I spun it by hand the RPM registered (side note, I got it up to 7000 rpm without assistance from the motor). Finally as I was about to give up, I made an odd contact and for a brief moment it spun up. It seemed that if I held the connecter in a particular way, it would spin, but as soon as it moved the smallest amount it would stop. The connector on the new fan was not what I would consider a snug fit, whereas the old fan had a snug fit. i noticed with the old fan if I spun it manually it would continue to spin indefinitely although at a very slow speed. I considered the possibility that the fan connector on the motherboard was making a bad connection, but I could pretty reliably get the old fan to spin at low RPM, and the new fan would not. I chose to overlook the fact that it worked perfectly and consistently when I first plugged the fan in, and decided the new fan was not making good contact.

I scoured the house looking for a 3-pin fan connecter I could cannibalize and shockingly found none. Left with no other choice I sacrificed both fans to create one SUPER FAN consisting of the new fan assembly and the old fan connector. During this process I remembered why it is I stick with software and stay away from hardware hacking. I know, it’s pretty grotesque. So I took my handy work and hooked it up, nothing. I fidgeted here and there, tried to make it connect the way it did before. Again, just as the hard drives are about to overheat (and cause the ReadyNAS to shut down), it made contact! The beatiful sound of 3000 RPM generated by the wonders of electricity. I set the fan gently in its approximate location and let it cool the hard drives down for an hour or two. When I came back it was still working, so I decided to reassemble the appartus without shutting it down. I assembled it one way, but the fan cable wasn’t long enough (I was trying to keep the cable taut so the bare union wouldn’t touch metal (I know, genius!). I tried a second way and it was long enough but I just barely couldn’t make the pieces fit back together. I took it apart and started on the third attempt when tragedy struck, it stopped spinning! I tried in vain to resuscitate my creation to no avail. Eventually the drives overheated and the device aborted it glorious mission.

I’ve checked the websites of the nearby hardware suppliers and nobody seems to carry the 92mm fan I need. I even probed a few friends known for their hardware hoarding compulsions with no luck. Tomorrow I may try again, but I think at this point it looks like resolution won’t come without talking to Infrant on Monday.

ReadyNAS rollercoaster

First a side note. When I was corresponding with the Infrant support guy, he asked for a copy of my invoice. I forwarded him a copy of the invoice along with thread (in case the ticket tracking system needed something in the body). In the process, I accidentally copied the sales guys at ExcaliberPC (where I bought the ReadyNAS last year). I noticed my mistake but i figured they would get the email, see it was regarding a purchase made last year, and delete it.  Much to my surprise, last night (Thursday) at around 10pm I got a call from a support guy at ExcaliberPC calling to check if everything was alright with the device and if there was anything he could do to help. Despite the time of the call, it was a pleasant surprise to see customer service handled so proactively. I am very impressed.

Now, for the roller coaster that was today’s adventure. When I got home from work, Leigha told me I had received a package and I knew it was the replacement fan. It never really crossed my mind that the replacement fan wouldn’t work, so I didn’t get around to replacing it until after 7pm. At around 7:30pm I had it all reassembled, turned it on, fan didn’t spin. Checked the control panel and no fan was detected.

It was at this precise moment that a lot of information came together.

1. Infrant is based in California where it was now 4:30pm
2. It’s Friday
3. I’m out of town all next week!
4. If I don’t get this working now, I won’t be able to troubleshoot remotely!!
5. I won’t have this fixed for like another 2 weeks!!!

This is around when panic set in and I launched a full assault on Infrant. One if by landline, two if by SMTP.

Now I’ll add another bit of information to the puzzle. Early versions of the ReadyNAS NV had a fan issue that required a little adapter. My unit was supposed to be affected by the issue, so when they shipped the fan they helpfully included the adapter, which I dutifully installed. As soon as I hung up after leaving voicemail, it occurred to me that maybe the adapter is unnecessary/faulty and I should try without it. I take the thing apart, remove the adapter, and (without reattaching all the screws this time) as the fan spins up, the phone rings. I talk to my tech guy (kudos for always talking to the same tech) and tell him it’s spinning now without the adapter. He tells me the adapter is only necessary for certain board/fan combinations. I’m happy so I apologize for assaulting their communications but explain the 5pm thing and the vacation thing and he confirms they close at 5pm.

At 4:fucking59pm PST (8pm EST) the thing stops spinning. The control panel says “out of spec” for the fan. RPM == 0. I assault again, but this time I’ve cried wolf and everyone is busy drinking beer so no return phone call. After a few minutes it seems to  restart itself. It’s certainly not comforting but if something goes wront it shuts itself down so I let it ride.

Subject: Unexpected enclosure status detected [nas0]
Date: Fri, 15 Jun 2007 21:14:47 -0360
Fan 1 has failed.

For extra insult, in the meantime I had gone back and added my cellphone to the list of emails to notify in case of failure. So it starts losing it’s mind as 10 more messages come in alarming of ever increasing hard drive temperatures, culminating with:

Subject: Disk temperature limit exceeded [nas0]
Date: Fri, 15 Jun 2007 21:42:00 -0360
The reported temperature of a disk in the NAS has exceeded maximum recommended temperature.  Shutting down the system to prevent hardware damage.
Disk on channel 2 reports a temperature of 61C / 141F.
Fan 0: 0 RPM

Since I’m sure they are closed over the weekend, I’ll spend my time searching the message boards for similar issues and tinkering with the fan adapter to see why it wouldn’t work.

Score:

ReadyNAS: 2
Infrant: 1
Pablo: 0

Interesting Screenshot

I was just about to start looking for somethingon the Costco site when I saw a smaller picture of this Dell Laptop. Before I even read the caption my first thought was “Neat, Costco is selling MacBook Pro’s”. I clicked through and found a Dell. I confirmed that sure enough that is the default OSX Tiger aqua blue background. I find it hard to believe that Apple would ship with stock art, so I think someone decided to co-opt the background for marketing purposes.

Why I paid extra for a NAS chassis

As you’ll see below, the main fan in my Infrant ReadyNAS NV died recently. It died on Wednesday and I didn’t notice until Saturday. Good thing it turned itself off before the drives overheated. Below is the transaction of emails I received before it turned itself off to prevent damage to the disks (this is an option, but I can’t remember if it’s on by default or not).

Subject: Unexpected enclosure status detected [nas0]
Date: Wed, 06 Jun 2007 05:25:07 -0360
Fan 1 has failed.

Subject: Unexpected enclosure event detected (nas0)
Date: Thu, 07 Jun 2007 02:32:56 -0360
Disk on channel 2 has reached warning level (55C/131F).
[Thu Jun 7 02:32:36 EDT 2007]

More emails after the jump but you get the idea. I just noticed the odd notation for the timezone (-0360 == -0400). Also the host monitoring I use (‘spong’) also caught the shutdown and emailed me. The reason I didn’t notice until Saturday is because I’ve been too busy to keep up with my personal e-mail.

Thursday, 06/07/07
——————————————————————————
red S 02:54 nas0 http no response from http server
red S 02:54 nas0 ping ping failed for 10.0.0.7

I submitted a ticket to Infrant support and they responded Monday, I provided them a copy of my invoice on Tuesday and they shipped out the replacement fan this morning (hoping to have it by Friday or Saturday before we leave town). It would have been nice to get the replacement part faster (or be given the option to upgrade the shipping) but it’s not usually critical to have access to the data on a day-to-day basis, just very very convenient.

It’s odd that I didn’t get a shutdown email from the NAS, but given the choice between sending an email and shutting down, I’d rather the device shut down. It’s clear from the message trail what happened. Also, I should configure spong to page me on NAS outages.

Continue reading Why I paid extra for a NAS chassis…

Where I’ve been

The last time I had the time, energy, and opinion to write something was December, nearly six months ago, and it ended an unprecedented 10-month streak of posts. Around that time the speed of life shifted into fifth gear and slammed the accelerator because although I’ve had things I’ve wanted to post aboud I rarely had the time or energy to do it.

In December I changed roles at work. I went from being a programmer to an architect, and I moved from Internet Operations in the CTO organization to the Product division (in Video). I’ve been doing video-y things since then. The move was dramatic in many ways; we spent nearly the entire month of January visiting our remote offices, meeting with our new colleagues and getting to know them, what they do, and how they do it. It was a whirlwind trip in and of itself, except at the end of the month of January, we closed on a house, a work in progress since Thanksgiving.

Combine the duties of a new job and the stress of a moving and a new house and you’ve got yourself a perfect storm.  In case you might think I’m over dramatizing the whole thing, consider the following. We were scheduled and locked to close on the house on Monday, January 29th. On Sunday the 21st, I flew to California. On Tuesday in California I was diagnosed with an ear infection and advised not to fly until Sunday the 28th. On Wednesday, my colleagues departed for the Tucson office, while I boarded the train that would take me across the country for the next three a half days. I arrived back in DC on the Saturday before the closing (incidentally, the following Tuesday I still had the same ear infection, so the gamble paid off).

I’ve had a lot of things on my mind, especially regarding work and video and I’d like to share them once I’m clearer on the blogging policy at work. Hopefully this post is a prelude to turning the slow drip of postings back on.