All posts by jrdalrymple

Maximum drainage technology

I’m hoping this is the lamest thing that happens today:

Motorola DROID RAZR, 2.4, 4.6mm, 0.001 sec, ISO 100
Motorola DROID RAZR, 2.4, 4.6mm, 0.02 sec, ISO 107
Motorola DROID RAZR, 2.4, 4.6mm, 0.001 sec, ISO 100
 
Motorola DROID RAZR, 2.4, 4.6mm, 0.001 sec, ISO 100
Motorola DROID RAZR, 2.4, 4.6mm, 0.001 sec, ISO 100
Motorola DROID RAZR, 2.4, 4.6mm, 0.001 sec, ISO 100
 
Motorola DROID RAZR, 2.4, 4.6mm, 0.002 sec, ISO 100
Motorola DROID RAZR, 2.4, 4.6mm, 0.04 sec, ISO 107
Motorola DROID RAZR, 2.4, 4.6mm, 0.001 sec, ISO 100
 

The contents is fine, I just need to decide if it’s worth fixing or if I should burn it and/or turn it into an unenclosed trailer. The barn doors still swing open and shut perfectly fine. I wish they didn’t, it would make my decision a lot easier.

I think this is the kind of lesson a guy only has to learn 1 time.

ACS Network Status Update – Sucks

If I spent 10 minutes writing about every poor experience I’ve had with Alaska Communications I’d be busy writing for over 24 hours. The number of people I’ve interacted with at that company that were actually pleasant to work with could easily be counted on 1 hand, and most of them worked in customer service in Anchorage. Today I’m going to just complain about one interaction with this company, my least favorite organization on the planet (no exaggeration).

The red circle was last Friday, ACS rolled a truck to *upgrade* the speed of my client’s Internet. The blue line is a continuous stream of data between my client and my servers in MN. It’s not obvious from the image above, but the speed *upgrade* took my client off the Internet for a total of 4 hours and 18 minutes during normal business hours. I suppose in the 1990s it was acceptable to have such a large outage for such a modest *upgrade*, but it’s 2014. I suggest that the upgrade was a modest one because they were attempting to go from a 3Mpbs business class service to a 7Mbps business class service. Not long ago I upgraded my low-cost and low-priority home Internet from 6Mbps to 50Mbps. There was no outage and the total amount of invested time on my part was about 15 minutes. In my mind this is how upgrades should happen in 2014.

The Friday outage was split into 2 parts. The first was a lengthy 3 hours and 3 minutes, during which time it was blackout for me. I had no knowledge of what was happening. I was E-mailed prior to the commencement of the upgrade asking if I had any advice. I replied with a basic structure of what my client’s router expects to see in order to reinitiate the connection to the ISP and the disclaimer that if he or she didn’t understand my E-mail then they should postpone the upgrade until we could work together on it. They proceeded so I assumed he knew what he was doing. I was eventually contacted by an individual back at the Alaska Communications call center, they needed my assistance (and understandably so) getting the office back online. I told them to reboot the router, at which point the PPPoE connection came up and all was fine, it seemed the technician did his job OKĀ  just my ddial PPP connection wasn’t coming online. A reboot is simple enough and something that Alaska Communications themselves wouldn’t hesitate to do with most customers, it just so happens my router is slightly more intimidating looking:

After the line came up I asked the technician to not touch anything further and leave the room. Although my client still wasn’t on the Internet, the remaining portion was up to me. I did my business then it was my time to leave work so I started my commute home. The connection stayed up for all of 15 minutes and 10 seconds. About 1 mile into my commute I got a text message saying it was down again. Normally I’d turn around to put myself back at a terminal, but this particular Friday I had a meeting after work that I didn’t want to miss. I continued on my way and did phone troubleshooting with staff at my client’s office.

After probably 45 minutes of great help from one of my client’s employees, we found the inability to connect even just using a laptop and Windows 7’s Internet Connection Wizard. I turned the problem back to Alaska Communications with a phone call back to the individual who had called me earlier. That ACS employee contacted my client’s staff, had them plug the router back in and it miraculously came online. I’m aggravated as I’m sure the ACS employee played it off like nothing was wrong the entire time when there is no doubt in my mind he did something on his end. There is no other explanation since we couldn’t even connect with a generic Windows 7 box. The 2nd outage while only a mere hour was big enough to make 3 9s system when measured over a year.

This brings us past the red circle above and into the orange circle. The Internet was online all of Friday evening, all of Saturday even though speeds dropped in the early morning as you can see, but the connection did stay up. The blue circle encompasses the Sunday outage, one which started about 30 minutes prior to the end of AK Daylight Savings Time. The outage lasted through the day, meaning for an entire day and then some my client could not send nor receive E-mails. Their servers were unreachable so no doubt some correspondence was bounced. ACS doesn’t have a 24 hour business unit, or at least not one with a listed phone number, so I had to wait until they opened at 8:00AM. The good news is even after they opened the technical support I was able to contact was utterly useless. “I can see you’re not online, but we can’t do anything about it without someone onsite.” Certainly I agree it would be nice to have someone onsite, but unfortunately it wasn’t in the cards. The people that would normally be there on a Sunday were out of town this particular weekend. The annoying part was that he didn’t even bother to look at connection logs or any such, they just gave up at “you’re not online.”

About 30 hours past, the connection was brought back to life Monday morning at 8:09AM AK time. This is what availability looked like for the week – Tuesday morning through today, note that the point when the *upgrade* started is clearly visible:

Since their Internet presence returned yesterday morning, my client hasn’t fallen back offline. Their bandwidth remains pretty random. At the end of the day, my only prayer is that I don’t have to work with ACS Personnel, ever again, in my life. They’re one thing I’ve been unable to leave behind by leaving AK, and I wish more than anything that I could change that.

HDS – Frustrated from day 1

If you know me well you know that I couldn’t care less about what badge your gear wears, as long as it works. On the motocross track I started on red then ended up carrying both yellow and blue in my trailer. I couldn’t tell you which color was my favorite, the all have their own special place in my heart and they all did their job well. The reason I never rode green, I didn’t like the folks who sold it. That fact will play into the remainder of my diatribe.

Paul (my direct peer at work) and myself are trying to make use of the fancy new machine that was set on our floor. HDS came in and configured up our VSP. The tech promised a day to set it up, it ended up being 3.5 days to do 2. I don’t blame the tech that made the promise as he isn’t the one who ended up doing the work; not his fault. That said, if it takes HDS 2 days to set up a paultry 30T (don’t ask how much paultry 30Ts cost!!!) usable array they’re never going to make it in datacenters the likes of a big organization like Microsoft, Google or Amazon. You’d never guess this based upon how much they love boasting how amazing their gear is. Mind you this is our first Hitachi box. We had to listen to endless beating of their own drum Hitachi staff did about how happy we’re going to be getting off IBM and on Hitachi.

Finally the tech is done with the initial setup and it’s ready for us to login to the box and start creating pools, carving LUNs and presenting disks. Probably the most exciting thing for a storage admin, logging onto a fresh array and digging in.

Direct quote from the Hitachi Storage Navigator User Guide:

To log in to Storage Navigator as the super user:
  1. Call your local service representative to obtain the superuser ID and default password.

Yup, here we are. We have an E-mail string going back and forth with pre-sales and the guys who spun the wrenches last week doing the initial config. Meanwhile, Paul and myself are seeing our deadline to have this array online creep closer and closer and the brick is sitting in the datacenter heating our cold air and being very green by wasting 100% of the electricity it’s consuming.

Thus far, I’m not leaping out of my chair with happiness because of our new vendor. If you happen to know the username and password to logon to my array, let me know. I certainly don’t.

Server 2012 “server specified requires restart” loop when installing WID

I was trying to create a RDS deployment, something I’ve done before without issues but this time when trying to install the necessary roles and features I ended up with the reboot loop described. Each time I tried to install, Server Manager complained “The request to add or remove features on the specified server failed. the operation cannot be completed because the server that you specified requires a restart.”

I narrowed the problem down to the WID (Windows Internal Database) feature installation causing the issue. From there I googled and found a MSDN page written partially in another language, hence my own English sharing of knowledge.

http://social.msdn.microsoft.com/Forums/ro-RO/e7e9bc17-14d1-43c9-809c-464f69b366cd/server-2012-windows-internal-database-error-during-installation

The post useful to me was the one by kswail about halfway down. Adjust your domain (or domain controller if appropriate) security policy to allow “NT SERVICE\MSSQL$MICROSOFT##WID” to log on as a service, a GPO setting that can be found under Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > User Rights Assignment. Simply adding the security principle mentioned above to that policy solved a problem that haunted me for 2 days.