Showers, SANs and Insurgency

I’m doomed not to be able to have a decent hot electric shower. Came home from an Easter weekend away to find it’s the second time it has stopped working. I expect it’s the electrics again rather than the shower. Looks like I’ll have to buy another, less powerful, shower and try and sell this one on ebay. Anyone want to buy a 10kw shower? In the meantime, bath time each morning. Just one more thing to irritate me.

Spent the weekend with my wife and children with her brother and his family and extended family. A great weekend, only spoiled by the damn shower when we got home! On Sunday we went to a first birthday party, which was actually more of an adult affair. There I met a neighbour, an American gentleman called Michael and his wife and child. He told me he was a journalist, and had been a foreign correspondant now writing his second book. I finally got the chance to sit down for a chat when we had to leave for the long journey back home. So I promised to look up his book when I got home.

After a quick google, I found out some more about this interesting gentleman. His name is Michael Goldfarb, his book is called “Ahmad’s War, Ahmad’s Peace“. Michael is a very, very accomplished journalist. After reading bios, I am sad that I didn’t have more time to talk with him. He has created a documentary for the “Inside Out” program on WBUR Boston on the subject, which, in my opinion, is a moving account of the start of the current war in Iraq from a personal perspective. Click here to visit that documentary, which includes the radio program. It’s an hour long. Take my advice: get yourself a clear hour, turn up the speakers on your PC, get comfy and listen to it.

At work, I’ve finally sorted out my SAN problem, no thanks to the hopeless vendor (two letters, known for printers). Brocade were on site earlier last week for another matter, and their technical guru sat down with me, looked at the problem for a few minutes, and said “hafailover”. I’d suggested that to my vendor over a week previous - perhaps it got lost in translation between here and India. So, I spent an hour raising the change ticket, another hour talking to our customers, then the following morning I hit the return key, crossed my fingers and hoped the 12000 wouldn’t die. It was as monumentous as when the clocks ticked over to 1st January 2000. No planes dropped out of the sky, I still owed the bank several body parts for a mortgage, and the problem went away.

Take note Brocade customers - Brocade are offering a supplementary support service, obviously at a cost. If you’ve bought your switches from HP, IBM or EMC, get them in now. Don’t delay!

Dark porcelain

Considering the grief I’ve had recently over dark fibre, Google’s April Fool really made me chuckle (http://www.google.com/tisp/). Enjoy.

Wood, Trees, D’oh!

Well, good news and bad. Good news, the DWDM links are working properly. Bad news, my inability to see the wood for the trees. Got so tied up looking at the quality of the links, someone said to me today “You’ve allocated too many buffer credits”. So, I turned off the long distance, hey presto, no more time outs. D’oh, back to the dummy’s book for me. Too many credits meant the destination port was being flooded. Now I need to test and tweak. Without LD, I’ve not got enough credits. We’ll have a diverse 22km route soon, maths says “1 credit per km” for a 2gb link. I though dynamic allocation (LD) mode on the Brocades would handle everything nicely for me, but obviously not. Need to find out why it thought the link was 30km. So, time for testing with static allocations. Nothing like sucking and seeing.

Bad news, still waiting for my 12000 vendor to come back to me and explain why one half still thinks the fabric is busy and stops me propagating fabric changes. Not impressed. I fear I’ll have to reboot, which means an emergency change. Perfect timing as our Change Management team has just introduced a new template, which means half a days just to fill out the change request for the outage. And of course, they’ve not publicised the change. I noted in the new template there’s not a section relating to hypocrisy.

No ends in sight

Firstly, thanks to Sangod for his comments. The morning after I wrote the piece I was showing my new spangly blog to a team mate, saying how I found other storage blogs useful, went to Sangod’s site and there was a reference to my post! It made me feel very special (not “Special Needs” as my co-workers commented!) However my joy was brief.

My long distance relationship is now over. It’s now shorter, thanks to having having both DWDM routes at 15km, which may occasionally diverge if we’re lucky. It’s an accident just waiting to happen. At least all my links are up, even if I did have to travel to the remote site to swap a tx/rx connection around. But those shiny green lights on the ports on the Brocade 48000s are as deceptive as Gordon Brown’s latest budget (don’t get me started on how he’s cut the basic rate of tax yet the lowest paid in our society will be worse off). The links are up, I have 2 x 2gb dynamically trunked paths on each fabric, the switches see each other, but can anyone see their remote storage? Our remote USP array can see the WWN of the remote servers, but any attempt to access that storage times out.

I see the ‘encoding errors outside of frame’ count go up faster than my blood pressure, which indicates to me that the connections are about as crap as the customer service from my ISP. We have DWDM links to another remote site provided by another telecoms company that are mostly reliable and after aeons of up time those ‘enc_out’ errors number are around 3,000, far less than the 300,000+ in a couple of hours seen on the new links. If anyone reads this and has any thoughts on this subject, I’d appreciate the input.

But that’s not the end of it. In another location I have one half of a Brocade 12000 director preventing us from propagating any configuration changes. Even an ‘alishow’ request is ignored with a ‘fabric busy’ message. The other half of the 12000 is fine. This leads me nicely into another area of irritation. Firstly, it irks me that I have to get support from the re-seller rather than direct from Brocade (though I believe if we pay them enough now they’ll help out), especially as our fabrics have built up over time so switches have come from different re-sellers. But then they re-badge them. To me, it’s a Brocade 12000. So to the re-seller, it should be a Brocade 12000, not an HP whatever or IBM thingumyjig (at least HDS supply them vanilla). Anyhoo, I’ve had a call logged for about 3 days now, still no solution, last I heard the problem is relaxing somewhere on the Indian sub-continent. At least if I end up power cycling the director then only fabric is affected and data can be reached on the other, unlike another site that had each half of the 12000 on a different fabric. It certainly hit the fan when that director crashed recently. Roll on hardware refresh time, I like the Brocade switches, but the 12000 is certainly not their best design.

On a personal note, got a letter today saying I could be a match for someone needing a bone marrow transplant, so some more test are needed. If it comes to it, donating marrow or stem cells is a tad more invasive than your average blood donation. I knew this when I signed up, but seeing this now in writing puts it into perspective. A little scary.

Long Distance Relationships are a pain

We’re currently commissioning a new data centre. It’s about 10km as the crow flies, but can you get decent DWDM links over that relatively short distance? A certain cable company in the news at the moment (I left them for Sky a couple of years ago) is providing the links, but one of them is 60km! At least the data will be well travelled. We’re trying to implement syncronous truecopy between our USP arrays, and you can’t do it with that. Now it looks like we’re going to end up with all our routes instead going over the shorter 15km link, so all it will take is a clumsy oaf with a digger and our SAN links will go down faster than Audley Harrison in his last fight.

At the moment we have two links up, one on each of our fabrics, and one of them is giving so many ‘out of frame’ errors that we cannot propogate fabric changes on it, and I’ve spent most of today trying in vain to get around it. It’s having to deal with crap like this that is out of our hands that makes everything else I’m working on late, and that really pisses me off.

Welcome

Welcome to my blog. My name is John. I have finally succumbed to the temptation to climb onto the virtual soap box, press the button on the virtual megaphone and whince as the feedback squeals like fingernails scraping a blackboard. Only you don’t really see blackboards any more. Certainly not in schools. It’s whiteboards, targets, stats, testing, more targets, while desperate teachers struggle to meet the needs of the children whilst drowning under the bureaucracy that is the norm under this bastard, corrupt, sleazy Labour government that is dragging this country (that’s England) down with its leaders as they prepare themselves for arrest and trial for selling peerages whilst crippling this country with “off the books” PFI debt that at some point is going to drop a large turd like the proverbial elephant in the corner of the room, and we are going to have to pay for the clean up. Well, rather those of us who work the long hours and bear the brunt of Gordon’s taxes are going to have to pay for it.

Now you’ve survived my first venting, as my title implies, I am round and irritated. I work in IT as an Enterprise Storage Specialist. Any posts here will be a combination of geek related storage items and my general irritation with the world around me, hopefully tinged with a hint of humour and irony.