Bandwidth – so near I can smell it

2 08 2008

It looks like my rantings managed to help somebody, which has kicked me in the pants to apologise to my blog for neglicting it. As per my earlier posts, long distance bandwidth has always been a problem. Well, hopefully that could now be cured. I have received the gift of Ciena CN2000s. They claim I should get at least a 4:1 compression of my fibre channel traffic overall. If that’s the case, when all is done I’ll have a creamy 32Gb of bandwidth, which brings me a small amount of pleasure.

After a bit of faffing with our DWDM supplier about OTR wavelengths, I now have a pair talking to each other. A couple of fibre links are connected but not enabled yet, ready for that fateful moment when I enable the ports on the switches and hopefully something will happen. For once, I’m not irritated. Well, except for the amount of time I’ve had to spend begging and offerring deviant favours to get the bandwidth.

More to come……


Stout Provisioning

13 02 2008

This is a copy of a question I’ve posed on the HDS forum. Highlights my sad state of mind, and that I can’t switch off when I’ve got a few days off for half term!

One of those odd questions that has been rattling around my brain recently has been about allocating pools for COW and Thin Provisioning. I’m anticipating trialling the former soon, and maybe the latter when I can get my hands on a USP-V.

I’m led to believe that the data is written to the pools in a very wide stripe, i.e. scattered all over it, sounding to me in a very similar fashion to the STK RVAs I was implementing in the mid to late 90s. So, the question is, what is the best practice for allocating the physical LDEVs that make up the pool, assuming Raid 5 7D+1P is the norm for the RGs? For a normal allocation to a host, I’d use a dispersed allocation, e.g. RG 1-1, 1-2, 1-3, 1-1, 1-2, 1-3, 1-1, 1-2, 1-3 in order, with a host based stripe on tape for good measure.

For a pool, would that still be best practice? Does the wide striping algorithm work in a similar fashion to a host based stripe, e.g. something that Veritas might do; or does it really matter?

Another thought in my head is about utilising thin provisioning pools to possibly enhance performance. My thought behind that is that you could use the principle of creating VDEVs across a large pool, but not over allocate the pool, so that you could have 3 layers of striping: a host stripe over a pool stripe over a physical stripe. I like to call that ‘Stout Provisioning’ (fans of Not the Nine O’Clock News will understand my meaning).

Could there be a benefit to that? Has anybody actually tried it? All opinions and experiences shared would be greatly appreciated.

No more virtual fubar

31 01 2008

To cut a long story short, my test data was ok. It was how Linux LVM handled the presented disks when I virtualised them, or to be more precise, the order in which they were presented. One of the tests I was doing was to see if it was important to re-advertise the disks after they were virtualised behind the NSC, so I had not presented them in the same order. When I re-virtualised the 9980 disks the third time, I presented them to the NSC in the same order as native to the linux host, and presented them as external Open-Vs. I presented them in that same order from the NSC to the Linux host, and Bob’s your relative, data was accessible and all was dandy. Brown trousers averted!

I then re-created the experiment from the USP-100 that I’m actually going to be migrating data from soon (before you jump up and down and shout ‘a USP to NSC/AMS – are you mad?’ – the answer to which is ‘occasionally’ – this is development data that should not have been there to start with). This time the presented disks are 14Gb Open-Vs, same size as formatted on the NSC and its production external storage on an AMS-1000. That worked a treat, and then I used shadowimage to replicate the data from the virtualised USP-100 through the NSC and out to the virtualised AMS, then re-presented that to the Linux host.

So there you have it. You don’t need to fork out for tiered storage manager (HDS – stop calling it TSM – it’s confusing, there’s already a TSM out there. I know it’s supposed to be called HTSM, with one of the characters in lower case, but everyone always forgets the ‘H’!) if you have a Shadowimage licence. You will need a couple of application outages to do a migration this way, but it can be done and timed to minimise the inconvenience to nothing.

Don’t forget the gotchas: present the old storage out as Open-Vs, and make sure your destination Open-Vs are the same size for Shadowimage to work; make sure you know whether or not the host O/S will need to have the virtualised old disks and subsequent new disks presented in the same order (probably not an issue with Veritas, for example); test, test and test again! I’ll be replicating my testing on AIX, Solaris and HP-UX, some with Veritas, others with native LVM. I’ll let you know how I get on.

Off to pick up the children now from nursery and after school clubs. Expensive business, childcare, and Gordon **** Brown just doesn’t help out enough with the tax breaks. It’s not so bad in the Midlands, but in London it cane cost £800 – £1000 per month for a child when the parents work full time, and a £55 / week tax break is a pathetic pittance. Oh how I wish he’d called that election a few months ago. Can’t wait to see him on the opposition benches. 

Life should be simple

27 01 2008

Just now I was chugging along with domestic life, masticating on the things on my mind, such as bill paying, feeding the family, making my code more efficient, when my 3 year old daughter nonchalantly ambles by wearing her Disney Princess dress that she got for Christmas and starts rummaging around her room. Then out she trots, happy as a pig in poop, because she’s found her tiara. I stop my thinking (I’ve been known to do that) and realise I’m jealous. Not of the tiara – it’s far too small for my large melon – but that life for her is so simple that her biggest worry is that she can’t find her tiara. I wish life could be that simple.

Talking of simple things, or things that should be simple, I’ve been lab testing array migration by externalising behind the NSC I look after. We already virtualise storage, but have never used it for migration. I recently emptied an old 9980 array, but that was by using host based mirroring. That work’s fine, but the problem is that I had to hand over the mirroring work to our Unix support team, who are so overwhelmed with work that it had to wait its turn. Perhaps being a control freak who does not like to have to rely on others to finish his work has something to do with it as well?

So now I’ve got the old 9980 to myself, I allocated a few LDEVs to my linux box, populated them with crud data, then decallocated it from the server and virtualised them behind the NSC. I’ve done this before in the lab with like for like storage, worked a treat. So I represent the now virtualised LDEVs back to the linus server, and it complains that the disk headers are fubar’d.

I de-virtualise (if there’s such a word) the storage and represent it native. Headers are still fubar’d. Oh dear. Going back through the process, I’m sure I didn’t cock up. I didn’t reformat the virtualised LDEVs. I did, however, present them from the NSC as Open-Vs, whereas on the 9980 they’re Open-Es. I did have the option of presenting them from the NSC as Open-Es, but I didn’t do that because a) I wasn’t expecting anything to be written to the 9980, and b) I wanted to try out using Shadowimage to migrate the data (I know there are other tools to do that, but they cost money). I’m going to redo this next week, this time I’ll virtualise as Open-Es and see what happens. This concerns me as I have seen other bloggers discuss how they’ve virtualised all sort of pre-existing data on a variety of arrays. Maybe I did cock something up? That’s the joy of testing I suppose. I’ve got to migrate data from a USP100 soon, so I’ll try some testing on that too. At least that should be like for like.

One final point before I head off with the family to the park – regarding my previous post, I’ve been asked how can a prospective employer contact me? Good point, seeing as I don’t publish my e-mail address (nothing personal, but I get enough spam). I moderate all posts that come in, so if you want to talk to me, just drop in a reply posting to this article. I’ll take the details, then delete the post so it won’t appear and will remain confidential. And please, no agents touting for CVs. More on that later…..

Been a while….

25 01 2008

It’s been a while since my last post. However, I have been getting lots of comments. Looking just now, it was around 35,000! Unfortunately, all spam. Another of life’s little irritations. I had to take down my union branch’s website forum recently because of spammers breaking through the defences. Went away for a two week holiday to Canada (lovely country, by the way) and found some of the site users most distressed at Viagra and various orifice probes being offerred as solutions to the questions they were posing.

I’ve been getting to grips more with our HDS kit recently. Tuning Manager is a bundle of laughs, isn’t it? As was pointed out to me, it was written in the Japanese market place, which is vendor driven rather than customer driven. Boy, does it show. And I enjoy some of their intuitive screens on Storage Navigator – who thought it was a bright idea when setting up Universal Replicator that right clicking on a large blank grey area of screen to get something done was intuitive?

On the domestic front, I’ve had to change the shower again. What is it with me and electric showers? They seem to last just long enough to be out of warranty and then *bang*. I’ve also been tracing my family tree, which has been really interesting, and at times my wife has had to suggest to me that I pause for a break, as she thinks I’ve been a tad obsessive with it at times. I think she may be right 🙂

I’ve got quite a lot of ranting to get out of my irritated system. And the odd question to pose. I’ll need to filter it out in small chunks.

Oh, and by the way, if anybody is looking for an experienced Storage professional to cover the Midlands area for a Professional Services Consultancy type role, let me know.

Floating Fag End Bins Spell Disaster

5 08 2007

Been busy for a while, lots to catch up on. Cast your mind back a few weeks to those balmy weeks in June when the Monsoon season first paid a visit to the UK. Needless to say, it got more than a tad moist around the data centre. After a few days of torrential downpour, the woodland at the back of the building had enough of the deluge and decided not to hold on to the rain water any more. We happen to be located in a bit of a dip. We noticed with initial amusement the water levels rising around the back of our building. As it steadily rose over a few minutes great mirth was had as the fag end bins from the smoking shelter decided now was their chance to make a break for freedom, and floated nonchalantly past the window.

Then an alarm rang. The bung in a conduit into one of our computer rooms had given out and water was gushing into it under the false floor. Proved that the water sensors worked! So we decided to start cleanly shutting down affected applications over as quickly as possible to fail over to the remote DR site. This was soon followed by numerous identical queries: ‘has your session hung?’

Smoke had been smelt in the affected computer room, so the EPO had been hit. I was soooo jealous. I’ve always wanted to hit the EPO! Always imagined a scene like the one in ‘Total Recall’ where air is gushing out of the terminal on Mars after Arnie’s fake head exploded, and the brave soldier hanging on for dear life hovers his hand dramatically over the big red button for precarious seconds until he slams it and the emergency air lock doors come down. Instead, it was “<sniff>… I smell burning” followed by a quick poke of a small red button. Real life is so much more boring than the movies!

To cut a long story short, nobody had much sleep for the next 3 days. Some of the techies pulled 24+ hour shifts to restore service. We were plied with pizza – catnip to a techie – and heroism ensued. Got my first real life experience of TSM backupsets as well. Never felt the urge to use them, but someone thought it would be a bright idea to take some remote site backupsets to speed up recovery at our striken site, and I was asleep when it was agreed to use them.

So, if I’ve learnt one thing from this, it’s to make sure that if anybody ever suggests using TSM backupsets for a speedier recovery again, I shall staple their mouth shut. They are a righteous pain in the arse, especially when you’re sleep deprived!

Since then I’ve also had more fun with Brocade’s answer to the MG Montego, the 12000 director. More to come on that soon……

It’s the software, stupid!

21 06 2007

During the past couple of weeks a couple of applications have fubar’d (I think that’s the correct spelling). And of course, we storage peeps are the scapegoats. Well, in each case, it’s definitely been a case of duff code. For starters, a Windows app (recently bought by HP) that shall remain nameless, but I will say its clusters failover by fiddling with SAN ports, toasted its log volumes. Completely buggered. Muggins was on call, so after over an hour of trawling switch and array logs, told them it was the app’s fault. Anyhoo, they recovered. Now the product support are trying to blame the SAN. I’ve put on my teflon coat and told them to write their application properly to handle problems. I’m a properly trained programmer, on a proper platform (that’s a mainframe to you script kiddies out there), and as part of your training you’re taught to make your code robust. Robust is never an adjective I’d associate with a Windows app. Windows has its place – on the desktop! Keep it out of the machine room and away from enterprise storage!

Then this week a couple of VMware servers managed to block several ports on one of our USPs by sending duff blocking instructions to it, which also caused great distress to several unix servers as a result. I don’t know the exact ins and outs, but once again badly written drivers cause the shit to hit the fan.

Talking of shit, I’ve just read this hilarious blog entry about the side effects of a new diet pill on sale in the US, which I caught by chance just now. It’s at and is a stark, oily brown warning to those of you, like me, who are of a round disposition, and who, unlike me, wish to change that shape to something more stick like by thinking you can pop a few pills. You can certainly pop a few pills, but in this case you’ll also pop, or to be more accurate, explode out of your arse!

And talking of stupid, I saw a great ad for a premium rate “fleece the gullible, stupid kiddies” text line. It’s very simple. Lonely? Want to know the initial of your soulmate? Well, text us a message, costing £1.50 (that’s $3 US), and we’ll text back a letter of the alphabet! I have to admire the audacity and genius of the person who came up with that. It’s a well known fact that poverty can be a great wealth generator for those who are willing to exploit those in need, but this is living proof that stupidity can be a great wealth generator too for those willing to exploit the clinically stupid!

But what came after that ad was even more priceless. The same company asks you to text them if you’re thinking about reproducing, and for the same price will send you a random name back for the offspring! If you’re even vaguely thinking of doing that, then stop right now, and don’t even think about reproducing. You’ve probably settled down to text for a baby’s name after getting the initial of your future soulmate, so for the sake of humanity, please, please, do not reproduce. All that will happen is that the gene pool will get even shallower, possibly evaporating to a small puddle.

I was in Brighton at the weekend for a union conference. Pretty mundane, but I saw a great t-shirt slogan, which I’m thinking as adopting as one of my mottos. I’ll leave you with this its poignant message:

“Get in Shape. Round is a shape”