This is a copy of a question I’ve posed on the HDS forum. Highlights my sad state of mind, and that I can’t switch off when I’ve got a few days off for half term!

One of those odd questions that has been rattling around my brain recently has been about allocating pools for COW and Thin Provisioning. I’m anticipating trialling the former soon, and maybe the latter when I can get my hands on a USP-V.

I’m led to believe that the data is written to the pools in a very wide stripe, i.e. scattered all over it, sounding to me in a very similar fashion to the STK RVAs I was implementing in the mid to late 90s. So, the question is, what is the best practice for allocating the physical LDEVs that make up the pool, assuming Raid 5 7D+1P is the norm for the RGs? For a normal allocation to a host, I’d use a dispersed allocation, e.g. RG 1-1, 1-2, 1-3, 1-1, 1-2, 1-3, 1-1, 1-2, 1-3 in order, with a host based stripe on tape for good measure.

For a pool, would that still be best practice? Does the wide striping algorithm work in a similar fashion to a host based stripe, e.g. something that Veritas might do; or does it really matter?

Another thought in my head is about utilising thin provisioning pools to possibly enhance performance. My thought behind that is that you could use the principle of creating VDEVs across a large pool, but not over allocate the pool, so that you could have 3 layers of striping: a host stripe over a pool stripe over a physical stripe. I like to call that ‘Stout Provisioning’ (fans of Not the Nine O’Clock News will understand my meaning).

Could there be a benefit to that? Has anybody actually tried it? All opinions and experiences shared would be greatly appreciated.

To cut a long story short, my test data was ok. It was how Linux LVM handled the presented disks when I virtualised them, or to be more precise, the order in which they were presented. One of the tests I was doing was to see if it was important to re-advertise the disks after they were virtualised behind the NSC, so I had not presented them in the same order. When I re-virtualised the 9980 disks the third time, I presented them to the NSC in the same order as native to the linux host, and presented them as external Open-Vs. I presented them in that same order from the NSC to the Linux host, and Bob’s your relative, data was accessible and all was dandy. Brown trousers averted!

I then re-created the experiment from the USP-100 that I’m actually going to be migrating data from soon (before you jump up and down and shout ‘a USP to NSC/AMS - are you mad?’ - the answer to which is ‘occasionally’ - this is development data that should not have been there to start with). This time the presented disks are 14Gb Open-Vs, same size as formatted on the NSC and its production external storage on an AMS-1000. That worked a treat, and then I used shadowimage to replicate the data from the virtualised USP-100 through the NSC and out to the virtualised AMS, then re-presented that to the Linux host.

So there you have it. You don’t need to fork out for tiered storage manager (HDS - stop calling it TSM - it’s confusing, there’s already a TSM out there. I know it’s supposed to be called HTSM, with one of the characters in lower case, but everyone always forgets the ‘H’!) if you have a Shadowimage licence. You will need a couple of application outages to do a migration this way, but it can be done and timed to minimise the inconvenience to nothing.

Don’t forget the gotchas: present the old storage out as Open-Vs, and make sure your destination Open-Vs are the same size for Shadowimage to work; make sure you know whether or not the host O/S will need to have the virtualised old disks and subsequent new disks presented in the same order (probably not an issue with Veritas, for example); test, test and test again! I’ll be replicating my testing on AIX, Solaris and HP-UX, some with Veritas, others with native LVM. I’ll let you know how I get on.

Off to pick up the children now from nursery and after school clubs. Expensive business, childcare, and Gordon **** Brown just doesn’t help out enough with the tax breaks. It’s not so bad in the Midlands, but in London it cane cost £800 - £1000 per month for a child when the parents work full time, and a £55 / week tax break is a pathetic pittance. Oh how I wish he’d called that election a few months ago. Can’t wait to see him on the opposition benches. 

Just now I was chugging along with domestic life, masticating on the things on my mind, such as bill paying, feeding the family, making my code more efficient, when my 3 year old daughter nonchalantly ambles by wearing her Disney Princess dress that she got for Christmas and starts rummaging around her room. Then out she trots, happy as a pig in poop, because she’s found her tiara. I stop my thinking (I’ve been known to do that) and realise I’m jealous. Not of the tiara - it’s far too small for my large melon - but that life for her is so simple that her biggest worry is that she can’t find her tiara. I wish life could be that simple.

Talking of simple things, or things that should be simple, I’ve been lab testing array migration by externalising behind the NSC I look after. We already virtualise storage, but have never used it for migration. I recently emptied an old 9980 array, but that was by using host based mirroring. That work’s fine, but the problem is that I had to hand over the mirroring work to our Unix support team, who are so overwhelmed with work that it had to wait its turn. Perhaps being a control freak who does not like to have to rely on others to finish his work has something to do with it as well?

So now I’ve got the old 9980 to myself, I allocated a few LDEVs to my linux box, populated them with crud data, then decallocated it from the server and virtualised them behind the NSC. I’ve done this before in the lab with like for like storage, worked a treat. So I represent the now virtualised LDEVs back to the linus server, and it complains that the disk headers are fubar’d.

I de-virtualise (if there’s such a word) the storage and represent it native. Headers are still fubar’d. Oh dear. Going back through the process, I’m sure I didn’t cock up. I didn’t reformat the virtualised LDEVs. I did, however, present them from the NSC as Open-Vs, whereas on the 9980 they’re Open-Es. I did have the option of presenting them from the NSC as Open-Es, but I didn’t do that because a) I wasn’t expecting anything to be written to the 9980, and b) I wanted to try out using Shadowimage to migrate the data (I know there are other tools to do that, but they cost money). I’m going to redo this next week, this time I’ll virtualise as Open-Es and see what happens. This concerns me as I have seen other bloggers discuss how they’ve virtualised all sort of pre-existing data on a variety of arrays. Maybe I did cock something up? That’s the joy of testing I suppose. I’ve got to migrate data from a USP100 soon, so I’ll try some testing on that too. At least that should be like for like.

One final point before I head off with the family to the park - regarding my previous post, I’ve been asked how can a prospective employer contact me? Good point, seeing as I don’t publish my e-mail address (nothing personal, but I get enough spam). I moderate all posts that come in, so if you want to talk to me, just drop in a reply posting to this article. I’ll take the details, then delete the post so it won’t appear and will remain confidential. And please, no agents touting for CVs. More on that later…..

It’s been a while since my last post. However, I have been getting lots of comments. Looking just now, it was around 35,000! Unfortunately, all spam. Another of life’s little irritations. I had to take down my union branch’s website forum recently because of spammers breaking through the defences. Went away for a two week holiday to Canada (lovely country, by the way) and found some of the site users most distressed at Viagra and various orifice probes being offerred as solutions to the questions they were posing.

I’ve been getting to grips more with our HDS kit recently. Tuning Manager is a bundle of laughs, isn’t it? As was pointed out to me, it was written in the Japanese market place, which is vendor driven rather than customer driven. Boy, does it show. And I enjoy some of their intuitive screens on Storage Navigator - who thought it was a bright idea when setting up Universal Replicator that right clicking on a large blank grey area of screen to get something done was intuitive?

On the domestic front, I’ve had to change the shower again. What is it with me and electric showers? They seem to last just long enough to be out of warranty and then *bang*. I’ve also been tracing my family tree, which has been really interesting, and at times my wife has had to suggest to me that I pause for a break, as she thinks I’ve been a tad obsessive with it at times. I think she may be right

I’ve got quite a lot of ranting to get out of my irritated system. And the odd question to pose. I’ll need to filter it out in small chunks.

Oh, and by the way, if anybody is looking for an experienced Storage professional to cover the Midlands area for a Professional Services Consultancy type role, let me know.

Been busy for a while, lots to catch up on. Cast your mind back a few weeks to those balmy weeks in June when the Monsoon season first paid a visit to the UK. Needless to say, it got more than a tad moist around the data centre. After a few days of torrential downpour, the woodland at the back of the building had enough of the deluge and decided not to hold on to the rain water any more. We happen to be located in a bit of a dip. We noticed with initial amusement the water levels rising around the back of our building. As it steadily rose over a few minutes great mirth was had as the fag end bins from the smoking shelter decided now was their chance to make a break for freedom, and floated nonchalantly past the window.

Then an alarm rang. The bung in a conduit into one of our computer rooms had given out and water was gushing into it under the false floor. Proved that the water sensors worked! So we decided to start cleanly shutting down affected applications over as quickly as possible to fail over to the remote DR site. This was soon followed by numerous identical queries: ‘has your session hung?’

Smoke had been smelt in the affected computer room, so the EPO had been hit. I was soooo jealous. I’ve always wanted to hit the EPO! Always imagined a scene like the one in ‘Total Recall’ where air is gushing out of the terminal on Mars after Arnie’s fake head exploded, and the brave soldier hanging on for dear life hovers his hand dramatically over the big red button for precarious seconds until he slams it and the emergency air lock doors come down. Instead, it was “<sniff>… I smell burning” followed by a quick poke of a small red button. Real life is so much more boring than the movies!

To cut a long story short, nobody had much sleep for the next 3 days. Some of the techies pulled 24+ hour shifts to restore service. We were plied with pizza - catnip to a techie - and heroism ensued. Got my first real life experience of TSM backupsets as well. Never felt the urge to use them, but someone thought it would be a bright idea to take some remote site backupsets to speed up recovery at our striken site, and I was asleep when it was agreed to use them.

So, if I’ve learnt one thing from this, it’s to make sure that if anybody ever suggests using TSM backupsets for a speedier recovery again, I shall staple their mouth shut. They are a righteous pain in the arse, especially when you’re sleep deprived!

Since then I’ve also had more fun with Brocade’s answer to the MG Montego, the 12000 director. More to come on that soon……

During the past couple of weeks a couple of applications have fubar’d (I think that’s the correct spelling). And of course, we storage peeps are the scapegoats. Well, in each case, it’s definitely been a case of duff code. For starters, a Windows app (recently bought by HP) that shall remain nameless, but I will say its clusters failover by fiddling with SAN ports, toasted its log volumes. Completely buggered. Muggins was on call, so after over an hour of trawling switch and array logs, told them it was the app’s fault. Anyhoo, they recovered. Now the product support are trying to blame the SAN. I’ve put on my teflon coat and told them to write their application properly to handle problems. I’m a properly trained programmer, on a proper platform (that’s a mainframe to you script kiddies out there), and as part of your training you’re taught to make your code robust. Robust is never an adjective I’d associate with a Windows app. Windows has its place - on the desktop! Keep it out of the machine room and away from enterprise storage!

Then this week a couple of VMware servers managed to block several ports on one of our USPs by sending duff blocking instructions to it, which also caused great distress to several unix servers as a result. I don’t know the exact ins and outs, but once again badly written drivers cause the shit to hit the fan.

Talking of shit, I’ve just read this hilarious blog entry about the side effects of a new diet pill on sale in the US, which I caught by chance just now. It’s at http://angryaussie.wordpress.com/2007/06/20/miracle-diet-pill-with-teeny-tiny-side-effect/ and is a stark, oily brown warning to those of you, like me, who are of a round disposition, and who, unlike me, wish to change that shape to something more stick like by thinking you can pop a few pills. You can certainly pop a few pills, but in this case you’ll also pop, or to be more accurate, explode out of your arse!

And talking of stupid, I saw a great ad for a premium rate “fleece the gullible, stupid kiddies” text line. It’s very simple. Lonely? Want to know the initial of your soulmate? Well, text us a message, costing £1.50 (that’s $3 US), and we’ll text back a letter of the alphabet! I have to admire the audacity and genius of the person who came up with that. It’s a well known fact that poverty can be a great wealth generator for those who are willing to exploit those in need, but this is living proof that stupidity can be a great wealth generator too for those willing to exploit the clinically stupid!

But what came after that ad was even more priceless. The same company asks you to text them if you’re thinking about reproducing, and for the same price will send you a random name back for the offspring! If you’re even vaguely thinking of doing that, then stop right now, and don’t even think about reproducing. You’ve probably settled down to text for a baby’s name after getting the initial of your future soulmate, so for the sake of humanity, please, please, do not reproduce. All that will happen is that the gene pool will get even shallower, possibly evaporating to a small puddle.

I was in Brighton at the weekend for a union conference. Pretty mundane, but I saw a great t-shirt slogan, which I’m thinking as adopting as one of my mottos. I’ll leave you with this its poignant message:

“Get in Shape. Round is a shape”

Now is the time to rejoice! I enabled ‘ISL R_RDY’ mode on the four ISLs between sites, enabled long distance with sufficient buffer credits for 30km at full packet size (my links are either 16km or 18km, depending on who I’m talking to) and sat back to watch the portperfshow. I thought I was going to wet my pants when I saw the speed on each link leap from 60mb/s to 150! I reset the port stats, and noticed that I was still getting a number of ‘out of buffer credits’ errors. I’m not sure if I can get stats on the average packet size going through (can anyone advise?), so I took a stab and increased to 40km and reset the stats again. My self pleasure went into overload as the speed topped 170mb/s!

To get a tape copy to run from the remote site to production, we deleted a couple of copy pool volumes in TSM and re-ran a copy storage pool with multiple drives, so I had tape traffic going in both directions, and soon the throughput was topping 200mb/s!

For once, I am happy, and needed a good hosing down to calm myself. I just need to get some long distance licences for the 4100s I have spare so I can move the other site links (we have a second remote site linked to another machine room) from the creaking 12000s and I will be truly joyful.

On the home front, so far there have been no further steaming gifts left in the living room, just the odd puddle!

Had an interesting meeting today (don’t say that often) with a man from Nortel who summed up my long distance ISL problem nicely - I need to use ISL R_RDY mode with the long distance settings on the ISLs across our DWDM links (fronted by Nortel MOTRs). Apparently the Brocade VC Link Init protocol is too sensitive to work with their kit, so I shall be trying that out early next week. It may even cure my lack of throughput, which I suspect is because I’m not getting a huge amount of full size packets going through the links, hence exhausting the buffer credits.

Also found out that with Condor ASIC equipped Brocade switches (4100, 48000 etc) I should not expect to see an even spread of throughput across ISL trunks, as theire algorithm loads a single link up to around 70% first before starting to spread out further, to cut out unneccessary frame splits and re-joins. Makes sense I suppose. 

There was a hint of a beige tint to the air when I got home tonight. My daughter has started potty training, and not long before I came home had decided that she couldn’t get to the potty in time, so left what the Big Yin (that’s Billy Connolly, one of the funniest people on the planet) may describe as a ‘wee beige jobby’ in the middle of the living room! At least she went and washed her hands afterwards!

Which leads nicely to the fact the I’m on call this weekend and already had my first call within half an hour of getting home

Thanks for spotting my typo, I’ve had a good chuckle. About the only chuckle this week as this week has been my week on the Operations Bridge and be on-call, where I get the pleasure of sitting in uncomfortable chairs during the day, having to respond to inane flapping and bleeting, and then get woken up in the middle of the night for more flapping. Plus I’ve been in pay negotiations, and to thanks us for all our hard work my employer wants to offer us a pay cut in real terms. Nice. Well, if they want a fight, they’re going to get one. This is a good advertisment for union membership - in parts of our business where there is a high percentage of union members, pay offers are high, and where it’s not, they’re low. Hopefully I’ll be persuading more non-union members to join over the next few weeks, it’s time for their free ride to come to an end if they want to get a decent pay deal.

As you might be able to tell, I have a pretty low opinion of those who won’t join, but are happy to take the benefits that are gained from the subs paid by members, and complain that they haven’t had their pay rise yet. It’s like people who say “I didn’t vote - I couldn’t be bothered” or “it doesn’t make a difference”. Democracy is a priviledge, not a right, and if you don’t vote, you have no right to complain. If you don’t like the choice, spoil your ballot paper, express your displeasure. I have a lot of respect for those who’ll do that - it’s democracy in action. People are dying around the world every day in places like Burma, China and Zimbabwe to have what we have.

Now I’ve finished venting, on the SAN side, we’ve worked around the ISL hit for now, though we’ve rigged up another server to try various load tests. I’ve tried the suggestion about changing trunk masters etc., but I still think there’s a problem. The loads across these ISLs are just not balancing, i.e. if I’ve got 100mb throughput, I’d expect to see close to 25mb on each of the four trunks, yet it seems to load mostly onto one connection. Strange. Next week shall see me pulling cables (orange ones ), adding more, changing trunks and generally hiding away in the server room.

On the plus side, I’ve got Brocade Fabric Manager to play with for the moment, which seems useful, though like any Java app it sucks resource like an Electrolux (for those of you not British or of my generation, there was an ad in the 80s for a hoover that said ‘Nothing sucks like an Electrolux’ - naturally in my youth we would use that as a challenge for the ladies my friends and I would date). Anyhoo, I digress, so returning from the dirt track to the highway next week we’ll also be trying out some tuning of AIX server fibre cards, as we seem to be maxing out the throughput of our new TSM servers prematurely when duplicating LTO3 tapes, and we believe got the TSM settings optimised.

So, thanks for the input. I’m off to bed soon (after the end of the Eurovision Song Contest) as my wife’s away this weekend so I’m looking after the children (5 and 2) on my own, and they’ll be wanting their breakfast bright and early. At least it puts my storage problems into perspective!

My shower is working again. It started working again the next day. Maybe a dodgy connection in the switch? Who cares, showers all round again.

I have a cold. My wife had it for two days, but took a day off work sick to aid recouperation, which did the trick. Then I caught it. Two weeks later I still have it. I’m too busy to be sick, same old story, so I just carry on at work as normal. At least I can share my misery. When I’m miserable, I want to share it. When I’m happy, I’m selfish, it’s all mine!

Over the past few weeks lots of new AIX servers have been going in. Almost without fail, for each one I’ll hear the whinge from the Unix guys that one of the partitions won’t log onto the SAN. “Must be a SAN problem” they cry. So each time I’ve gone into the server room, have seen no lights, so checked the cables, flipped the ends, tried other ports and even thrown 30m cables across the room to check whether or not it’s the patch panels, and every time I’ve turned round and said “it’s your server - check the fibre card”. And each time I’ve been right. Shit like this just never happenned when I worked with mainframes. Looks to me like we’re getting more and more shoddy server fibre cards emerging from wherever they come from in the Far East. IBM need to get their act together.

However, a curious performance problem hit recently. Chronic performance from a new server/app on one of our shiny new USPs. We did the usual checks, dispersed allocation across RAID groups, port contention etc. Noticed that the filesystem was not striped. So that was sorted. But problem was still there, seeing 30ms responses. Head scratches all round. It was noted that the server was connected to one 48000, the storage on another, both linked by a under-utilised 8gb trunk. I beefed the trunk to 16gb, still bad. To cut a long story short, we proved the point the ISL was the bottleneck, server and storage on same switch = 3ms response. Now that has really baffled me. Our normal standards are to try and host servers and their storage as best we can on the same switch, though it’s not always possible, but I’ve never seen latency like this. The switches are close to each other, linked by 9m cables between ISLs, and I’d expect no more than 20 micro seconds each way latency, not milli-seconds! Been hitting the books again, but baffled once again. If any of you kind techies out who might be taking your valuable time to enjoy my rantings have any thoughts, I’d appreciate any insight, as I have appreciated the comments I’ve received to date.