Long Distance Relationships are a pain

20 03 2007

We’re currently commissioning a new data centre. It’s about 10km as the crow flies, but can you get decent DWDM links over that relatively short distance? A certain cable company in the news at the moment (I left them for Sky a couple of years ago) is providing the links, but one of them is 60km! At least the data will be well travelled. We’re trying to implement syncronous truecopy between our USP arrays, and you can’t do it with that. Now it looks like we’re going to end up with all our routes instead going over the shorter 15km link, so all it will take is a clumsy oaf with a digger and our SAN links will go down faster than Audley Harrison in his last fight.

At the moment we have two links up, one on each of our fabrics, and one of them is giving so many ‘out of frame’ errors that we cannot propogate fabric changes on it, and I’ve spent most of today trying in vain to get around it. It’s having to deal with crap like this that is out of our hands that makes everything else I’m working on late, and that really pisses me off.




3 responses

21 03 2007

DWDM is a great technology – allowing 4-8 different signals to travel down the same link.

The down side is when you get, say 8 channels going down a 60km link, you’ve created a very wide path indeed.

But you’ve not fixed the latency problem. Under ideal circumstances latency over fibrechannel is about 2ms per kilometer.

2ms per k at 60k is 120ms. That’s each way, there is a return trip as well for each ACK transmission.

Now when you add multiple data paths, the only thing that changes is now instead of having one I/O outstanding, waiting for it’s ACK, you’ve got four or eight.

60k is more than twice what I as an engineer would recomend without some sort of repeater, especialy when you consider that optical cable is not an “ideal” transmission medium.

The speed of light has some profound implications for networking technology. Light, or electromagnetic radiation, travels at 299,792,458 meters per second in a vacuum. Within a copper conductor the propagation speed is some three quarters of this speed, and in a fibre optic cable the speed of propagation is slightly slower, at two thirds of this speed.

At 2/3 the speed of light, latency is actually closer to 3ms/km.

22 03 2007

Slight problem with my math. .0002 seconds is 2 microseconts, not 2 miliseconds.

(I re-posted this comment on my blog and someone corrected me. 😉

25 04 2007

Hi John,

Of course it all depends on how much latency you and your customer considers acceptable but Ive worked on a TrueCopy synchronous environment with two twin tailed data centres. One link was around 90KM and the other being 120KM. If I remember right response times averaged around 20ms.

We were using McData 6140 directors and 2 x Nishan 4100 IPS connected to two Cisco ONS DWDM boxes. The solution was quite susceptible to problems with the DWDM link but that was really down to the nature of synchronous replication and not so much to the distance between sites.

The customer did not require extreme performance from the storage but could not tolerate missing data at the remote site in the event of a disaster. So this long distance solution served them quite well (most of the time).

PS. I was told that telcos often lay their fibre alongside rail tracks in this country (Egnalnd).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: