Received: from cernxchg46.cern.ch ([137.138.137.154]) by cernxchg08.cern.ch with Microsoft SMTPSVC(6.0.3790.1830); Tue, 24 Mar 2009 14:13:26 +0100 Received: from cernfe10.cern.ch ([137.138.140.34]) by cernxchg46.cern.ch with Microsoft SMTPSVC(6.0.3790.1830); Tue, 24 Mar 2009 14:13:26 +0100 Received: from cernmx11.cern.ch (137.138.142.198) by cernfe10.cern.ch (137.138.140.34) with Microsoft SMTP Server (TLS) id 8.1.340.0; Tue, 24 Mar 2009 14:13:25 +0100 Received: from mailgate.doe.gov (205.254.128.11) by cernmxgwlb.cern.ch (137.138.142.186) with Microsoft SMTP Server id 8.1.340.0; Tue, 24 Mar 2009 14:13:24 +0100 X-WSS-ID: 0KH0I22-01-2D3-02 X-M-MSG: Received: from hqwss.hr.doe.gov (hqmms4.hr.doe.gov [205.254.132.8]) by mailgate.doe.gov (Tumbleweed MailGate 3.6.1) with ESMTP id 2E65C1D99585; Tue, 24 Mar 2009 09:13:14 -0400 (EDT) Received: from [146.138.8.206] by hqwss.hr.doe.gov with ESMTP (US Dept of Energy SMTP Relay (Email Firewall v6.3.2)); Tue, 24 Mar 2009 09:12:45 -0400 X-Server-Uuid: 7BA904BC-BA52-4865-A377-BC9866E73167 Received: from SCVEXCH1.scdoe.local ([146.138.8.202]) by scnrelay1p.scdoe.local with Microsoft SMTPSVC(6.0.3790.3959); Tue, 24 Mar 2009 09:12:35 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-Class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: US LHCNet RFP and request to DOE for 2009 Date: Tue, 24 Mar 2009 09:12:34 -0400 Message-ID: In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: US LHCNet RFP and request to DOE for 2009 Thread-Index: AcmokbSFywQX+DdKSMWwrj8QuwS9xwDf/WUQABwf+GA= References: From: "Boehnlein, Amber" To: "Gordon, Howard A" , "Gonzalez, Saul" , "Crawford, Glen" , "Petravick, Don" CC: Michael Tuts , "Pripstein, Moishe" , "Whitmore, James J." , Jim Shank , "Rajagopalan, Srini" , "Ernst, Michael" , , , Daniel Marlow , Lothar Bauerdick , Ian Fisk , Harvey Newman Return-Path: Amber.Boehnlein@science.doe.gov X-OriginalArrivalTime: 24 Mar 2009 13:12:35.0379 (UTC) FILETIME=[32BDB830:01C9AC82] X-WSS-ID: 65D603C82TC213992-02-01 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C9AC82.3243B540" Received-SPF: None (cernmxgwlb.cern.ch: Amber.Boehnlein@science.doe.gov does not designate permitted sender hosts) X-CERN-Reverse-DNS: mailgate.doe.gov Keywords: CERN SpamKiller Note: -5 ------_=_NextPart_001_01C9AC82.3243B540 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thank you for your input. =20 ________________________________ From: Gordon, Howard A [mailto:gordon@bnl.gov]=20 Sent: Monday, March 23, 2009 6:50 PM To: Boehnlein, Amber; Gonzalez, Saul; Crawford, Glen; Petravick, Don Cc: Michael Tuts; Pripstein, Moishe; Whitmore, James J.; Jim Shank; Rajagopalan, Srini; Ernst, Michael; smckee@umich.edu; butler@fnal.gov; Daniel Marlow; Lothar Bauerdick; Ian Fisk; Harvey Newman Subject: FW: US LHCNet RFP and request to DOE for 2009 =20 Memo To: DOE Office of High Energy Physics From: Mike Tuts and Howard Gordon Subject: USLHCnet Supplementary Funding Date: March 16, 2009 We are writing to support a supplemental request from USLHCnet to allow them to select a more reliable vendor for their network infrastructure. The recent USLHCnet RFP resulted in 11 responses from vendors. Planning for USLHCnet infrastructure, which provides the primary connection from the CERN Tier-0 center to the USATLAS and USCMS Tier-1 centers, includes just-in-time provisioning of needed bandwidth for renewed LHC operation and path/vendor diversity across the Atlantic. This is well matched to USATLAS needs for a reliable, resilient service that is critical to USATLAS operations. While the responses to the RFP are nominally for the same technical capabilities (number of circuits and bandwidth) there are significant differences between the various vendors in terms of their corresponding Service Level Agreements (SLA's) and the historical reliability (see USLHCnet RFP report, Annex B). The lowest cost vendor, Global Crossing, listed (which also matches the USLHCnet proposed budget) is also the one with the worst SLA and reliability. Global Crossing provides circuits currently used in USLHCNet and their circuits have been significantly less reliable than all other vendors and its SLA is the least favorable in terms of motivating the vendor to provide a highly reliable service. Records associated with link availability monitoring have shown that its two USLHCnet circuits have had link availability of 97.67% and 96.56% during the past 6 months. For a network link this is extremely poor performance and as a result, and despite path redundancy incorporated into the network architecture the U.S. ATLAS Tier-1 center at BNL experienced three major outages or service level impairments since January 2008, lasting between 12 and 24 hours. The three incidents and their durations are listed below 1.) January 23 & 24, 2008; BNL lost connectivity to the Tier-0 center at CERN and most of the Tier-1 centers for 22 hours 2.) November 20, 2008; BNL's bandwidth to CERN was reduced for 14 hours to less than 50% of the nominal capacity 3) November 22, 2008; BNL's bandwidth to CERN was reduced for 18 hours to less than 50% of the nominal capacity With the central infrastructure components associated with the ATLAS-wide production service (PanDA) operated at BNL, ATLAS production lost the ability to fully utilize the computing and storage resources deployed at sites that are part of the ATLAS worldwide distributed computing facility. Losing connectivity between the PanDA submission hosts located in the regions or clouds and the central services means, while jobs that had started to run prior to the outage can continue to run, no new production and distributed analysis jobs can be launched until the connectivity is restored. Results from running jobs finishing during the outage are eventually lost if the outage is taking substantially longer than 12 hours following job completion.=20 Contrary to Global Crossing, network services providers like Level3 and T-Systems are offering their circuits at the availability and reliability level that allows the Tier-1 center to meet its obligations as outlined in the WLCG MoU (a version as of March 2009 can be found at http://lcg.web.cern.ch/LCG/MoU/Goettingen/MoU-Goettingen-4MAR09.pdf. The table on page A3.4 outlines the obligations of Tier-1 centers regarding network service levels in detail). According to the MoU a complete loss of connectivity or a degradation of the service quality by more than 50% is not supposed to account for more than 2% over the period of one year or 7 days in total. As indicated above the implication for the U.S. ATLAS facilities could be significant. Simultaneous loss of links to the Tier-0 center at CERN can result in loss of the complete service. With circuit diversity (path and vendor) this risk is reduced, but with link unavailability rates of ~2% there is still a non-negligible chance for complete service interruption. Even if the Tier-0 to Tier-1 service remains operational, loss of a link can degrade the available bandwidth because of competition/congestion introduced along the alternate path. While the ATLAS production system is designed to buffer through short outages, the facility operations can significantly degrade during outages or slow-downs introduced by circuit outages. This degradation can rapidly worsen as the outage or service impairment duration increases. For such a critical service USATLAS requests more reliable vendors with a more advantageous SLA in place to help ensure a robust Tier-0 to Tier-1 data transfer service. Therefore we fully support USLHCnet's supplemental request to allow them to select such vendors. If that would not be approved U.S. ATLAS would be willing to support its share of the supplementary request from Operations Program funds. --------------- Appendix: Outage related messages On 11/21/08 2:50 PM, "Harvey Newman" wrote: Dear Michael, Very good. I am glad to see the performance restored. Following up on what Artur said: We lost three Global Crossing circuits at once. The entire design of our circuits is predicated on physical diversity. This not supposed to be possible. SURFNet also was hit. Global Crossing is not telling us the whole story. We are older and wiser after this experience, and will act upon it in the course of evaluating our 2009 RFP responses, in January. Best regards Harvey Ernst, Michael wrote: > Artur, > This morning Nikolay modified the routing MED in Vienna such that traffic destined to CNAF is taking the path from NY via Frankfurt instead of going via the USLHCNET/Vienna circuit. > > This has improved the situation dramatically (see below). > > We hope network engineers will be able to sort out the problem with that circuit soon. > > -- > Michael Two days later there was one more outage affecting several circuits provisioned by USLHCNet.This one happened to last for >15 hours.=20 -----Original Message----- From: Artur Barczyk [mailto:Artur.Barczyk@cern.ch]=20 Sent: Saturday, November 22, 2008 12:23 AM To: WAN FNAL; John Bigrow; Ernst, Michael; Bradley, W. Scott; ESnet OPS; DANTE Network Operations Centre; USLHCNet NOC Subject: multiple outage again Hi, USLHCNet is experiencing a dual outage again. Global Crossing technician has arrived at the GC PoP in New York. Loops placed towards our equipment in NY and Chicago indicate it's the same failure we had two days ago. The "repaired" fibre deteriorated within few hours to basically unusable. We have also a ticket open with Ciena. Some VCs which should be and are reported as operational are in fact not. FNAL primary path is down, as well as BNL backup path. I have shut down ESnet-GEANT vlan 2611, as it was not operational. We are treating this as a very serious situation, for both Global Crossing and Ciena. I will update you on the progress of repairs. Cheers, Artur =20 ------_=_NextPart_001_01C9AC82.3243B540 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Re: US LHCNet RFP and request to DOE for 2009

Thank you for your = input.

 

From: Gordon, Howard A [mailto:gordon@bnl.gov]
Sent: Monday, March 23, = 2009 6:50 PM
To: Boehnlein, Amber; = Gonzalez, Saul; Crawford, Glen; Petravick, = Don
Cc: Michael Tuts; = Pripstein, Moishe; Whitmore, James J.; Jim Shank; Rajagopalan, Srini; Ernst, = Michael; smckee@umich.edu; butler@fnal.gov; Daniel Marlow; Lothar Bauerdick; Ian Fisk; Harvey Newman
Subject: FW: US LHCNet = RFP and request to DOE for 2009

 

Memo
To: DOE Office of High Energy Physics
From: Mike Tuts and Howard Gordon
Subject: USLHCnet Supplementary Funding
Date: March 16, 2009

We are writing to support a supplemental request from USLHCnet to allow = them to select a more reliable vendor for their network infrastructure.

The recent USLHCnet RFP resulted in 11 responses from vendors. =  Planning for USLHCnet infrastructure, which provides the primary connection from = the CERN Tier-0 center to the USATLAS and USCMS Tier-1 centers, includes just-in-time provisioning of needed bandwidth for renewed LHC operation = and path/vendor diversity across the Atlantic.  This is well matched to USATLAS needs for a reliable, resilient = service that is critical to USATLAS operations.

While the responses to the RFP are nominally for the same technical capabilities (number of circuits and bandwidth) there are significant differences between the various vendors in terms of their corresponding = Service Level Agreements (SLA's) and the historical reliability (see USLHCnet = RFP report, Annex B).  The lowest cost vendor, Global Crossing, =  listed (which also matches the USLHCnet proposed budget) is also the one with = the worst SLA and reliability. =  Global Crossing provides circuits currently used in USLHCNet and their circuits = have been significantly less reliable than all other vendors and its = SLA is the least favorable in terms of = motivating the vendor to provide a highly reliable service.  

Records associated with link availability monitoring have shown that its = two USLHCnet circuits have had link availability of 97.67% and 96.56% during = the past 6 months.  For a network link this is extremely poor = performance and as a result, and despite path redundancy incorporated into the network architecture the U.S. ATLAS Tier-1 center at BNL experienced three major outages or service level impairments since January 2008, lasting between = 12 and 24 hours. The three incidents and their durations are listed below
1.) January 23 & 24, 2008; BNL lost connectivity to the Tier-0 = center at CERN and most of the Tier-1 centers for 22 hours
2.) November 20, 2008; BNL’s bandwidth to CERN was reduced for 14 = hours to less than 50% of the nominal capacity
3) November 22, 2008; BNL’s bandwidth to CERN was reduced for 18 = hours to less than 50% of the nominal capacity

With the central infrastructure components associated with the = ATLAS-wide production service (PanDA) operated at BNL, ATLAS production lost the = ability to fully utilize the computing and storage resources deployed at sites = that are part of the ATLAS worldwide distributed computing facility. Losing = connectivity between the PanDA submission hosts located in the regions or clouds and = the central services means, while jobs that had started to run prior to the = outage can continue to run, no new production and distributed analysis jobs can = be launched until the connectivity is restored. Results from running jobs finishing during the outage are eventually lost if the outage is taking substantially longer than 12 hours following job completion.

Contrary to Global Crossing, network services providers like Level3 and T-Systems are offering their circuits at the availability and = reliability level that allows the Tier-1 center to meet its obligations as outlined in the = WLCG MoU (a version as of March 2009 can be found at http://lcg.web.cern.ch/LCG/MoU/Goettingen/MoU-Goettingen-4MAR09.pdf.= The table on page A3.4 outlines the obligations of Tier-1 centers = regarding network service levels in detail). According to the MoU a complete loss = of connectivity or a degradation of the service quality by more than 50% is = not supposed to account for more than 2% over the period of one year or 7 = days in total.

As indicated above the implication for the U.S. ATLAS facilities could = be significant.  Simultaneous loss of links to the Tier-0 center at = CERN can result in loss of the complete service. With circuit diversity (path and vendor) this risk is reduced, but with link unavailability rates of ~2% = there is still a non-negligible chance for complete service interruption. =  Even if the Tier-0 to Tier-1 service remains operational, loss of a link can = degrade the available bandwidth because of competition/congestion introduced = along the alternate path. While the ATLAS production system is designed to buffer = through short outages, the facility operations can significantly degrade during = outages or slow-downs introduced by circuit outages.  This degradation can = rapidly worsen as the outage or service impairment duration increases.

For such a critical service USATLAS requests more reliable vendors with = a more advantageous SLA in place to help = ensure a robust Tier-0 to Tier-1 data transfer service.   Therefore we = fully support USLHCnet's supplemental request to allow them to select such = vendors. If that would not be approved U.S. ATLAS would be willing to support its = share of the supplementary request from Operations Program funds.
---------------

Appendix:
Outage related messages

On 11/21/08 2:50 PM, "Harvey Newman" <Harvey.Newman@cern.ch> = wrote:

Dear Michael,

Very good. I am glad to see the performance restored.

Following up on what Artur said: We lost three Global Crossing circuits = at
once. The entire design of our circuits is predicated on physical
diversity.
This not supposed to be possible. SURFNet also was hit.

Global Crossing is not telling us the whole story. We are older and = wiser
after this experience, and will act upon it in the course of evaluating = our
2009 RFP responses, in January.

Best regards
  Harvey

Ernst, Michael wrote:
> Artur,
>  This morning Nikolay modified the routing MED in Vienna such that traffic destined to CNAF is taking the path from NY via = Frankfurt instead of going via the = USLHCNET/Vienna circuit.
>
> This has improved the situation dramatically (see below).
>
> We hope network engineers will be able to sort out the problem with = that circuit soon.
>
> --

>    Michael

Two days later there was one more outage affecting several circuits = provisioned by USLHCNet.This one happened to last for >15 hours.

-----Original Message-----
From: Artur Barczyk [mailto:Artur.Barczyk@cern.ch]
Sent: Saturday, November 22, 2008 12:23 AM
To: WAN FNAL; John Bigrow; Ernst, Michael; Bradley, W. Scott; ESnet = OPS;
DANTE Network Operations Centre; USLHCNet NOC
Subject: multiple outage again

Hi,

USLHCNet is experiencing a dual outage again. Global Crossing
technician has arrived at the GC PoP in New York. Loops placed towards
our equipment in NY and Chicago indicate it's the same failure
we had two days ago. The "repaired" fibre deteriorated = within
few hours to basically unusable.

We have also a ticket open with Ciena. Some VCs which should be and
are reported as operational are in fact not.
FNAL primary path is down, as well as BNL backup path.
I have shut down ESnet-GEANT vlan 2611, as it was not operational.

We are treating this as a very serious situation, for both Global
Crossing and Ciena.

I will update you on the progress of repairs.

Cheers,
Artur



 

------_=_NextPart_001_01C9AC82.3243B540--