System Announcements

Current operations announcements - Recent changes in systems, software, and support

Scheduled May 9, 2008 down time cancelled
04/30/2008 - The normally scheduled MPP2 downtime for May 9th will not occur. Queues will be kept open and the machine will be available.
Scheduled NETWORK downtime: Saturday, 5/17/2008 through Sunday, 5/18/2008
04/30/2008 - As part of PNNL's ongoing commitment to provide reliable, secure, and high-performance networking systems, the PNNL Network infrastructure will be upgraded over the weekend. There will be no access to MSCF computers between Saturday, May 17, through Sunday, May 18th. During this time users of the MSCF will not have network access. The network outage will not affect running jobs. Jobs waiting in the queue will start as nodes are available.
DTEMP Data will be lost when MPP2 is turned off.
04/30/2008 - MPP2 will be unplugged after users have been migrated to the Phase-1 Chinook system. Any data on /dtemp will be lost. Please be sure to copy your files to your local file system or the NWfs Archive. The exact date will be announced when available.
Next Scheduled MPP2 downtime: April 11, 2008
04/10/2008 - System administrators will stop the queues at 2:00 p.m. (Pacific Daylight Time) on Friday, April 11th. The system will probably be available by Friday evening. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime several switch components will be replaced. Please send any concerns or any questions to mscf-consulting@emsl.pnl.gov
Scheduled March 14, 2008 down time cancelled
03/14/2008 - The normally scheduled MPP2 downtime for March 14th will not occur. Queues will be kept open and the machine will be available.
Scheduled NETWORK downtime: March 14, 2008 down time
03/14/2008 - As part of PNNL's ongoing commitment to provide reliable, secure, and high-performance networking systems, the PNNL Network infrastructure will be upgraded this weekend. Various systems will be affected at different times between Saturday, March 15, beginning at 6:00 a.m. through Monday, March 17 at 3:00 a.m. (PDT). During this time users of MPP2 may not have network access to MPP2e. The network outage will not affect running jobs. Jobs waiting in the queue will start as nodes are available.
Next Scheduled MPP2 downtime: Feb. 8, 2008
02/07/2008 - System administrators will stop the queues at 2:00 p.m. (Pacific Standard Time) on Friday, February 8th. The system will probably be available by Friday evening. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime a new image will be installed with a few bug fixes. Please send any concerns or any questions to mscf-consulting@emsl.pnl.gov.
NWChem Version 5.1 now default version on MPP2
01/30/2008 - Version 5.1 is now the default version of NWChem on MPP2. The "llnw" script will use version 5.1. For those who need to run version 5.0, the "llnw50" script is still available. All users are encouraged to use version 5.1.
Next Scheduled MPP2 downtime: Jan. 11, 2008
01/08/2008 - System administrators will stop the queues at 8:00 a.m. (Pacific Standard Time) on Friday, January 11th. The system will probably be down for at least 12 hours. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 0800 hours Friday will not be started until the outage is complete. During this downtime special testing will take place. Please send any concerns or any questions to mscf-consulting@emsl.pnl.gov.
Scheduled December 14, 2007 down time cancelled
12/12/2007 - The normally scheduled MPP2 downtime for December 14th will not occur after all. Queues will be kept open and the machine will be available. We anticipate a January 11th downtime.
Next Scheduled MPP2 downtime: Dec. 14, 2007
12/10/2007 - System administrators will stop the queues at 8:00 a.m. (Pacific Standard Time) on Friday, December 14th. The system will probably be down for about 12 hours. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 0800 hours Friday, will not be started until the outage is complete. During this downtime special testing will take place. Please send any concerns or any questions to mscf-consulting@emsl.pnl.gov.
MPP2 is Back on Line
11/13/2007 - MPP2 is began running batch jobs early this morning. The log in nodes are both available as of 5:00 a.m. (PST). Not all nodes are available for very large jobs. We continue to replace components as necessary to restore full functionality. Again, we apologize for any inconvenience this causes.
MPP2 Still Not Available
11/12/2007 - Electrical Power has been restored to EMSL however not all of the high voltage breakers in the computer room are functioning properly. We are re-routing some of the power in an attempt to get at least part of MPP2 up and running until we obtain a replacement breaker. We apologize for any inconvenience this causes.
Next Scheduled MPP2 downtime: Nov. 9-12, 2007
10/30/2007 - System administrators will stop the queues at 2:00 p.m. (Pacific Standard Time) on Friday, November 9th. The system will probably be down for about 72 hours. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime some security patches will be applied. Please send any concerns or any questions to mscf-consulting@emsl.pnl.gov.
Scheduled October 12, 2007 down time cancelled
10/10/2007 - The normally scheduled MPP2 downtime for October 12th will not occur. Queues will be kept open and the machine will be available. We will be having a scheduled power outage Nov. 9th -12th 2007 during which time MPP2 will be turned off.
MPP2e has a new IP address
10/02/2007 - PNNL is having occasional network connectivity problems causing mpp2e to be unavailable to outside users. We have decided to move mpp2e to a new IP address which will be more reliable. It will take a while to propagate the new IP address throughout the internet. If the standard hostname "mpp2e.emsl.pnl.gov" does not work, users may connect using the new IP address: 130.20.63.104 until sometime after 2000 hours (PDT) when the hostname should again work. If you have any problems, please send an e-mail message to mscf-consulting@emsl.pnl.gov.
Scheduled September 14, 2007 down time cancelled
9/12/2007 - The normally scheduled MPP2 downtime for September 14th will not occur. Queues will be kept open and the machine will be available.
Scheduled August 10, 2007 down time cancelled
8/09/2007 - The normally scheduled MPP2 downtime for August 10th will not occur. Queues will be kept open and the machine will be available.
Scheduled MPP2 downtime: Friday, July 27, 2007
7/26/2007 - The circuit breaker that controls the chilled water for cooling the MPP2 computer room has been tripping frequently over the last couple of weeks. An electrical outage beginning early Saturday morning requires that MPP2 be shutdown during the cooling system outage. Therefore, system administrators will stop the queues at 9:30 p.m. (Pacific Daylight Time) on Friday, July 27th. The system will be unavailable until early Saturday afternoon. Jobs submitted during this week which would not complete by 2130 hours Friday will not be started until the outage is complete. Please report any problems or any questions to mscf-consulting@emsl.pnl.gov.
Scheduled July 13, 2007 down time cancelled
7/12/2007 - The normally scheduled MPP2 downtime for July 13th will not occur. Queues will be kept open and the machine will be available.
Next Scheduled MPP2 downtime: Friday, June 8, 2007
6/6/2007 - System administrators will stop the queues at 2:00 p.m. (Pacific Daylight Time) on Friday, June 8th. The system will probably be down for the full 24 hours. The Lustre file system /DTEMP will be reformatted and all data on that file system will be lost forever. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime some MLIB libraries will be updated and security patches will be applied. Please report any problems or any questions to mscf-consulting@emsl.pnl.gov.
MPP2 back on line
5/31/2007 - MPP2 is again available and the queues are open. The offending switch with bad ports will be replaced during the scheduled June 8th outage.
Unscheduled Outage on MPP2
5/31/2007 - We are currently having a problem with the MPP2 core switch as of 12:30 p.m. (PDT). The queues have been stopped while we work at correcting the problem. Some jobs were affected by this problem. Users can request a refund after MPP2 is back on line. As always, if you have any questions send an e-mail to mscf-consulting@emsl.pnl.gov.
DTEMP to be REFORMATTED: Friday, June 8, 2007
5/22/2007 - Since the kernel and software stack upgrade, performance on the /dtemp file system has been significantly less than before the upgrade. We have identified the problem but to implement the fix we will need to reformat /dtemp. ALL data that has not been saved either on local disk storage or copied to NWfs will be lost during the June 8th scheduled monthly outage. Please send questions to mscf-consulting@emsl.pnl.gov.
MPP2 was down Friday, May 4th from 4:25 pm until 10:40 pm (PDT)
5/07/2007 - A power bump caused one of the UPS systems to blow circuit breakers which cascaded into one third of the nodes loosing power. Running jobs were lost. Refunds are available upon request. The UPS was repaired over the weekend and the system is back to normal. More thorough testing will be done during this Friday's scheduled outage. We apologize for any inconvenience. Please report any problems, questions, or refund requests to mscf-consulting@emsl.pnl.gov.
Codes no longer fail with Signal 7 Bus errors
4/18/2007 - The problem with codes failing with a BUS error "killed by signal 7 was identified and corrected last night. We apologize for any inconvenience. If you still encounter this problem please send e-mail to mscf-consulting@emsl.pnl.gov.
Some Codes Failing with Signal 7 Bus errors
4/16/2007 - Since our MPP2 outage on Friday, April 13th, we have uncovered a problem with relocation of some library files that is causing codes like NWChem and others to fail with a BUS error "killed by signal 7). We are working on a solution and will post when a fix has been implemented. We apologize for any inconvenience.
Next Scheduled MPP2 downtime: Friday, April 13, 2007
4/11/2007 - This month's MPP2 outage will begin early Friday morning instead of the afternoon. System administrators will stop the queues at 7:00 a.m. (Pacific Daylight Time) on Friday, April 13th. The system will be down for a full 24 hours. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 700 hours Friday will not be started until the outage is complete. During this downtime some power changes will be made in the computer room and security patches will be applied. Please report any problems or any questions to mscf-consulting@emsl.pnl.gov.
Scheduled NETWORK Outage: Saturday, March 17th, 8am to 6pm (PDT)
3/13/2007 - Due to continuing upgrades to the data center, the PNNL Network infrastructure will again be unavailable on Saturday, 17-March-2007 beginning at 8:00 a.m. and lasting until 6:00 p.m. (PDT). This NETWORK outage will not affect jobs running on MPP2. You just will not be able to log onto MPP2 to view results or submit new jobs during the ten hour time window (0800 2007-03-17 to 1800 2007-03-17 PDT).
MPP2 Up and Running
3/12/2007 - We have successfully completed the software stack upgrade on MPP2. We are now running under Ret Hat Linux 2.6 with the 9.0 Intel compiler and associated libraries. The new default module is pnnl_env/2.00. We encourage you to recompile your code with the new compiler since we have noticed a decrease in runtimes and an increase is I/O speed. Please see our "Tips" web page (button at top of this web page) for how to run old executables under the old environment. As always, please report any problems or any questions to mscf-consulting@pnl.gov.
Scheduled NETWORK Outages: Saturday, March 10th, 8am to 6pm (PST)
3/7/2007 - As part of PNNL's ongoing commitment to provide reliable, secure, and high-performance networking systems, the PNNL Network infrastructure will be unavailable beginning on Saturday, 10-March-2007, at 8:00 a.m. until 6:00 p.m. (PST) to increase network connection speeds and improve accessibility between EMSL and PNNL networks. This NETWORK outage should not affect jobs running on MPP2. You just will not be able to log onto MPP2 or any other PNNL computer to view results or submit new jobs during the five hour time window (0800 2007-03-10 to 1300 2007-03-10 PST) with intermittent service interruptions until 1800 hours PST.
DTEMP to be Reformatted: Monday, March 12, 2007
3/6/2007 - As part of the kernel and software stack upgrade, the /dtemp file system will be reformatted. ALL DATA ON /DTEMP WILL BE REMOVED. Any data on /dtemp you want saved needs to be moved to either your local disk storage or to NWfs.
Next Scheduled MPP2 Downtime: March 12th
3/5/2007 - Instead of the usual second Friday of the month MPP2 downtime, we will be postponing the downtime until the following Monday. We will be upgrading the system kernel to provide support for more recent versions of compilers, debuggers and software. We have tested the new kernel on the test cluster and found the current executables run well and many show a significant increase in local disk I/O rates. System administrators will stop the queues at 5:00 a.m. (Pacific Daylight Time) on Monday, March 12th. The system should be available by 12:00 noon. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 0500 hours Monday will not be started until the outage is complete. During this downtime the /dtemp file system will be reformatted. ALL data on /dtemp will be DELETED. You will need to back up any /dtemp data you want to keep either to local disk storage or to NWfs. Data on the /home file system will remain unaffected. Please report any problems or any questions to mscf-consulting@pnl.gov
Next Scheduled MPP2 Downtime: February 9th
2/6/2007 - System administrators will stop the queues at 2:00 p.m. (Pacific Standard Time) on Friday, February 9th. The system may be available by late Saturday morning. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime firmware upgrades will be made. WE EXPECT TO DELETE ALL FILES ON "/DTEMP" DURING THE March or April OUTAGE! Please be sure to copy your files to your local file system or the NWfs Archive before that time. Please report any problems or any questions to mscf-consulting@pnl.gov
MPP2 is Back On-line
1/22/2007 - MPP2 is once again available for log in. We apologize for any inconvenience this has caused.
MPP2 Unavailable
1/22/2007 - The PNNL Network infrastructure was upgraded on Sunday morning January 21st. from 6:00-10:00 am (PDT). This NETWORK outage has had an unanticipated effect on MPP2. Jobs are running but it is not possible to log on. We are working to restore access as soon as possible.
Next Scheduled MPP2 Downtime: January 12
1/9/2007 - System administrators will stop the queues at 2:00 p.m. (Pacific Standard Time) on Friday, January 12th. The system may be available by Saturday morning. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime the kernel will be updated which restores the local /scratch disks to their read and write speeds that existed before the memory upgrade. Also tests will be run on Lustre file system /dtemp. Please report any problems or any questions to mscf-consulting@pnl.gov.
MPP2 is Back On-line
12/21/2006 - MPP2 is back on line. The queue is will be inactive for only a short period of time. You can log on and get access to your files. Thank you for your patience during this time.
MPP2 Suffering from Power Surges
12/21/2006 - Freezing rain last night has caused intermittent power loss and surges. We are working to get MPP2 back to normal.
MPP2e Restored to Service
12/20/2006 - Mpp2e was returned to service yesterday evening around 7:30 p.m. (PST). External users can now gain access to mpp2 through mpp2e. We apologize for any inconvenience this may have caused.
MPP2e Temporarily Off-Line
12/19/2006 - We have had to take mpp2e off-line. This does not affect jobs running on mpp2 or people who can log directly onto mpp2. Users who are outside the EMSL firewall will not be able to access mpp2 until mpp2e is back in service. We expect to have mpp2e restored later this afternoon. We apologize for any inconvenience this is causing.
Next Scheduled MPP2 Downtime: December 8th
12/5/2006 - System administrators will stop the queues at 2:00 p.m. (Pacific Standard Time) on Friday, December 8th. The system may be available by Friday evening. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 1400 hours Friday will not be started until the outage is complete. During this downtime the Brocade switches will be updated and security patches will be applied. Please report any problems or any questions to mscf-consulting@pnl.gov
Scheduled Complete Network Outage: Friday 11/24/2006, 8am to Monday 11/27/2006, 6am
11/21/2006 - As part of PNNL's ongoing commitment to provide reliable, secure, and high-performance networking systems, the PNNL Network infrastructure will be unavailable beginning on Friday, 24-November-2006, at 8:00 a.m. through Monday, 27-November-2006, until 6:00 a.m. (PST). This NETWORK outage should not affect jobs running on MPP2. You just will not be able to log onto MPP2 or any other PNNL computer to view results or submit new jobs during the SEVENTY hour time window (08:00 2006-11-24 - 06:00 2006-11-27 PST). So be sure to submit your jobs before the outage.
MPP2 Downtime for November 10, 2006 has been cancelled
11/10/2006 - The normally scheduled MPP2 downtime for November 10th has been cancelled. Queues will be kept open and the machine will be available.
MPP2 Partially Unavailable
10/20/2006 - During the last two weeks, memory on each of the 980 nodes was upgraded to 10 GBytes per node for a total of 9.7 TBytes of RAM on the system. This was completed during last weeks scheduled down time. Early yesterday morning (Thursday, Oct. 19) one of the core switches failed and caused many but not all jobs to stop running. Operations folks began working the problem at 5:00 a.m. PDT to fix the core switch and return the system to full function. By 2:30 p.m. most of MPP2 was working again and the queues were opened. We anticipate a "burn in" time for the new memory modules over the next several weeks. Please report any job failures or other problems to mscf-consulting@pnl.gov and thank you for your patience.
Scheduled Network Downtime: October 15th
10/13/2006 - As part of PNNL's ongoing commitment to provide reliable, secure, and high-performance networking systems, on Sunday, 15-Oct-2006, from 8 am-noon (PDT), we will be performing network upgrades, and therefore, all network services will be unavailable. This NETWORK outage should not affect jobs running on MPP2. You just will not be able to log onto MPP2 to view results or submit new jobs during the four hour time window (08:00 - 12:00 PDT).
Next Scheduled MPP2 Downtime: October 13th
10/11/2006 - System administrators will stop the queues at 2:00 p.m. (Pacific Daylight Time) on Friday, October 13th. The system may be available by Friday evening. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 14:00 hours Friday will not be started until the outage is complete. During this downtime memory upgrades will be completed and security patches will be applied. Please report any problems or any questions to mscf-consulting@pnl.gov.
Next Scheduled MPP2 Downtime: September 8th
9/6/2006 - System administrators will stop the queues at 2:00 p.m. (Pacific Daylight Time) on Friday, September 8th. The system will probably not be available until Saturday afternoon, September 9th. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 14:00 hours Friday will not be started until Saturday. During this downtime some hardware maintenance will be done, the kernel will be updated as well as continued testing of the Lustre file system. Please report any problems or any questions to mscf-consulting@pnl.gov.
Scheduled Network Downtime: August 19-21
8/16/2006 - Due to extensive power and cooling upgrades to the PNNL data center, network services to the EMSL and PNNL servers will be unavailable. MPP2 will be unavailable for 48 hours from Saturday, August 19th from 3:00 am until 3:00 am (PDT) Monday, August 21st. This NETWORK outage should not affect jobs running on MPP2. You just will not be able to log onto MPP2 to view results or submit new jobs during the above time.
Next Scheduled MPP2 Downtime: August 4th
8/4/2006 - The August scheduled downtime will be one week earlier this month to upgrade the MPP2 network firmware. MPP2 will be unavailable until the work is completed. System administrators will stop the queues at 2:00 p.m. (Pacific Daylight Time) on Friday, August 4th. The system may be available by Friday evening. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 14:00 hours Friday will not be started until Saturday.
Scheduled Network Downtime: July 22nd
7/20/2006 - As part of PNNL's continued commitment to provide reliable, secure, and high-performance networks, the PNNL Network infrastructure will be upgraded this weekend. MPP2 will be unavailable for 2 hours on Saturday, July 22nd from 10:00-12:00 pm (PDT). This NETWORK outage should not affect jobs running on MPP2. You just will not be able to log onto MPP2 to view results or submit new jobs during the above time (22:00-24:00 hours PDT).
Next Scheduled MPP2 Downtime: July 14th
7/11/2006 - System administrators will stop the queues at 2:00 p.m. (Pacific Daylight Time) on Friday, July 14th. The system will probably not be available until Saturday afternoon, July 15th. Users will not be able to login during this outage window. Jobs submitted during this week which would not complete by 14:00 hours Friday will not be started until Saturday. During this downtime some hardware maintenance will be done as well as extensive testing of the Lustre file system. Please report any problems or any questions to mscf-consulting@pnl.gov.