This is an archived page of the 2000 conference


Linux Supercluster Users Conference

Keynote, Applications Track, Clusters, Building a Computational Grid, & Networking Abstracts


Title Linux and Commodity PC Clusters: The HPCC Phase Change
Author Dan Reed
Author Inst NCSA, University of Illinois
Presenter Daniel A. Reed
Abstract A revolution is underway! Open source software and commodity PC clusters bridge the HPC computing divide, uniting both developers of community application codes and system software researchers. The phase change from proprietary systems to commodity HPC clusters with extraordinarily high performance is here -- ride the exponential!
Title Linux and Linux Clusters at IBM Clusters: The HPCC PHase Change
Author Dave Turek
Author Inst IBM
Presenter Dave Turek
Abstract IBM has been heavily engaged in Linux activities for over two years now. This presentation will review IBM's Linux strategy and associated Linux activities, including recent announcements covering the creation of an Open Source Development Laboratory in Portland as well as the investments made in Europe and Asia for Linux porting and development centers. It will also address our more recent activies with Linux clusters and give a perspective on how we expect technology in this space to evolve and the solutions and services IBM will supply.

Applications Track Abstracts:

Applications Session I
Title Comparing Clusters and Supercomputers for Lattice QCD
Author Steven Gottlieb
Author Inst Indiana University
Presenter Steven Gottlieb
Abstract The MILC family of codes for studying Quantum Chromodynamics (QCD) are highly portable and have been run on supercomputers such as the Cray T3E/D, Origin 2000, IBM SP, Intel Paragon and HP Exemplar. More recently, they have been run on clusters such as Roadrunner, Los Lobos, the NT Supercluster at NCSA and the Teracluster at LLNL. This talk will describe the potential performance bottleneck and benchmarks on several of the systems mentioned above. Prospects for future clusters may be discussed.
Title Project Vista Azul: The Development of a Hybrid UNIX "Hypercluster"
Author Thomas More Jackman
Author Inst IBM Research
Presenter Thomas More Jackman
Abstract Currently in progress is a collaborative research endeavor between The University of New Mexico and IBM to construct a heterogeneous infrastructure to do concurrent scientific and visual computing. It is called Project Vista Azul. The project seeks to create a unified computing resource from a hybrid of technologies which includes a cluster of IBM SP nodes running the AIX version of UNIX and another cluster of IA32 Netfinity nodes running Redhat Linux. This heterogeneous composite of hardware, software, and networking has been referred to as a "hypercluster". The project seeks to investigate the merits of leveraging the best of proprietary and commodity hardware and software to do distributed computation and scalable visualization and to address the issues of interoperability between the components. It is seen as a prototype for the productive integration of the Linux operating system into existing UNIX environments. Included in Project Vista Azul is a novel technology, called the Scalable Graphics Engine, which is used for providing synchronous, high bandwidth streaming of rendered pixels from nodes participating in distributed visualization out to an immersive system of displays.
Applications Session II
Title Automatically Tuned Linear Algebra Software (ATLAS)
Author Clint Whaley
Author Inst University of Tennesee, Knoxville
Presenter Clint Whaley
Abstract Geophysical models of seismic sound wave propagation, solid state convection in the This talk will describe the ATLAS (Automatically Tuned Linear Algebra Software) project, as well as the fundamental principles that underly it. ATLAS is an instantiation of a new paradigm in high performance library production and maintenance, which we term AEOS (Automated Empirical Optimization of Software); this style of library management has been created in order to allow software to keep pace with the incredible rate of hardware advancement inherent in Moore's Law. ATLAS is the application of this new paradigm to linear algebra software, with the present emphasis on the Basic Linear Algebra Subprograms (BLAS), a widely used, performance-critical, linear algebra kernel library. Further information is available at:
Title DynaProf and PAPI: An Object Code Instrumentation System for Dynamic Profiling
Author Phil Mucci
Author Inst IBM ACTC/University of Tennessee, Knoxville
Presenter Phil Mucci
Abstract This talk will introduce PAPI and an associated tool called DynaProf. PAPI, the Perfomance Application Programming Interface is a portable library that gives the developer sophisticated access to the hardware performance counters present on a variety of Microprocessors, from Pentiums to Powers. Statistical profiling and sampling is supported as well as traditional aggregate counts. See for more information.

DynaProf is a tool that uses DynInst from The University of Maryland. DynInst allows the developer to directly modify object code through a C++ interface. DynaProf uses DynInst to insert modular instrumentation at key points in the code. This instrumentation is in the form of loadable modules to generate profiles based on wallclock time, hardware counters or sampling of the PC for input to a variety of reporting tools.
Title The Interfacial Behavior of Liquid Water
Author Paul Alsing, Will Matthews, Cara Slutter, Evangelos Coutsias, John McIver
Author Inst University of New Mexico
Presenter Paul Alsing
Abstract The important properties of water stem from its unique structure and the way this structure allows water to interact with electrically charged molecules. Such properties are thought to arise from the ability of water to form tetrahedrally coordinated hydrogen bonds. Water is unusual as both a liquid and a solvent. The insertion of nonpolar solutes into water is strongly unfavorable, called the ''hydrophobic effect'', while polar molecules dissolve freely and are characterized as ''hydrophilic''. It is thought that the physics of the hydrophilic and hydrophobic reaction is strongly affected by interfacial curvature. This work investigates the surface arrangement and orientation of water molecules near flat and curved hydrophobic surfaces. The effects imposed by the hydrophobic surface on surface arrangement and orientation of water molecules represent the balance between the tendencies of water to maximize the number of hydrogen bonds on the one hand and maximize the packing density of molecules on the other.

To facilitate the investigation of water our research utilizes a 3D version of the parallel force decomposition molecular dynamics algorithm [1,2] to model the effects of a large number of water molecules interacting amongst themselves and the hydrophobic surface. MD simulations involve repeated calculations of pair-wise interactions between constituent atoms. These calculations can be performed independent of each other and are thus ideally suited for parallelization. We use the recently developed force decomposition algorithm wherein the pair-wise force interactions of the form Fi,j are evenly distributed over available processors. Here, Fi,j is the force on particle i due to particle j. In contrast to atom decomposition and spatial decomposition techniques, this method affords ease of load balancing and performs well for intermediate number of atoms even for irregular geometries. The use of MPI communication splitting routines allows independent collective communication between user defined groups of processors. This feature is used to split the processors into row communicators (the ith index of the force) and column communicators (the jth index of the force). The interactions between the molecules are calculated in a separate module and can be of van der Waals or any other user specified type. For our model of water we use a Hartree-Fock pair potential [3] and the hydrophobic surface is represented as a potential formed from integrating the Lennard-Jones interaction of a single water molecule with all the molecules in the interfacial surface [4]. In this work we investigate (1) water between two infinite slabs in the z-direction with periodic boundary conditions in the x and y directions, and (2) hydrophobic spheres of various radii and of uniform density imbedded in the water.

Molecular dynamics simulations tend to produce copious amounts positional data in exquisite detail. Visualization therefore becomes an important tool, not only to observe results but also to monitor and analyze the real time calculations. In this talk we will describe our work in utilizing OpenDX [5] (IBM's Data Explorer, now open source) as a tool transfer data to and from an executing simulation for real- time visualiztion and computational steering. We have developed DX TCP/IP import and export modules that allow data to be operated upon and visualized in either DX itself or a third party visualization environment. We demonstrate the later by using these modules to visualize our MD simulation in DX and in Flatland [6]. Flatland is an open development shell for the creation of interactive, immersive visualizations. New data objects are easily added to the environment, allowing the scientist to locomote around the data from within a virtual vessel. This allows the researcher to directly interact with the data, and to control any simulation in real time that is producing the data.

  1. S. Plimpton, Fast Parallel Algorithms for Short-Range Molecular Dynamics, J. Comp. Phys. 117, 1
    (1995); S. Plimpton and B. Hendrickson, A New Parallel Method for Molecular Dynamics
    Simulation of Macromolecular Systems, J. Comp. Chem. 17, 326 (1996).
  2. P.M. Alsing, E. Coutsias and J. McIver, "The Interfacial Behavior of Liquid Water Near
    Hydrophobic Surfaces: a Parallel Force Decomposition Molecular Dynamics Code," in High
    Performance Computing in Engineering VI, Eds. M. Ingber, C.A. Brebbia and H. Power, WIT
    Press, Southhampton, Boston (2000).
  3. F. Franks, Ed. Water: A Comprehensive Treatise, Plenum Press, N.Y., 1972-1982, Vol 6, p317.
  4. C. Y. Lee, J.A. McCammon and P.J. Rossky, The Structure of Liquid Water at an Extended
    Hydrophobic Surface, J. Chem. Phys. 80, 4448 (1984).
  6. Flatland was created at the University of New Mexico's
    Albuquerque High Performance Computing Center by Dr. Thomas P. Caudell, Electrical
    Engineering and Computing Engineering Department,
Applications Session III
Title High-Resolution WeatherForecasting on Linux Superclusters
Author Daniel B. Weber
Author Inst University of Oklahoma
Presenter Daniel B. Weber
Abstract The history and future potential of storm-scale weather prediction with ARPS: Are we on the verge of high-resolution continental scale weather prediction using cluster technology?

The speaker will present a brief history regarding the development of the Advanced Regional Prediction System (ARPS) including the forecasting joys and failures behind the web-page glamour. The second half of the presentation will focus on the details of running a forecast system with an emphasis on the performance and optimization issues. Compiler developers and microprocessors designers are encouraged to attend.
Title Binary Black Hole Simulations on Clusters
Author David Neilson
Author Inst University of Texas
Presenter David Neilson
Abstract With the advent of gravitational wave detectors (LIGO, VIRGO, GEO, TAMA, LISA), understanding physical systems which exhibit strong gravitational fields is not only of mere mathematical interest, but also of fundamental astrophysical importance. Prior knowledge of the waveforms that would be measured by these detectors would expedite the search and characterization of these strongly gravitating systems. This possibility opens up a new window to the universe through which researchers will be able to obtain answers to important questions, ranging from properties of the sources that produced the waves to the very origin of our universe. Prime candidates for producing detectable sources of gravitational waves of sufficent strength waves that could be detected are binary systems containing neutron stars and/or black holes. The dynamics of strong non-linear fields associated with these systems can only be accurately modeled by solving Einstein equations in their full generality. Owing to the involved nature of the equations and the generality of the prolbem, numerical techniques must be used to obtain the solutions. We describe our efforts aimed towards obtaining an simulation of binary black hole systems and our usage of RoadRunner and LosLobos for such a task.
Title EIGER on Cplant
Author Joseph D. Kotulski, Roy E. Jorgenson, William A. Johnson, and Larry Warne
Author Inst Sandia National Laboratories
Presenter Joseph D. Kotulski
Abstract EIGER ( Electromagnetic Interactions GEneRalized) is an electromagnetics code that solves the integral equation description of Maxwell s Equations. It is formulated in the frequency-domain and designed to integrate a variety of analysis methods into a single package. The code suite consists of a pre-processor, computational engine, and post-processor. EIGER is written in Fortran 90 and uses object-oriented design that results in a computational engine that is general, flexible, and extendable. The capabilities to treat surfaces, wires, wire-surface junctions, multi-layer media, periodic structures, and dielectric media are present as well as the flexibility to choose different integral equation operators. In addition coupling models have been introduced that enable the analysis of thin-slot coupling into cavities.

The solution process consists of generating a matrix equation using the Method of Moments. This complex-dense matrix is solved using LU factorization configured and optimized for parallel platforms.

The solution to some representative problems that expose the code capabilities will be presented as well as performance on the Cplant parallel platform.

Clusters Abstracts:

Clusters I
Title Scalable Cluster Interconnect: Overview and Technology
Author Charles L. Seitz
Author Inst Myricom, Inc.
Presenter Charles L. Seitz
Abstract Physics and technology govern the speed of "tightly coupled" computing devices. Within a given technology, the smaller the device, the faster and more efficiently can it function. In the interests of performance, efficiency, and reliability, large computing systems must necessarily be composed of parts that can operate relatively independently (concurrently), and can tolerate some latency in the communication between these parts. Computing clusters apply this "principle of locality" at the scale of parts that are programmable computers, referred to as hosts or nodes, tied together with a message-passing network. Clusters are scalable in two senses. They are arbitrarily scalable in the number of host computers, and are thus open-ended in aggregate peak performance. One does not need to reduce the clock rate of the host computers as the diameter of a cluster grows. However, the utilization or efficiency of the hosts may be limited by the way in which a computation is distributed amongst processes -- the same principle coming into play at the higher level of processes as parts. Clusters also scale well with technology. Inasmuch as the cluster architecture and distributed programming respect the principle of locality, clusters will directly and easily exploit future advances in circuit technology.
Title The Linux/IA64 Project
Author Khalid Aziz
Author Inst Hewlett Packard Co.
Presenter Khalid Aziz
Abstract IA-64 is the next generation high performance 64-bit architecture designed jointly by HP and Intel. This talk will focus on HP Labs' effort on porting Linux to IA-64 architecture. We will take a look at the IA-64 architecture highlights and the Linux/ia64 project. We will delve into IA-64 Linux kernel design and take a look at the tools used during the porting process.
Clusters II
Title Cplant Architecture
Author Rolf Riesen
Author Inst Sandia National Laboratories
Presenter Rolf Riesen
Abstract The Computational Plant (Cplant(tm)) project at Sandia National Laboratories started out as a small research project with a 96-node cluster. Over the last three years new hardware has been added and we are about to deploy a ~1600-node Linux cluster later this Fall. In this talk I will describe the hardware architecture we have developed and the design decisions we have made to scale a cluster to this size. The talk will go into details on how the hardware and system software is organized so it can be managed as a single machine even though it consists of thousands of off-the-shelf components.
Title Clusters for a Multi-User, Production-Computing Environment
Author Robert Eades
Author Inst Pacific Northwest National Laboratory
Presenter Robert Eades
Abstract The Pacific Northwest National Laboratory is active in developing and porting so ftware to increase the production quality computing capabilities for clusters in a multi-user environment. The primary PNNL Linux cluster system, a 194-processo r system, is located in the Molecular Science Computing Facility. Cluster software development activities encompass
  1. system infrastructure software (scheduling, resource allocation management, etc.),
  2. parallel programming tools and libraries (Global Arrays),
  3. parallel scientific applications software (NWChem, NWGrid/NWPhys, etc.), and
  4. problem solving environments (Ecce).
    An overview of these activities will be presented, along with performance results.
Clusters III
Title The Flatland Visualization Environment
Author Thomas P. Caudell
Author Inst University of New Mexico
Presenter Thomas P. Caudell
Abstract Flatland is a visualization/virtual reality application development tool developed at the University of New Mexico. It allows software authors to construct and users to interact with arbitrarily complex graphical and aural representations of scientific data sets and complex software systems. Flatland is written in C/C++ and uses the standard OpenGL graphics language extensions to produce the 3D graphics. In addition, Flatland uses the standard GLUT library for window, mouse, and keyboard management, and therefor supports any type of display technology. Flatland is multithreaded and uses dynamically linked libraries (DLL) to load applications that construct or modify its virtual environment (VE). In addition, Flatland is currently being parallelized to run on Linux clusters, and to interact with the NSF Access Grid. This talk will give an overview of the features Flatland , give examples of applications that use Flatland, and discuss current research issues.
Title PowerPC Linux
Author Gregory P. Rodgers
Author Inst IBM
Presenter Gregory P. Rodgers
Abstract Of the many architectures that Linux runs on, PowerPC is the most scaleable platform. Linux on PowerPC spans from embedded processors through high end 64-bit SMP supercomputers. The variety of features supported by PowerPC Linux is evident in the many Linux configuration options found in the arch/ppc branch of the Linux source tree. This talk will sort through the key features of PowerPC Linux. These features include support for POWER3 and POWER4 that will be available with Linux 2.4.

Building a Computational Grid

Title Clustering Ohio: From the TeraFlop to the Desktop
Author Al Stutz
Author Inst Ohio Supercomputer Center
Presenter Al Stutz
Abstract OSC has been active in cluster computing since 1994, building a total four clusters so far with the largest system (132 processors) being shown on the exhibit floor of Supercomputing 1999 in Portland, Oregon. OSC develops hardware acquisition plans in concert with the Ohio user community's needs. OSC and the Ohio user community have recently set a more aggressive direction in cluster computing. A new program has been created to encourage faculty throughout the state to build a local clusters in their labs to compliment the central cluster at OSC. This concept centers around a state wide software licensing program and a hardware grant program.

OSC will present several user success stories, including experiences and performance metrics.


Title Portals 3.0: A High-Performance, Message-Passing Layer Supporting Application-Bypass
Author Arthur B. (Barney) Maccabe
Author Inst University of New Mexico
Presenter Arthur B. (Barney) Maccabe
Abstract OS-Bypass involves putting the policies of the host operating system on the network interface card (NIC) so that communication need not incur the costs associated with interrupting the host processor. Application-Bypass takes OS-Bypass to the next step, putting the policies of the application on the NIC. This talk will review the common themes that have evolved in high-performance networking over the past several years, including zero-copy message passing and OS-bypass. We will then present examples motivating Application-Bypass and describe the Portals 3.0 interface that has been designed explicitly to support Application-Bypass.
Title MPICH on Clusters: Future Directions
Author Rajeev Thakur
Author Inst Argonne National Laboratory
Presenter Rajeev Thakur
Abstract We are redesigning and reimplementing MPICH from scratch to support newer networking technologies such as Myrinet and VIA, multithreading, MPI-2 functionality, and also for better performance and scalability of existing communication methods such as TCP and shared memory. Key to the new design is a new internal abstract-device interface for MPICH called ADI-3. This talk will describe the design of the new version of MPICH and provide an update on the implementation status.
Title M-VIA and MVICH: Status and Future Plans
Author Michael L. Welcome
Author Inst NERSC Future Technologies Group
Presenter Michael L. Welcome
Abstract The Virtual Interface Architecture (VIA) is an industry standard that enables high-performance communication on a cluster. It provides user-level zero-copy data transfers, enabling low latency and high bandwidth communication. M-VIA is a modular implementation of VIA for Linux that provides high performance and ease of portability to new network interfaces. MVICH is an MPICH-based implementation of MPI using VIA at the transport layer. This talk will discuss the current status of both software projects and indicate future directions.

Both M-VIA and MVICH are available under a BSD-style license. For more information and instruction on how to download the current versions, please see: