What is the objective of TopCrunch?

The objective of TopCrunch is to track the aggregate performance trends of high performance computer systems and engineering software. Instead of using a synthetic benchmark, actual engineering software applications are used with real data and are run on high performance computer systems.

Who runs TopCrunch?

Prof. David Benson at UCSD runs TopCrunch, with support from his students, post-docs, research engineers, and web site support from the Jacobs School of Engineering.

How do I contact TopCrunch?

For web site related problems, click on “webmaster” at the bottom of the page. For issues associated with the benchmarks themselves, contact dbenson@ucsd.edu.

Who funds TopCrunch?

TopCrunch is supported by DARPA HPCS through a subcontract from the USC Information Sciences Institute.

Why do you use production codes for the benchmarks?

The objective is to track the aggregate computing performance available to scientists and engineers. Synthetic benchmarks are usually more highly optimized than production codes because they are much smaller and simpler codes, and therefore don’t reflect real world performance. Research codes are generally not available to the broad range of people we would like to see perform the benchmarks, lack the breadth we are seeking, and the limited resources of the developers means that they can’t provide the support required for non-specialists to use them.

How are the benchmark codes chosen?

The benchmarks are chosen to reflect the types of calculations performed in the mechanical and aerospace communities. Therefore, codes associated with structural dynamics (LS-DYNA), fluid flow (CTH), and materials science (SPaSM) have been chosen. These codes have different challenges to address in terms of domain decomposition, message passing, load balancing, and dynamic memory allocation that makes the comparison of their relative scaling interesting. Additional codes may be added in the future.

How are the benchmarks problems chosen?

The benchmark problems are chosen to reflect current engineering practice in the real world, and to have a structure that allows them to be scaled up as computer performance grows. The problems are not intended to be optimal analyses, i.e., the fastest possible choice of options to achieve a particular solution, because engineers rarely have time to optimize their analyses in real life. For example, the accuracy of the stress distribution in a structural element increases with the number of Gaussian quadrature points, but at the expense of speed. For a given level of accuracy, there is, therefore, a choice that maximizes speed. It is, however, common engineering practice to use more points than absolutely necessary because an inaccurate solution will require rerunning the analysis, which effectively doubles its cost and more than doubles the wall clock time to get an acceptable answer. The same general observation holds true for many other analysis choices to be made.

How do I obtain the benchmark codes?

The codes are available directly from their authors, and are not available on this site. The contact information is given on the code description pages.

How do I run the benchmark problems?

The general command lines required for executing a program are given on the code description pages, along with pointers to additional documentation for the codes.

Can I change the problem data?

No! Aside from the number of processors, no changes are permitted. As discussed, the problems are chosen to represent current industrial practice, and are not optimized for performance. Our goal is to track software and hardware performance, not analyst performance!

How do I obtain technical support for running the problems?

Technical support for running the codes is available through the software providers. If a particular problem downloaded from this site doesn’t run, contact David Benson.

How do I submit the benchmark results?

Mail the ASCII output files from the analyses, which contain the timings, to Administrator. The entire file must be submitted, not just the timings.

How does TopCrunch prevent people from cheating?

We can’t. We try to avoid obvious cheating by requiring the ASCII output files, but we recognize that a determined cheater can readily edit the output. We reserve the right to withdraw or not post results that appear suspicious, and to require benchmarkers to provide additional proof such as the binary restart and plot files. But we explicitly don’t warranty the accuracy of the benchmark results. If you feel that results posted here are incorrect, please contact the Administrator.

Can I run the benchmark problems with my own code?

As one objective of the site is to track software advances, competition between codes is welcomed. However, in the interests of fairness, if you wish to have your results posted, you must

  1. Run a model that is identical in all respects to the benchmark model. This includes, but is not limited to, element formulation, the number of integration points, time step scaling, material models, and domain decomposition (including the specification of contact surfaces).
  2. Be willing to supply us with access to the results for verification and provide a version of the code to a platform of our choice under our control to run your data file and program independently at no cost to us.

Should I buy my computer based on the posted results?

We recognize that these results may be useful to some users in making basic decisions about their computing requirements. However, be aware of the following caveats:

  1. The performance of a computer is the result of many aspects of the system, including the chip, the available memory, the connection between the processors (e.g., Gigabit Ethernet vs. Myrinet), the version of the operating system, the version of MPI, and the particular version of the benchmark code.
  2. The performance is also subject to the particular circumstances under which it was performed, e.g., a benchmark run in a single user mode will usually be faster than one that was run with other users on the system.
  3. We explicitly don’t warranty the accuracy of the numbers posted on this site.