HPCC – Overview

About: Welcome to the HPC Center (HPCC)

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The High-Performance Computing Center (HPCC) provides state-of-the-art research computing infrastructure and training accessible to all UCR researchers and affiliates at low cost. Currently, it supports over 150 research groups with more than 800 active users. Its resources are also heavily used for instructing undergraduate and graduate classes in a wide range of computational, statistical, life science and engineering disciplines.

News

This year the HPCC was awarded an MRI equipment grant (#2215705) by NSF for the acquisition of a Big Data HPC Cluster in the total amount of $942,829. For details see here.

Quick start

The following lists the most frequently visted pages of the HPCC site. They can also be accessed via the navigation system outlined below.

Navigating and searching this site

Top menu located in bar on top of each page provides links to the main content categories
Section menu to the left links to subpages of each main category
Table of content to the right links to sections within each page

Gain access

User account creation for accessing HPCC’s infrastructure
Latest recharging rates
Log in instructions

HPC usage

Usage instructions are provided in the manual section
To efficiently navigate the Manuals pages, use the Manual dropdown in the menu bar on the top of this site.
Event schedule for workshops and user meetings

Infrastructure description

Infrastructure description of HPCC’s clusters and parallel storage systems
Facility description document for grant applications and related purposes

Help and contacts

For Support please consider asking on our Forums.
For requesting user accounts, password help, or software requests, please email support@hpcc.ucr.edu.
You can also join our Slack at ucr-hpcc.
Contact information

About: Access

Mon, 01 Jan 0001 00:00:00 +0000

User account requests

To create new user or lab accounts, please follow these instructions:

Please email user account requests to support@hpcc.ucr.edu. Include the full name, NetID and email address of both users and PI. Users need to be members of the PI’s group. Preferentially, user account requests should come from the corresponding PI directly. If the request comes from a new user then the PI needs to be CC’ed in the email exchange.
If a PI’s lab is not registered yet, please provide in the same email a COA (formerly FAU) required to pay for the annual subscription fee, the email of your financial advisor, and optionally any additional data storage (see here). If additional storage is needed, mention how much and the COA to be used for the additional recharge.

After receiving the access information for a new account, users want to follow the login instructions here.

Recharging rates

HPCC’s recharging rate structure is outlined below. A more formal summary is available in the most recent Recharging Rate PDF here.

PI-based Registration Fee

An annual registration fee of $1,000 gives all members of a UCR lab access to our high-performance computing infrastructure. The registration provides access to the following resources:

Over 16,000 CPU cores (60% AMD and 40% Intel), ~230,000 cuda cores (Nvidia A100, P100 and K80 GPUs), ~5PB parallel GPFS-based disk space, 512GB-1TB of memory/node, etc. More details are available on the hardware pages.
Over 1000 software packages and community databases. Details are available on the software page.
Free attendance of workshops offered by HPCC staff
Free consultation services (up to 1 hour per month)
Note: there is no extra charge for CPU usage but each user and lab have CPU quotas of 384 and 768 CPU cores, respectively. Computing jobs exceeding these quotas can be submitted but will stay in a queued state until resources within the quota limits become available.

Big data storage

For data storage the HPCC uses a central parallel GPFS storage system that scales to many thousands of TBs. This high-availability storage system is directly attached (mounted) to all its CPU and GPU nodes, meaning users can immediately process their data with high-performance computing hardware without moving them from one location (e.g. a data archival system) to another.

Rented big data storage
- Standard user accounts have a storage quota of 20 GB. To gain access to much larger storage pools, PIs have the option to rent or own storage space.
- Storage rental option
  - $1000 per 10TB of usable and backed up storage space per year. Smaller units than 10TB are also available (e.g. 100GB units). For details see here. In comparison, the maintenance cost for the same amount of owned storage is $260 per year (see below).
  - Since the HPCC backs up all user data and uses snapshotting as an additional data security measure, 10TB of usable backed up space is the equivalent of almost 30TB of raw disk space. Thus, the cost for rented storage is $33.33 for 1TB/yr raw disk space. «««< HEAD
  - User account and big data backups are performed monthly and stored long-term, or as long as users maintain their storage subscriptions and/or owned hard drives are not older than seven years. To prevent the accumulation of unwanted data which is very costly, any data deleted by users in their user account or bigdata will also be removed from the backup system. To retrieve recent unwanted changes to data, the snapshotting system can be used. This allows retrieval of recently deleted files.
    =======
  - User account and big data backups are performed monthly and stored long-term, or as long as users maintain their storage subscriptions and/or owned hard drives are not older than seven years. To prevent the accumulation of unwanted data which is very costly, any data deleted by users in their user account or bigdata will also be removed from the backup system. To retrieve recent unwanted changes to data, previous snapshots can be used assuming data was there before the snapshot. This allows retrieval of recently deleted files. More info on snapshots can be found here.

f1a24833b7dd64bb693e75e18dc093f8a07fe41a * The rented storage pool can be shared among all user accounts of a registered lab.

Ownership models

Owned big data storage
- A lab/PI purchases storage hardware (e.g. hard drives) according to the specifications of the facility. Owned hard drives will be added to the facility’s parallel GPFS storage systems including production and backup storage. There is no extra charge for the additional storage infrastructure required for operation, including hard drive enclosures (servers) and high-speed network. The annual support fee for owned disk storage is $260 per 10TB of usable and backed up storage space. Since we back everything up to a secondary server room and use snapshotting as an additional data security measure, 10TB of usable backed up space is the equivalent of almost 30TB of raw disk space. Thus, the maintenance cost for owned storage is $8.67 for 1TB/yr raw disk space. Note, owned storage space is only available to the users of a PI’s group or those a PI wishes to give access to.
- The owned storage pool can be shared among all user accounts of a registered lab.
- Owned storage can be attractive for labs with storage needs above 40TBs. For smaller amounts the rental option is often a better and more flexible choice (e.g. available within a few days).
- Due to the pooled nature of our storage, PIs should not expect the ability of retrieving the individual drives purchased for cluster use, nor that their data is stored on their exact drives. GPFS is a shared pool of storage, and data will end up stored across multiple disks.

Computer nodes
- A lab/PI purchases compatible computer nodes (e.g. with supported network cards). Examples of popular high-density architecture are quad node systems shown here. A quad node system includes 4 nodes where each node can be configured, for example, with two 64 core AMD or Intel chips (providing 128 cores per node or 512 cores per quad node system), 1,024GB of RAM, 2TB SSD and NDR-IB interconnect (additional example). Similar options are available for GPU nodes.
- Nodes are administered under a priority queueing system that gives users from an owner lab priority and also increases that lab’s overall CPU quota (see above) by the number of owned CPU cores.
- Owned computer nodes are an attractive solution for labs requiring 24/7 access to hundreds of CPU cores with no or only minor waiting times in queue.

Software install

Registered users can email software install requests to HPCC’s issue tracking system @ support@hpcc.ucr.edu. Install requests are addressed in the order received. Simple installs are addressed within 1 to a few days. Complex installs may take longer.

Department cluster membership with owned computing nodes

This option addresses the need of department-level HPC access where the standard PI-based membership is not practical, e.g. provide cluster access to large number of undergraduate students in classes. Under this model a department purchases computer nodes that will be administered similarly as described above under the Ownership model. Due to the large number of expected users from departments, the CPU quota per user is usually lower compared to the PI-based model.

Using HPCC cluster for classes

To use the HPCC cluster for teaching UCR classes, please coordinate with the systems administrators (support@hpcc.ucr.edu) at least 4 weeks prior to the start of a class so that there is enough time for planning. Details that need be discussed includes the number of user accounts required, special software requirements, creation of a class-specific Slurm partition, data storage reservations, as well as other needs that may vary for different classes.

External user accounts

Accounts for external customers can only be granted if a lab has a strong affiliation with UC Riverside, such as a research collaboration with UCR researchers. Both the corresponding UCR PI and external collaborator need to maintain an HPCC subscription. External accounts are subject to an annual review and approval process. To be approved, the external and internal PIs have to complete this External Usage Justification.

Trial Accounts

If you’re not sure if the HPCC might work for your use case, we can offer limited-compute accounts to allow for you to test the performance and compatability of your software. A limited number of trial accounts can be created for a lab, with access to limited compute compared to a standard account. Please reach out to HPCC support if you’re interested in a trial account.

Facility description

The latest hardware/facility description (e.g. for grant applications) is available here.

About: Activity Report

Mon, 01 Jan 0001 00:00:00 +0000

Summary report

Do note that just because resources appear to be idle might not mean that they can be immeditally allocated to a job. There are many factors at play, such as priority, requested runtimes, number of resources requested, etc. that contribute to how jobs are queued. If you’re concerned about your jobs being queued for extended periods of time, please see the Queue Policies page or reach out to support.

About: Rates

Mon, 01 Jan 0001 00:00:00 +0000

Facility description

Facility description (e.g. for grant applications)

Recharging rates

Purchased Disk Information

To achieve our desired balance of performance and density in our storage systems we must place some restrictions on the lifetime of purchased storage. Details can be found in our Owned Storage Information document

Ownership Models

For more information relating to owned storage, see the Ownership models section of the Access page.

External user accounts

About: Contact

Mon, 01 Jan 0001 00:00:00 +0000

Support

For cluster support, see the Help and Contacts section of our Introduction.

Facility

Please do not contact the System Administrators directly for general cluster/software support. Our support email (support [at] hpcc.ucr.edu) will make sure the most eyes see the ticket and prevent it from getting lost in cluttered email inboxes.

Austin Leong, Sr. HPC Systems Administrator
Emerson Jacobson, HPC Systems Administrator
Thomas Girke, Director of HPC Center

Advisory Board (executive committee)

The responsibilities of the Advisory Board are outlined here.

Faculty members
- Jason E Stajich (Microbiology & Plant Pathology)
- Wenxiu Ma (Statistics)
- Stefano Lonardi (CSE)
- Mark Alber (Mathematics)
- Adam Godzik (Biomedical Sciences)
- Laura Sales (Physics)
- Ahmed Eldawy (CSE)
- Xinping Cui (Statistics)
- Leonard Mueller (Chemistry)
HPC expert staff members from UCR
- Keith Richards-Dinger (Earth Sciences)
- Victor Hill (CS)
- Bill Strossman (C&C)
External members from academia and industry
- One of each to be added here.

Office location and mailing address

1208/1207 Genomics Building (Google Map) 3401 Watkins Drive University of California Riverside, CA 92521

Server rooms

Genomics

HPCC’s main server room is in the Genomics Building, Rm 1120A.

CoLo server room

School of Medicine CoLo

Help

For questions or requesting new user accounts please email support@hpcc.ucr.edu.

About: Acknowledgement of Facility

Mon, 01 Jan 0001 00:00:00 +0000

Acknowledgement in Publications

We appreciate that you have chosen our facility to support your research and would like to remind you to add the following statement to acknowledge the High-Performance Computing Center in your publications and presentations.

Computations were performed using the computer clusters and data storage resources of the HPCC, which were funded by grants from NSF (MRI-2215705, MRI-1429826) and NIH (1S10OD016290-01A1).

Your success is important to us and we would appreciate if you could share with us the references or URLs to any publications or presentations that used our facilty.

Grants

About:

Mon, 01 Jan 0001 00:00:00 +0000

#— #type: docs #linkTitle: Overview slides #title: Overview slides #weight: 6 #—