SCRP has different types of storage tailoring to different needs. Placing your files in correct locations is crucial to getting the best performance out of the cluster.

Basics

We recommend the following rule in placing your files:

File Type Location Quota
Scripts and results ~/ 2/20/50GBa
Large datasets ~/large-data 20/500/1000GBb
Dataset backups ~/archive 20/1000/5000GBb
Temporary files /tmp 0.3-3.7TB (shared)

aQuotas for undergraduates/postgraduates/faculty and staff respectively.
bQuotas for taught postgraduates/research postgraduates/faculty and staff respectively.

Different locations have different properties:

Location Shared across Nodes Backup Performance
~/ Yes Daily Moderate
~/large-data Yes No High
~/archive Yes No Moderate
/tmp No No Very high

Quota

To check ~/ and ~/archive directories’ remaining disk quota, type:

 quota -s

To check ~/large-data directory’s remaining disk quota, type:

beegfs-ctl --getquota --uid $USER

What if I Need More Space? Additional quota can be granted on a case by case basis. Please contact support.

Backup

There are two backup directories while we transit from the old HDD-based backup to the new flash-based one.

New backups since June 30 are mounted under /backup on each node:

  • Hourly backups: /backup/hourly.[0-22]
  • Daily backups: /backup/daily.[0-5]
  • Weekly backups: /backup/weekly.[0-6]
  • Monthly backups: /backup/monthly.[0-3]

Old backups are mounted under /backup-old on each node:

  • Daily backups: /backup/daily.[0-5]
  • Weekly backups: /backup/weekly.[0-6]
  • Monthly backups: /backup/monthly.[0-3]

Configuration Details

~/ is your home directory, which you can also access by specifying the full path /home/users/username or the environment variable $HOME. It is exported from scrp-control-2 using Network File System (NFS), supported by two SSDs in RAID 1 configuration with a total usable capacity of 2.5TB.

  • Full Path: /home/users/username
  • Aliases: ~/, $HOME
  • Speed: 1100MB/s read and 850MB/s write.
  • Quota: 2/20/50GB for undergraduates/postgraduates/faculty respectively.
  • Purging Policy: Files are deleted when user is no longer affiliated with CUHK.

~/large-data is a shared storage space backed by a flash-based parallel file system (BeeGFS on six Intel P4510 8TB SSDs, paired with six Infiniband FDR connections). Each file in the directory is splitted and distributed across multiple servers, allowing for very high read and write speeds. Even though it appears under your home directory, it is only a soft link to the parallel storage’s real location /data/users/username, which means the folder is not included in the daily backup. The total usable capacity 22TB.

  • Full Path: /data/users/username
  • Aliases: ~/large-data, $LARGE_DATA
  • Speed: Up to 6GB/s read and 3GB/s write per node. Maximum aggregate throughout is 19.2GB/s.
  • Quota: 20/500/1000GB for taught postgrauduates/research postgraduates/faculty respectively
  • Purging Policy: Files owned by either UG or TPG students are automatically deleted at the end of each term.

~/archive is a large storage space suitable for storing a second copy of your data. Even though it appears under your home directory, it is only a soft link to the archive’s real location /archive/users/username, which means the folder is not included in the daily backup. It is exported from scrp-login using NFS, supported by a 12-disk RAID 6 HDD array with a total usable capacity of 91TB.

  • Full Path: /archive/users/username
  • Aliases: ~/archive, $ARCHIVE
  • Speed: 500-700MB/s read and 200-300MB/s write.
  • Quota: 20/1000/5000GB for taught postgrauduates/research postgraduates/faculty respectively.
  • Purging Policy: Files are deleted when user is no longer affiliated with CUHK.

/tmp is a shared storage space intended for temporary files generated during a job. The directory is not shared across nodes, meaning that files you generate in one node is not accessible from another node.

  • Full Path: /tmp
  • Aliases: $TMP
  • Speed: 2300MB/s read and 1200MB/s write on compute nodes. 500MB/s read and 200MB/s write on login nodes.
  • Quota:
    • scrp-login-[1-2] 300GB shared.
    • scrp-node-[1-3]: 1.9TB shared.
    • scrp-node-[4-5]: 800GB shared.
    • scrp-node-[7]: 3.7TB shared.
  • Purging Policy: Bi-monthly.