Skip to content
Neurodesk

CVMFS

Install the CernVM File System (CVMFS)

Section titled “Install the CernVM File System (CVMFS)”

To begin, install CVMFS. Follow the official quickstart.

An example installation for Ubuntu in Windows Subsystem for Linux (WSL) looks like this:

Terminal window
sudo apt-get install lsb-release
wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb
sudo dpkg -i cvmfs-release-latest_all.deb
rm -f cvmfs-release-latest_all.deb
sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install cvmfs

Ubuntu 24.04 might have an issue with this, so try installing dependencies manually:

Terminal window
sudo apt install libattr1=1:2.5.2-1build1 libuuid1=2.39.3-9ubuntu6

Configure CVMFS

Section titled “Configure CVMFS”

Once installed, create the keys and configure the servers to use:

Terminal window
sudo mkdir -p /etc/cvmfs/keys/ardc.edu.au/




echo "-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwUPEmxDp217SAtZxaBep
Bi2TQcLoh5AJ//HSIz68ypjOGFjwExGlHb95Frhu1SpcH5OASbV+jJ60oEBLi3sD
qA6rGYt9kVi90lWvEjQnhBkPb0uWcp1gNqQAUocybCzHvoiG3fUzAe259CrK09qR
pX8sZhgK3eHlfx4ycyMiIQeg66AHlgVCJ2fKa6fl1vnh6adJEPULmn6vZnevvUke
I6U1VcYTKm5dPMrOlY/fGimKlyWvivzVv1laa5TAR2Dt4CfdQncOz+rkXmWjLjkD
87WMiTgtKybsmMLb2yCGSgLSArlSWhbMA0MaZSzAwE9PJKCCMvTANo5644zc8jBe
NQIDAQAB
-----END PUBLIC KEY-----" | sudo tee /etc/cvmfs/keys/ardc.edu.au/neurodesk.ardc.edu.au.pub


echo "CVMFS_USE_GEOAPI=yes" | sudo tee /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf


echo 'CVMFS_SERVER_URL="http://cvmfs-geoproximity.neurodesk.org/cvmfs/@fqrn@;http://cvmfs.neurodesk.org/cvmfs/@fqrn@;http://s1osggoc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1sampa-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1brisbane-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1nikhef-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1perth-cvmfs.openhtc.io/cvmfs/@fqrn@"' | sudo tee -a /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf


echo 'CVMFS_KEYS_DIR="/etc/cvmfs/keys/ardc.edu.au/"' | sudo tee -a /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf


echo "CVMFS_HTTP_PROXY=DIRECT" | sudo tee  /etc/cvmfs/default.local
echo "CVMFS_QUOTA_LIMIT=5000" | sudo tee -a  /etc/cvmfs/default.local


sudo cvmfs_config setup

You can use the list above, but you can also pick a subset of servers that are close to you or fit your use case better. To understand what to choose, we use the following CVMFS server setup.

These CVMFS Stratum 1 servers are hosted by the Open Science Grid and every server has a Cloudflare CDN alias that is geo-located through the Maxmind GeoAPI service in the CVMFS client:

  • Illinois, USA: s1fnal-cvmfs.openhtc.io:8080 → cvmfs-s1fnal.opensciencegrid.org:8000
  • Nebraska, USA: s1osggoc-cvmfs.openhtc.io:8080 → cvmfs-s1goc.opensciencegrid.org:8000
  • New York, USA: s1bnl-cvmfs.openhtc.io:8080 → cvmfs-s1bnl.opensciencegrid.org:8000
  • Oxford, UK: s1ral-cvmfs.openhtc.io:8080 → cvmfs-egi.gridpp.rl.ac.uk:8000
  • Netherlands, Europe: s1nikhef-cvmfs.openhtc.io:8080 → cvmfs01.nikhef.nl:8000

This server is currently down:

  • Sao Paulo, Brazil: s1sampa-cvmfs.openhtc.io:8080 → sampacs01.if.usp.br:8000

These CVMFS Stratum 1 servers are hosted by ARDC Nectar Cloud and also have a Cloudflare CDN alias:

  • Brisbane, Queensland, Australia: s1brisbane-cvmfs.openhtc.io → cvmfs-brisbane.neurodesk.org
  • Sydney, New South Wales, Australia: s1sydney-cvmfs.openhtc.io → cvmfs-sydney.neurodesk.org
  • Melbourne, Victoria, Australia: s1melbourne-cvmfs.openhtc.io → cvmfs-melbourne.neurodesk.org
  • Perth, Western Australia, Australia: s1perth-cvmfs.openhtc.io → cvmfs-perth.neurodesk.org

This CVMFS Stratum 1 server is hosted by AWS:

  • Frankfurt, Germany: cvmfs-frankfurt.neurodesk.org → ec2-3-72-92-91.eu-central-1.compute.amazonaws.com

This CVMFS Stratum 1 server is hosted by Jetstream:

  • Indiana, USA: cvmfs-jetstream.neurodesk.org → 149.165.172.188

We have one geolocation-steered domain, cvmfs-geoproximity.neurodesk.org:

  • 153.02, -27.46 → cvmfs-brisbane.neurodesk.org
  • 151.2073, -33.8678 → cvmfs-sydney.neurodesk.org
  • 115.86, -31.95 → cvmfs-perth.neurodesk.org
  • -88.30, 41.84 → cvmfs-s1fnal.opensciencegrid.org
  • -96.66, 40.83 → cvmfs-s1goc.opensciencegrid.org
  • -1.26, 51.75 → cvmfs-egi.gridpp.rl.ac.uk
  • 4.90, 52.37 → cvmfs01.nikhef.nl
  • 8.68, 50.11 → ec2-3-72-92-91.eu-central-1.compute.amazonaws.com
  • -46.63, -23.54 → sampacs01.if.usp.br
  • -86.45, 39.22 → cvmfs-jetstream.neurodesk.org
  • 145.13, -37.92 → cvmfs-melbourne.neurodesk.org

Every location has a health check attached to it and doesn’t forward to it if the destination is not working.

We also have 3 direct URLs without CDNs that are geolocation-steered:

cvmfs1.neurodesk.org:

  • South America → sampacs01.if.usp.br
  • North America → cvmfs-s1fnal.opensciencegrid.org
  • Default → cvmfs-brisbane.neurodesk.org
  • Europe → ec2-3-72-92-91.eu-central-1.compute.amazonaws.com
  • Asia → cvmfs-perth.neurodesk.org

cvmfs2.neurodesk.org:

  • North America → cvmfs-s1goc.opensciencegrid.org
  • Europe → cvmfs01.nikhef.nl
  • Default → cvmfs-s1goc.opensciencegrid.org

cvmfs3.neurodesk.org:

  • North America → cvmfs-s1bnl.opensciencegrid.org
  • Asia → cvmfs-brisbane.neurodesk.org
  • Default → cvmfs-s1bnl.opensciencegrid.org
  • Oceania → cvmfs-perth.neurodesk.org

These servers are currently NOT working and are NOT YET mirroring our repository (we are waiting for RAL to come back online, then the others will mirror that):

  • Swinburne, Australia: s1swinburne-cvmfs.openhtc.io:8080 → cvmfs-s1.hpc.swin.edu.au:8000
  • China: s1ihep-cvmfs.openhtc.io:8080 → cvmfs-stratum-one.ihep.ac.cn:8000

For WSL users

Section titled “For WSL users”

You need to run this for each new WSL session:

Terminal window
sudo cvmfs_config wsl2_start

Test if the connection works:

Terminal window
sudo cvmfs_config chksetup


ls /cvmfs/neurodesk.ardc.edu.au


sudo cvmfs_talk -i neurodesk.ardc.edu.au host info


cvmfs_config stat -v neurodesk.ardc.edu.au

For Ubuntu 22.04 users

Section titled “For Ubuntu 22.04 users”

If configuring CVMFS returns the following error:

Terminal window
Error: failed to load cvmfs library, tried: './libcvmfs_fuse3_stub.so' '/usr/lib/libcvmfs_fuse3_stub.so' '/usr/lib64/libcvmfs_fuse3_stub.so' './libcvmfs_fuse_stub.so' '/usr/lib/libcvmfs_fuse_stub.so' '/usr/lib64/libcvmfs_fuse_stub.so'
./libcvmfs_fuse3_stub.so: cannot open shared object file: No such file or directory
/usr/lib/libcvmfs_fuse3_stub.so: cannot open shared object file: No such file or directory
/usr/lib64/libcvmfs_fuse3_stub.so: cannot open shared object file: No such file or directory
./libcvmfs_fuse_stub.so: cannot open shared object file: No such file or directory
libcrypto.so.1.1: cannot open shared object file: No such file or directory
/usr/lib64/libcvmfs_fuse_stub.so: cannot open shared object file: No such file or directory




Failed to read CernVM-FS configuration

A temporary workaround is:

Terminal window
wget https://mirror.umd.edu/ubuntu/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb
dpkg -i libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb

Install Singularity / Apptainer

Section titled “Install Singularity / Apptainer”

For example, for Ubuntu/Debian install apptainer:

Terminal window
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt-get update
sudo apt-get install -y apptainer
sudo apt-get install -y apptainer-suid

Or for Ubuntu/Debian install Singularity:

Terminal window
export VERSION=1.18.3 OS=linux ARCH=amd64 && \
    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
    rm go$VERSION.$OS-$ARCH.tar.gz


echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
    source ~/.bashrc


go get -d github.com/sylabs/singularity


export VERSION=v3.10.0 # or another tag or branch if you like && \
    cd $GOPATH/src/github.com/sylabs/singularity && \
    git fetch && \
    git checkout $VERSION # omit this command to install the latest bleeding edge code from master


export VERSION=3.10.0 && # adjust this as necessary \
    mkdir -p $GOPATH/src/github.com/sylabs && \
    cd $GOPATH/src/github.com/sylabs && \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
    tar -xzf singularity-ce-${VERSION}.tar.gz && \
    cd ./singularity-ce-${VERSION} && \
    ./mconfig --without-seccomp --without-conmon


./mconfig --without-seccomp --without-conmon && \
    make -C ./builddir && \
    sudo make -C ./builddir install


export PATH="/usr/local/singularity/bin:${PATH}"

Use of Neurodesk CVMFS containers

Section titled “Use of Neurodesk CVMFS containers”

The containers are available in /cvmfs/neurodesk.ardc.edu.au/containers/ and can be started with:

Terminal window
singularity shell /cvmfs/neurodesk.ardc.edu.au/containers/itksnap_3.8.0_20201208/itksnap_3.8.0_20201208.simg

Make sure that SINGULARITY_BINDPATH includes the directories you want to work with:

Terminal window
export SINGULARITY_BINDPATH='/cvmfs,/mnt,/home'

For WSL users

Section titled “For WSL users”

The home directory might not be supported. Avoid mounting it with:

Terminal window
singularity shell --no-home /cvmfs/neurodesk.ardc.edu.au/containers/itksnap_3.8.0_20201208/itksnap_3.8.0_20201208.simg

Or configure it permanently:

Terminal window
sudo vi /etc/singularity/singularity.conf

Set:

Terminal window
mount home = no

Install module system

Section titled “Install module system”
Terminal window
sudo yum install lmod

or

Terminal window
sudo apt install lmod

Use of containers in the module system

Section titled “Use of containers in the module system”

Configuration for module system

Section titled “Configuration for module system”

Create the new file /usr/share/module.sh with the following content. Update the version (here YOURLMODVERSION_HERE) with your Lmod version, for example 6.6 (Ubuntu 20.04/22.04) or 8.6.19 (Ubuntu 24.04):

Terminal window
# system-wide profile.modules                                          #
# Initialize modules for all sh-derivative shells                      #
#----------------------------------------------------------------------#
trap "" 1 2 3


case "$0" in
    -bash|bash|*/bash) . /usr/share/lmod/YOURLMODVERSION_HERE/init/bash ;;
       -ksh|ksh|*/ksh) . /usr/share/lmod/YOURLMODVERSION_HERE/init/ksh ;;
       -zsh|zsh|*/zsh) . /usr/share/lmod/YOURLMODVERSION_HERE/init/zsh ;;
          -sh|sh|*/sh) . /usr/share/lmod/YOURLMODVERSION_HERE/init/sh ;;
                    *) . /usr/share/lmod/YOURLMODVERSION_HERE/init/sh ;;  # default for scripts
esac


trap - 1 2 3

Make the module system usable in the shell

Section titled “Make the module system usable in the shell”

Add the following lines to your ~/.bashrc file, or to /etc/bash.bashrc for a global install:

Terminal window
if [ -f '/usr/share/module.sh' ]; then source /usr/share/module.sh; fi


if [ -d /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules ]; then
        # export MODULEPATH="/cvmfs/neurodesk.ardc.edu.au/neurodesk-modules"
        module use /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/*
else
        export MODULEPATH="/neurodesktop-storage/containers/modules"
        module use $MODULEPATH
        export CVMFS_DISABLE=true
fi


if [ -f '/usr/share/module.sh' ]; then
        echo 'Run "ml av" to see which tools are available - use "ml <tool>" to use them in this shell.'
        if [ -v "$CVMFS_DISABLE" ]; then
                if [ ! -d $MODULEPATH ]; then
                        echo 'Neurodesk tools not yet downloaded. Choose tools to install from the Application menu.'
                fi
        fi
fi

Restart the current shell or run:

Terminal window
source ~/.bashrc

Use containers in the module system

Section titled “Use containers in the module system”
Terminal window
export SINGULARITY_BINDPATH='/cvmfs,/mnt,/home'
module use /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/*
ml fsl
fslmaths

Troubleshooting and diagnostics

Section titled “Troubleshooting and diagnostics”
Terminal window
# Check servers
sudo cvmfs_talk -i neurodesk.ardc.edu.au host probe
sudo cvmfs_talk -i neurodesk.ardc.edu.au host info


# Change settings
sudo touch /var/log/cvmfs_debug.log.cachemgr
sudo chown cvmfs /var/log/cvmfs_debug.log.cachemgr
sudo touch /var/log/cvmfs_debug.log
sudo chown cvmfs /var/log/cvmfs_debug.log


sudo vi /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf
echo -e "\nCVMFS_DEBUGLOG=/var/log/cvmfs_debug.log" | sudo tee -a /etc/cvmfs/default.local
cat /etc/cvmfs/default.local
sudo cvmfs_config umount
sudo service autofs stop
sudo mount -t cvmfs neurodesk.ardc.edu.au /cvmfs/neurodesk.ardc.edu.au
# check if new settings are applied correctly:
cvmfs_config showconfig neurodesk.ardc.edu.au


cat /var/log/cvmfs_debug.log
cat /var/log/cvmfs_debug.log.cachemgr