Research Guides: Coding and Cookies: Spatial Data Science in the Cloud (Alpine HPC) using Python

Materials

The following resources have been assembled to take your python skills to the next level so you can:

Effectively work with geospatial data
Break-up your analysis and run several instances of your program
Move your analysis to an HPC and retrieve results

Resources:

Introduction to Spatial Data Science
Python
- Learning Data Analysis in Python Notebooks
- Interactive Geospatial Python Notebooks: These notebooks have been prepared for those new to geospatial data science in python. Special attention has been given to using interactive plots for choosing both points of interest (POI) and areas of interest (AOI) to customize analysis location.
HPC
- CSU Alpine 101
- Linux 101
- Set-up Virtual Environment on HPC -
1. Enter the shell (Clusters > >_Alpine Shell Access)
2. Navigate to your project folder (use :"cd /projects/{user name}"), replacing {user name} to match yours
3. Clone Interactive Geospatial Python Notebooks (type: "git clone https://github.com/GeospatialCentroid/interactive_geospatial_python.git" in shell and press enter)
4. Create the virtual environment
  1. Type "acompile" and press enter to switch to a compile node
  2. Type "module load anaconda" and press enter to load the anaconda module
  3. Then type "conda create -n geospatial -c conda-forge -y jupyterlab numpy matplotlib xarray rasterio geopandas rioxarray earthpy descartes xarray-spatial pystac-client python-graphviz"
    - You can use "conda info --envs" to verify your environment exists
  4. Activate your new environment (use: "conda activate geospatial" in shell)
  5. With your environment activated (the command line should now start with "(geospatial)") install one more library with "pip install papermill" so we can call our notebook from the command line and pass parameters
- Run jobs using Slurm (the HPC job scheduler)
  1. Duplicate and edit the job_template.sh file with "cp job_template.sh job_template{year}.sh"
    1. Open the new file with "vi job_template{year}.sh"
      - Enter edit mode, press the Esc key then 'i'
      - Change the 'year' value used in the file (two locations)
      - Update the email address
      - Save and close the file, press the Esc key then type ':wq' followed by the Enter key
  2. Use the command "sbatch job_template{year}.sh" to submit the job
  3. Then use "squeue --user={user_id}@colostate.edu --long" to check the status of the job
    - See Useful Slurm commands for more information

Additional Resources: