GIScience (Geographic Information Science)

Geographic Information System (GIS) software has been in existence since the 1980's and since then over 80 raster and vector file formats have been created. As many of these file formats are proprietary, specific software is required to work with them limiting their reuse. In an effort to promote the interoperability of geospatial data, a few open standards have emerged. The following document focuses on these open file formats and provides a brief explanation of each. Links to resources on how to create and open each file format have been included, along with an example file for each format.

The file formats have been broken-down based on the types of data they can contain - either raster, vector, or both. For a brief explanation of raster and vector data types, please visit https://libguides.colostate.edu/gis/geospatial_data_basics

Table of contents

Raster

GeoTIFF
Hierarchical Data Format (HDF)
Network Common Data Format (NetCDF)

Vector

ESRI Shapefile
Comma Separated Values (CSV) and Tab Separated files (TSV)
Geographic Markup Language (GML)
Keyhole Markup Language (KML)
GeoJSON

Both Raster and Vector

GeoPackage

 

Raster

GeoTIFF

The combination of a Tag Image File Format (TIFF) with geospatial (i.e., location) information produces a GeoTIFF. The extension for these files is either .tif or .tiff, and upon first glance look like standard image files. The inclusion of geospatial information as part of TIFF files can be determined using a GIS or programmatically using GDAL.

Creation

• QGIS - https://docs.qgis.org/3.4/en/docs/user_manual/plugins/core_plugins/plugins_georeferencer.html then https://freegistutorial.com/how-to-export-map-to-geotiff-format-on-qgis/
• ArcGIS Pro - https://michaelminn.net/tutorials/arcgis-pro-digitizing/index.html

Opening

• QGIS - https://gis.stackexchange.com/questions/31827/opening-tiff-dems-in-qgis
• ArcGIS Pro- https://firms.modaps.eosdis.nasa.gov/usfs/tutorials/ba-wv/ Start at Step 5
• Google Earth - https://www.google.com/earth/outreach/learn/importing-geographic-information-systems-gis-data-in-google-earth/
• Python - https://towardsdatascience.com/reading-and-visualizing-geotiff-images-with-python-8dcca7a74510
• R – https://datacarpentry.org/r-raster-vector-geospatial/01-raster-structure/

Example file

2016 riparian vegetation within the Colorado River Basin https://mountainscholar.org/handle/10217/186040

Hierarchical Data Format (HDF)

HDF files are very versatile allowing researchers to store images, tables, graphs, and even documents within one file. Identified by files with the extension .hdf, .he5, and .h5, these files are comparable to a folder with a root level containing child folders and files within. These self-describing files can store multi-dimensional data and are useful for storing general-purpose scientific data. HDF5 is the latest version which is much more structured than the previous.

Creation

• Python - https://www.christopherlovell.co.uk/blog/2016/04/27/h5py-intro.html
• R - https://www.neonscience.org/resources/learning-hub/tutorials/hdf5-intro-r

Opening

• QGIS - https://youtu.be/XAClP8ERmsA
• ArcGIS - https://hdfeos.org/software/arcgis.php
• Python - https://docs.h5py.org/en/stable/
• R – https://www.bioconductor.org/packages/devel/bioc/vignettes/rhdf5/inst/doc/rhdf5.html

Example File

OMI Level 2 O2-O2 Cloud Data https://gamma.hdfgroup.org/ftp/pub/outgoing/opendap/data/HDF5/zoo/OMI-Aura_L2-OMCLDO2_2009m1001t0941-o27729_v003-2009m1001t155925.h5

Network Common Data Format (NetCDF)

Developed in 1980 for sharing atmospheric science data, the NetCDF file format has gained popularity across many scientific disciplines. NetCDF files have the file extension .netcdf, .nc, and .nc4. They can be used to store large numeric multi-dimensional datasets with a time-series and spatial component, making them great for computational model outputs. Like the Hierarchical Data Format (HDF), NetCDF files are self-describing ensuring that documentation accompanies the data when shared. A few organizations using NetCDF include NOAA, NASA and USGS.

Creation

• Python - https://www.esri.com/arcgis-blog/products/arcgis/data-management/creating-netcdf-files-for-analysis-and-visualization-in-arcgis/
• R - https://pjbartlein.github.io/REarthSysSci/netCDF.html#create-and-write-a-netcdf-file

Opening

• QGIS - https://ereefs.aims.gov.au/ereefs-aims/help/how-to-open-a-NetCDF-file-with-ArcMap-and-QGIS
• ArcGIS Pro - https://pro.arcgis.com/en/pro-app/2.8/help/data/multidimensional/reading-netcdf-data-as-a-raster-layer.htm
• Python - https://github.com/Unidata/netcdf4-python
• R – https://pjbartlein.github.io/REarthSysSci/netCDF.html#reading-restructuring-and-writing-netcdf-files-in-r

Example File

Understanding the response of tropical ascent to warming using an energy balance framework https://mountainscholar.org/handle/10217/199724

Vector

ESRI Shapefile

Developed in the early 1990's, the shapefile format is used to store vector data in either points, lines, or polygons. The main shapefile with extension .shp must be accompanied by at least two other files with extensions .shx and .dbf. Often more than 3 files encompass a single shapefile, all with the same name but different extensions (denoting the different types of information stored in each). This file format requires field names shorter than 11 characters and file sizes less than 2 GB. When sharing these data, it is important to package all accompanying files in zip so they remain together.

Creation

• QGIS - https://docs.qgis.org/3.22/en/docs/training_manual/create_vector_data/create_new_vector.html
• ArcGIS - http://sites.tufts.edu/gis/files/2013/11/Creating-and-Editing-a-new_shape_file.pdf

Opening

• QGIS - https://docs.qgis.org/2.8/en/docs/user_manual/working_with_vector/supported_data.html
• ArcGIS Pro - https://pro.arcgis.com/en/pro-app/latest/help/data/shapefiles/working-with-shapefiles-in-arcgis-pro.htm
• Python - https://github.com/Toblerity/Fiona
• R – https://datacarpentry.org/r-raster-vector-geospatial/06-vector-open-shapefile-in-r/

Example File

Snow persistence grids and snow zone shapefiles for the western United States https://mountainscholar.org/handle/10217/171907

Comma Separated Values (CSV) and Tab Separated files (TSV)

CSV or TSV files are plain text with columns of data separated by commas (or tabs) and each line represents a new record. Having a field or fields with census tracts, zip codes, or latitude and longitude values allows each record to be spatially aware. These files can be opened in any text editor, spreadsheet software or using programmatic methods. These files can be identified by the extension .csv or .tsv. It is best to include column names in the top line of these files, though it could be omitted (known as head-less files). Beyond a column header, documentation for these files generally remains separate. Efforts like the ICARTT standard (file extension .ict) have reserved a portion in the top of these files to more thoroughly describe each column of data, offering a self-describing option.

Creation

• QGIS - https://opensourceoptions.com/blog/export-qgis-attribute-table-to-excel-and-csv/ and be sure to create a column for both latitude and longitude https://medium.com/@samath/qgis-adding-latitude-and-longitude-to-your-attribute-table-80d4ddbdc02b
• ArcGIS Pro - https://pro.arcgis.com/en/pro-app/latest/help/data/tables/export-tables.htm and be sure to create a column for both latitude and longitude https://support.esri.com/en/technical-article/000008944

Opening

• QGIS - http://www.qgistutorials.com/en/docs/importing_spreadsheets_csv.html
• ArcGIS Pro - https://pro.arcgis.com/en/pro-app/2.8/get-started/create-points-from-a-table.htm
• Python - https://docs.python.org/3/library/csv.html
• R - https://swcarpentry.github.io/r-novice-inflammation/11-supp-read-write-csv/

Creation and Opening of ICARTT

• Python - https://pypi.org/project/icartt/

Example File

2016 NETCARE Amundsen campaign ship_s position data SHIPSPOSN_Amundsen_20160714_R0_NETCARE.ict

Geographic Markup Language (GML)

GML is an Extensible Markup Language (XML) structure for storing geographic features. Initially released in 2000, the GML format provided a way to standardize geospatial data being shared online. GML files can have the extension of either .gml, or .xml, and as with most XML structures, there is a separate schema file to describe the elements used in the data and the data itself. The GML schema offers metadata for coordinate reference systems, units of measure, and feature descriptors (for primitive objects like points, lines, polygons, and even curves). The schema is also extensible allowing for new feature descriptions of real-world objects like roads, bridges, etc. for use within a specific community. In 2007 GML became an ISO standard, with the official number ISO 19136:2007. GML is the default encoding for all INSPIRE (Infrastructure for Spatial Information in Europe, https://inspire.ec.europa.eu/about-inspire/563) spatial data themes. Though generally not considered for its ability to support raster data, GML files can also be used to georeference jpg2000 images (https://www.ogc.org/standards/gmljp2)

Creation

• QGIS - https://maps.cga.harvard.edu/qgis_2/wkshop/export_GIS.php

Opening

• QGIS - https://plugins.qgis.org/plugins/gml_application_schema_toolbox/
• ArcGIS Pro - https://support.esri.com/en/technical-article/000009150
• Python - https://networkx.org/documentation/stable/reference/readwrite/gml.html
• R – https://r-spatial.github.io/sf/articles/sf2.html

Example File

INSPIRE dataset: Airspace Area https://inspire-geoportal.ec.europa.eu/download_details.html?view=downloadDetails&resourceId=%2FINSPIRE-f670705f-f4e9-11e6-81e4-52540023a883_20220601-135202%2Fservices%2F1%2FPullResults%2F201-220%2Fdatasets%2F9&expandedSection=metadata

Keyhole Markup Language (KML)

KML is a derivative of GML used for styling of geographic information, and includes a specific set of features for use in Google Earth and Google Maps. Developed by Keyhole Inc. for use in their Keyhole Earth Viewer, Google acquired the company in 2004 and a year later released Google Earth. Features from the platform were also incorporated into Google Maps. These files are identified by the .kml extensions and a compressed version of KML has a .kmz extension. KML became an OGC standard in 2008 and as part of this standard, a camera view can be used to control what the user sees, add image overlays, captions and icons, differentiating it from other geospatial data standards.

Creation and Opening

• Python - https://pythonhosted.org/pykml/
• R - https://github.com/briatte/tidykml

Example File

Real-Time Earthquakes https://services.google.com/earth/kmz/realtime_earthquakes_n.kmz

GeoJSON

Leveraging JavaScript Object Notation (JSON), the GeoJSON file format has several defined geospatial datatypes used for easily sharing data online. Compared to XML, JSON files are smaller in size as they rely on key-value pairs instead of a tag-based structure. Recognized by the extension .json and .geojson, GeoJSON files can include vector data with one or more point, line, polygon, multi-point, and multi-line features. All GeoJSON files have the same geographic coordinate reference system (WGS84) simplifying the visualization of these data. Most web maps have libraries to view GeoJSON files, making it a great choice for sharing and visualizing data online.

Creation and Opening

• Python - https://pypi.org/project/geojson/
• R – https://r-graph-gallery.com/325-background-map-from-geojson-format-in-r.html

Example File

Native Territories https://native-land.ca/wp-json/nativeland/v1/api/index.php?maps=territories

Both Raster and Vector

GeoPackage

Built upon the open-source database SQLite, GeoPackages provide a platform-independent, self-describing, and compact format that supports both raster tile matrix sets and vector data. These geospatial files have the extension .gpkg and can store many geospatial layers together in one compact file. Beyond sharing data with others, geopackages are useful for field data collection when internet access is intermittent or non-existent as data can be stored locally and later synced to a remote system.

Creation

• QGIS - https://learn.openwaterfoundation.org/owf-learn-geopackage/using-geopackage/qgis/ 
• ArcGIS - https://learn.openwaterfoundation.org/owf-learn-geopackage/using-geopackage/arcgis/

Opening

• QGIS - https://freegistutorial.com/how-to-create-a-geopackage-on-qgis-3-2/
• ArcGIS Pro - https://pro.arcgis.com/en/pro-app/latest/help/data/databases/work-with-sqlite-databases-in-arcgis-pro.htm
• Python - https://github.com/Toblerity/Fiona
• R - https://r-spatial.github.io/sf/

Example File

Global Urban Street Networks GeoPackages https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/E5TPDQ