Skip to content

Transferring Data with GoCommands and Command Line

GoCommands is a command-line tool developed by CyVerse. Unlike iCommands, Gocommands is portable which does not require any installation and runs on any modern OSes, such as MacOS, Linux, and Windows.

This section will cover the basics of GoCommands.

Things to remember about GoCommands
  • This is a command line tool, operated in a terminal.
  • Windows users must use PowerShell or Command Prompt.
  • You must have a CyVerse account or other iRODS Zone account to authenticate.

Download GoCommands

Visit GoCommands Releases webpage. Download a package file bundled with tar.gz or zip for your target system's OS and architecture.

Uncompress the package file using tar zxvf <package file>.

Here are commands to download and uncompress the package file for a target system's OS and architecture.

Linux AMD64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-linux-amd64.tar.gz | tar zxvf -

Linux ARM64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-linux-arm64.tar.gz | tar zxvf -

MacOS AMD64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-darwin-amd64.tar.gz | tar zxvf -

MacOS ARM64 (M1/M2)

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-darwin-arm64.tar.gz | tar zxvf -

Windows AMD64 with Command Prompt

curl -L -s -o gocmdv.txt https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt && set /p GOCMD_VER=<gocmdv.txt
curl -L -s -o gocmd.zip https://github.com/cyverse/gocommands/releases/download/%GOCMD_VER%/gocmd-%GOCMD_VER%-windows-amd64.zip && tar zxvf gocmd.zip && del gocmd.zip gocmdv.txt

Windows AMD64 with PowerShell

curl -o gocmdv.txt https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt ; $env:GOCMD_VER = (Get-Content gocmdv.txt)
curl -o gocmd.zip https://github.com/cyverse/gocommands/releases/download/$env:GOCMD_VER/gocmd-$env:GOCMD_VER-windows-amd64.zip ; tar zxvf gocmd.zip ; del gocmd.zip ; del gocmdv.txt

Run GoCommands using iCommands configuration

GoCommands can read iCommands configuration file ~/.irods/irods_environment.json. To create iCommands configuration file using GoCommands, type below:

./gocmd init

When prompted, enter the values below:

host name port # username zone password
data.cyverse.org 1247 CyVerse Username iplant CyVerse Password

If you already have iCommands' configuration file, you don't need any steps to do.

To display current configuration, type below:

./gocmd env

Run GoCommands using a YAML configuration file

GoCommands can read a configuration file in YAML format. Create a YAML file using an editor and type-in following:

irods_host: "data.cyverse.org"
irods_port: 1247
irods_user_name: "<username>"
irods_zone_name: "iplant"
irods_user_password: "<password>"

Give -c or --config flag to provide the configuration file.

./gocmd -c config.yaml env

Missing Values

GoCommands will prompt if there is any missing values in the YAML configuration file. You can omit irods_user_password field if you do not want to have the password in clear text in the configuration file.


Anonymous Access to the CyVerse Data Store

You can access public data in the CyVerse Data Store with GoCommands using:

  • Username: anonymous
  • Password: <leave blank>

Upload Files/Folders from Local Computer to Data Store

Warning

When uploading your data to the Data Store, you should not upload files/folders with names containing spaces (e.g., experiment one.fastq) or special characters (e.g., ~ `` ! @ # $ % ^ & * ( ) + = { } [ ] | : ; " ' < > , ? / and \).

The Apps on the Discovery Environment and most command line applications will typically not tolerate these characters.

For long file/folder names, we recommend the use of underscores (e.g., experiment_one.fastq) instead of spaces.

  1. Upload a directory using the put subcommand.
./gocmd put --progress /local_directory /iplant/home/cyverse_username/destination_folder
# This command will output the progress as it uploads your local directory

There are several optional arguments that the put subcommand can take:

./gocmd put -R # for specifying resource server

./gocmd put --progress # to display the progress of the upload

./gocmd put -f # to force the upload and overwrite

Download Files/Folders from Data Store to Local Computer

  1. Download a file using the get subcommand.
./gocmd get --progress /iplant/home/cyverse_username/target_file /local_destination
# This command will output the progress as it downloads to your local machine

There are several optional arguments that the get subcommand can take:

./gocmd get -R # for specifying resource server

./gocmd get --progress # to display the progress of the download

./gocmd get -f # to force the download and overwrite

Additional Sub-Commands

In addition to the commands above, there are several frequently used subcommands, most of which follow the Linux paradigm:

subcommand description
help Display help
env Display current configuration
init Initialize iCommands Configuration
cd Change current working directory
pwd Display current working directory
ls List data-objects or collections in a collection
cp Copy iRODS data-objects or collections to a target collection
mv Move iRODS data-objects or collections to a target collection
rm Remove iRODS data-objects or collections
rmdir Remove iRODS collections
mkdir Make iRODS collections
get Download iRODS data-objects or collections
put Upload local files or directories to a target collection
bput Upload local files or directories to a target collection by bundling and transferring in parallel (optimzied for many small file upload)
bun Extract iRODS data-objects in a bundle file format (*.tar) to a target collection
sync Sync local directory with iRODS collection
passwd Change user password
copy-sftp-id Upload SSH public key to iRODS for SFTP access

Last update: 2023-05-17