Skip to content

Transferring Data with GoCommands and Command Line

GoCommands is a command-line tool developed by CyVerse. Unlike iCommands, Gocommands is portable which does not require any installation and runs on any modern OSes, such as MacOS, Linux, and Windows.

This section will cover the basics of GoCommands.

Things to remember about GoCommands
  • This is a command line tool, operated in a terminal.
  • Windows users must use PowerShell or Command Prompt.
  • You must have a CyVerse account or other iRODS Zone account to authenticate.

Download GoCommands

Here are commands to download and uncompress the package file for a target system's OS and architecture.

Linux AMD64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-linux-amd64.tar.gz | tar zxvf -

Linux ARM64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-linux-arm64.tar.gz | tar zxvf -

MacOS AMD64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-darwin-amd64.tar.gz | tar zxvf -

MacOS ARM64 (M1/M2)

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-darwin-arm64.tar.gz | tar zxvf -

Windows AMD64 with Command Prompt

curl -L -s -o gocmdv.txt https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt && set /p GOCMD_VER=<gocmdv.txt
curl -L -s -o gocmd.zip https://github.com/cyverse/gocommands/releases/download/%GOCMD_VER%/gocmd-%GOCMD_VER%-windows-amd64.zip && tar zxvf gocmd.zip && del gocmd.zip gocmdv.txt

Windows AMD64 with PowerShell

curl -o gocmdv.txt https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt ; $env:GOCMD_VER = (Get-Content gocmdv.txt)
curl -o gocmd.zip https://github.com/cyverse/gocommands/releases/download/$env:GOCMD_VER/gocmd-$env:GOCMD_VER-windows-amd64.zip ; tar zxvf gocmd.zip ; del gocmd.zip ; del gocmdv.txt

Run GoCommands using iCommands configuration

GoCommands can read iCommands configuration file ~/.irods/irods_environment.json. To create iCommands configuration file using GoCommands, type below:

./gocmd init

When prompted, enter the values below:

host name port # username zone password
data.cyverse.org 1247 CyVerse Username iplant CyVerse Password

If you already have iCommands' configuration file, you don't need any steps to do.

To display current configuration, type below:

./gocmd env

Run GoCommands using a YAML configuration file

GoCommands can read a configuration file in YAML format. Create a YAML file using an editor and type-in following:

irods_host: "data.cyverse.org"
irods_port: 1247
irods_user_name: "<username>"
irods_zone_name: "iplant"
irods_user_password: "<password>"

Give -c or --config flag to provide the configuration file.

./gocmd -c config.yaml env

Missing Values

GoCommands will prompt if there is any missing values in the YAML configuration file. You can omit irods_user_password field if you do not want to have the password in clear text in the configuration file.

Anonymous Access to the CyVerse Data Store

You can access public data in the CyVerse Data Store with GoCommands using:

  • Username: anonymous
  • Password: <leave blank>

Upload Files/Folders from Local Computer to Data Store

Warning

When uploading your data to the Data Store, you should not upload files/folders with names containing spaces (e.g., experiment one.fastq) or special characters (e.g., ~ `` ! @ # $ % ^ & * ( ) + = { } [ ] | : ; " ' < > , ? / and \).

The Apps on the Discovery Environment and most command line applications will typically not tolerate these characters.

For long file/folder names, we recommend the use of underscores (e.g., experiment_one.fastq) instead of spaces.

  1. Upload a directory using the put subcommand.
./gocmd put --progress /local_directory /iplant/home/cyverse_username/destination_folder
# This command will output the progress as it uploads your local directory

There are several optional arguments that the put subcommand can take:

./gocmd put -R # for specifying resource server

./gocmd put --progress # to display the progress of the upload

./gocmd put -f # to force the upload and overwrite

Download Files/Folders from Data Store to Local Computer

  1. Download a file using the get subcommand.
./gocmd get --progress /iplant/home/cyverse_username/target_file /local_destination
# This command will output the progress as it downloads to your local machine

There are several optional arguments that the get subcommand can take:

./gocmd get -R # for specifying resource server

./gocmd get --progress # to display the progress of the download

./gocmd get -f # to force the download and overwrite

Upgrade GoCommands

GoCommands can self-upgrade. Run the upgrade command to upgrade to the latest release of GoCommands available.

./gocmd upgrade # check a new release and upgrade the gocmd binary

If you don't want to upgrade GoCommands, but to check if there is a new release available: 

./gocmd upgrade --check # just check a new release and display

Additional Sub-Commands

In addition to the commands above, there are several frequently used subcommands, most of which follow the Linux paradigm:

subcommand description
help Display help
env Display current configuration
init Initialize iCommands Configuration
cd Change current working directory
pwd Display current working directory
ls List iRODS files or directories in a iRODS directory
cat Show content of an iRODS file
cp Copy iRODS files or directories to a target iRODS directory
mv Move iRODS files or directories to a target iRODS directory
rm Remove iRODS files or directories
rmdir Remove iRODS directories
mkdir Make iRODS directories
get Download iRODS files or directories
put Upload local files or directories to a target iRODS directory
bput Upload local files or directories to a target iRODS directory by bundling and transferring in parallel (optimized for many small file upload)
bun Extract iRODS files in a bundle file format (*.tar) to a target iRODS directory
bclean Clear temporary files used by bput
sync Sync a local directory with an iRODS directory
lsticket List iRODS tickets
mkticket Create an iRODS ticket
modticket Update an iRODS ticket
rmticket Remove iRODS tickets
passwd Change user password
svrinfo Display server information
ps Display server processes
copy-sftp-id Upload SSH public key to iRODS for SFTP access
Last update: 2023-11-30