Transferring Data with GoCommands and Command Line¶

GoCommands is a command-line tool developed by CyVerse. Unlike iCommands, Gocommands is portable which does not require any installation and runs on any modern OSes, such as MacOS, Linux, and Windows.

This section will cover the basics of GoCommands.

Things to remember about GoCommands

This is a command line tool, operated in a terminal.
Windows users must use PowerShell or Command Prompt.
You must have a CyVerse account or other iRODS Zone account to authenticate.

Download GoCommands¶

Visit GoCommands Releases webpage. Download a package file bundled with tar.gz or zip for your target system's OS and architecture.

Uncompress the package file using tar zxvf <package file>.

Here are commands to download and uncompress the package file for a target system's OS and architecture.

Linux AMD64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-linux-amd64.tar.gz | tar zxvf -

Linux ARM64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-linux-arm64.tar.gz | tar zxvf -

MacOS AMD64

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-darwin-amd64.tar.gz | tar zxvf -

MacOS ARM64 (M1/M2)

GOCMD_VER=$(curl -L -s https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt); \
curl -L -s https://github.com/cyverse/gocommands/releases/download/${GOCMD_VER}/gocmd-${GOCMD_VER}-darwin-arm64.tar.gz | tar zxvf -

Windows AMD64 with Command Prompt

curl -L -s -o gocmdv.txt https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt && set /p GOCMD_VER=<gocmdv.txt
curl -L -s -o gocmd.zip https://github.com/cyverse/gocommands/releases/download/%GOCMD_VER%/gocmd-%GOCMD_VER%-windows-amd64.zip && tar zxvf gocmd.zip && del gocmd.zip gocmdv.txt

Windows AMD64 with PowerShell

curl -o gocmdv.txt https://raw.githubusercontent.com/cyverse/gocommands/main/VERSION.txt ; $env:GOCMD_VER = (Get-Content gocmdv.txt)
curl -o gocmd.zip https://github.com/cyverse/gocommands/releases/download/$env:GOCMD_VER/gocmd-$env:GOCMD_VER-windows-amd64.zip ; tar zxvf gocmd.zip ; del gocmd.zip ; del gocmdv.txt

Run GoCommands using iCommands configuration¶

GoCommands can read iCommands configuration file ~/.irods/irods_environment.json. To create iCommands configuration file using GoCommands, type below:

./gocmd init

When prompted, enter the values below:

host name	port #	username	zone	password
data.cyverse.org	1247	CyVerse Username	iplant	CyVerse Password

If you already have iCommands' configuration file, you don't need any steps to do.

To display current configuration, type below:

./gocmd env

Run GoCommands using a YAML configuration file¶

GoCommands can read a configuration file in YAML format. Create a YAML file using an editor and type-in following:

irods_host: "data.cyverse.org"
irods_port: 1247
irods_user_name: "<username>"
irods_zone_name: "iplant"
irods_user_password: "<password>"

Give -c or --config flag to provide the configuration file.

./gocmd -c config.yaml env

Missing Values

GoCommands will prompt if there is any missing values in the YAML configuration file. You can omit irods_user_password field if you do not want to have the password in clear text in the configuration file.

Anonymous Access to the CyVerse Data Store¶

You can access public data in the CyVerse Data Store with GoCommands using:

Username: anonymous
Password: <leave blank>

Upload Files/Folders from Local Computer to Data Store¶

Warning

When uploading your data to the Data Store, you should not upload files/folders with names containing spaces (e.g., experiment one.fastq) or special characters (e.g., ~ `` ! @ # $ % ^ & * ( ) + = { } [ ] | : ; " ' < > , ? / and \).

The Apps on the Discovery Environment and most command line applications will typically not tolerate these characters.

For long file/folder names, we recommend the use of underscores (e.g., experiment_one.fastq) instead of spaces.

Upload a directory using the put subcommand.

./gocmd put --progress /local_directory /iplant/home/cyverse_username/destination_folder
# This command will output the progress as it uploads your local directory

There are several optional arguments that the put subcommand can take:

./gocmd put -R # for specifying resource server

./gocmd put --progress # to display the progress of the upload

./gocmd put -f # to force the upload and overwrite

Download Files/Folders from Data Store to Local Computer¶

Download a file using the get subcommand.

./gocmd get --progress /iplant/home/cyverse_username/target_file /local_destination
# This command will output the progress as it downloads to your local machine

There are several optional arguments that the get subcommand can take:

./gocmd get -R # for specifying resource server

./gocmd get --progress # to display the progress of the download

./gocmd get -f # to force the download and overwrite

Additional Sub-Commands¶

In addition to the commands above, there are several frequently used subcommands, most of which follow the Linux paradigm:

subcommand	description
`help`	Display help
`env`	Display current configuration
`init`	Initialize iCommands Configuration
`cd`	Change current working directory
`pwd`	Display current working directory
`ls`	List data-objects or collections in a collection
`cp`	Copy iRODS data-objects or collections to a target collection
`mv`	Move iRODS data-objects or collections to a target collection
`rm`	Remove iRODS data-objects or collections
`rmdir`	Remove iRODS collections
`mkdir`	Make iRODS collections
`get`	Download iRODS data-objects or collections
`put`	Upload local files or directories to a target collection
`bput`	Upload local files or directories to a target collection by bundling and transferring in parallel (optimzied for many small file upload)
`bun`	Extract iRODS data-objects in a bundle file format (*.tar) to a target collection
`sync`	Sync local directory with iRODS collection
`passwd`	Change user password
`copy-sftp-id`	Upload SSH public key to iRODS for SFTP access

Last update: 2023-05-17