Data Management using GoCommands¶

GoCommands offers a variety of commands to help you manage your data in the Data Store. In the Data Store, file and directory are treated as data objects and collections, respectively. It's perfectly fine to consider these terms interchangeable.

Display the Current Working Collection¶

In the Data Store, the current working collection is equivalent to the concept of a current working directory in traditional file systems. You can display or change your current working collection using GoCommands.

gocmd pwd

By default, after configuring GoCommands, your current working collection is set to your home directory, which is typically located at:

/<Zone Name>/home/<Username>

Note: Paths in the Data Store always start with the zone name /iplant.

Change the Current Working Collection¶

Change to a specific collection using an absolute path:
```
gocmd cd /iplant/home/myUser/mydata
```
This changes your current working collection to /iplant/home/myUser/mydata.
Use a relative path from your current location: Assuming your current working collection is /iplant/home/myUser:
```
gocmd cd mydata
```
Return to your home collection:
```
gocmd cd "~"
```
Note: The ~ must be quoted to prevent shell expansion by your local shell. Without quotes, it will expand to your local machine's home directory instead of your Data Store home directory.
Move up one level:
```
gocmd cd ..
```

List Data Objects (files) and Collections (directories)¶

List the content of a collection:
```
gocmd ls /iplant/home/myUser/mydata
```
This will display the data objects and collections in the /iplant/home/myUser/mydata collection:
```
/iplant/home/myUser/mydata:
    file1.bin
    file2.bin
    C- /iplant/home/myUser/mydata/subdir1
```
The C- prefix indicates that the item is a collection (directory).
List the content of the current working collection:
```
gocmd ls
```
List the contents of a collection in long format with additional details:
```
gocmd ls -l /iplant/home/myUser/mydata
```
This command will show the data objects and collections within /iplant/home/myUser/mydata, along with their additional details:
```
/iplant/home/myUser/mydata:
    myUser  0   demoRes1;rs1    436 2024-04-02.13:36    &   file1.bin
    myUser  1   demoRes2;rs2    436 2024-04-02.13:36    &   file1.bin
    myUser  0   demoRes1;rs1    700 2024-04-02.17:15    &   file2.bin
    myUser  1   demoRes2;rs2    700 2024-04-02.17:15    &   file2.bin
    C- /iplant/home/myUser/mydata/subdir1
```
- Each output line for a data object represents a replica. If iRODS is configured to create multiple replicas, you will see one line for each replica of the data object. For example, if two replicas are created, two lines will be displayed for each file.
- Each line is shown in the owner replica_id resource_server size creation_time replica_state name format.
- Possible replica states are:
  - &: Good
  - X: Stale
  - ?: Unknown
List the contents of a collection with their access control lists:
```
gocmd ls -A /iplant/home/myUser/mydata
```
This command will show the data objects and collections within /iplant/home/myUser/mydata, along with their access control lists (ACLs):
```
/iplant/home/myUser/mydata:
        ACL - g:rodsadmin#iplant:own    myUser#iplant:own
        Inheritance - Disabled
    file1.bin
        ACL - g:rodsadmin#iplant:own    myUser#iplant:own
    file2.bin
        ACL - g:rodsadmin#iplant:own    myUser#iplant:own
    C- /iplant/home/myUser/mydata/subdir1
```
- The g: prefix in the ACL username indicates that the user is a group.
- The ACL is displayed in the username#zone:access_level format.
- Most common access levels are:
  - read_object: Allows read access to the data object or collection.
  - modify_object: Allows modification (write) of the data object or collection.
  - own: Grants ownership of the data object or collection.

Make a Collections (directories)¶

Create a new collection:

gocmd mkdir /iplant/home/myUser/newCollection

Create parent collections if they do not exist:
```
gocmd mkdir -p /iplant/home/myUser/parentCollection/newCollection
```
This command creates the newCollection along with its parent collection parentCollection if it does not already exist.

Upload Data Objects (files) and Collections (directories) to the Data Store¶

Warning

When uploading your data to the Data Store, avoid using:

Spaces in names (e.g., experiment one.fastq)
Special characters: ~ `` ! @ # $ % ^ & * ( ) + = { } [ ] | : ; " ' < > , ? / and \

These may cuase issues with Discovery Environment Apps and command-line applications.

Recommendation: Use underscores for long names (e.g., experiment_one.fastq).

Upload a single file:
```
gocmd put /local/path/file.txt /iplant/home/myUser/
```
This command uploads the file /local/path/file.txt to /iplant/home/myUser/, creating /iplant/home/myUser/file.txt in the Data Store.
Upload a directory and its contents:
```
gocmd put /local/dir /iplant/home/myUser/
```
This command uploads the contents of the directory /local/dir to /iplant/home/myUser/dir in the Data Store. The uploaded files and subdirectories will be placed within the /iplant/home/myUser/dir folder.

Upload with progress bars:

gocmd put --progress /local/path/largefile.dat /iplant/home/myUser/

Force upload:
```
gocmd put -f /local/path/largefile.dat /iplant/home/myUser/
```
This command overwrites the existing file in the Data Store without prompting.
Upload and verify checksum:
```
gocmd put -K /local/path/important_data.txt /iplant/home/myUser/
```
This command uploads the file and verifies its integrity by calculating a checksum during transfer.
Upload only different or new contents:
```
gocmd put --diff /local/dir /iplant/home/myUser/
```
This command uploads only files that are different or don't exist in the destination. It compares file sizes and checksums to determine which files need updating.
Upload via iCAT:
```
gocmd put --icat /local/dir /iplant/home/myUser/
```
This command uses iCAT as a transfer broker, useful when direct access to the resource server is unstable.
Upload via resource server:
```
gocmd put --redirect /local/dir /iplant/home/myUser/
```
This command bypasses the iCAT server for data transfer, directly accessing the specified resource server for optimized performance.

Download Data Objects (files) and Collections (directories) From the Data Store¶

Download a data object to a specific local path:
```
gocmd get /iplant/home/myUser/file.txt /local/path/file_new_name.txt
```
This command downloads the data object /iplant/home/myUser/file.txt and saves it as /local/path/file_new_name.txt.
Download a collection to a specific local path:
```
gocmd get /iplant/home/myUser/dir /local/path/
```
This command downloads the collection /iplant/home/myUser/dir and its contents to /local/path. A new directory named dir will be created under /local/path, resulting in /local/path/dir containing all the downloaded files and subdirectories.

Download with progress bars:

gocmd get --progress /iplant/home/myUser/largefile.dat /local/dir/

Force download:
```
gocmd get -f /iplant/home/myUser/largefile.dat .
```
This command overwrites the local file without prompting if it already exists.
Download and verify checksum:
```
gocmd get -K /iplant/home/myUser/important_data.txt .
```
This command downloads the file and verifies its integrity by calculating the checksum after download and comparing it with the original in the Data Store. This ensures data consistency and detects any corruption during transfer.
Download only different or new contents:
```
gocmd get --diff /iplant/home/myUser/dir /local/dir
```
This command downloads the source collection to the local directory, transferring only files that are different or don't exist locally. It compares file sizes and checksums to determine which files need updating, making the transfer more efficient by skipping unchanged files.
Download via iCAT:
```
gocmd get --icat /iplant/home/myUser/dir /local/dir
```
This command uses iCAT as a transfer broker, which is useful when direct access to the resource server is unstable. It ensures reliable data transfer by routing through the iCAT server.
Download via resource server:
```
gocmd get --redirect /iplant/home/myUser/dir /local/dir
```
This command bypasses the iCAT server for data transfer, directly accessing the specified resource server. It optimizes performance for large files by direct connection to the resource server.
Download with wildcard:
```
gocmd get -w /iplant/home/myUser/dir/file*.txt /local/dir
```
This command downloads all data objects matching the pattern "file*.txt" from the specified collection to the local directory.

Remove Data Objects (files) or Collections (directories) From the Data Store¶

Remove a single data object:
```
gocmd rm /iplant/home/myUser/file.txt
```

Remove an empty collection:

gocmd rmdir /iplant/home/myUser/emptyCollection

Remove a collection and its contents recursively:

gocmd rm -r /iplant/home/myUser/parentCollection

Force remove a collection and its contents recursively:
```
gocmd rm -rf /iplant/home/myUser/parentCollection
```

Move/Rename Data Objects (files) or Collections (directories)¶

Rename a data object:

gocmd mv /iplant/home/myUser/oldfile.txt /iplant/home/myUser/newfile.txt

Move a data object to a different collection:

gocmd mv /iplant/home/myUser/file.txt /iplant/home/myUser/subcollection/

Rename a collection:

gocmd mv /iplant/home/myUser/oldcollection /iplant/home/myUser/newcollection

Move multiple data objects with wildcard:
```
gocmd mv -w /iplant/home/myUser/*.txt /iplant/home/myUser/targetcollection/
```
This command will move all data objects with the .txt extension from the /iplant/home/myUser/ collection to the targetcollection. The asterisk (*) wildcard matches any number of characters in the filename.

You can use more specific wildcard patterns for precise file selection, such as file*.txt to move all text files starting with "file".

Additional Resources¶

For detailed information on GoCommands, refer to the GoCommands GitHub repository. The following command-specific documentation is available:

init: Initialize GoCommands configuration
env: Display or modify environment variables
passwd: Change user password
cd and pwd: Change and print working directory
ls: List directory contents
touch: Create empty files or update timestamps
mkdir: Create directories
rm: Remove files or directories
rmdir: Remove directories
mv: Move or rename files and directories
cp: Copy files or directories
cat: Display contents of a file
get: Download files from the Data Store
put: Upload files to the Data Store
bput: Bulk upload files to the Data Store
sync: Synchronize local and remote directories
chmod: Change access permission of files or directories
chmodinherit: Change access permission inheritance of directories
lsmeta: List metadata of data objects, collections, resources, or users in iRODS
addmeta: Add metadata to data objects, collections, resources, or users in iRODS
rmmeta: Remove metadata from data objects, collections, resources, or users in iRODS
copy-sftp-id: Configure SFTP Public-key Authentication
svrinfo: Display server information
ps: Display current iRODS sessions
upgrade: Upgrade GoCommands
copy-sftp-id: Copy SFTP identity file to the Data Store for SFTP public-key authentication