Data Management using iCommands¶

iCommands offers a variety of commands to help you manage your data in the Data Store. In the Data Store, file and directory are treated as data objects and collections, respectively. It's perfectly fine to consider these terms interchangeable.

Display the Current Working Collection¶

In the Data Store, the current working collection is equivalent to the concept of a current working directory in traditional file systems. You can display or change your current working collection using iCommands.

ipwd

By default, after configuring iCommands, your current working collection is set to your home directory, which is typically located at:

/<Zone Name>/home/<Username>

Note: Paths in the Data Store always start with the zone name /iplant.

Change the Current Working Collection¶

Change to a specific collection using an absolute path:
```
icd /iplant/home/myUser/mydata
```
This changes your current working collection to /iplant/home/myUser/mydata.
Use a relative path from your current location: Assuming your current working collection is /iplant/home/myUser:
```
icd mydata
```
Return to your home collection:
```
icd "~"
```
Note: The ~ must be quoted to prevent shell expansion by your local shell. Without quotes, it will expand to your local machine's home directory instead of your Data Store home directory.
Move up one level:
```
icd ..
```

List Data Objects (files) and Collections (directories)¶

List the content of a collection:
```
ils /iplant/home/myUser/mydata
```
This will display the data objects and collections in the /iplant/home/myUser/mydata collection:
```
/iplant/home/myUser/mydata:
    file1.bin
    file2.bin
    C- /iplant/home/myUser/mydata/subdir1
```
The C- prefix indicates that the item is a collection (directory).
List the content of the current working collection:
```
ils
```
List the contents of a collection in long format with additional details:
```
ils -l /iplant/home/myUser/mydata
```
This command will show the data objects and collections within /iplant/home/myUser/mydata, along with their additional details:
```
/iplant/home/myUser/mydata:
    myUser  0   demoRes1;rs1    436 2024-04-02.13:36    &   file1.bin
    myUser  1   demoRes2;rs2    436 2024-04-02.13:36    &   file1.bin
    myUser  0   demoRes1;rs1    700 2024-04-02.17:15    &   file2.bin
    myUser  1   demoRes2;rs2    700 2024-04-02.17:15    &   file2.bin
    C- /iplant/home/myUser/mydata/subdir1
```
- Each output line for a data object represents a replica. If iRODS is configured to create multiple replicas, you will see one line for each replica of the data object. For example, if two replicas are created, two lines will be displayed for each file.
- Each line is shown in the owner replica_id resource_server size creation_time replica_state name format.
- Possible replica states are:
  - &: Good
  - X: Stale
  - ?: Unknown
List the contents of a collection with their access control lists:
```
ils -A /iplant/home/myUser/mydata
```
This command will show the data objects and collections within /iplant/home/myUser/mydata, along with their access control lists (ACLs):
```
/iplant/home/myUser/mydata:
        ACL - g:rodsadmin#iplant:own    myUser#iplant:own
        Inheritance - Disabled
    file1.bin
        ACL - g:rodsadmin#iplant:own    myUser#iplant:own
    file2.bin
        ACL - g:rodsadmin#iplant:own    myUser#iplant:own
    C- /iplant/home/myUser/mydata/subdir1
```
- The g: prefix in the ACL username indicates that the user is a group.
- The ACL is displayed in the username#zone:access_level format.
- Most common access levels are:
  - read_object: Allows read access to the data object or collection.
  - modify_object: Allows modification (write) of the data object or collection.
  - own: Grants ownership of the data object or collection.

Make a Collections (directories)¶

Create a new collection:

imkdir /iplant/home/myUser/newCollection

Create parent collections if they do not exist:
```
imkdir -p /iplant/home/myUser/parentCollection/newCollection
```
This command creates the newCollection along with its parent collection parentCollection if it does not already exist.

Upload Data Objects (files) and Collections (directories) to the Data Store¶

Warning

When uploading your data to the Data Store, avoid using:

Spaces in names (e.g., experiment one.fastq)
Special characters: ~ `` ! @ # $ % ^ & * ( ) + = { } [ ] | : ; " ' < > , ? / and \

These may cuase issues with Discovery Environment Apps and command-line applications.

Recommendation: Use underscores for long names (e.g., experiment_one.fastq).

Upload a single file:
```
iput /local/path/file.txt /iplant/home/myUser/
```
This command uploads the file /local/path/file.txt to /iplant/home/myUser/, creating /iplant/home/myUser/file.txt in the Data Store.
Upload a directory and its contents:
```
iput /local/dir /iplant/home/myUser/
```
This command uploads the contents of the directory /local/dir to /iplant/home/myUser/dir in the Data Store. The uploaded files and subdirectories will be placed within the /iplant/home/myUser/dir folder.

Upload with progress output:

iput -P /local/path/largefile.dat /iplant/home/myUser/

Force upload:
```
iput -f /local/path/largefile.dat /iplant/home/myUser/
```
This command overwrites the existing file in the Data Store without prompting.
Upload and verify checksum:
```
iput -K /local/path/important_data.txt /iplant/home/myUser/
```
This command uploads the file and verifies its integrity by calculating a checksum during transfer.
Upload via resource server:
```
iput -I /local/dir /iplant/home/myUser/
```
This command bypasses the iCAT server for data transfer, directly accessing the specified resource server for optimized performance.
Upload with connection auto-renewal:
```
iput -T /local/dir /iplant/home/myUser/
```
Renews connections every 10 minutes to prevent failures due to connection or firewall issues.

Download Data Objects (files) and Collections (directories) From the Data Store¶

Download a data object to a specific local path:
```
iget /iplant/home/myUser/file.txt /local/path/file_new_name.txt
```
This command downloads the data object /iplant/home/myUser/file.txt and saves it as /local/path/file_new_name.txt.
Download a collection to a specific local path:
```
iget /iplant/home/myUser/dir /local/path/
```
This command downloads the collection /iplant/home/myUser/dir and its contents to /local/path. A new directory named dir will be created under /local/path, resulting in /local/path/dir containing all the downloaded files and subdirectories.

Download with progress output:

iget -P /iplant/home/myUser/largefile.dat /local/dir/

Force download:
```
iget -f /iplant/home/myUser/largefile.dat .
```
This command overwrites the local file without prompting if it already exists.
Download and verify checksum:
```
iget -K /iplant/home/myUser/important_data.txt .
```
This command downloads the file and verifies its integrity by calculating the checksum after download and comparing it with the original in the Data Store. This ensures data consistency and detects any corruption during transfer.
Download via resource server:
```
iget -I /iplant/home/myUser/dir /local/dir
```
This command bypasses the iCAT server for data transfer, directly accessing the specified resource server. It optimizes performance for large files by direct connection to the resource server.
Download with connection auto-renewal:
```
iget -T /iplant/home/myUser/dir /local/dir
```
Renews connections every 10 minutes to prevent failures due to connection or firewall issues.

Remove Data Objects (files) or Collections (directories) From the Data Store¶

Remove a single data object:
```
irm /iplant/home/myUser/file.txt
```

Remove an empty collection:

irmdir /iplant/home/myUser/emptyCollection

Remove a collection and its contents recursively:
```
irm -r /iplant/home/myUser/parentCollection
```
Force remove a collection and its contents recursively:
```
irm -rf /iplant/home/myUser/parentCollection
```

Move/Rename Data Objects (files) or Collections (directories)¶

Rename a data object:

imv /iplant/home/myUser/oldfile.txt /iplant/home/myUser/newfile.txt

Move a data object to a different collection:

imv /iplant/home/myUser/file.txt /iplant/home/myUser/subcollection/

Rename a collection:

imv /iplant/home/myUser/oldcollection /iplant/home/myUser/newcollection

Additional Resources¶

For detailed information on iCommands, refer to the iRODS iCommands Docs.