Accessing Data Store¶
The Data Store is more than a place to save your files – it is a way to manage the life cycle of your data. From creation to publication to beyond, there are a number of practices to ensure that the integrity and value of your data are maintained.
In this lesson we will practice other ways of accessing the Data Store, plus how to make your data publicly available.
Discovery Environment Interface¶
DE interface allows uploading and downloading one file at a time. It works best for transfer of small files < 2GB.
From the DE interface, go to “Upload -> Simple Upload from Desktop” to upload a file. For downloading a file go to “Download -> Simple Download”.
Sample data for practice /iplant/home/shared/iplantcollaborative/example_data/FOSS_2020/foss_sampledata
iCommands is a collection of tools developed by the iRODS project, which is the technology that supports the CyVerse Data Store. Using iCommands is the most flexible way to interact with the Data Store.
iCommands provides command line access to the Data Store, so it can be included in scripts to automate data upload and download. Unfortunately, the latest iCommands cannot be installed on most Windows operating systems, but participants with Windows computers can do this exercise using Atmosphere (which will be covered in tomorrow’s lessons). If you are running Window 10, you can run iCommands on the Linux subsystem.
Follow along with the Using iCommands quick start to:
- Install and configure iCommands
- Upload a file to your home folder (iput)
- Download a file to your desktop (iget)
In addition, we will use iCommands to:
- Create a new folder in your home directory (imkdir)
$ imkdir newdir
- Move a file from your home directory to the new folder (imv)
$ imv file_name newdir/file_name
- Navigate to a public folder (icd)
$ icd /iplant/home/shared/imicrobe/camera
- List files in a directory
- Copy a public file to the newly created folder (icp)
$ icp camera_projects/CAM_PROJ_AcidMine.csv /iplant/home/$username/newdir/CAM_PROJ_AcidMine.csv $ icd /iplant/home/$username/ $ils newdir $ils -A newdir
Here is the full documentation of iCommands.
Cyberduck is a free 3rd party software tool that allows you to drag-and-drop files between your local computer (or a remote server) and the Data Store. Cyberduck can also be used to rename files, and browse other shared or public Data Store locations.
Follow along with the CyberDuck quick start to:
- Install and configure CyberDuck
- Upload a file to your CyVerse home directory
- Navigate to a public folder
WebDAV is an extension to the HTTP protocol that allows users to remotely edit and manage files. CyVerse has added support for WebDAV to the Data Store. This means users can access their home and public folders in the CyVerse Data Store from their local computers using web browsers and other WebDAV enabled applications such as common operating system file managers. With WebDAV, users can copy files between local computer and the Data Store as easily as if they were copying them between two folders on their computer.
Follow along with the WebDAV quick start to access CyVerse data via WebDAV services.
Which method to choose?¶
|Criteria||Cyberduck||iCommands||DE Interface||DE WebDAV|
|Ease of use||2||4||1||3|
|Works best for?||Multiple small files||Large files||Small files < 2GB||Small files < 2GB|
|Allows to open/edit files?||No||No||Yes||Yes|
CyVerse Data Commons¶
The Data Commons provides services throughout CyVerse to manage, organize, preserve, publish, discover, and reuse data.
For data that are not stable or permanent, you can request a Community Released Folder.
For an overview see Publishing data on the CyVerse Data Commons.