Adding files to the dataset (Extraordinary procedure)

Prior Authorization

Before uploading files, CSUC must authorize the deposit as an extraordinary upload.

Once authorized, you will receive an email containing:

The upload link to access the space where files will be deposited
The password to access the space
The public link to view and download the dataset files

Procedure 1 – Web Interface

Once you have received the upload link and password, you can access the space created by CSUC via your browser.

From the web interface, you can:

Manually upload files
Create the desired folder structure

Important: Once the upload is complete, please get in touch with your institution’s support service to proceed with publishing the dataset.

Procedure 2 – Rclone

What is Rclone?

A command-line tool for managing files across more than 70 cloud storage providers
Equivalent to rsync, cp, mv, mount, ls for the cloud
Automatically resumes interrupted transfers

Rclone Installation Guide (Linux)

Step 1: Visit the official page: https://rclone.org/
Step 2: Installation

Ubuntu/Debian

curl https://rclone.org/install.sh | sudo bash

CentOS/RHEL/Fedora

curl https://rclone.org/install.sh | sudo bash

Arch Linux

sudo pacman -S rclone

You can also download it directly from: https://rclone.org/downloads/

Rclone Installation Guide (MacOS)

Step 1: Visit the official page: https://rclone.org/
Step 2: Installation

Homebrew

brew install rclone

MacPorts

sudo port install rclone

You can also download it directly from: https://rclone.org/downloads/

Rclone Installation Guide (Windows)

Step 1: Visit the official page: https://rclone.org/
Step 2: Installation

Chocolatey

choco install rclone

Scoop

scoop install rclone

You can also download it directly from: https://rclone.org/downloads/

Configuration for Uploading Files

Step 3: Run the configuration in Terminal or PowerShell:

rclone config

Steps:

Select n to create a new configuration
Enter a name for the remote (e.g., owncloud_upload)
Choose WebDAV as the remote type
Enter the server URL
- From the link https://disc.csuc.cat/s/ornFnBuUVtQDIjU, extract the URL in the format:
  https://disc.csuc.cat/remote.php/dav/public-files/ornFnBuUVtQDIjU
Choose Other as the vendor
Username → public
Password → As received in the CSUC email
Leave bearer_token empty (press Enter)
Choose n for advanced configuration
Save the configuration

File Upload Instructions

NOTE: The SYNC command will generate the same directory structure that exists within the specified path.

Basic upload:

rclone sync ./dataset_local/ "owncloud_upload:/" --progress --log-level INFO

Upload with checksum verification:

rclone sync ./dataset_local/ "owncloud_upload:/" --progress --log-level INFO --checksum

Bandwidth limitation:

rclone sync ./dataset_local/ "owncloud_upload:/" --progress --log-level INFO --bwlimit=1M

Useful options:

Opció	Descripció
`--progress`	Shows transfer progress
`--log-level INFO`	Provides detailed logs
`--transfers=4`	Parallel transfers
`--checkers=8`	Parallel checkers
`--bwlimit=1M`	Limits bandwidth to 1MB/s

Upload Verification

List remote content:

rclone ls "owncloud_upload:/"

Check local vs remote content:

clone check ./dataset_local/ "owncloud_upload:/" --one-way

Automation Script (Linux/MacOS)

#!/bin/bash
script_descarga.sh
 
REMOTE_NAME="owncloud"
REMOTE_PATH="/"
LOCAL_DIR="./dataset_local"
 
echo "Iniciant càrrega des de $REMOTE_NAME:$REMOTE_PATH"
rclone sync "$LOCAL_DIR" "$REMOTE_NAME:$REMOTE_PATH" --progress --log-level INFO --transfers=4 --checkers=8
 
if [ $? -eq 0 ]; then
echo "Càrrega completada amb èxit"
else
echo "Error durant la càrrega - pots reanudar executant la mateixa comanda"
exit 1
fi

Automation Script (Windows)

$remoteName = "owncloud_upload"
$remotePath = "/"
$localDir = "./dataset_local"

Write-Host "Iniciant càrrega cap a $remoteName`:$remotePath"
rclone sync $localDir "$remoteName`:$remotePath" --progress --log-level INFO --checksum

if ($LASTEXITCODE -eq 0) {
  Write-Host "Càrrega completada amb èxit"
} else {
  Write-Host "Error durant la càrrega - pots reexecutar la mateixa comanda"
  exit 1
}

References and Tips

Rclone official page (https://rclone.org/)
OwnCloud WebDAV API documentation (https://owncloud.dev/apis/)
Rclone Manual (https://rclone.org/docs/)
WebDAV configuration with Rclone (https://rclone.org/webdav/)

Consells útils:

Always use --progress --log-level INFO to monitor the process
In case of interruption, rerun the command: Rclone resumes automatically and does not duplicate already uploaded files
For large datasets, consider using --bwlimit to avoid saturating your connection
Verify integrity with rclone check

Space shortcuts

Page tree

What is Rclone?

NOTE: The SYNC command will generate the same directory structure that exists within the specified path.