Uploading your data

Uploading your data

This feature is available to Owners and Members.
A data upload or download does not require a running Virtual Machine!
If you're uploading a folder with many subfolders, make a zip file!

In your workspace, you can upload a single file, a collection of files or even complete folders at a time. To ensure integrity and auditability of data, every upload is in principle wrapped up in a container, with a default title indicating the date, time and uploader. After uploading, you commit the container to the workspace, ensuring the data is stored in your workspace. To find the folder structure of the data drive, go to the Files tab in the workspace. Here, you can upload files and folders, which is explained below. 

Even though you can already create folders in the Files tab view, drag and dropping files in those folders does not result in an upload to that specific folder. Files above 5 MB will always be placed in the Inbox folder. From there you can distribute it to the appropriate folder.

Anyone who is a member of a workspace is allowed to upload data. However, not everyone has direct permission to download data! 
In the DRE web portal, you can choose to upload your files and folders via drag-and-drop, or via an upload pop-up. Alternatively, you can use Azure Storage Explorer to upload files to the workspace. Find the instructions for each option below.

To quickly scroll to the most suitable uploading options, see which situation fits your needs the best:
  1. I want to upload a small file below 5 MB - Drag & Drop or Upload menu.
  2. I want to upload a folder (or multiple) of small size - Drag & Drop.
  3. I want to upload a file larger than 5 MB - Drag & Drop or Upload menu.
  4. I want to upload large datasets - Azure Storage Explorer.
  5. I want to upload a folder with many subfolders - please read Folders with many subfolders. After that choose the preferred option.

How to upload files and folders through drag-and-drop

  1. In the workspace, open the Files tab.
  2. Drag-and-drop the files/folders you want to upload into the Files tab.
  3. Wait for the upload to finish.

  1. Commit the transfer to the workspace by pressing OK.

The files and folders will now be uploaded to the workspace. You will receive an e-mail when the upload is finished.

How to upload file(s) through the upload menu

By uploading through the upload menu, you cannot upload folders. For this, you need to upload through drag-and-drop or through the Azure Storage Explorer.
  1. In the workspace, open the Files tab.
  2. Click on the Upload-icon (red square) in the menu bar on top.

  1. By default, you create a new container folder for your upload. If you want your files to be uploaded to an already existing container, select this folder under Upload inbox.
  2. When creating a new container folder, you can edit the folder title under Create new. By default, the title contains data, time and uploader information.
  3. Click on Select and upload, select the files/folder you want to upload from the opened file explorer, then click on Open.
  4. When the upload is finished, click on Close
  5. Commit the transfer by pressing OK
The files will now be uploaded to the workspace.

How to upload files and folders through Azure Storage Explorer

Access to the workspace as Owner or Member.
Downloaded and installed Azure Storage Explorer: https://azure.microsoft.com/nl-nl/features/storage-explorer/
Corresponding SAS URI.
  1. In the workspace, navigate to the Files tab.
  2. Click on the Upload icon and select Use External Program.
  3. At the bottom of the pop-up a shared access signature URI (SAS URI) will be created.

  1. Copy the provisioned SAS URI.
  2. Open Azure Storage Explorer and click on the power plug icon (red circle).
  3. Click on the option 'Blob container' (blue circle).

  1. Select the Shared Access Signature (SAS) option and click on Next.

  1. Paste the copied SAS URI in the bottom field, and make note of what your container name is. Then click on Next and then Connect.
  2. In the dropdown system on the left, select the container (red square) and click on Upload (blue square).

  1. Choose the option file or folder in the next screen, browse for your specific file or folder, and click on Upload. You might be getting a warning about AzCopy. You can click Yes here to start the upload. You can check the progress under Activities in the bottom of the screen.
  2. After the upload is finished, you need to commit the upload. Navigate to your workspace and go to the data inbox manager (red square) and commit your uploads from there.

  1. Select the container you used to upload your data, select your data and click on Import to Workspace. This process might take a while, depending on the size of folders/files you have uploaded. Wait until you receive an e-mail that the upload has finished before using the files.

Important: do not import the workspace before your upload has completed! When you close the upload dialogue in the portal, it will automatically ask you to import the inbox. Do not click on OK before the upload is done! Rather just decline. 

What is a container?

The dialog for uploading data allows you to select a container for uploading. A container is a temporary storage folder that serves as a location where data is stored while you are uploading it. Once you are done uploading, you must commit the container to your workspace, i.e. you copy whatever is in the container to your workspace. Usually you want to create a new container every time you upload something, but if you wish to continue a previous upload (e.g. because you had to stop it half-way) you may choose to select an existing container here. Usually, though, you want to create a new container. You can rename the container if you like, this can be useful to keep different uploads apart from each other. After committing you will find the container in the Inbox folder.

It is good practice to delete the timestamped container folder when you have moved over the files to another persistent folder to avoid duplications of files.

Uploading details

Once you are done uploading, you can close the window. A dialog window will ask you if you want to commit the upload (i.e. move the uploaded data to your workspace). If you choose 'no', you can continue your upload later and commit them, or go to the upload inbox manager and commit the upload there. if you choose 'yes', simply wait and the upload will appear in your /data/inbox folder after a while.

Continue uploading
Do you want to continue the upload at a later time, or even another day? You can stop the upload by closing the window (ignoring any warning). Do not choose 'yes' when closing the dialog! When you want to continue uploading, go to this upload dialog, select the upload container where you started the upload, and upload the same files. Assuming the files didn't change, the uploader will notice you already uploaded part of the data and continue where it left off. If the files did change, it will warn you and ask what you want to do with it.

Remember that upload containers have a limited lifespan. Do not assume you can leave your data on them for a long time, they are scheduled to be cleared after about one week (although in reality they may last longer).

How much data can uploaded onto the Files tab?

By default the fileshare (data-drive/Z-drive) is limited to 5 TiB. Mostly because anDREa code is not yet updated to enable large file shares. For more statistics about the fileshare: https://docs.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets.

In case you need more storage, please contact the (Core) Support Team and discuss your needs. For large amount of storage, alternatives from Microsoft Azure (such as blob containers) other than the fileshare might be more suitable for you.

It is likely users will never reach this number as it takes a long time to write (60 MiB/sec by default or 300 MiB/sec for large file shares, but that can only be reached with in parallel read/write, and Windows is notoriously not doing that). Limits for different tiers (premium, hot, cold/archive) of the fileshare apply, but they have to be switched on manually and come with their own drawbacks.

 I/O limit is being worked on in SMB v3.x

Folders with many subfolders: folders missing

Large file uploads or folders with many subfolders use a different uploading mechanism, that is not visible for the user. Sometimes it can happen that certain folders are missing, especially when there are many subfolders. One of the solutions is to create a .zip file containing all of the folders and uploading it through one of the options listed above.

    • Related Articles

    • Uploading your data

      Both Owners and Members can upload files to the workspace. Tips: A data upload or download does not require a running Virtual Machine. If you're uploading a folder with many subfolders, make a zip file. To work in your DRE workspace, you of course ...
    • Downloading and transferring files via data requests

      Introduction Just as on researchenvironment.org, members with the role Researcher/Member have to create a download request to download data files from the Files tab. Owners of the workspace will receive an e-mail and can approve or deny this request ...
    • Blob storage

      Version: 2022-11-14 This service is available to Accountables and owners of a workspace. What is Azure Blob storage? Azure Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that does not adhere to a ...
    • ZorgTTP ‘DRM’ (Doel- en Receive Module) software in DRE workspace

      Following a use case from Acute Zorgregio Oost (AZO), anDREa, AZO and ZorgTTP collaborated on making ZorgTTP software available in a workspace. We would like to thank AZO and ZorgTTP sincerely for making this solution and help item available. ZorgTTP ...
    • [Resolved] Data upload & download issue

      What happened At 10:04 on 2022-11-07 it was first reported that upon download request, the status of the request remains at file copy pending, causing downloads not to come through. Cause The Azure Function that is responsible for handling copy ...