Uploading your data
This feature is available to workspace members all except the Data reader
A data upload or download does not require a running Virtual Machine!
If you're uploading a folder with many subfolders, make a zip file!
In your workspace, you can upload a single file, a collection of files or even complete folders at a time. To ensure integrity and auditability of data, every upload is in principle wrapped up in a container, with a default title indicating the date, time and uploader. After uploading, you commit the container to the workspace, ensuring the data is stored in your workspace. To find the folder structure of the data drive, go to the Files tab in the workspace. Here, you can upload files and folders, which is explained below.
Even though you can already create folders in the Files tab view, uploading files directly to those folders is not possible. Files will always be placed in the Inbox folder on the data drive (Z:). From there you can distribute them to the appropriate folder.
Anyone who is a member of a workspace is allowed to upload data (except for Data Readers). However, not everyone has direct permission to download data!
In the DRE web portal, you can choose to upload your files and folders via drag-and-drop, or via an upload pop-up. Alternatively, you can use Azure Storage Explorer to upload files to the workspace. Find the instructions for each option below.
To quickly scroll to the most suitable uploading options, see which situation fits your needs the best:
- I want to upload a small file below 5 MB - Drag & Drop or Upload menu.
- I want to upload a folder (or multiple) of small size - Drag & Drop.
- I want to upload a file larger than 5 MB - Drag & Drop or Upload menu.
- I want to upload large datasets - Azure Storage Explorer.
- I want to upload datasets with >1000 files - contact your (C)ST member
How to upload files and folders through drag-and-drop
- In the workspace, open the Files tab.
- Click on Upload file
- Select Direct Upload
- Drag-and-drop the files you want to upload into the field.
- Wait for the upload to finish. Click on Upload Files
The files and folders will now be uploaded to the workspace. You will receive an e-mail when the upload is finished. You will find them under the Data > Inbox in a time stamped folder (UTC time)
How to upload file(s) through the upload menu
By uploading through the upload menu, you cannot upload folders. For this, you need to upload through the Azure Storage Explorer.
In some versions of Microsoft Edge and Google Chrome, the upload menu gives an error. This is a known issue (see Can't open this file error while uploading data). The drag-and-drop method detailed above does work in those cases.
- In the workspace, open the Files tab.
- Click on Upload file
- Select Direct Upload
- Click to select files you want to upload.
- Wait for the upload to finish. Click on Upload Files
The files and folders will now be uploaded to the workspace. You will receive an e-mail when the upload is finished. You will find them under the Data > Inbox in a folder with a title containing data, time (UTC time).
How to upload files and folders through Azure Storage Explorer
WARNING
SAS Keys have a strict 7-day validity period from the date of creation. While access will be granted if the key is used within this timeframe, any attempts to use the key after the 7-day window has expired will result in an unauthorized error message.
⚠️ Please do not share the SAS Key! ⚠️
Requirements
Access to the workspace as Accountable, Privileged member, Advanced member, Standard member
Downloaded and installed Azure Storage Explorer (see Installing Azure Storage Explorer)
Corresponding SAS URI.
- In the workspace, navigate to the Files tab.
- Click on the Upload icon and select Large Files.
- On the next page of the pop-up a shared access signature URI (SAS URI) will be created.
- Copy the provisioned SAS URI.
- Open Azure Storage Explorer and click on the power plug icon (red circle).
- Click on the option 'Blob container' (blue circle).
- Select the Shared Access Signature (SAS) option and click on Next.
- Paste the copied SAS URI in the bottom field, and make note of what your container name is. Then click on Next and then Connect.
- In the dropdown system on the left, select the container (red square) and click on Upload (blue square).
- Choose the option file or folder in the next screen, browse for your specific file or folder, and click on Upload. You might be getting a warning about AzCopy. You can click Yes here to start the upload. You can check the progress under Activities in the bottom of the screen.
- After the upload is finished, you need to commit the upload. Navigate to your workspace and if you have your popup still open please click on Import to workspace
This button can also be found in the Files tab > Import files
- Select the container you used to upload your data, select your data and click on Import to Workspace. This process might take a while, depending on the size of folders/files you have uploaded. Wait until you receive an e-mail that the upload has finished before using the files.
Important: do not import the workspace before your upload has completed! When you close the upload dialogue in the portal, it will automatically ask you to import the inbox. Do not click on OK before the upload is done! Rather just decline.
What is a container?
The dialog for uploading data allows you to select a container for uploading. A container is a temporary storage folder that serves as a location where data is stored while you are uploading it. Once you are done uploading, you must commit the container to your workspace, i.e. you copy whatever is in the container to your workspace. Usually you want to create a new container every time you upload something, but if you wish to continue a previous upload (e.g. because you had to stop it half-way) you may choose to select an existing container here. Usually, though, you want to create a new container. You can rename the container if you like, this can be useful to keep different uploads apart from each other. After committing you will find the container in the Inbox folder.
It is good practice to delete the timestamped container folder when you have moved over the files to another persistent folder to avoid duplications of files.
Uploading details
Once you are done uploading, you can close the window. A dialog window will ask you if you want to commit the upload (i.e. move the uploaded data to your workspace). If you choose 'no', you can continue your upload later and commit them, or go to the upload inbox manager and commit the upload there. if you choose 'yes', simply wait and the upload will appear in your /data/inbox folder after a while.
Continue uploading
Do you want to continue the upload at a later time, or even another day? You can stop the upload by closing the window (ignoring any warning). Do not choose 'yes' when closing the dialog! When you want to continue uploading, go to this upload dialog, select the upload container where you started the upload, and upload the same files. Assuming the files didn't change, the uploader will notice you already uploaded part of the data and continue where it left off. If the files did change, it will warn you and ask what you want to do with it.
Remember that upload containers have a limited lifespan. Do not assume you can leave your data on them for a long time, they are scheduled to be cleared after about one week (although in reality they may last longer).
How much data can be uploaded onto the Files tab?
By default the fileshare (data-drive/Z-drive) is limited to 5 TiB. Mostly because anDREa code is not yet updated to enable large file shares. For more statistics about the fileshare: https://docs.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets.
In case you need more storage, please contact the (Core) Support Team and discuss your needs. For large amount of storage, alternatives from Microsoft Azure (such as blob containers) other than the fileshare might be more suitable for you.
I/O limit is being worked on in SMB v3.x
Folders with many subfolders: folders missing
Large file uploads or folders with many subfolders use a different uploading mechanism, that is not visible for the user. Sometimes it can happen that certain folders are missing, especially when there are many subfolders. One of the solutions is to create a .zip file containing all of the folders and uploading it through one of the options listed above.
Installing Azure Storage Explorer
Some institutes block their own users from installing software on company-managed PCs. Because research data is often originally stored on company-managed servers before it is uploaded to the DRE, it is necessary to have access to Azure Storage Explorer on a company-managed PC. Ask your (C)ST member or IT department about the possibilities.
To install Azure Storage Explorer manually, go to Azure Storage Explorer – cloud storage management | Microsoft Azure and select your operating system under the button 
(or 
). Download the install file and run it on your PC. Next, follow the steps above to start your upload.
IMPORTANT: Do not click on 
, as this leads you through the process of creating an Azure subscription. Azure subscriptions for myDRE are created for you by your institute, there is no need to do this yourself.
Related Articles
Uploading your data
All workspace members (except Data reader) can upload data to the workspace Tips: A data upload or download does not require a running Virtual Machine. If you're uploading a folder with many subfolders, make a zip file. To work in your myDRE ...[Resolved] Data upload issues
Issue description We have received some reports about data uploads not showing up in the portal and Z-drive. We are investigating the issue and will update this announcement when new information emerges. Event log 2023-11-22 @ 16:25 Test upload ...Can't open this file error while uploading data
Problem: Some users are reporting that when they add files in the Upload menu from their computer or use drag and drop functionality, using Microsoft Edge or Google Chrome they get an error “can’t open files in this folder because it contains system ...Blob storage
Version: 2022-11-14 Updated: 2024-08-12 This service is available to Accountables and Privileged members of a workspace. Blob storage in myDRE is restricted to the context of one workspace What is Azure Blob storage? Azure Blob storage is optimized ...Zipping files for quick up-/download
When you want to up- or download a lot of files or whole folder structures at once to/from your workspace, compressing the data into a single file (e.g., zip) is highly recommended. There are a few things you need to take into account when doing ...