Larger file uploads from bulk storage/HPC with winscp/scp

Larger file uploads from bulk storage/HPC with winscp/scp

When you want to upload files from bulk storage/HPC to your DRE workspace, you might run into the issue that you don't have the DRE portal available. Therefore, you cannot use the usual upload mechanics.
Of course, copying large files to another environment, to then upload them to your DRE workspace, is inefficient, costly and takes up too much time.
Luckily, there is at least one alternative. Please continue reading below.

Pulling data from a DRE VM with winscp/scp

Instead of 'pushing' data from storage to a DRE workspace such as when you use the DRE web portal, you can also set up your workspace and VMs to 'pull' data from other locations . 
  1. First add an External access rule to the storage location. You need to specify the IP address and the port. For sftp, used by winscp, the portnumber is 22. The IP is of course dictated by the storage location. 
    1. After creation, the External access rule still needs to be turned on by checking the mark!
    2. An External access rule allows outbound connections to be made from the VM to specific IP addresses on specific ports.
  2. If you're using a Windows VM, install winscp. Winscp will be included in the standard UMC Utrecht image, but for now, read how to self-service install software.
  3. From within the VM, use winscp/scp to log in and then to download data from the storage location.

Important: by pulling data this way, there is not 'logging' of the data being upload to the workspace. There's only logging of the External Access rule being created and turned on.

  



    • Related Articles

    • Software on Windows VMs (UMCU)

      This article is only for UMCU workspaces! To work with data on your Windows VM in a DRE workspace, you of course need software. For UMC Utrecht workspaces, there are a few different ways in which software ends up on a Windows VM: Some software comes ...

    • How to remove the red crosses in the Software Store

      This article is only for UMCU workspaces! Introduction By default, the icons of applications within the Software Store on the UMCU template will not be visible, instead a red cross will be shown. This is caused by the missing connection to a blob ...