[Mitigated] Proxy server showing unhealthy status
What happened?
Our monitoring system alerted us on 2023-12-20 at 11:45 (GMT+1) that our proxy server returned an unhealthy status. This might affect your experience with reaching the Z: drive and domain allowlisting.
We've immediately started the investigation and will update here. We apologize for any inconvenience this might cause you.
Event log
Update 4: 2023-12-20 @ 12:40 (GMT+1)
The cause has been identified: logs are getting cycled normally, however AzCopy stores it logs in a different location which does not get cleaned up. This has accumulated and clogged the disk, in turn causing the proxy instances to return an unhealthy status and failure.
The development team of anDREa will work on a fix. In the meantime, the two newly added healthy instances function normally and additional alerting has been configured to help prevent this from happening again until the fix is live.
For now, this issue has been marked as Mitigated.
Update 3: 2023-12-20 @ 12:17 (GMT+1)
Two new healthy proxy instances have been added to mitigate, while the development team of anDREa debugs the other instances. Users should be able to reach their Z: drive and use domain allowlisting (again).
The alert will remain active until the cause has been identified and resolved.
Update 2: 2023-12-20 @ 12:06 (GMT+1)
A potential cause has been identified, the anDREa development team is working on a solution.
Update 1: 2023-12-20 @ 11:57 (GMT+1)
Received an alert from our monitoring system with:
"Fired:Sev0 Azure Monitor Alert Squid Fatal Error - andrea on andrea-log-analytics ( microsoft.operationalinsights/workspaces ) at 12/20/2023 10:44:38 AM"
Investigation has started; no user reports yet. More updates will follow as we know more.
Related Articles
Proxy configurations for connecting to a server or blob
Version: 2022-11-02 Update: 2022-11-14 Introduction Domain allowlisting is a selfservice feature that is now available on myDRE. As the name implies, domains on the allowlist can be accessed from within the Workspace (you can request specific domain ...Turn Proxy Off/On on Linux VM
1. Log in to the Linux VM 2. Write command sudo nano /etc/environment press enter and enter your password 3. comment out all the content that you find in there e.g #http_proxy="http://127.0.0.1:8080" 4. Press ctrl + x to exit, save with y and press ...Proxy configurations Anaconda
First version: 2022-10-31 Latest version: 2023-11-16 Last change(s): Clarification in the introduction; clarifying instructions; added screenshots. This is a community effort. The article was created by the anDREa Support Team in their spare time. If ...Shiny Server in a myDRE Workspace
Version = 2023-02-02 This instruction is used to get Shiny Server up and running in a Workspace. For your actual workloads you might need other size VMs. Also there is much more to be configured, but being able to get Shiny up and running is a good ...Solution to drive/client/server issues in a Workspace
Introduction Some of the workspaces have manually configured the proxy to allow domain whitelisting. Earlier, we received messages concerning the visibility of the I: and Z: drives when the proxy was configured. We have updated the article Manual ...