Skip to main content

Variables and secrets

Variables and secrets, such as credentials and configurations, are often critical for computational work. Furthermore, such variables and secrets often need to be secured and shared among teams. Deep Origin supports the following types of variables and secrets, making it easy for you to organize them, share them, and install them into workflows and workstations.

  • Environment variables: Key-value pairs, such as for usernames and passwords for accessing relational databases.
  • Configuration files: Files for customizing programs such as JupyterLab.
  • AWS profile: Profile names, access keys, and secret keys such as for uploading and downloading data to and from S3 buckets.
  • GPG private keys: Keys for encrypting, decrypting, and signing messages such as for signing Git commits.
  • HTTP credentials for Git: Usernames and password/tokens for cloning, pushing, and pulling Git repositories.
  • SSH private keys: Private keys for access remote computers via SSH.

Variables and secrets can be defined at the user account and organization levels. We recommend using user secrets to manage variables and secrets that represent you as an individual, such as GitHub personal access tokens (PATs). We recommend using organization secrets to manage variables and secrets that should be shared with your entire organization, such as an environment variable for a shared license for a database.

Follow these steps to manage variables and secrets, share them with your organization, and install them into your workflows and workstations.

  1. Create a variable or secret. For example, create an AWS IAM profile for accessing an AWS S3 bucket, and save the access key ID and secret access key.
  2. Add the variable to Deep Origin.
    1. To add a user variable or secret, navigate to your user account. To add a variable or secret to an organization, navigate to the organization.
    2. Within your user account or organization, open the "Variables and secrets" tab.
    3. Click the "Create" button.
    4. Complete the on screen form by selecting the type of variable or secret you wish to add and entering its value. See below for detailed information about each type of variable and secret.
    5. Click the "Save" button.
  3. Install the variable into a workstation.
    1. Create and connect to a terminal for a workstation. For example, connect to a workstation via SSH or launch a terminal within JupyterLab, RStudio Server, or VS Code.
    2. To pull variables and secrets from the Deep Origin OS and install them into a workstation, run the Deep Origin command line interface (CLI) within the workstation by executing deeporigin variables install. See below for detailed information about how the Deep Origin CLI installs each type of variable and secret into workstations.
    3. Open the prompted link in your web browser to sign your workstation into Deep Origin.
    4. To install environment variables and encrypted environment variables into the current shell, execute set -o allexport && source ~/.deeporigin/variables.env && set +o allexport. Note, running services such as JupyterLab, RStudio Server, and VS Code must be restarted to access the installed variables and secrets.

Supported types of credentials and configurations​

Deep Origin supports the types of variables and secrets outlined below. For assistance with additional types of credentials and configurations, please contact support.

Environment variables​

Environment variables are shell environment variables. For example, the key LANG and value es_ES.UTF-8 can be used to set the language of Bash shells within your workstation to Spanish.

The Deep Origin CLI installs shell environment variables into Bash shells. Within workstations, their values are installed into ~/.deeporigin/variables.env.

Encrypted environment variables​

Encrypted environment variables are shell environment variables whose values are encrypted within the Deep Origin OS. For example, MYSQL_PWD=**** can be used to capture a password for accessing a MySQL database.

The Deep Origin CLI installs encrypted environment variables similarly to environment variables.

Files​

File variables are plain text files. For example, ~/.ipython/profile_default/ipython_config.py can be used to configure IPython settings such as its shell colors.

The Deep Origin CLI installs file variables to the filenames that you specify within the Deep Origin web application. The Deep Origin CLI supports the tilde (~) shortcut, automatically substituting it for the home directory of the current user, typically /home/bench-user/.

Encrypted files​

Encrypted files are plain text files whose values are encrypted within the Deep Origin OS. For example, ~/.mylogin.cnf can be used to capture a username and password for accessing a MySQL database.

The Deep Origin CLI installs encrypted files similarly to file variables.

AWS profiles​

AWS profiles capture the name, access key, secret key, and default region of AWS profiles.

Information about creating AWS profiles is available in the AWS documentation.

The Deep Origin CLI installs AWS profiles using the AWS CLI. In turn, the AWS CLI installs profiles into ~/.aws/config and ~/.aws/credentials.

Git HTTP credentials​

Git HTTP credentials capture usernames and passwords or tokens for cloning, pushing, and pulling Git repositories over HTTP/HTTPS.

Information about creating access tokens for GitHub is available in the GitHub documentation.

Within workstations, the Deep Origin CLI installs Git credentials into ~/.git-credentials.

GPG private keys​

GPG private keys are a popular tool for decrypting and signing messages. For example, GPG keys can be used to sign Git commits.

To create a GPG key and add it to Deep Origin, follow these steps:

  1. Install GPG:
    1. Ubuntu: Run apt install gpg.
    2. MacOS, Windows, and other OSes: Download and install the executables available at https://gnupg.org/.
  2. Create a key:
    1. Execute gpg --default-new-key-algo rsa4096 --gen-key.
    2. Follow the onscreen instructions.
  3. Copy the ID of your key:
    1. Execute gpg --list-keys.
    2. Find the ID for your key within the output and copy it.
  4. Export your key to a file:
    1. Execute gpg --output {filename} --armor --export-secret-key {id}, replacing id with the ID copied in the previous step and filename with the path you would like to export the key to.
    2. Copy the contents of the exported file.
  5. Within the Deep Origin web application, enter the contents of the key file copied in the previous step.

Within workstations, the Deep Origin CLI installs GPG keys by executing gpg --batch --import.

SSH private keys​

SSH private key secrets are credentials for accessing remote machines via SSH. SSH private keys are often stored at locations such as ~/.ssh/id_rsa.

Within workstations, the Deep Origin CLI installs SSH private keys by saving them to the filenames you specify in the Deep Origin web application and adding those filenames to your SSH configuration (~/.ssh/config).

Precedence of user and organization variables and secrets​

User variables and secrets have precedence over organization variables and secrets. For example, if your account has an environment variable with the key KEY and value MY-VALUE and your organization has an environment variable with the same key and value ORG-VALUE, MY-VALUE will be installed into your workstations.

Configuring a workstation to automatically synchronize variables and secrets with the Deep Origin OS​

To configure your workstation to regularly pull your variables and secrets from the Deep Origin OS and install them into your workstation, within the workstation run deeporigin variables auto-install --time [interval-in-min]. For example, run deeporigin variables auto-install --time 5 to pull and install your variables and secrets every five minutes.

To disable regularly updating your variables and secrets, within the workstation run deeporigin variables auto-install --disable.

Uninstalling variables and secrets from a workstation​

To remove the variables and secrets installed into a workstation, within the workstation run deeporigin variables uninstall. This will undo the installation of each type of variable and secret from your workstation.

Overwriting local changes to your variables and secrets within a workstation​

By default, the Deep Origin CLI command will not overwrite or remove variables and secrets whose values you have modified locally within your workstation. To overwrite and remove locally modified variables and secrets, run the Deep Origin CLI with the --overwrite option:

  • Install variables and secrets: deeporigin variables install --overwrite.
  • Synchronize variables and secrets: deeporigin variables auto-install --time [interval-in-min] --overwrite.
  • Uninstall variables and secrets: deeporigin variables uninstall --overwrite.

Security of secrets​

Deep Origin secures your secret attributes by encrypting them, and only exposing their decrypted values to workstations. As a result, the values of the secret attributes cannot be read within the Deep Origin web application.

Deep Origin encrypts the following attributes of each type of variable and secret:

  • Environment variable: None
  • Encrypted environment variable: Value
  • File: None
  • Encrypted file: Value
  • AWS profile: Secret access key
  • Git HTTP credentials: Password/token
  • GPG private key: Value
  • SSH private key: Value