From Vulnerabilities to Vault: How We Stopped Hardcoding Secrets and Started Using Hashicorp Vault

We recently migrated our infrastructure from Kubernetes to HashiCorp Nomad.

Soon after, we encountered service discovery issues and integrated Consul to address them.

At this point, we were feeling a bit more relaxed, knowing that we could dedicate less time to infrastructure and focus more on product development, as we don’t have a separate resource dedicated to DevOps.

However, we encountered another challenge—a Nomad port was open on the Node, and a friendly person helped us identify this vulnerability.

They were able to access it directly using the server's IP address, without needing a domain.

As a result, we had to quickly rotate all our payment keys, AWS credentials, and other secrets.

The Problem: Hardcoded Secrets and Security Risks

In some areas, our team had hardcoded keys directly into the codebase.

This meant there wasn’t a single place to manage these secrets, and updating them across our systems was a manual, error-prone process.

It was clear we needed a solution that would allow us to change a secret in one place, and have it propagate automatically throughout the system.

Initially, we considered using AWS Secrets Manager alongside our GitLab CI/CD pipeline to inject environment variables during container deployments.

However, this approach required passing a large number of variables as arguments to nomad job run from the project's .gitlab-ci.yml, which cluttered our configuration and reduced system readability.

That’s when we thought, Why not go full HashiStack? We decided to explore Vault.

Exploring Secret Management Options

AWS Secrets Manager

AWS Secrets Manager offers capabilities for managing and rotating secrets such as database credentials, API keys, and other sensitive information.

• Access Management: AWS Secrets Manager relies on AWS IAM (Identity and Access Management) for managing permissions. For granular control, additional tools like Chamber maybe required. Chamber leverages AWS IAM roles and policies to restrict access by namespace, simplifying the process of partitioning secrets across different environments.
• Cross-Cloud Limitations: While effective within AWS (our existing infra), use in multi-cloud environments.
• Cost of service: $0.40 per secret per month.

HashiCorp Vault

Hashicorp Vault provides a robust, identity-based security solution designed to authenticate and authorize access to secrets automatically, supporting a wide range of infrastructures.

• Access Control: Vault offers RBAC (Role-Based Access Control) for user logins, enabling administrators to manage human access to secrets. For machine access, Vault provides policy-driven access with leases, allowing for precise control over which applications can access secrets and for how long. This includes options like automatic key rotation and expiration.
• Dynamic Secrets Generation: Vault can dynamically generate AWS IAM credentials, providing temporary access that’s tightly controlled by policies and easily revoked if needed.
• Cross-Environment Compatibility: Vault supports cross-region, cross-cloud, and cross-datacenter replication, making it a seamless choice for organizations with multi-cloud setups.
• Cost of service: Free.

Why We Chose HashiCorp Vault

Vault offered the ideal solution: it allowed Nomad to request sensitive information, such as payment secrets or AWS keys, directly from Vault.

Vault would verify if Nomad was authorized to access those secrets and promptly deliver the data.

This way, we could centralize key management, automate secret updates, and improve security by avoiding hardcoded keys.

We developed a proof of concept, where we demonstrated Vault's ability to manage key rotation and securely store secrets in one place, accessible across all our projects.

This not only simplified secret management but also ensured consistency and security for our entire infrastructure stack.

Getting Started with HashiCorp Vault

As with most HashiCorp products, Vault comes with comprehensive and beginner-friendly documentation to help you get started quickly. The official "Getting Started" guide provides step-by-step instructions for installation, basic setup, and creating your first secrets. You can find it here: Vault Getting Started Guide.

This guide will walk you through the initial setup, from installing Vault on your system to configuring secure access and creating policies. Whether you’re deploying locally or in production, HashiCorp’s documentation makes it easy to understand the process and best practices for integrating Vault into your infrastructure.

Retrieving Secrets can be done in many ways

HashiCorp Vault provides multiple methods to retrieve secrets, allowing flexibility in choosing the best method for your specific use case.

Using the Vault CLI

The Vault CLI is a straightforward and commonly used tool to interact with secrets in Vault. Here are some key examples:

1. Create or Update a Secret
   To store or update a key-value pair (e.g., foo=bar) in the secret mount:

   vault kv put -mount=secret foo bar=baz
   

2. Retrieve a Secret
   To read the value back:

   vault kv get -mount=secret foo
   

3. Get Metadata for a Secret Key
   Retrieve metadata about a key (such as creation time, version count):

   vault kv metadata get -mount=secret foo
   

4. Retrieve a Specific Version of a Secret
   Access a specific version if versioning is enabled:

   vault kv get -mount=secret -version=1 foo
   

For more information, see the CLI documentation on secrets engines.

Using REST APIs

Vault’s REST API allows you to interact programmatically, making it ideal for automation and integration in scripts or applications.

1. Retrieve a Secret

   curl \
       --header "X-Vault-Token: hvs.blah" \
       --request GET \
       https://myvault.com/v1/kv/data/aws/s3 | jq -r ".data.data"
   

2. Store a Secret
   To store data in JSON format, for example, {"foo": "bar"}:

   curl \
       --header "X-Vault-Token: hvs.blah" \
       --request POST \
       --data '{ "data": {"foo": "bar"} }' \
       https://myvault.com/v1/kv/data/aws/s3 | jq -r ".data"
   

Refer to the Vault API documentation for a detailed guide on accessing secrets via REST.

Using Nomad with Vault Integration

Vault integrates seamlessly with Nomad, allowing secrets to be securely accessed during job deployments. Here’s a high-level guide:

1. Define Vault Policies for Nomad
   Write a Vault policy to specify which paths Nomad jobs can access:

   tee nomad-job-policy.hcl <<EOF
   path "auth/token/lookup" {
     capabilities = ["update"]
   }
   path "kv/" {
     capabilities = ["read"]
   }
   EOF
   
   
   vault policy write nomad-job nomad-job-policy.hcl
   

2. Configure Vault Authentication for Nomad
   Assign a Vault token role for Nomad:

   vault write auth/token/roles/nomad-cluster policies="nomad-job"
   

3. Inject Vault Secrets in Nomad Jobs
   Use environment variables from Vault in the Nomad job file, leveraging job templating.

For a more detailed tutorial, visit the Vault-Nomad integration guide.

Using Vault in a Nomad Job File (Example)

1. Dynamic Secret Injection
   The template block retrieves secrets from Vault and creates a .env file in the container. This keeps sensitive data out of the job file, injecting secrets like S3_ACCESS_KEY and MONGO_USER_PASSWORD securely at runtime.

2. Automatic Secret Management
   Vault’s templating ensures the latest secrets are populated in the container each time the job runs. This setup supports secret rotation without code changes.

3. Secure Environment Configuration
   Using templating, environment variables are securely configured from Vault. Secrets are automatically updated in .env, reducing risks and making secure deployments.

job "service" {
  datacenters = ["dc1"]
  type        = "service"
  priority    = 1

  group "backend" {
    network {
      port "http" { to = 1337 }  // Map to Docker port 1337
    }

    task "service-task" {
      driver = "docker"

      config {
        image = "${var.image_name}"
        auth {
          username = "${var.registry_user}"
          password = "${var.registry_password}"
        }
        ports = ["http"]

        volumes = [
          "secrets/.env:/code/.env"  // Mounting the .env file into the working directory of the Docker container.

        ]
      }

      vault {
        policies = ["nomad-job"]
      }

      template {
        destination = "secrets/.env"  // Path where the .env file will be created
        data = <<EOF
        {{ with secret "main/aws/s3" }}
        S3_ACCESS_KEY="{{ .Data.data.access_key }}"
        S3_SECRET_KEY="{{ .Data.data.secret_key }}"
        {{ end }}

        {{ with secret "main/aws/ses" }}
        SES_ACCESS_KEY="{{ .Data.data.access_key }}"
        SES_SECRET_KEY="{{ .Data.data.secret_key }}"
        {{ end }}


        {{ with secret "main/payment/razor-pay" }}
        RAZORPAY_TEST_ACCESS_KEY="{{ .Data.data.test_access_key }}"
        RAZORPAY_TEST_SECRET_KEY="{{ .Data.data.test_secret_key }}"
        {{ end }}

        {{ with secret "main/db/mongo" }}
        MONGO_HOST="{{ .Data.data.mongo_host }}"
        MONGO_PORT="{{ .Data.data.mongo_port }}"
        MONGO_USER_NAME="{{ .Data.data.mongo_user_name }}"
        MONGO_USER_PASSWORD="{{ .Data.data.mongo_password }}"
        {{ end }}

        EOF
      }
      service {
        name      = "service-com"
        tags      = ["service.com"]
        port      = "http"
      }
    }
  }
}

variable "image_name" {
  type = string
}

variable "registry_user" {
  type = string
}

variable "registry_password" {
  type = string
}

Wrapping Up: Securing the Future of Our Infrastructure with Vault

Transitioning from hardcoded secrets to HashiCorp Vault was more than just a security upgrade—it was a shift in our approach to managing infrastructure.

Vault, combined with Nomad and Consul, has transformed how we think about security, offering peace of mind and flexibility in a fast-paced DevOps landscape.

Whether you're a growing startup or a mature enterprise, implementing a robust secrets management solution like Vault can streamline your workflows and ensure that your secrets are as secure as your code.

So, if you’re still hardcoding, maybe it’s time to make the leap. After all, Vaulting your secrets isn’t just good for security—it’s good for sanity, too.

Happy Vaulting!

Stay ahead of the curve! Subscribe for a weekly dose of insights on
development, IT, operations, design, leadership and more.