cloudscaping

Tag Management with Azure Policy

2020-06-23T00:00:00+00:00

Following on from my previous post about managing tags using Azure Policy and Terraform, there was some interest in how the Policies worked for both enforcing the tags, and making them inherit down from the Resource Group to the deployed resources.

This is actually deceptively simple as we can utilise a few built-in Policies to do the work for us!

The first Policy is called “Require a tag on resource groups”, and has the Definition ID /providers/Microsoft.Authorization/policyDefinitions/96670d01-0a4d-4649-9c89-2d3abc0a5025.

The policyRule of this Definition is as follows:

"policyRule": {
      "if": {
        "allOf": [
          {
            "field": "type",
            "equals": "Microsoft.Resources/subscriptions/resourceGroups"
          },
          {
            "field": "[concat('tags[', parameters('tagName'), ']')]",
            "exists": "false"
          }
        ]
      },
      "then": {
        "effect": "deny"
      }
    }

The logic here is:

If what is being deployed is a Resource Group
And a particular named tag doesn’t exist
Then deny the deployment

You can assign this same policy multiple times, with one instance for each tag to be enforced.

The Policy for the inheritance is called “Inherit a tag from the resource group”, and has Definition ID /providers/Microsoft.Authorization/policyDefinitions/cd3aa116-8754-49c9-a813-ad46512ece54.

The policyRule of this Definition is as follows:

"policyRule": {
      "if": {
        "allOf": [
          {
            "field": "[concat('tags[', parameters('tagName'), ']')]",
            "notEquals": "[resourceGroup().tags[parameters('tagName')]]"
          },
          {
            "value": "[resourceGroup().tags[parameters('tagName')]]",
            "notEquals": ""
          }
        ]
      },
      "then": {
        "effect": "modify",
        "details": {
          "roleDefinitionIds": [
            "/providers/microsoft.authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c"
          ],
          "operations": [
            {
              "operation": "addOrReplace",
              "field": "[concat('tags[', parameters('tagName'), ']')]",
              "value": "[resourceGroup().tags[parameters('tagName')]]"
            }
          ]
        }
      }
}

This uses a more interesting effect known as “Modify”, which can alter resources as they are being deployed.

As such, the logic is:

If the resource being deployed doesn’t have a named tag matching one defined in the containing Resource Group
And the Resource Group tag doesn’t have an empty value
Then modify the resource to include the Resource Group’s tag value

As above, you can assign this Policy multiple times and specify the tag names to use in each case to match those enforced by the first Policy defined above.

To keep this all together, I’d recommend creating a Policy Initiative called “Tag Governance” or similar and use that to group the Definitions and Assignments together, as that will make administration much more consistent.

If you want to take this further, there’s no reason why you can’t create your own Policy Definitions to alter the behaviour. Next time, I’ll show an example of how we can do that using Terraform.

Tag Management with Azure Policy and Terraform

2020-06-18T00:00:00+00:00

When deploying resources via Terraform, especially as part of a CI/CD pipeline, one of the main pain points I’ve seen from a Governance perspective is how best to manage tagging. For example, say I want to define the following tags for filtering and cost management purposes:

business_owner
cost_centre
application_name

When we’re creating an individual resource, defining these tags is easy enough:

resource "azurerm_resource_group" "tf_tags_rg" {
  name     = "tftags"
  location = "North Europe"

  tags = {
    business_owner = "benhu"
    cost_centre = "12345"
    application_name = "myapp"
  }
}

The problem is that we have to define the same tags for every resource we create. This is fine if there’s a handful, but what if there are tens or hundreds?

I’ve seen people extract the tags to a separate variables file, which is definitely better, but you still then have to remember “tags = var.tags” in all your resources. What if there was a way to define tags on a Resource Group, and then have everything automatically inherit…

In Azure, there is something called Azure Policy, and this is generally the mechanism used for tag management if using native tooling:

First Policy blocks creation of a Resource Group if Tags are not defined
Second Policy auto-filters tags from the Resource Group down to the child resources

However, if you are deploying via Terraform, it won’t be aware of these tags as they are applied by the Azure Policy engine and therefore won’t be in the Terraform state. You could do some manual importing, but this is not sustainable in the long run.

Luckily, there’s a pretty cool way you can do this using some custom lifecycle logic in your modules.

Let’s look at our example from earlier. We’ll create an Azure Policy and link it to our Subscription to enforce these tags, as well as link some built-in Policies to inherit these tags to resources. Note, you can create and apply this Policy using Terraform… but that’s for another time.

To manage the tag lifecycle, we then need to add some code to our Terraform modules. Here, I’ve got a module that creates an Azure Storage Account, but uses the lifecycle keyword in Terraform to ignore changes to specific Tags:

# append random string to storage account for uniqueness
resource "random_string" "stg_suffix" {
  length = 8
  special = false
  upper = false
}

resource "azurerm_storage_account" "stg_acct" {
  name = "${var.name}${random_string.stg_suffix.result}"
  location = var.location
  resource_group_name = var.rg_name
  account_tier = var.type
  account_replication_type = var.replication

  tags = {
      business_owner = "placeholder"
      cost_centre = "placeholder"
      application_name = "placeholder"
  }

  lifecycle {
      ignore_changes = [
          tags["business_owner"],
          tags["cost_centre"],
          tags["application_name"]
      ]
  }
}

We can then call this code as per below. Note that we only define Tags on the Resource Group, and not on the Storage Account itself:

resource "azurerm_resource_group" "tf_tags_rg" {
  name     = "tftags"
  location = "North Europe"

  tags = {
    business_owner = "benhu"
    cost_centre = "12345"
    application_name = "myapp"
  }
}

module "tf_tags_stg" {
  source = "./stg_tags"
  
  name = "bhtftag"
  location = azurerm_resource_group.tf_tags_rg.location
  rg_name = azurerm_resource_group.tf_tags_rg.name
  type = "Standard"
  replication = "LRS"
}

We see that the Resource Group gets created with the required Tags, the Storage Account is created by Terraform using the placeholder tags, and finally the Storage Account inherits the tags as per the defined Policies:

If we want to then update the Storage Account from LRS to GRS, we can see that only the replication type is being changed, and not the Tags due to the lifecycle policy we defined in the module:

Deploying a Linux VM with CMK Disk Encryption using Terraform

2020-06-16T00:00:00+00:00

Continuing the recent Terraform theme, I’ve also been working on an example of how to deploy a VM in Azure using the new method of Disk Encryption with Customer Managed Keys.

This is comprised of a few key bits of functionality:

A Disk Encryption Set to contain the disks to be encrypted
An Azure Key Vault to store the encryption keys, as well as access policies for the Disk Encryption Set and (optionally) the user deploying the code

This uses version 0.12 of the Terraform syntax, and was tested with version 2.13.0 of the Azure Provider. You’ll need a Service Principal with the necessary rights in order to create these resources; once you have the SP, fill in the details in the provider section.

provider "azurerm" {
  # whilst the `version` attribute is optional, we recommend pinning to a given version of the Provider
  version = "=2.13.0"
  features {}

  client_id = "client_id"
  subscription_id = "subscription_id"
  tenant_id = "tenant_id"
  client_secret = "client_secret"
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "tfencrypt_rg" {
  name     = "tfencrypt"
  location = "North Europe"
}

resource "azurerm_virtual_network" "tfencrypt_vnet" {
  name                = "tfencrypt_vnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.tfencrypt_rg.location
  resource_group_name = azurerm_resource_group.tfencrypt_rg.name
}

resource "azurerm_subnet" "tfencrypt_subnet" {
  name                 = "tfencrypt_subnet"
  resource_group_name  = azurerm_resource_group.tfencrypt_rg.name
  virtual_network_name = azurerm_virtual_network.tfencrypt_vnet.name
  address_prefixes       = ["10.0.0.0/24"]
}

resource "azurerm_network_interface" "tfencrypt_nic" {
  name                = "tfencrypt_nic"
  location            = azurerm_resource_group.tfencrypt_rg.location
  resource_group_name = azurerm_resource_group.tfencrypt_rg.name

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = azurerm_subnet.tfencrypt_subnet.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_key_vault" "tfencrypt_kv" {
  name                        = "des-tfencrypt-keyvault"
  location                    = azurerm_resource_group.tfencrypt_rg.location
  resource_group_name         = azurerm_resource_group.tfencrypt_rg.name
  tenant_id                   = data.azurerm_client_config.current.tenant_id
  enabled_for_disk_encryption = true
  soft_delete_enabled         = true
  purge_protection_enabled    = true
  sku_name                    = "standard"
}


resource "azurerm_key_vault_access_policy" "tfencrypt_kvuserpol" {
  key_vault_id = azurerm_key_vault.tfencrypt_kv.id

  tenant_id = data.azurerm_client_config.current.tenant_id
  object_id = data.azurerm_client_config.current.object_id

  key_permissions = [
    "get",
    "create",
    "delete"
  ]
}

resource "azurerm_key_vault_key" "tfencrypt_kvkey" {
  name         = "des-tfencrypt-key"
  key_vault_id = azurerm_key_vault.tfencrypt_kv.id
  key_type     = "RSA"
  key_size     = 2048

  depends_on = [
    azurerm_key_vault_access_policy.tfencrypt_kvuserpol
  ]

  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
}

resource "azurerm_disk_encryption_set" "tfencrypt_des" {
  name                = "tfencrypt_des"
  resource_group_name = azurerm_resource_group.tfencrypt_rg.name
  location            = azurerm_resource_group.tfencrypt_rg.location
  key_vault_key_id    = azurerm_key_vault_key.tfencrypt_kvkey.id

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_key_vault_access_policy" "tfencrypt_kvdiskpol" {
  key_vault_id = azurerm_key_vault.tfencrypt_kv.id

  tenant_id = azurerm_disk_encryption_set.tfencrypt_des.identity.0.tenant_id
  object_id = azurerm_disk_encryption_set.tfencrypt_des.identity.0.principal_id

  key_permissions = [
    "get",
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
}

resource "azurerm_linux_virtual_machine" "tfencrypt_vm" {
  name                  = "tfencrypt"
  location              = azurerm_resource_group.tfencrypt_rg.location
  resource_group_name   = azurerm_resource_group.tfencrypt_rg.name
  network_interface_ids = [azurerm_network_interface.tfencrypt_nic.id]
  size               = "Standard_F2s"
  admin_username = "benhu"
  admin_password = "Password1234!"
  disable_password_authentication = false

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "18.04-LTS"
    version   = "latest"
  }

  os_disk {
    name          = "tfencrypt_osdisk"
    caching       = "ReadWrite"
    storage_account_type = "Premium_LRS"
    disk_encryption_set_id = azurerm_disk_encryption_set.tfencrypt_des.id
  }
}

Deploying the Azure Monitor Agent for Linux using Terraform

2020-06-05T00:00:00+00:00

Hashicorp Terraform is a very popular tool for deploying and managing resources, both in a cloud environment or on-premises. The support in Azure for Terraform is excellent, but I had a bit of trouble getting the Azure Monitor agent installed as a VM Extension, so thought I would share my working code here.

provider "azurerm" {
  version = "=2.13.0"
  features {}

  client_id = "client_id"
  subscription_id = "subscription_id"
  tenant_id = "tenant_id"
  client_secret = "client_secret"
}

resource "random_string" "tfazmon_lga_suffix" {
  length = 6
  special = false
  upper = false
}

resource "azurerm_resource_group" "tfazmon_rg" {
  name     = "tfazmonitor"
  location = "North Europe"
}

resource "azurerm_virtual_network" "tfazmon_vnet" {
  name                = "tfazmonitor"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.tfazmon_rg.location
  resource_group_name = azurerm_resource_group.tfazmon_rg.name
}

resource "azurerm_subnet" "tfazmon_subnet" {
  name                 = "tfazmonitor"
  resource_group_name  = azurerm_resource_group.tfazmon_rg.name
  virtual_network_name = azurerm_virtual_network.tfazmon_vnet.name
  address_prefixes       = ["10.0.0.0/24"]
}

resource "azurerm_network_interface" "tfazmon_nic" {
  name                = "tfazmonitor"
  location            = azurerm_resource_group.tfazmon_rg.location
  resource_group_name = azurerm_resource_group.tfazmon_rg.name

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = azurerm_subnet.tfazmon_subnet.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_log_analytics_workspace" "tfazmon_lga" {
  name                = "bhtfazmonitor${random_string.tfazmon_lga_suffix.result}"
  location            = azurerm_resource_group.tfazmon_rg.location
  resource_group_name = azurerm_resource_group.tfazmon_rg.name
  sku                 = "PerGB2018"
  retention_in_days   = 180
}

resource "azurerm_virtual_machine" "tfazmon_vm" {
  name                  = "tfazmonitor"
  location              = azurerm_resource_group.tfazmon_rg.location
  resource_group_name   = azurerm_resource_group.tfazmon_rg.name
  network_interface_ids = [azurerm_network_interface.tfazmon_nic.id]
  vm_size               = "Standard_F2s"

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "18.04-LTS"
    version   = "latest"
  }

  storage_os_disk {
    name          = "tfazmonitor"
    caching       = "ReadWrite"
    create_option = "FromImage"
    managed_disk_type = "Premium_LRS"
  }

  os_profile {
    computer_name  = "tfazmonitor"
    admin_username = "benhu"
    admin_password = "Password1234!"
  }

  os_profile_linux_config {
    disable_password_authentication = false
  }
}

resource "azurerm_virtual_machine_extension" "tfazmon_ext" {
  name                 = "OmsAgentForLinux"
  virtual_machine_id   = azurerm_virtual_machine.tfazmon_vm.id
  publisher            = "Microsoft.EnterpriseCloud.Monitoring"
  type                 = "OmsAgentForLinux"
  type_handler_version = "1.12"
  auto_upgrade_minor_version = true

  settings = <<SETTINGS
    {
        "workspaceId": "${azurerm_log_analytics_workspace.tfazmon_lga.workspace_id}"
    }
SETTINGS

    protected_settings = <<PROTECTEDSETTINGS
    {
        "workspaceKey": "${azurerm_log_analytics_workspace.tfazmon_lga.primary_shared_key}"
    }
PROTECTEDSETTINGS
}

Automating Azure Bastion using PowerShell Functions

2019-11-29T00:00:00+00:00

Inspired by a conversation with a colleague about “Just In Time” access to workloads, I decided to investigate how to automate provisioning and de-provisioning Azure Bastion on weekdays only to reduce the overall running costs.

Here’s a quick overview of my (self-imposed) requirements for the solution:

Deploy new at 08:30 each weekday, remove at 18:00
Minimise running costs for automation
No passwords to manage
No hard-coding of values in code
Keep same public IP. In theory, Bastion can create a new Public IP for you each time, but it helps to keep this static for e.g. whitelisting purposes

Step 1 - Creating the Static Resources

In order for an Azure Bastion host to be created, it requires a few pre-requisites:

Resource group in which to live
Virtual Network in a supported region (e.g. West Europe)
Subnet of a suitable size (at least /27) in that virtual network with the name AzureBastionSubnet
Public IP in the same region as the virtual network with the Standard SKU

Therefore, the first step is to create these resources. I chose to use the Azure CLI from the Cloud Shell to do this, but use whatever method works for you:

az group create --name bh-bastion --location westeurope
az network vnet create --name bh-bastion-vnet --resource-group bh-bastion --location westeurope --address-prefixes '10.0.0.0/16' --subnet-name AzureBastionSubnet --subnet-prefixes '10.0.255.0/24'
az network public-ip create --name bh-bastion-pip --resource-group bh-bastion --location westeurope --allocation-method Static --sku Standard

Make a note of the names and Resource Group for these, as we’ll need them later.

Step 2 - Creating the Function App

Now that we have our “landing zone” for the Bastion host, we need a way of deploying and destroying it on a regular basis. There are numerous different ways of doing this, but one that meets all of the requirements is doing this using PowerShell in an Azure Function App. PowerShell support went GA recently, and Azure Functions have some excellent built-in features such as Timer triggers and Managed Identities that align with our goals. We can also run this on a Consumption plan, so we only pay for the time the commands are running: perfect for requirement #2.

We can also assign some variables as Application Settings in the Function to tick requirement #4.

Creating a new Function App is also pretty straightforward:

az storage account create --name bhstgfuncbastion --location westeurope --resource-group bh-bastion --sku Standard_LRS
az functionapp create --name bhfuncbastion --resource-group bh-bastion --consumption-plan-location westeurope --name bhfuncbastion --storage-account bhstgfuncbastion --os-type Windows --runtime powershell --disable-app-insights true
az functionapp config appsettings set --name bhfuncbastion --resource-group bh-bastion --settings "BASTION_VNET_NAME=bh-bastion-vnet" "BASTION_VNET_RG=bh-bastion" "BASTION_PIP_NAME=bh-bastion-pip" "BASTION_PIP_RG=bh-bastion" "BASTION_NAME=bh-bastion" "BASTION_RG=bh-bastion"

Step 3 - Setting up Identity and Access Control

For our Function App to be able to perform actions within Azure, it needs to have permission to deploy and remove resources from the Bastion resource group. Functions has an option to assign a Managed Identity, which is an identity for the Function App itself that exists in Azure Active Directory and can be combined with Role Based Access Control to grant permissions as required; this also ticks off requirement #3.

In our case, let’s assign the identity and then give it Contributor access over the Bastion Resource Group.

In an ideal world, I would use the following command:

az functionapp identity assign --name bhfuncbastion --resource-group bh-bastion --role Contributor --scope $(az group show --name bh-bastion --query 'id' -o tsv)

However, this doesn’t currently work; see this GitHub Issue.

As a workaround, you can enable the Managed Identity through the Portal:

Browse to your Function App in the Azure Portal
Click on Platform features
Click on Identity, which is under Networking
Change the Status to On
Hit Save, and click Yes when prompted

Copy the object ID shown on the screen, and we can assign the Contributor role as follows:

az role assignment create --assignee <object_id_from_function_app> --role Contributor --scope $(az group show --name bh-bastion --query 'id' -o tsv)

Step 4 - Creating the Create/Remove Functions

In terms of creating the Functions themselves, the easiest way I’ve found to do this is either using the built-in editor in the Azure Portal, or by using Visual Studio Code. If you end up doing any extensive work with Functions I highly recommend the latter option, but for now let’s work in the Portal.

For the Create Function:

Browse to your Function App in the Azure Portal
On the left hand side, under the main Function drop down, click on Functions, then on the + New function button at the top
From the templates, choose Timer trigger
Give the Function a suitable name e.g. Create_Azure_Bastion
For the Schedule, enter “0 30 8 * * 1-5”. This is cron syntax for “08:30 on days 1-5” i.e. Monday to Friday

You should now be in the in-Portal editor. Replace the code with the PowerShell located here, and hit Save.

For the Remove Function:

Browse to your Function App in the Azure Portal
On the left hand side, under the main Function drop down, click on Functions, then on the + New function button at the top
From the templates, choose Timer trigger
Give the Function a suitable name e.g. Remove_Azure_Bastion
For the Schedule, enter “0 0 18 * * 1-5”. This is cron syntax for “18:00 on days 1-5” i.e. Monday to Friday

As before, replace the code with the PowerShell located here and hit Save.

Step 5 - Increase Function App Timeout

One final (optional) item: in my testing, I occasionally saw the Removal take longer than the default 5 minute timeout supported by Functions (never the creation, oddly enough!). To workaround this, we can change the default timeout for our Functions to 10 minutes instead.

This is achieved by editing the host.json file for your Function App. This can be done locally using Visual Studio Code and the Function Core Tools, but you can also do this in the Portal:

Browse to your Function App in the Azure Portal
Click on Platform features
Click on App Service Editor, which is under Development Tools. This should open a new tab, allowing us to edit the files in the Function App
Click on host.json, which should then open in the right hand pane
Add the following code as a new line: “functionTimeout”: “00:10:00”
The file automatically saves after you make a change, so you can now close the tab

Conclusion

There we have it: one Function App, two Functions, and all of our requirements nicely met.

From a potential saving perspective, I did some rough calculations using the Azure Pricing Calculator:

Running Bastion for 730 hours (roughly 24/7 for 1 month) in West Europe comes to £51.69
With the Create/Remove method, we run for 9.5 hours/day, weekdays only, and if we say 21 weekdays per month, this is 199.5 hours, which totals £14.16

That’s over 70% saving!

Also, total running cost of our Functions? Free. Even if they run for the maximum of 10 minutes each time. Not bad for 5 lines of PowerShell.

On-Demand, Serverless SFTP

2019-04-11T00:00:00+00:00

SSH File Transfer Protocol or SFTP has been around for over 20 years, and still remains a great way to transfer files securely, specifically allowing the use of SSH keys to ensure that only the right users can perform the necessary operations.

One such scenario is the ability to receive files from known third-parties. This would generally involve having a server running 24/7, and possibly left open to the internet all the time as well. Even in the cloud, having a VM (or two for HA) switched on full time for such a simple task can quickly rack up costs.

So I thought about creating an on-demand SFTP service. This could easily be done in a VM by just starting and stopping it, but VMs are quite heavyweight for just receiving files; let’s see if we can create an on-demand SFTP service with no VMs required!

Containers are a good choice as a stand-in for VMs: their start times are generally much faster, and they can contain (pun intended) just the services we require; in our case, OpenSSH.

Microsoft Azure provides Azure Container Instances (ACI) for this purpose, which is an incredibly easy way to run a container image: just give it an image from Docker Hub and you’re pretty much set.

However, since ACI abstracts away from the underlying infrastructure running these containers, we need to use some kind of remote file system for persistent storage; enter Azure Files.

ACI provides native support for mounting Azure Files shares within running containers, so this is perfect for my requirements.

Since I started writing this blog post, this usage of ACI has been made into a formal Azure Sample, so a reference template can be found here.

Getting started

I’ll be doing all of this in the Azure CLI, which you can download from here or access via the Azure Cloud Shell. Also, ACI isn’t available in all of the Azure regions yet, so I’ll be using West Europe; make sure you check availability in your region of choice.

Firstly, let’s create a new Resource Group to hold the resources we’re going to be creating:

az group create --name bh-rg-sftp-we-01 --location westeurope

Next, we need to create the storage account. This will be used to host our file shares to be mounted into the containers at runtime:

az storage account create --name bhstgsftpwe01 --location westeurope --resource-group bh-rg-sftp-we-01 -sku Standard_LRS --kind Storage

Finally, let’s create the file share that we’ll mount into the ACI:

connection_string = $(az storage account show-connection-string -n <storage-account> -g <resource-group> --query 'connectionString' -o tsv)
az storage share create --name upload --quota 1024 --connection-string $connection_string

Choosing our image

One of the best things about containers is the incredible community that has grown as part of this movement. Central to this is the work that Docker have done to maintain central image repositories such as Docker Hub to enable people to upload and share images that they have created.

With this in mind, I decided to use one of these pre-existing images for our SFTP server; specifically the sftp image created by atmoz, which is available here.

There are a number of different ways to get the user/password details into this particular container, one of which is by using the SFTP_USERS environment variable. This needs to be in the format username:password:userID e.g. ftpuser:Password1234:1001. This environment variable will be composed of various parameters passed to a template at deployment time.

Creating the ACI template

I decided to use an Azure Resource Manager (ARM) template to create the resources required for this project. This will allow us to define the infrastructure in code, and take advantage of all the usual version control and Continuous Integration/Continuous Deployment (CI/CD) pipelines. Note that you can use any other method of your choice for deploying the ACI e.g. Azure CLI, PowerShell, Terraform etc.

The template schema for creating ACIs is pretty straightforward, so let’s focus on the sections responsible for mounting the Azure File Share. First, defining the mount point into the ACI:

"containers": [
    {
        "name": "[variables('sftpContainerName')]",
        "properties": {
            ...
            ...
                "volumeMounts": [
                    {
                        "mountPath": "[concat('/home/', parameters('sftpUser'), '/upload')]",
                        "name": "sftpvolume",
                        "readOnly": false
                    }
                ]
            }
    }
]

Here, we are telling the ACI to mount a share (yet to be defined) called “sftpvolume” at the location /home//upload. The username is defined as a parameter of the template, and so is dynamically set at deployment time.

The next section defines the volume to be mounted:

 "volumes": [
    {
        "name": "sftpvolume",
        "azureFile": {
            "readOnly": false,
            "shareName": "[parameters('existingFileShareName')]",
            "storageAccountName": "[parameters('existingStorageAccountName')]",
            "storageAccountKey": "[listKeys(variables('storageAccountId'),'2018-02-01').keys[0].value]"
        }
    }
]

This section of the template defines the volume we referenced previously. It is of type “azureFile” i.e. an Azure Files share, and we pass the share name and storage account names as parameters at deployment time. The storage account key is required to access this type of volume, so the listKeys() function is used to extract this on the fly (as long as the user deploying the template has permissions to do so).

Once we have this template defined, we can deploy it via the Azure CLI:

az group deployment create --resource-group bh-rg-sftp-we-01 --template-file aci-sftp.json --existingFileShareName xxx --existingStorageAccountName yyy ... etc.

Once deployed, connect via SFTP, copy some files, and they will appear in your Azure Files share. :)

Remember to delete the ACI once you have finished copying files to ensure you don’t rack up unnecessary costs:

az aci delete --name bhsftpaci01 --resource-group bh-rg-sftp-we-01

Taking it further

There are a few potential ways to extend this implementation, or to customise it to meet your requirements. For example:

Use SSH instead of passwords: this can be achieved by mounting a second Azure Files share into /home/username/.ssh/keys ** example here
Create your own custom image with e.g. custom host keys defined
Use the ACI Logic Apps connector to create the ACI as part of a wider workflow
Run the container image in a Kubernetes cluster with the Azure Files share defined as a persistent volume

One key thing to remember is that the ACI should not be a long-lived object: if you need something to always be available, consider deploying a VM to fulfil that role. However, since the SFTP service only really needs to be online for the period of the transfer, it is well worth considering the value that this on-demand SFTP can bring.

CI/CD with Kubernetes on ACS - Part 3 - Configuring VSTS

2017-08-24T00:00:00+00:00

Part 1 - Introduction
Part 2 - Creating ACS & ACR
Part 3 - Configuring VSTS (this post)
Part 4 - Kubernetes-ifying Application
Part 5 - Build Definition
Part 6 - Release Definition
Part 7 - Wrap-up

Getting Started with VSTS

Our next step is to get our Visual Studio Team Services (VSTS) environment ready to go. If you haven’t got a VSTS account already, head over to https://www.visualstudio.com/team-services/ and sign up for free.

Once you have a VSTS account, create a new project; keep the default settings for now.

The first thing we’ll need to do is to authorise VSTS with GitHub so that it can access your repositories. Note that it is perfectly appropriate, and arguably preferable, to use VSTS itself to host your code, but mine is in GitHub so this step is required!

In your project, click on the little cog at the top to go to Settings, then select Services and New Service Endpoint -> GitHub. You can choose to either autorise VSTS entirely, or pre-create a Personal Access Token in GitHub for VSTS to use. I chose to “Grant Authorization”, and gave the endpoint a suitable name.

Add ACR to VSTS

Next, we’ll need to create another Service Endpoint for our ACR. In the same Settings -> Services screen as before, choose New Service Endpoint -> Docker Registry.

Choose a suitably descriptive name for the connection, and enter your full ACR name in the “Docker Registry” box i.e. myacr-microsoft.azurecr.io.

I hope you noted down the appID and Password from Part 2, because you need to enter these as the Docker ID and Password respectively.

Add SSH to VSTS

The final service endpoint we need is an SSH connection to our k8s master. Choose New Service Endpoint -> SSH to create the connection.

The Host name should be the DNS name associated with your Master nodes; in my example it is bhk8s.westeurope.cloudapp.azure.com.

The User name will be azureuser, and you can get the Private Key by SSH-ing into the k8s master and copying it from there (it will be in ~/.ssh/id_rsa by default).

Create the VSTS Build Agent

This step is optional, but I found having a custom VSTS build agent useful to understand what was going on and for debugging purposes. In addition, depending on your requirements you may need some additional software that isn’t installed on the Hosted agents, or you may burn through your free build/release time quite quickly.

Anyhow, if you do decide to create your own agent, the first step is to create a new agent pool, which can be done by going to Settings -> Agent queues -> New queue…

Once you have the queue/pool configured, you will need to create a VSTS Personal Access Token for you agent to use. The full steps are here, and you will need to assign permissions for “Agent Pools (read, manage)” for the token. Make a note of the token for later.

Finaly, it is time to set up the build agent. I used a new Ubuntu 16.04 VM (running in Azure, natch). I didn’t do anything special apart from installing latest updates and installing Docker.

You’ll also need to install a few pre-reqs for the build agent:

sudo apt-get install -y libunwind8 libcurl3

The full steps for installing the agent are here. The documentation is pretty good and up-to-date, but remember to configure it to join the correct Agent Queue that we created earlier.

Now we’ve got VSTS sorted, the next step is to turn the existing Docker Compose file into a Kubernetes YAML; join me in Part 4 for that.

Extending Storage Spaces Volumes

2017-04-22T00:00:00+00:00

One of the cool features introduced in Windows Server 2012 was Storage Spaces, which is the ability to create Storage Pools that span multiple physical disks; it is sort of like software RAID on steroids. Storage Spaces was further enhanced in 2012 R2, and some additional polish has been added in 2016.

The most useful application of Storage Spaces in recent times is the ability to go beyond the disk size limit in public clouds such as Azure and AWS by adding multiple virtual hard drives to your VMs and creating a Storage Pool over the top.

However, what happens when you want to dynamically extend a virtual disk created on top of a Storage Pool? Do you need to shut everything down, and potentially stop one of your business critical applications from running? Happily, no: you can dynamically expand the size of your virtual disks using the following PowerShell:

First, create the Storage Pool. This command gets all available data disks attached to the VM and adds them to the pool:

New-StoragePool –FriendlyName StoragePool1 -StorageSubSystemFriendlyName 'Windows Storage*' -PhysicalDisks (Get-PhysicalDisk -CanPool $True)

Next, create a virtual disk on top of the Storage Pool and format it to get a partition:

$Disks = Get-StoragePool –FriendlyName StoragePool1 -IsPrimordial $False | Get-PhysicalDisk
New-VirtualDisk –FriendlyName VirtualDisk1 -ResiliencySettingName Simple –NumberOfColumns 2 –UseMaximumSize –Interleave 256KB -StoragePoolFriendlyName StoragePool1
Get-VirtualDisk –FriendlyName VirtualDisk1 | Get-Disk | Initialize-Disk –Passthru | New-Partition –AssignDriveLetter –UseMaximumSize | Format-Volume –AllocationUnitSize 64KB

A few important things to note about the above commands:

ResiliencySettingName = Simple: this is equivalent to RAID 0 i.e. write the data over multiple disks, no additional copies required
NumberOfColumns = 2: the number of disks over which to write data simulataneously. By specifying 2 here, we’ll need to add new disks in multiples of 2 to be able to add them to the Pool
Interleave = 256KB: how much data is written to each column simultaneously; adjust to suit your workload
AllocationUnitSize = 64KB: choose the best size for your workload

I highly recommend checking out this article on Storage Spaces design to help choose the best configuration for you.

Once we have the pool configured and a volume created, we can extend it further by adding more virtual disks; remember that the number of disks you will need to add depends on the Resiliency and Column settings you chose above.

Add some more disks to your VM without shutting it down, and they should appear inside the OS. We can add these disks to the pool and then confirm the available size for disks:

Get-StoragePool -FriendlyName StoragePool1 | Add-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Get-StoragePool -FriendlyName StoragePool1 | Get-VirtualDiskSupportedSize -ResiliencySettingName Simple

Next, we need to increase the virtual disk to take up the additional space in the pool:

$disksize = (Get-VirtualDisk -FriendlyName VirtualDisk1).Size
$poolsize = (Get-VirtualDiskSupportedSize -StoragePoolFriendlyName StoragePool1 -ResiliencySettingName Simple).VirtualDiskSizeMax
Resize-VirtualDisk -FriendlyName VirtualDisk1 -Size ($disksize + $poolsize)

Finally, we can extend the required partitions to take up the available space:

$VirtualDisk = Get-VirtualDisk VirtualDisk1
$Partition = $VirtualDisk | Get-Disk | Get-Partition | Where PartitionNumber -Eq 2
$Partition | Resize-Partition -Size ($Partition | Get-PartitionSupportedSize).SizeMax 

Hope this helps!

CI/CD with Kubernetes on ACS - Part 1 - Introduction

2017-04-12T00:00:00+00:00

Part 1 - Introduction (this post)
Part 2 - Creating ACS & ACR
Part 3 - Configuring VSTS
Part 4 - Kubernetes-ifying Application
Part 5 - Build Definition
Part 6 - Release Definition
Part 7 - Wrap-up

I read an article recently that discussed implementing Continuous Integration & Continuous Deployment (CI/CD) using Visual Studio Team Services (VSTS) to a Docker Swarm cluster on Azure Container Service (ACS); acronyms-ahoy!

This seemed like an excellent use of all the associated technologies, but it got me thinking: how would this work with Kubernetes as the orchestrator rather than Docker Swarm?

As a relative newcomer to container technologies and orchestrators it was a bit of an arduous task, but I finally managed to get my Build and Release pipelines sorted and deploying a (very simple) ASP.NET Core application to a Kubernetes cluster hosted on ACS.

So, starting from the beginning, what you will need to follow along with this series:

An Azure account (free trial should be sufficient if you don’t have one already)
A Visual Studio Team Services account
A Github account
Some knowledge of git & Linux command line
It will be helpful to understand the basics of Kubernetes, but I’ll try to explain stuff as I go along

I used the Azure CLI v2, which you can either install locally or run as a Docker container, but Azure PowerShell should work just fine as well. You can get the Azure CLI v2 from here.

When you’re ready to go, check out Part 2!

CI/CD with Kubernetes on ACS - Part 2 - Creating ACS and ACR

2017-04-12T00:00:00+00:00

Part 1 - Introduction
Part 2 - Creating ACS & ACR (this post)
Part 3 - Configuring VSTS Part 4 - Kubernetes-ifying Application
Part 5 - Build Definition
Part 6 - Release Definition
Part 7 - Wrap-up

Install Azure CLI

The first step is to get up and running with Azure and the Azure CLI. You can get a free trial for Azure here, which will give you more than enough credit to get started. If you already have an Azure subscription, make sure that you have the necessary permissions to create Service Principals as this is required by Kubernetes for configuring the Azure resources.

You can get the Azure CLI v2 from here.

Once you are set up with an Azure account and the CLI, open up a terminal/command prompt and run:

az login

This will prompt you with a code, and you will need to open a web browser to http://aka.ms/devicelogin, enter the code, and login to your Azure account. Once this is complete, you will then be authenticated from the CLI.

Alternatively, depending on your account security settings, you can login using:

az login -u <username> -p <password>

Create the ACS Cluster

Next step is to create a new Resource Group in Azure to contain all of the resources we are going to be creating:

az group create --name k8scicd --location westeurope

Note: I’m using West Europe for everything as part of this series, as it contains all of the services that are required. Double check the Azure services by region page to help choose a suitable region for you.

Now we have our Resource Group, it’s time to create our Azure Container Service (ACS) cluster:

az acs create  --name k8scicd --resource-group k8scicd  --orchestrator-type kubernetes --dns-prefix bhk8s --verbose

This command creates the ACS cluster using k8s as the orchestrator and configures the DNS prefix for connecting to the cluster later; this DNS prefix must be unique as it is associated with some public IPs.

Once the cluster has been created, you’ll need to install kubectl, which is the CLI for k8s. You can do that by running the following:

sudo az acs kubernetes install-cli

This installs kubectl to /usr/local/bin/kubectl by default, but you can change it with the –install-location parameter if required.

To check that your k8s cluster is working, run:

kubectl get nodes

Assuming everything has gone to plan, you should get a list back of the nodes in your ACS cluster.

I’ll be honest: when I was writing this up, this command failed for me with the warning “Private key file is encrypted”. After a bit of troubleshooting, I worked out that it was because I had an old k8s config file lurking around that was trying to connect me to a different cluster.

To resolve this, I removed the existing config and then copied the new config down from my k8s cluster:

rm ~/.kube/config
scp azureuser@bhk8s.westeurope.cloudapp.azure.com:.kube/config $HOME/.kube/config

You now have a k8s cluster up and running!

Create the ACR

Before we move on, let’s set up our Azure Container Registry (ACR). To do this, we create a storage account, and than an ACR:

az storage account create --name bhacrstg --location westeurope --resource-group k8scicd --sku Standard_LRS
az acr create --name bhacr --location westeurope --resource-group k8scicd --storage-account-name bhacrstg

Another command, another error: this time it was “The subscription is not registered to use namespace ‘Microsoft.ContainerRegistry’”. As ACR has only recently become Generally Available, your subscription might not have been updated to register with the Resource Provider that supports it. Easily solved though:

az provider register --namespace Microsoft.ContainerRegistry
az provider show --namespace Microsoft.ContainerRegistry | grep registrationState

registrationState should change to “registered” when complete, which might take a couple of minutes.

Finally, we need to be able to push and pull container images from our ACR. To do this, we need a new Service Principal:

az ad sp create-for-rbac --scopes /subscriptions/<subcription_id>/resourcegroups/k8scicd/providers/Microsoft.ContainerRegistry/registries/bhacr --role Owner --password <password>

Note down the appID and name that are generated by this command as we will need them in the next step, which is adding these credentials to k8s:

kubectl create secret docker-registry bhacr --docker-server=https://<ACR_name>-microsoft.azurecr.io --docker-username=<appID> --docker-password=<password> --docker-email=<any_email>

Make sure you remember these settings as we will use them again as (spoliers!) part of the build definition in VSTS.

Now that you’re up and running with ACS & ACR, check out Part 3 to prepare VSTS.