Learn how to - Automate Bitbucket Tasks via Terraform

Learn how to – Automate Bitbucket Tasks via Terraform.

Everything involving a huge team requires a high level of standardisation, order, and consensus about how tools within an organisation will be used in order to foster best practices and hence improve staff productivity. Whenever there are engineers, you can bet that there are tools around them. And no matter how powerful a tool is built, it cannot flourish if there is lack of good choreography and orchestration on the platform. Bitbucket is one of those powerful tools that requires some form of intervention that will help in how a team will handle the organisation of projects, groups, users, default reviewers, branch permissions, branching models and much more. Without good way of working, these aspects of Bitbucket can suddenly be haphazard, difficult to manage and therefore attracting lesser and lesser productivity. So what to do about it?

One way to do it is to communicate to the entire team about the expectations and minimum rules that everyone must abide by. This can work of the culture of the organisation is spick and span and everyone works like a well lubricated engine. In case you are still working on your culture, then this is going to be a tall order and another option will have to be brought up. This is the path of automation that will help streamline how your team will work. By introducing automation, big wins can be bagged way into the middle of the night. Checks can be put in place to ensure that any action deviating from the stipulated measures are logged and the relevant parties can be reached out to about the same. Automation, thus far, is the best way to go hence the inception of this project to automate Bitbucket platform. The scope of work involved the following:

Importing all resources manually created to terraform state
Create a standard repository permissions template that can be easily applied to any repository
Enforce merge checks
Create relevant groups and users within them via terraform
Create new projects and migrate repositories in wrong projects to them
Create process flow that will guide anyone on how to add or modify records
Document the entire journey.

This is a brief document that details how we are going to use Terraform to automate most of the tasks in Bitbucket such as creating groups, adding users in them, repositories, branch permissions, branch models, default reviewers and others in each of the repositories. I give tribute to two terraform providers listed below for the heavy-lifting.

That said, Let’s fasten our space regalia and roll into the stars.

Step 1: Prepare your Bitbucket Modules

Since you may be having multiple projects in Bitbucket, it makes sense to organise them into separate modules so that it instantly impresses your heart in manoeuvering through the directories and files. It also becomes easier to add new records in future as well as provide a good representation of what lies in your environment. We will be consolidating our Bitbucket Terraform files into the following directory tree.

├── bitbucket_automation
│ └── bitbucket
│     └── learnhowto_modules
│         ├── business_intel
│         │ └── repositories
│         ├── human_resource
│         │ └── repositories
│         ├── support
│         │ └── repositories
│         ├── infrastructure
│         │ └── repositories
│         ├── datalabs
│         │ └── repositories
│         ├── infosec
│         │ └── repositories

So let us create this folder structure then warm their rooms with terraform files later. The idea behind it is to organise the projects representing each team in Bitbucket as modules then add repositories in each of the projects modules as sub-modules. This way, it will be easy to organise groups per project/per team and then add users in them per team. We can also easy manage default reviewers per project and also manage repositories with utmost ease. Let us create the directories thus:

cd ~
mkdir -p bitbucket_automation/bitbucket/learnhowto_modules
cd bitbucket_automation/bitbucket/learnhowto_modules
mkdir {business_intel,human_resource,support,infrastructure,datalabs,infosec}

Then you can easily create the “repositories” directory (sub-modules) in each project directory as follows

cd bitbucket_automation/bitbucket/learnhowto_modules
mkdir {business_intel/repositories,human_resource/repositories,support/repositories,infrastructure/repositories,datalabs/repositories,infosec/repositories}

And now that the structure of our projects looks like what we have above, we are perfectly ready to furnish their rooms with our lively files.

Step 2: Where is the Cumin and Black Pepper?

Here, we will add the most important files to season this project. The first file is the main file that will link every directory (modules and sub-modules) together. Navigate into the “bitbucket” directory and add the “main.tf” file with the contents that follow:

$ vim ~/bitbucket_automation/bitbucket/main.tf
##################################################################################

 provider "vault" {
   token   = var.geeks_token
   alias   = "geeks_vault"
   address = var.geeks_vault_address
}

################################## The Projects ##################################

#1. Bitbucket Business Intel Project Module

 module "business_intel" {
  source                     = "./learnhowto_modules/business_intel"
  providers = {
    vault   = vault.geeks_vault
  }
}

#2. Bitbucket Human Resource Project Module

 module "human_resource" {
  source                     = "./learnhowto_modules/human_resource"
  providers = {
    vault   = vault.geeks_vault
  }
}

#3. Bitbucket Support Project Module

 module "support" {
  source                     = "./learnhowto_modules/support"
  providers = {
    vault   = vault.geeks_vault
  }
}

#4. Bitbucket Infrastructure Project Module

 module "infrastructure" {
  source                     = "./learnhowto_modules/infrastructure"
  providers = {
    vault   = vault.geeks_vault
  }
}

#5. Bitbucket Datalabs Project Module

 module "datalabs" {
  source                     = "./learnhowto_modules/datalabs"
  providers = {
    vault   = vault.geeks_vault
  }
}

#6. Bitbucket InfoSec Project Module

 module "infosec" {
  source                     = "./learnhowto_modules/infosec"
  providers = {
    vault   = vault.geeks_vault
  }
}
## This is the bucket where tfstate for CloudSQL will be stored in GCP

terraform {
  backend "gcs" {
    bucket      = "geeks-terraform-state-bucket"
    prefix  = "terraform/bitbucket_automation_state"
  }
}

As you can see, we have added Vault provider that will aid in fetching Bitbucket Api authentication secrets stored in the Vault server. We have added each project as a module inside the “learnhowto_modules” directory.

Note also that we are using GCP Bucket backend to store terraform’s state. Ensure that a bucket with the name given already exists or you can create it if it is not there yet. You will find the state in a the “terraform” sub-directory called “bitbucket_automation_state”

Within the “bitbucket” folder, we need to update vault provider variables as follows.

$ vim ~/bitbucket_automation/bitbucket/vars.tf
variable "geeks_token" {
  type        = string
  default     = "infra@ownyourlife.com.ng"
}

variable "geeks_vault_address" {
  type        = string
  default     = "//vault.ownyourlife.com.ng:8200"
}

That will take care of vault’s server address and token variables that Terraform will use to connect to Vault.

With those coonfigurations, we have connected every project module together but terraform does not have visibility into the “repositories” sub-modules. This is the part that we enable that next.

Navigate into each project module and create another “main.tf” file that will light up the path for terraform to see the “repositories” sub-module. You can create one file then copy it into each project directory since it is similar in stature.

$ cd ~
$ vim ~/bitbucket_automation/bitbucket/learnhowto_modules/business_intel/main.tf

terraform {
  required_providers {
    vault = {
      source = "vault"
      version = "~> 3.0"
    }
  }
}


module "repositories" {
  source                     = "./repositories"
}

Then copy the file into each project’s directory at once as follows:

cd ~/bitbucket_automation/bitbucket/learnhowto_modules/
echo "human_resource/ support/ infrastructure/ datalabs/ infosec/“ | xargs -n 1 cp -v business_intel/main.tf

Step 3: Create Must files we need in each project

Now, in each project, there are different repositories within that we are interested in importing as well as creating so that we can manage them all via terraform in future. This is going to be the hardest part of it all especially if you already have many repositories manually created before. Once the “main.tf” file in each project module is well done, we will need another special file that will reside in each other “repositories” directory in each project directory. This is the file that will have the Bitbucket providers and how their credentials will be fetched from Vault. Let us get into one of the directories then copy it into each of the other like we did before. We are going to call it “provider.tf”.

$ cd ~/bitbucket_automation/bitbucket/learnhowto_modules/
$ vim business_intel/provider.tf

# Get credentials from Vault

data "vault_generic_secret" "bitbucket" {
  path = "bitbucket/credentials"
}

provider "bitbucket" {
  alias        = "drfaust92"
  username  = data.vault_generic_secret.bitbucket.data["bitbucket_username"]
  password  = data.vault_generic_secret.bitbucket.data["bitbucket_password"]
}

provider "bit-bucket" {
  alias     = "zahiar"
  username  = data.vault_generic_secret.bitbucket.data["bitbucket_username"]
  password  = data.vault_generic_secret.bitbucket.data["bitbucket_password"]
}

terraform {
  required_providers {
    bit-bucket = {
      source   = "zahiar/bitbucket"
      version  = "1.0.2"
    }
    bitbucket  = {
      source   = "DrFaust92/bitbucket"
      version  = "2.15.0"
    }
  }
}

Then copy the file into each of the “repositories” directory at once as follows:

cd ~/bitbucket_automation/bitbucket/learnhowto_modules/
echo "infrastructure/repositories infosec/repositories datalabs/repositories human_resource/repositories support/repositories" | xargs -n 1 cp -v business_intel/repositories/provider.tf

The secret here is an “App password” Bitbucket credential that can authenticate to the Bitbucket API and be able to apply the terraform tasks we declare. Make sure this “App password” has admin privileges to avoid problems. So we should already have the username and password for this user created in Vault within the “bitbucket/credentials” path. This is what the (data “vault_generic_secret” “bitbucket”) will be fetching.

We will need a variables file that will have our users to be added in the groups we will create per project as well as the designated default reviewers per project or per repository depending on your needs. Let us move ahead and create the variables file with sample “Business Intel” content added as follows.

$ vim vars.tf

######################################################
# Business Intel BLOCK
######################################################

## business-intel-frontend Members
variable "business_intel_frontend" {
  type        = list(string)
  default     = []
  description = "Bitbucket business_intel_frontend group members"
}

## business-intel-backend Members
variable "business_intel_backend" {
  type        = list(string)
  default     = ["{0370a95a-5a58-4815-8d97-d85f9f31a205}", "{40db8a17-544b-43a1-9403-d85f9f31a205}", "{0570a95c-8b20-4077-9c61-d85f9f31a205}"]
  description = "Bitbucket business_intel_backend group members"
}

## business-intel-admins Members
variable "business_intel_admins" {
  type        = list(string)
  default     = ["{0370a95a-5a58-4815-8d97-d85f9f31a205}", "{40db8a17-544b-43a1-9403-d85f9f31a205}", "{0570a95c-8b20-4077-9c61-d85f9f31a205}"]
  description = "Bitbucket business_intel_admins group members"
}

## sre-admins Members list data type
variable "business_intel_list" {
  type        = list(string)
  default     = ["{0370a95a-5a58-4815-8d97-d85f9f31a205}", "{40db8a17-544b-43a1-9403-d85f9f31a205}", "{0570a95c-8b20-4077-9c61-d85f9f31a205}"]
  description = "Bitbucket business_intel group members"
}

## Workspace ID
variable "geeks_workspace_id" {
  type        = string
  default     = "{0370a95a-5a58-4815-8d97-d85f9f31a205}"
  description = "ComputingForGeeks Bitbucket Workspace"
}
## Workspace Name
variable "geeks_workspace_name" {
  type        = string
  default     = "geeksteam"
  description = "ComputingForGeeks Bitbucket Workspace"
}

##UUID of the users above and their readable names
#June: {0370a95a-5a58-4815-8d97-d85f9f31a205}
#April: {40db8a17-544b-43a1-9403-d85f9f31a205}
#Tech: {0570a95c-8b20-4077-9c61-d85f9f31a205}

Bitbucket API expects users’ UUID when they are being added to a group! This can be found by executing a “GET” Request to this following endpoint: “https://api.bitbucket.org/2.0/workspaces/{workspace}/members”. The authorisation is the same “App password” credential you can create under you Bitbucket Profile. You can do this by clicking on your profile icon at the top right of Bitbucket interface, then clicking on “Personal Settings” from the menu that will drop down. You will find “App Passwords” item on the left pane on the page that will ensue. Click on it and you will be ushered to a page where you can create an “App passord”. Remember to give it admin privileges since Terraform needs them.

Click Profile Icon

Automate Bitbucket Tasks via Terraform 1

Click App Passwords

Automate Bitbucket Tasks via Terraform 2

Step 4: Create repositories files

Like we mentioned before, this is the most challenging part of the entire process because we have to manually add each repository and its settings then import them to terraform state. We will do a demo of the same here. Suppose we need to add “user-service” repository in Human Resource Project, we would navigate to the “human_resource” project directory, get into the “repositories” folder then add a file for the repository, for example:

$ cd ~/bitbucket_automation/bitbucket/learnhowto_modules/human_resource
$ vim user-service.tf

############################ User Service ############################ 

resource "bitbucket_repository" "user-service" {
  provider         = bit-bucket.zahiar
  workspace        = var.geeks_workspace_name
  name             = "user-service"
  project_key      = "US"
  is_private       = true
  fork_policy      = "no_public_forks"
  enable_pipelines = true
  description      = "In-house user service solution"
}
## Default reviewers for this repository. (It Accepts UUID only)
resource "bitbucket_default_reviewer" "human-resource-default-reviewers-user-svce" {
  provider        = bit-bucket.zahiar
  count           = length(var.default_reviewers)
  workspace       = var.geeks_workspace_name
  repository      = "user-service"
  user            = var.default_reviewers[count.index]

    depends_on = [
    bitbucket_repository.user-service,
  ]
}
############################ Master Branch ############################ 
## Master Branch Restriction -1
resource "bitbucket_branch_restriction" "master-min-approvals-for-merge-to-happen-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "require_approvals_to_merge"
  value      = 2
}


## Master Branch Restriction -2
resource "bitbucket_branch_restriction" "master-none-allowed-to-delete-the-branch-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "delete"
}

## Master Branch Restriction -3
resource "bitbucket_branch_restriction" "master-restrict-pushes-to-some-users-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "push"
}

## Master Branch Restriction -4
resource "bitbucket_branch_restriction" "master-min_number_of_apprvls_frm_default_reviewers-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "require_default_reviewer_approvals_to_merge"
  value      = 2
}

## Master Branch Restriction -5
resource "bitbucket_branch_restriction" "master-min_number_of_passed_builds_to_merge-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "require_passing_builds_to_merge"
  value      = 1
}
## Master Branch Restriction -6
resource "bitbucket_branch_restriction" "master-require_all_tasks_to_be_completed-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "require_tasks_to_be_completed"
}
## Master Branch Restriction -7
resource "bitbucket_branch_restriction" "master-restrict-merges-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "restrict_merges"
  groups     = ["team-leads", "Administrators"]
}

## Master Branch Restriction -8
resource "bitbucket_branch_restriction" "master-force-restriction-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "master"
  kind       = "force"
}

############################ Develop Branch ############################ 
## Develop Branch Restriction -1
resource "bitbucket_branch_restriction" "develop-min-approvals-for-merge-to-happen-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "require_approvals_to_merge"
  value      = 3
}

## Develop Branch Restriction -2
resource "bitbucket_branch_restriction" "develop-none-allowed-to-delete-the-branch-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "delete"
}

## Develop Branch Restriction -3
resource "bitbucket_branch_restriction" "develop-restrict-pushes-to-some-users-rsvc" {
   provider   = bit-bucket.zahiar
  #provider        = bitbucket.drfaust92
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "push"
  groups     = ["team-leads"]
}

## Develop Branch Restriction -4
resource "bitbucket_branch_restriction" "develop-min_number_of_apprvls_frm_default_reviewers-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "require_default_reviewer_approvals_to_merge"
  value      = 3
}
## Develop Branch Restriction -5
resource "bitbucket_branch_restriction" "develop-min_number_of_passed_builds_to_merge-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "require_passing_builds_to_merge"
  value      = 1
}

## Develop Branch Restriction -6
resource "bitbucket_branch_restriction" "develop-require_all_tasks_to_be_completed-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "require_tasks_to_be_completed"
}
## Develop Branch Restriction -7
resource "bitbucket_branch_restriction" "develop-restrict-merges-rsvc" {
  provider        = bitbucket.drfaust92
  owner      = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "restrict_merges"
  users      = ["{74824a4f-70b8-4ab0-ade9-e4e52210ecf3}","{2b5f7e64-acb3-4d68-81c6-4ef760d132ea}"]
  #groups     = ["Team Leads", "Administrators"]
}

## Develop Branch Restriction -8
resource "bitbucket_branch_restriction" "develop-force-restriction-rsvc" {
  provider   = bit-bucket.zahiar
  workspace  = var.geeks_workspace_name
  repository = "user-service"
  pattern    = "develop"
  kind       = "force"
}
*/
####################################################################

# The repository's branching model here.
resource "bitbucket_branching_model" "user-service" {
  provider        = bitbucket.drfaust92
  owner           = var.geeks_workspace_name
  repository      = bitbucket_repository.routing-service.name

  development {
    use_mainbranch = false
    name           = "develop"
  }
  production {
    use_mainbranch = true
    enabled        = true
    name           = "master"
  }

  branch_type {
    enabled = true
    kind    = "bugfix"
    prefix  = "bugfix-"
  } 

  branch_type {
    enabled = true
    kind    = "feature"
    prefix  = "feature-"
  }

  branch_type {
    enabled = true
    kind    = "hotfix"
    prefix  = "hotfix-"
  }

  branch_type {
    enabled = true
    kind    = "release"
    prefix  = "release-"
  }  
}

If you observe the file, you will notice that it has the “user-service” repository, default reviewers configuration, several branch permissions and a branching model at the end. This is the configuration that will be applied to Bitbucket in case they do not already exist. In case the configuration already exists, we will have to import them to terraform state so that henceforth, they can be managed by terraform. We can import the repository as follows:

$ terraform import module. human_resource.module.repositories.bitbucket_repository.user-service "geeksteam/user-service"

We would import the branch permissions import the repository as follows:

$ terraform import module.human_resource.module.repositories.bitbucket_branch_restriction.master-min_number_of_passed_builds_to_merge-rs "geeksteam/user-service/10718612"

Where “10718612” is the branch restriction id that you can get from sending a “GET” request to the following Bitbucket API endpoint “https://api.bitbucket.org/2.0/repositories/geeksteam//branch-restrictions”. You will get sample JSON response that looks like below with the id as the last key and its value.

   {
      "kind": "require_default_reviewer_approvals_to_merge",
      "users": [],
      "links": {
        "self": {
          "href": "https://api.bitbucket.org/2.0/repositories/geeksteam/user-service/branch-restrictions/10718612"
        }
      },
      "pattern": "master",
      "value": 2,
      "branch_match_kind": "glob",
      "groups": [],
      "type": "branchrestriction",
      "id": 10718612
    },

Step 6: Add groups and users

To demonstrate how to add groups and including users in those groups, we will go ahead with adding supply chain groups and then add users in them. Navigate into “human_resource” directory and add the groups file as follows:

$ cd ~/bitbucket_automation/bitbucket/learnhowto_modules/human_resource
$ vim groups.tf

# human-resource-backend
resource "bitbucket_group" "human-resource-backend" {
  provider        = bitbucket.drfaust92
  workspace       = var.geeks_workspace_id
  name            = "human-resource-backend"
  auto_add        = false
  permission      = "write"
}

## Add group members to the human-resource-backend above
resource "bitbucket_group_membership" "human-resource-backend-members" {
  count           = length(var.human_resource_backend)
  provider        = bitbucket.drfaust92
  workspace       = var.geeks_workspace_id
  group_slug      = bitbucket_group.supply-chain-backend.name
  uuid            = var.human_resource_backend[count.index]

    depends_on = [
    bitbucket_group.human-resource-backend,
  ]
}

# human-resource-frontend group
resource "bitbucket_group" "human-resource-frontend" {
  provider        = bitbucket.drfaust92
  workspace       = var.geeks_workspace_id
  name            = "human-resource-frontend"
  auto_add        = true
  permission      = "write"
}

## Add group members to the human-resource-frontend group above
resource "bitbucket_group_membership" "human-resource-frontend" {
  count           = length(var.human_resource_frontend)
  provider        = bitbucket.drfaust92
  workspace       = var.geeks_workspace_id
  group_slug      = bitbucket_group.human-resource-frontend.name
  uuid            = var.human_resource_frontend[count.index]

    depends_on = [
    bitbucket_group.human-resource-frontend,
  ]
}

With this, you can see how resources can be added into each directory in an orderly manner. Every repository resides in a project and each repository has resources of its own such as the branch permissions, branching models, groups and users and much more. They can all be added courtesy of the providers shared in the first part of this guide.

From here, you can simply do the terraform plan and apply if the plan matches what you had in mind.

$ terraform plan -refresh=false -out modify_records -var geeks_token=${TF_VAR_geeks_token}
$ terraform apply -auto-approve -parallelism=5 modify_records

“geeks_token” is Vault token that will allow secrets to be retrieved. Create an environment variable named TF_VAR_geeks_token

Last Ballad

There we go good people. Once you add all of the repositories and importing them to terraform, you will be ready to administer them all from a single source of terraform scripts which will make your work streamlined and orderly. It will also be easy to know what changed and at what point it did change. Last but not least, the terraform scripts will serve as documentation of the entire Bitbucket infrastructure. One glance and you can tell what lies in your Bitbucket space.

Cart

Learn how to – Automate Bitbucket Tasks via Terraform