Skip to main content

Terraform

1. Introduction​

Terraform is an open-source infrastructure as code software tool created by HashiCorp. It allows users to define infrastructure for a variety of cloud providers, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP), using a declarative configuration language. Here are some common concepts related to Terraform:

  1. Infrastructure as code: With Terraform, you can define your infrastructure using code rather than manually configuring resources through a web interface. This allows you to version control your infrastructure and make it easier to automate the provisioning and management of resources.

  2. Modules: A Terraform module is a self-contained package of Terraform configurations that are used to create reusable components. Modules can be shared and used by multiple Terraform configurations, making it easier to manage and maintain complex infrastructure.

  3. Providers: A provider is a plugin that tells Terraform how to connect to and manage a particular cloud provider's API. Terraform includes a number of built-in providers, and users can also create custom providers.

  4. State: Terraform stores the current state of your infrastructure in a state file. This file is used to determine what resources have been created and what changes are necessary to bring your infrastructure into the desired state.

  5. Variables: Variables allow you to parameterize your Terraform configurations, making them more flexible and easier to reuse. You can define variables and assign them values in a variety of ways, such as through environment variables or command-line arguments.

  6. Outputs: Outputs allow you to extract and display information about your infrastructure after it has been created. This can include things like the IP addresses of created resources or the URL of a created website.

2.Static Infrastructure VS Dynamic Infrastructure​

Static Infrastructure relies on static fleet of standardized infrastructure, provisioned for long periods of time and dedicated to users.

This is kind of hetrogeneous and private,

Traditional Approach :

  • Manual Provisioning
  • Fixed set of resources
  • Workflow requires ticketing & queues

Dynamic Infrastructure is hetrogeneous and frequently provisioned, short lived and automated provisioning on-demand.

Terraform Approach :

  • Infrastructure as code
  • Embrace diversity with providers
  • Enable self service infrastructure

3. Datatypes in Terraform​

info

Terraform has several built-in data types that you can use to define your infrastructure code. Here are some of the most commonly used data types in Terraform:

String: The string data type represents a sequence of characters. Strings are enclosed in double quotes ("") or single quotes ('').

Number: The number data type represents a numeric value, either integer or floating-point. Numbers can be expressed in decimal, hexadecimal, or octal format.

Boolean: The boolean data type represents a logical value, either true or false. Boolean values are often used to control flow and make decisions in Terraform code.

List: The list data type represents an ordered collection of values, all of the same type. Lists are enclosed in square brackets ([ ]). Lists can be used to define multiple values for a single resource property.

Map: The map data type represents a collection of key-value pairs. Maps are enclosed in curly braces (). Maps can be used to define a set of related properties for a single resource.

Set: The set data type represents an unordered collection of unique values. Sets are enclosed in curly braces (). Sets can be used to define a collection of resources or properties that should be unique within a certain scope.

Object: The object data type represents a complex data structure that can contain multiple properties of different types. Objects are defined using curly braces () and key-value pairs.

4. Terraform Syntax​

info

Terraform has a simple and easy-to-understand syntax that uses declarative language to define infrastructure as code.

  • Blocks: Terraform code is organized into blocks, which are defined by a block type and a block label. For example, a resource block starts with the "resource" block type and a label that identifies the type of resource being defined.

  • Arguments: Blocks contain arguments that define the properties and attributes of a resource. Arguments are defined using a key-value syntax, where the key is the name of the attribute and the value is the value of the attribute.

  • Interpolation: Terraform supports interpolation, which allows you to reference values from other parts of your code. Interpolation is done using the $ syntax, where the expression inside the curly braces is evaluated at runtime.

  • Variables: Terraform allows you to define variables that can be used to parameterize your code. Variables are defined using the "variable" block type, and can be set in a separate file or passed in as command-line arguments.

  • Outputs: Terraform allows you to define outputs that represent values that are produced by your code. Outputs are defined using the "output" block type and can be used to retrieve information about your infrastructure.

  • Providers: Terraform uses providers to interact with different cloud providers and other external systems. Providers are defined using the "provider" block type and are configured with provider-specific settings and credentials.

5. Terraform State​

info

Terraform state is a critical component in Terraform. It is a file that keeps track of the current state of your infrastructure resources, including their configuration and relationships with other resources. The state file is used by Terraform to create, update, and delete resources based on the desired state defined in your Terraform code.

Advantages of Terraform State:​

Accurate Tracking: Terraform state file accurately tracks the current state of your infrastructure resources. It ensures that Terraform only applies the necessary changes to your resources to bring them to the desired state defined in your code.

Collaboration: Terraform state file enables collaboration and sharing of infrastructure code between team members. It ensures that every team member has access to the current state of the infrastructure, which makes it easier to work together on a project.

Reusability: Terraform state file makes it easy to reuse code and modules across different projects. You can use the same infrastructure code across multiple environments, and the state file ensures that each environment is configured correctly.

Automation: Terraform state file allows you to automate the provisioning, updating, and deletion of infrastructure resources. You can define the desired state of your infrastructure resources in code and let Terraform handle the rest.

Disadvantages of Terraform State:​

Complexity: Terraform state can become complex and difficult to manage when dealing with large and complex infrastructure environments. It requires careful planning and management to ensure that the state file is accurate and up-to-date.

Security: Terraform state file contains sensitive information, such as access keys and passwords, which can be a security risk if not properly managed. It is important to encrypt and secure the state file to prevent unauthorized access.

Version Control: Terraform state file is not designed to be stored in version control systems like Git. It is recommended to use a remote backend to store and manage the state file.

6. Terraform Modules​

info

Terraform modules are a way to organize and encapsulate Terraform code into reusable components. A module can contain one or more resources and can be used to represent a piece of infrastructure, a logical component, or a specific function.

Modules can be used to simplify the management of complex infrastructure environments, reuse code across different projects, and enable collaboration between teams. They also provide a way to abstract away details and complexity, making it easier to understand and manage infrastructure code.

Here are the main features and benefits of Terraform modules:

Encapsulation: Modules allow you to encapsulate Terraform code into a single unit, making it easier to understand and manage. You can define the inputs and outputs of a module, which provides a clear separation of concerns between different parts of your code.

Reusability: Modules can be reused across different projects and environments, which can save time and effort. You can define a module once and reuse it multiple times, without having to duplicate code or modify it for each environment.

Abstraction: Modules provide a way to abstract away details and complexity, making it easier to manage infrastructure code. You can create a module that represents a piece of infrastructure or a logical component, and then use it in different parts of your code without having to worry about the underlying details.

Collaboration: Modules can be shared and used by different teams, which enables collaboration and sharing of best practices. You can create a module that represents a specific function or workflow and share it with other teams, which can improve the consistency and quality of your infrastructure code.

Overall, Terraform modules are a powerful feature that can simplify the management of complex infrastructure environments, promote code reuse and collaboration, and enable abstraction and separation of concerns in your Terraform code.

7. Terraform Remote State Lookup Implementation​

info
modules/topi-role-creation/main.tf
------

locals{

    topic_arn = (var.region == "useast-2") ? join("/", [replace(var.mask_cluster_arn,"cluster","topic"),format("*.%s", var.topic_name)]) : join("/",[replace(var.mask_cluster_arn,"cluster","topic"), var.topic_name])

    msk_group_arn = join("/", [replace(var.mask_cluster_arn,"cluster","group"),"*"])

    glue_registry_arn = (var.region == "us-east-1") ? module.ebeh_msk_glue_us_east_1[0].remote_data_output.registry_arn: module.ebeh_msk_glue_us_east_2[0].remote_data_output.registry_arn

}

module "ebeh_msk_glue_us_east_1" {
    count = (var.region == "us-east-1") ? 1 : 0
    source = "git::https://gitlab.com/r-financial/enterprise-business-event-hub/fulcrum-common.git//remote-state-lookup?ref=v1"
    region = var.region
    environment = var.environment
    state_key_name = "ebeh_msk_cluster"
}

module "ebeh_msk_glue_us_east_2" {
    count = (var.region == "us-east-2") ? 1 : 0
    source = "git::https://gitlab.com/r-financial/enterprise-business-event-hub/fulcrum-common.git//remote-state-lookup?ref=v1"
    region = var.region
    environment = var.environment
    state_key_name = "ebeh_msk_cluster"
}

data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "topic_consumer_policy" {

    statement {
        effect = "Allow"
        actions = [
            "kafka-cluster: Connect"
        ]
        resources = [var.msk_cluster_arn]
    }
    statement {
        effect = "Allow"
        actions = [
            "kafka-cluster: ReadData",
            "kafka-cluster: DescribeTopic"
        ]
        resources = [var.msk_topic_arn]
    }
    statement {
        effect = "Allow"
        actions = [
            "kafka-cluster: DescribeGroup",
            "kafka-cluster: AlterGroup"
        ]
        resources = [var.msk_group_arn]
    }
    statement {
        effect = "Allow"
        actions = [
            "glue: GetSchema",
            "glue: ListSchemas",
            "glue: GetSchemaByDefinition",
            "glue: GetSchemaVersion",
            "glue: ListSchemaVersions",
            "glue: GetSchemaVersionsDiff",
            "glue: GetSchemaVersionMetaData",
            "glue: GetTags"

        ]
        resources = ["*"]
    }
    statement {
        effect = "Allow"
        actions = [
            "glue: GlueRegistry"
        ]
        resources = [local.glue_registry_arn]
    }
    
    statement {
        effect = "Allow"
        actions = [
            "glue: ListRegistries"
        ]
        resources = ["*"]
    }
}


data "aws_iam_policy_document" "topic-consumer-trust-policy" {
    
    statement {
        sid = ""
        effect = "Allow"
        principals = [
            type = "AWS"
            identifiers = var.task_execution_consumer_role_arn
        ]
        actions = ["sts:AssumeRole"]
    }
}

module "iam-topic-consumer-role" {

    source = "module-registry.services.rmavuluri.com/rmavuluri/iam/aws"
    version = "~ > 3.0"
    count = length(var.task_consumer_execution_role_arn) == 0 ? 0 : 1
    namespace = var.namespace
    name = "${var.topic_name}-r
    region = var.region
    trust_policy = data.aws_iam_policy_document.topic-consumer-trust-policy.json
    policy = [
        data.aws_iam_policy_document.topic-consumer-policy.json
    ]
    tags = var.tags
}



main.tf
-------

data "aws_caller_identity" "current" {}

data "terraform_remote_state" "ebeh_msk_glue" {
    backend = "s3"
    config = {
        bucket = "ally-${var.region}-${data.aws_caller_identity.current.account_id}-tf-states"
        key = "env:/${var.environment}/ebeh_msk_cluster/terraform.tfstate"
        region= var.region
    }
    defaults = null
}

module "topic-roles" {
    source = "./modules/topic-role-creation"
    for_each = local.topics
    topic_name = each.key
    environment = var.environment
    namespace = module.namespace.lower_short_name
    region = var.region
    msk_cluster_arn = data.terraform_remote_state.ebeh_msk_glue.outputs.msk_cluster_arn
    task_execution_consumer_role_arn = each.value.consumer_execution_role
    tags = module.namespace.tags
    depends_on =[
        null_resource.create_topics
    ]    
}