Variable unpacking for easy module resource parameter matching

[drewmullen]

Explanation / setup

Say I have a module that allows users to set some specific values on resources. The common approach is to define a variable for each parameter

root module

variable "vpc_region" {}
variable "vpc_cidr" {}
# ... etc

module "vpc" {
   region = var.vpc_region
   cidr   = var.vpc_cidr
}

module

Then inside the module code we map each var to the resource. This also requires redefining the variables

variable "vpc_region" {}
variable "vpc_cidr" {}
# ... etc

resource "aws_vpc" "m" {
  region = var.vpc_region
  cidr_block    = var.vpc_cidr
}

With 2 variables this isnt too bad but when done in reality this can get quite complicated.

Slightly better way

Another option is to use a single variable and then map to the resource parameters in the module.

root module

variable "vpc_attributes" {
  type = object({
    cidr   = string
    region = string
  })
}

module "vpc" {
  vpc_attributes = var.vpc_attributes
}

module

This method reduces the amount of variables you need to define in the module. However this still requires:

• setting each parameter on the resource
• handling for null or missing keys

variable "vpc_attributes" {}

resource "aws_vpc" "m" {
  region = var.vpc_attributes["region"]
  cidr_block    = var.vpc_attributes["cidr"]
}

The big idea

What if we had a way of packing terraform variables as known parameter keys of a resource. Then unpack them into the resource in the module code itself. The keys and types must match for this to work.

root module

variable "vpc_attributes" {
  type = object({
    cidr_block   = string
    region = string
    instance_tenancy = string
  })
}

module "vpc" {
   vpc_attributes = var.vpc_attributes
}

module

variable "vpc_attributes" {}

resource "aws_vpc" "m" {
  var.vpc_attributes...
  
  enable_dns_hostnames = true
}

The syntax is expansive the same way most programming languages have a form of variadic functions. Also note that other parameters can be set that are not necessarily in the variable. Also note that the vpc_attributes us using custom object types to enforce the same naming convention as the aws_vpc resource expects.

This aligns nicely with the spirit of variable custom object definitions with default values. Ultimately this would make for cleaner and more maintainable terraform modules.

[jbardin]

This is basically a resource version of #32184. It's not an exact duplicate though, because the fact that it's about resources is important, and makes it that much more difficult to design a solution.

Aside from the fact that HCL doesn't have a grammar for this, probably the biggest hurdle would be that resources still use blocks to structure their configuration, and you cannot assign a value directly to a block. Because a block's underlying type is mostly hidden from users, it's not clear what would be required to reliably assign a value to a block if it were allowed. There's probably a number of questions just about blocks themselves, which we either need to design, or design a way to communicate to the user what to do. Just off the top of my head:

• Does the block value need to be a single object, set, list, or map (representing a labeled block)? The syntax is ambiguous, and relies on the schema to decode it correctly.
• Blocks in configuration can be partially filled in, and the details of that are dealt with during decoding, but the assignment of objects values requires the type to match. We'd have to derive a way to synthesize structural types with nested optional attributes and convert the values on the fly, or force users to strictly list all attributes which would mostly defeat the purpose of the shortcut.
• Even with some leeway in type conversions, the object structure would need to match very closely with the resource structure, but we don't have the tools to dynamically operate on complex structures in the configuration while remaining within a strict type specification.
• Do we need to break compatibility with older providers which may not be able to handle more dynamic data assigned to blocks?

We have some experience trying to paper over this conceptual mismatch in the opposite direction. Back in 0.12 providers were given a temporary option to treat an attribute as a block in configuration in order to maintain compatibility with some legacy config. Like many temporary workarounds, the SchemaConfigModeAttr flag remains to this day, and still causes problems with configurations where the differences between blocks and attributes leak through the abstraction.

[drewmullen]

Thanks @jbardin. For modules, I believe Martins response I think is correct in how module interfaces should be handled. Custom object types are powerful and accomplish most of what the request is seeking.

Regarding complex structures in resources, I had begun to consider this as well. For standard types that are nested, theres no reason the user cannot match the resource parameters when primitive types are used. For block types, this could be a new primitive within terraform primitive types; It would only be useful for object expansion.

To play nicely with providers, I had imagined that Terraform would expand the variables and add them to the resource data the same as it does today. Ideally, no provider code would have to change... provider devs still could unpack the same as today.

resp.Diagnostics.Append(req.Config.Get(ctx, &data)...)

Could the var.x... expression be just "simple" syntactic sugar?

[jbardin]

Oh yes, I don't doubt that new provider development could work correctly, given that their type system is very close to Terraform's. It's the legacy providers still using the SDK that are the real impediment, because the SDK relies on many assumptions about the values it will see, and its type system can't handle complex structures, or even null consistently. Providers are on the other end of the protobuf interface though, so their implementation isn't directly tied to how we construct the values, we just have to construct them in such a way that it's valid for all working implementations. Giving users a way to construct values that might not be understood by some providers without the guardrails that the language syntax has can lead to hard to diagnose bugs and confusing behavior.

this could be a new primitive within terraform primitive types

FWIW the cty type system is not part of Terraform, it comes from the zclconf project, though we have contributed significantly upstream. That said though, there isn't anything about blocks that can't be represented by cty already, it's what we use to represent all values within Terraform. The mismatch is more at the language layer. Blocks are a special, flexible syntax to fill in certain data structures as defined by the schema; but how that's done isn't visible to users, so coming up with a consistent way for users to discover, define, and apply the data based on those hidden rules is a likely challenge.

[apparentlymart]

I think it's debatable whether it would be intuitive/usable enough in practice, but it seems at least technically possible to try to map an arbitrary cty.Value to a hcl.BodySchema using the same rules the JSON syntax uses to map JSON values to hcl.BodySchema.

In particular that would mean that the shape of the provided data structure would not need to exactly match the object type it ultimately becomes after decoding. It would just need to match one of the several supported patterns for declaring blocks as nested JSON objects, and would then get transformed to match the implied type of the schema after the fact.

I think cty's concepts of object and tuple types are similar enough to JSON's object and array types that the same rules could work. Those rules for the JSON syntax were just a best effort to codify a subset of what the original HCL decoder did while avoiding many of the rough edges that had made the legacy-SDK providers misbehave, and that's a very similar problem statement than what this thread seems to be talking about.

I think the bigger problem is that hcl.Body.Content and hcl.Body.PartialContent don't have hcl.EvalContext, so they cannot evaluate expressions in today's API. Getting this done would probably require a trick similar to how dynamic blocks are implemented, where you first "expand" the body, which attaches a hcl.EvalContext to it so that when you subsequently call hcl.Body.Content it can evaluate the for_each expressions. What's requested here is, I think, essentially a built-in version of that idea with its own special syntax instead of just being implemented in terms of HCL's syntax as an extension.

[jbardin]

Thanks @apparentlymart! Yes, the rough edges around the JSON syntax are part of what I was thinking of. The JSON syntax being targeted only for machine generated configuration means only the developer building the tool to generate the config needs to learn how the structures translate, but introducing those ideas to all users feels like an entirely different scope.

And yeah, I didn't get to the evaluation portion of the problem ;) I expected it would be something like dynamic, but adding those layers to the decoding process has also proven problematic, as every layer added to decoding of blocks has possible interactions with the other layers. I know I've seen problems with the decoding overhead in general, as well as poor interactions between blocktoattr and dynamic.