Skip to main content
AWS Messaging 0.9.1

Kinesis Stream Module

View SourceRelease Notes

This module makes it easy to deploy a Kinesis stream

Shard Sizing​

Kinesis streams achieve scalability by using shards. This module allows you to either specify number_of_shards directly or to specify the average_data_size_in_kb, records_per_second and number_of_consumers variables and the module will calculate the proper number of shards that should be used based on AWS best practices.

incoming_write_bandwidth_in_kb = average_data_size_in_kb * records_per_second

outgoing_read_bandwidth_in_kb = incoming_write_bandwidth_in_kb * number_of_consumers

number_of_shards = max(incoming_write_bandwidth_in_kb/1000, outgoing_read_bandwidth_in_kb/2000)

Encryption​

Kinesis streams support server-side encryption as described in the Kinesis SSE documentation. It can be switched on retrospectively for existing streams with no interruptions (although only new data will be encrypted).

To enable encryption, set the following parameter

encryption_type = "SSE"

This will use the default AWS service key for Kinesis, aws/kinesis.

If you need to use a custom key, see the master key module as well as documentation on user-generated KMS master keys for further information on how to create them. You can specify one using

kms_key_id = "alias/<my_cmk_alias>"

Examples​

Here are some examples of how you might deploy a Kinesis stream with this module:

module "kinesis" {
  source = "git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/kinesis?ref=v0.0.1"

  name = "my-stream"
  retention_period = 48

  number_of_shards = 1
  shard_level_metrics = [
    "IncomingBytes",
    "IncomingRecords",
    "IteratorAgeMilliseconds",
    "OutgoingBytes",
    "OutgoingRecords",
    "ReadProvisionedThroughputExceeded",
    "WriteProvisionedThroughputExceeded"
  ]
 
}
module "kinesis" {
  source = "git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/kinesis?ref=v0.0.1"
  name = "my-stream"
  retention_period = 48
  
  average_data_size_in_kb = 20
  records_per_second = 10
  number_of_consumers = 10
  
  shard_level_metrics = [
      "ReadProvisionedThroughputExceeded",
      "WriteProvisionedThroughputExceeded"
    ]
}

Sample Usage​

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S KINESIS MODULE
# ------------------------------------------------------------------------------------------------------

module "kinesis" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/kinesis?ref=v0.9.1"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The name of the Kinesis stream.
  name = <INPUT REQUIRED>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The average size of the data record written to the stream in kilobytes (KB),
  # rounded up to the nearest 1 KB
  average_data_size_in_kb = 0

  # The type of encryption to use (can be KMS or NONE)
  encryption_type = "NONE"

  # A boolean that indicates all registered consumers should be deregistered from
  # the stream so that the stream can be destroyed without error.
  enforce_consumer_deletion = false

  # ID of the key to use for KMS
  kms_key_id = "alias/aws/kinesis"

  # The number of Amazon Kinesis Streams applications that consume data concurrently
  # and independently from the stream, that is, the consumers
  number_of_consumers = 0

  # A shard is a group of data records in a stream. When you create a stream, you
  # specify the number of shards for the stream.
  number_of_shards = null

  # The number of data records written to and read from the stream per second
  records_per_second = 0

  # Length of time data records are accessible after they are added to the stream.
  # The maximum value of a stream's retention period is 168 hours. Minimum value is
  # 24.
  retention_period = 24

  # The additional shard-level CloudWatch metrics to enable
  shard_level_metrics = []

  # A map of key value pairs to apply as tags to the Kinesis stream.
  tags = {}

}

Reference​

Required​

namestringrequired

The name of the Kinesis stream.

Optional​

The average size of the data record written to the stream in kilobytes (KB), rounded up to the nearest 1 KB

0
encryption_typestringoptional

The type of encryption to use (can be KMS or NONE)

"NONE"

A boolean that indicates all registered consumers should be deregistered from the stream so that the stream can be destroyed without error.

false
kms_key_idstringoptional

ID of the key to use for KMS

"alias/aws/kinesis"
number_of_consumersnumberoptional

The number of Amazon Kinesis Streams applications that consume data concurrently and independently from the stream, that is, the consumers

0
number_of_shardsnumberoptional

A shard is a group of data records in a stream. When you create a stream, you specify the number of shards for the stream.

null
records_per_secondnumberoptional

The number of data records written to and read from the stream per second

0
retention_periodnumberoptional

Length of time data records are accessible after they are added to the stream. The maximum value of a stream's retention period is 168 hours. Minimum value is 24.

24
shard_level_metricslist(string)optional

The additional shard-level CloudWatch metrics to enable

[]
Details

   Possible Values:
  
   shard_level_metrics = [
     "IncomingBytes",
     "IncomingRecords",
     "IteratorAgeMilliseconds",
     "OutgoingBytes",
     "OutgoingRecords",
     "ReadProvisionedThroughputExceeded",
     "WriteProvisionedThroughputExceeded"
   ]

tagsmap(string)optional

A map of key value pairs to apply as tags to the Kinesis stream.

{}