Contents

Notes: Kinesis Agent for Windows

Recently I have to study Kinesis agent for Windows, so here we are. Actually AWS documents are pretty good for most of the time, this post is just some kind of crash course.

I strongly recommend to go through the official document if you have some time to spend.

What can Kinesis agent for Windows do?

  • Gathers, parses, transforms, streams:
    • logs
    • events
    • metrics
  • To:
    • Kinesis Data Streams (stream to customized consumers)
    • Kinesis Data Firehose (stream to storage services, e.g., Amazon S3, Amazon RedShift, Amazon ElasticSearch Service, and Splunk)
    • CloudWatch (For metrics)
    • CloudWatch Logs (For log storage/search)
    • …You can do analysis/visualizations/alarms with aforementioned services afterwards.

Insallation

Make sure .Net Framework 4.6 or later is already installed.

There are 2 ways to install agent:

  1. Use AWS System Manager

  2. Download From S3 and Install via Powershell

    • Latest version:

      1
      
      Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/InstallKinesisAgent.ps1'))
      
    • Specific version:

      1
      2
      
      Invoke-WebRequest "https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/InstallKinesisAgent.ps1" -OutFile InstallKinesisAgent.ps1
      .\InstallKinesisAgent.ps1 -version "1.0.0.115"
      
  3. Interact with service:

    1
    2
    3
    
    Start-Service -Name AWSKinesisTap
    Stop-Service -Name AWSKinesisTap
    Get-Service -Name AWSKinesisTap
    

Components

https://docs.aws.amazon.com/kinesis-agent-windows/latest/userguide/images/KinesisAgentPipes.png
(credit: Amazon Kinesis Agent for Microsoft Windows Concepts)
  • Source: Where data like log, events, and metrics are gathered.

    • Can optionally transform data.
  • Sink: Where to send the data:

    • Supported target services:

      • Kinesis Data Streams (stream to customized consumers)
      • Kinesis Data Firehose (stream to storage services, e.g., Amazon S3, Amazon RedShift, Amazon ElasticSearch Service, and Splunk)
      • CloudWatch (For metrics)
      • CloudWatch Logs (For log storage/search)
    • Can optionally transform data. e.g., text to json or xml.

    • Can use sink decorations to enhance the data.

    • Can config sink queueing when connection is unstable.

  • Pipe: 1 pipe associates 1 source and 1 sink.

    • Can optionally filter out data by using FilterPattern before sending to the sink.

    • Using multiple pipes can achieve many sources to one sink or one source to many sinks.

Configurations

  • appsettings.json: Located in %PROGRAMFILES%\Amazon\AWSKinesisTap.

  • Disable telemetrics if necessary.

  • Sources, Sinks, and Pipes are defined in saparated array respectively.

  • Example appsettings.json for security events:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    
    {
      "Sources": [
        {
          "Id": "SecurityLog",
          "SourceType": "WindowsEventLogSource",
          "LogName": "Security"
        }
      ],
      "Sinks": [
        {
          "Id": "KinesisSecurityEventSink",
          "SinkType": "KinesisStream",
          "StreamName": "KinesisSecurityEventStream",
          "Format": "json",
          "Region": "us-west-2"
        }
      ],
      "Pipes": [
        {
          "Id": "SecurityLogPipe",
          "SourceRef": "SecurityLog",
          "SinkRef": "KinesisSecurityEventSink"
        }
      ],
      "Telemetrics": {
        "off": "true"
      }
    }
    
  • More examples

Further Readings