Notes: Kinesis Agent for Windows

Recently I have to study Kinesis agent for Windows, so here we are. Actually AWS documents are pretty good for most of the time, this post is just some kind of crash course.

I strongly recommend to go through the official document if you have some time to spend.

What can Kinesis agent for Windows do?

#

• Gathers, parses, transforms, streams:
  • logs
  • events
  • metrics
• To:
  • Kinesis Data Streams (stream to customized consumers)
  • Kinesis Data Firehose (stream to storage services, e.g., Amazon S3, Amazon RedShift, Amazon ElasticSearch Service, and Splunk)
  • CloudWatch (For metrics)
  • CloudWatch Logs (For log storage/search)
  • …You can do analysis/visualizations/alarms with aforementioned services afterwards.

Insallation

#

Make sure .Net Framework 4.6 or later is already installed.

There are 2 ways to install agent:

1. Use AWS System Manager

2. Download From S3 and Install via Powershell

   • Latest version:

     Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/InstallKinesisAgent.ps1'))
     

   • Specific version:

     Invoke-WebRequest "https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/InstallKinesisAgent.ps1" -OutFile InstallKinesisAgent.ps1
     .\InstallKinesisAgent.ps1 -version "1.0.0.115"
     

3. Interact with service:

   Start-Service -Name AWSKinesisTap
   Stop-Service -Name AWSKinesisTap
   Get-Service -Name AWSKinesisTap
   

Components

#

(credit: Amazon Kinesis Agent for Microsoft Windows Concepts)

• Source: Where data like log, events, and metrics are gathered.

  • Can optionally transform data.

• Sink: Where to send the data:

  • Supported target services:

    • Kinesis Data Streams (stream to customized consumers)
    • Kinesis Data Firehose (stream to storage services, e.g., Amazon S3, Amazon RedShift, Amazon ElasticSearch Service, and Splunk)
    • CloudWatch (For metrics)
    • CloudWatch Logs (For log storage/search)

  • Can optionally transform data. e.g., text to json or xml.

  • Can use sink decorations to enhance the data.

  • Can config sink queueing when connection is unstable.

• Pipe: 1 pipe associates 1 source and 1 sink.

  • Can optionally filter out data by using FilterPattern before sending to the sink.

  • Using multiple pipes can achieve many sources to one sink or one source to many sinks.

Configurations

#

• appsettings.json: Located in %PROGRAMFILES%\Amazon\AWSKinesisTap.

• Disable telemetrics if necessary.

• Sources, Sinks, and Pipes are defined in saparated array respectively.

  • Do check the source declarations for how to parse data the way you want.

• Example appsettings.json for security events:

  {
    "Sources": [
      {
        "Id": "SecurityLog",
        "SourceType": "WindowsEventLogSource",
        "LogName": "Security"
      }
    ],
    "Sinks": [
      {
        "Id": "KinesisSecurityEventSink",
        "SinkType": "KinesisStream",
        "StreamName": "KinesisSecurityEventStream",
        "Format": "json",
        "Region": "us-west-2"
      }
    ],
    "Pipes": [
      {
        "Id": "SecurityLogPipe",
        "SourceRef": "SecurityLog",
        "SinkRef": "KinesisSecurityEventSink"
      }
    ],
    "Telemetrics": {
      "off": "true"
    }
  }
  

• …More examples

Further Readings

#

• Official document
• Build More Reliable and Secure Windows Services Using Amazon Kinesis Agent for Microsoft Windows
• Collect, parse, transform, and stream Windows events, logs, and metrics using Amazon Kinesis Agent for Microsoft Windows | Noise
• Manage centralized Microsoft Exchange Server logs using Amazon Kinesis Agent for Windows