Amazing log management with ElasticSearch and Kibana in .net

View Github Repository Open presentation in a new window

emilcardell

See all presentation from emilcardell

Amazing log management with ElasticSearch and Kibana in .net

0 0

deliberate-logging

Deliberate Logging Slides

On Github emilcardell / deliberate-logging

Amazing log management with ElasticSearch and Kibana in .net

Speaker Emil Cardell | @emilcardell

How is your application doing?

Behaving the way you designed it?

Why are these questions important?

  • Discovering errors
  • Achieving goals
  • Deliberate decisions

What's my goal?

Selfie

  • Run obstacle courses
  • Huge love for coding
  • Full stack C#/Node
  • Competitive gaming

Data at our disposal?

External resources

  • User analytics
  • Operational (CPU/Memory...)
  • System log and events

Internal resources

  • Error logs
  • Application events
  • Communication events
This talk is about harnessing the internal resources.

Collect the data?

Elasticsearch

http://www.elasticsearch.org/

Describe Elasticsearch => Persistent document oriented search. How do install and use it. Runs on JVM. Mention docker. 
POST /fruit/_search

{  
   filter:{  
      bool:{  
         must:[  
            {  
               term:{  
                  Quantity:10
               }
            },
            {  
               term:{  
                  Name:"banana"
               }
            }
         ]
      }
   },
   aggs:{  
      quantityAggs:{  
         terms:{  
            field:"Quantity"
         }
      },
      nameAggs:{  
         terms:{  
            field:"Name"
         }
      }
   }
}

Make data useful to us?

VISUAL ALL THE THINGS!

Kibana

http://www.elasticsearch.org/overview/kibana/

UI over Elasticsearch. Timeseries based. Old version plain html/JS. New runs on JVM.

How is your application doing?

  • Number of errors
  • Number of important events
  • Server stats

Behaving the way you designed it?

  • Changes in number of errors
  • Changes number of relevant events
  • Response times
  • Communication events
  • Changes in server stats

How should we design our system?

Your browser does not support SVG
Your browser does not support SVG
Your browser does not support SVG
Your browser does not support SVG

Now what?

LogStash

 

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

LogStash

LogFlow

It's a framework for collecting log data, transforming the data and storing it.

Why LogFlow?

  • Full power of the C#/CLR
  • Plug in based model
  • Fluid restricted DSL
  • Persistent state

Other tools?

More log collecting

Statsd

Real time

Graphana

Graphite or InfluxDB

Alerts and updates

Nagios

Sensu

Hubbot

Start collecting data

BE DELIBERATE!

Don't be a slave to randomness

CODE AWESOME STUFF!

Big thanks to LogFlow co creator

Roger Wilson | @captain_jinx

Links

Elasticsearch - http://www.elasticsearch.org/

Kibana - http://www.elasticsearch.org/overview/kibana/

LogFlow - https://github.com/LogFlow/LogFlow

Statsd - https://github.com/etsy/statsd/

Graphana - http://grafana.org/

Graphite - http://graphite.wikidot.com/

InfluxDB - http://influxdb.com/

Nagios - http://www.nagios.org/

Sensu - http://sensuapp.org/

Hubbot - https://hubot.github.com/

Farewell

Emil Cardell | @emilcardell

https://github.com/LogFlow/LogFlow