Collecting OpenAM logs with logstash

Logstash is a general purpose log collector that can read, transform and collect various logs.

The following logstash configuration will collect OpenAM Access logs. The default target here is Elastic Search - which is document oriented no-sql database optimized for text search (perfect for log files).

In a future blog I will show you how you can use Kibana to makes some sexy charts of your access data.

 file {  
   type => amAccess  
   start_position => beginning  
   path => "/path_to_your_install/openam/openam/log/amAuthentication.access"  
  }  
 }  
 filter {  
     if [type] == "amAccess" {  
     csv {  
         columns => [time,Data,LoginID,ContextID, IPAddr, LogLevel,  
             Domain, LoggedBy, MessageID, ModuleName, NameID, HostName]  
         separator => " "      
      }  
      date {  
             match => ["dateTime", "yyyy-MM-dd HH:mm:ss"]        
      }  
      geoip {   
           database => "/path_to_your/GeoIP.dat"  
           source => ["IPAddr"]  
         }  
  }  
 }  

Here is an upstart config file to start logstash:

 # logstash - indexer instance  
 #  
 description   "logstash indexer instance"  
 start on virtual-filesystems  
 stop on runlevel [06]  
 respawn  
 respawn limit 5 30  
 limit nofile 65550 65550  
 # set HOME to point to where you want the embedded elasticsearch  
 # data directory to be created and ensure /opt/logstash is owned  
 # by logstash:adm  
 env HOME=/opt/logstash  
 #env JAVA_OPTS='-Xms512m -Xmx512m'  
 chdir /opt/logstash  
 setuid ubuntu  
 setgid ubuntu  
 #setuid logstash  
 #setgid adm  
 console log  
 # for versions 1.1.1 - 1.1.4 the internal web service crashes when touched  
 # and the current workaround is to just not run it and run Kibana instead  
 script  
     exec /opt/java/bin/java -jar logstash.jar agent -f /opt/logstash/access.conf --log /opt/logstash/log.out   
 end script