Logstash configuration for collecting OpenAM and OpenIDM logs

Following on to my previous posting, here is a logstash configuration that collects logs from both OpenAM and OpenIDM, and feeds them into elastic search:

 input {  
  file {  
   type => idmRecon  
   start_position => beginning  
   path => "/opt/openidm/audit/recon.csv"  
  }  
  file {  
   type => idmActivity  
   start_position => beginning  
   path => "/opt/openidm/audit/activity.csv"  
  }  
  file {  
   type => amAccess  
 #  start_position => beginning  
   path => "/opt/openam/openam-config/openam/log/amAuthentication.*"  
  }  
 }  
 filter {  
     if [type] == "idmRecon" {  
         csv {  
             columns => [  
                     "idX","action","actionId","ambiguousTargetObjectIds","entryType","message","reconciling","reconId",  
                     "rootActionId","situation","sourceObjectId","status","targetObjectId","timestamp"  
                     ]  
         }  
         date {  
             match => ["timestamp", "ISO8601"]  
         }  
     }  
     if [type] == "idmActivity" {  
         csv {  
             columns => [  
             "_id","action","activityId","after","before","changedFields","message","objectId","parentActionid",  
             "passwordChanged","requester","rev","rootActionId","status","timestamp"  
             ]  
         }  
         date {  
             match => ["timestamp", "ISO8601"]  
         }  
     }  
     if [type] == "amAccess" {  
     csv {  
         columns => [time,Data,LoginID,ContextID, IPAddr, LogLevel,  
             Domain, LoggedBy, MessageID, ModuleName, NameID, HostName]  
         separator => " "  
      }  
      date {  
             match => ["time", "yyyy-MM-dd HH:mm:ss"]  
      }  
      geoip {   
        database => "/usr/share/GeoIP/GeoIP.dat"  
        source => ["IPAddr"]  
      }     
   }  
 }  
 output {  
  # Use stdout in debug mode again to see what logstash makes of the event.  
  stdout {  
   debug => true  
   codec => rubydebug  
  }  
  elasticsearch { embedded => true }  
 }  

Now we can issue elastic search queries across all of the data sets. Here is a very simple Kibana dashboard showing events over time and their source:

While this configuration is quite basic, it allows us to find and correlate events of interest across OpenAM and OpenIDM.

Try searching for a sample user "fred" by entering the string into the top search box. You will see all OpenAM and OpenIDM events that contain this string in any field. You can of course build more specific queries - but the default free form search does an excellent job.