Collecting OpenAM logs with logstash


Logstash is a general purpose log collector that can read, transform and collect various logs.

The following logstash configuration will collect OpenAM Access logs. The default target here is Elastic Search - which is document oriented no-sql database optimized for text search (perfect for log files).

In a future blog I will show you how you can use Kibana to makes some sexy charts of your access data. 

 file {  
   type => amAccess  
   start_position => beginning  
   path => "/path_to_your_install/openam/openam/log/amAuthentication.access"  
  }  
 }  
 filter {  
     if [type] == "amAccess" {  
     csv {  
         columns => [time,Data,LoginID,ContextID, IPAddr, LogLevel,  
             Domain, LoggedBy, MessageID, ModuleName, NameID, HostName]  
         separator => " "      
      }  
      date {  
             match => ["dateTime", "yyyy-MM-dd HH:mm:ss"]        
      }  
      geoip {   
           database => "/path_to_your/GeoIP.dat"  
           source => ["IPAddr"]  
         }  
  }  
 }


Here is an upstart config file to start logstash: 

 # logstash - indexer instance  
 #  
 description   "logstash indexer instance"  
 start on virtual-filesystems  
 stop on runlevel [06]  
 respawn  
 respawn limit 5 30  
 limit nofile 65550 65550  
 # set HOME to point to where you want the embedded elasticsearch  
 # data directory to be created and ensure /opt/logstash is owned  
 # by logstash:adm  
 env HOME=/opt/logstash  
 #env JAVA_OPTS='-Xms512m -Xmx512m'  
 chdir /opt/logstash  
 setuid ubuntu  
 setgid ubuntu  
 #setuid logstash  
 #setgid adm  
 console log  
 # for versions 1.1.1 - 1.1.4 the internal web service crashes when touched  
 # and the current workaround is to just not run it and run Kibana instead  
 script  
     exec /opt/java/bin/java -jar logstash.jar agent -f /opt/logstash/access.conf --log /opt/logstash/log.out   
 end script

Comments

Post a Comment

Popular posts from this blog

Introducing ds-operator, the ForgeRock Directory Services Operator for Kubernetes

Image
ForgeRock Directory Services 7.0 was a big achievement for the Grenoble Directory team.  It is the only "Kubernetes native" directory where you can add a new replica using kubectl: kubectl scale sts/ds-idrepo --replicas=3 The 7.0 deployment is assembled using standard Kubernetes primitives such as StatefulSets , Persistent Volume Claims, and Services.  This is all built and orchestrated using Skaffold and Kustomize .  An emerging pattern in the Kubernetes world is the use of Custom Resources and Operators .  Broadly speaking, a custom resource is the declaration of the desired system state, and the operator's job is to observe the current state and bring the system into alignment with the declared state: source: https://blog.container-solutions.com/kubernetes-operators-explained The Kubernetes API server (the thing that responds to your kubectl commands) can be extended to handle new custom types.  A custom resource definition (CRD) describes to the API se...
Read more

OAM R2 REST APIs for Policy Management

Image
Oracle Access Manager 11g R2 provides several new REST APIs. This continues a trend to expose key functionality via Web Services. The OAM Mobile and Social service provides APIs for Authentication, Authorization  and User Profile services.  I will cover those APIs in a future article (have a look here for examples) - but today I want to focus on the  policy management APIs. The Policy Administration API  enables to you to interact with OAM to create a variety of Policy objects such as Application Domains, Resources, AuthN Schemes, and AuthN/AuthZ policies. The policy model is shown below: For example, if you want to retrieve all of the resources in an Application Domain you can perform a GET against the /resource URI: curl -u USER:PASSWORD http://<SERVER>:<PORT>/oam/services/rest/11.1.2.0.0/ssa/policyadmin/resource?appdomain="IAM Suite" Note: The port above is where the OAM Admin Server is deployed (often 7001). It ...
Read more

Automating OpenDJ backups on Kubernetes

Image
Kubernetes   StatefulSets  are designed to run "pet" like services such as databases.   ForgeRock's OpenDJ LDAP server is an excellent fit for StatefulSets as it requires stable network identity and persistent storage. The ForgeOps project contains a Kubernetes Helm chart to deploy DJ to a Kubernetes cluster. Using a StatefulSet, the cluster will auto-provision persistent storage for our pod. We configure OpenDJ to place its backend database on this storage volume. This gives us persistence that survives container restarts, or even restarts of the cluster. As long as we don't delete the underlying persistent volume, our data is safe. Persistent storage is quite reliable, but we typically want additional offline backups. The high level approach to accomplish this is as follows: Configure the OpenDJ container to support scheduled backups to a volume. Configure a Kubernetes volume to store the backups. Create a sidecar container that archives the backups...
Read more