Collecting OpenAM logs with logstash


Logstash is a general purpose log collector that can read, transform and collect various logs.

The following logstash configuration will collect OpenAM Access logs. The default target here is Elastic Search - which is document oriented no-sql database optimized for text search (perfect for log files).

In a future blog I will show you how you can use Kibana to makes some sexy charts of your access data. 

 file {  
   type => amAccess  
   start_position => beginning  
   path => "/path_to_your_install/openam/openam/log/amAuthentication.access"  
  }  
 }  
 filter {  
     if [type] == "amAccess" {  
     csv {  
         columns => [time,Data,LoginID,ContextID, IPAddr, LogLevel,  
             Domain, LoggedBy, MessageID, ModuleName, NameID, HostName]  
         separator => " "      
      }  
      date {  
             match => ["dateTime", "yyyy-MM-dd HH:mm:ss"]        
      }  
      geoip {   
           database => "/path_to_your/GeoIP.dat"  
           source => ["IPAddr"]  
         }  
  }  
 }


Here is an upstart config file to start logstash: 

 # logstash - indexer instance  
 #  
 description   "logstash indexer instance"  
 start on virtual-filesystems  
 stop on runlevel [06]  
 respawn  
 respawn limit 5 30  
 limit nofile 65550 65550  
 # set HOME to point to where you want the embedded elasticsearch  
 # data directory to be created and ensure /opt/logstash is owned  
 # by logstash:adm  
 env HOME=/opt/logstash  
 #env JAVA_OPTS='-Xms512m -Xmx512m'  
 chdir /opt/logstash  
 setuid ubuntu  
 setgid ubuntu  
 #setuid logstash  
 #setgid adm  
 console log  
 # for versions 1.1.1 - 1.1.4 the internal web service crashes when touched  
 # and the current workaround is to just not run it and run Kibana instead  
 script  
     exec /opt/java/bin/java -jar logstash.jar agent -f /opt/logstash/access.conf --log /opt/logstash/log.out   
 end script

Comments

Post a Comment

Popular posts from this blog

Introducing ds-operator, the ForgeRock Directory Services Operator for Kubernetes

Image
ForgeRock Directory Services 7.0 was a big achievement for the Grenoble Directory team.  It is the only "Kubernetes native" directory where you can add a new replica using kubectl: kubectl scale sts/ds-idrepo --replicas=3 The 7.0 deployment is assembled using standard Kubernetes primitives such as StatefulSets , Persistent Volume Claims, and Services.  This is all built and orchestrated using Skaffold and Kustomize .  An emerging pattern in the Kubernetes world is the use of Custom Resources and Operators .  Broadly speaking, a custom resource is the declaration of the desired system state, and the operator's job is to observe the current state and bring the system into alignment with the declared state: source: https://blog.container-solutions.com/kubernetes-operators-explained The Kubernetes API server (the thing that responds to your kubectl commands) can be extended to handle new custom types.  A custom resource definition (CRD) describes to the API server the syntax
Read more

Automating OpenDJ backups on Kubernetes

Image
Kubernetes   StatefulSets  are designed to run "pet" like services such as databases.   ForgeRock's OpenDJ LDAP server is an excellent fit for StatefulSets as it requires stable network identity and persistent storage. The ForgeOps project contains a Kubernetes Helm chart to deploy DJ to a Kubernetes cluster. Using a StatefulSet, the cluster will auto-provision persistent storage for our pod. We configure OpenDJ to place its backend database on this storage volume. This gives us persistence that survives container restarts, or even restarts of the cluster. As long as we don't delete the underlying persistent volume, our data is safe. Persistent storage is quite reliable, but we typically want additional offline backups. The high level approach to accomplish this is as follows: Configure the OpenDJ container to support scheduled backups to a volume. Configure a Kubernetes volume to store the backups. Create a sidecar container that archives the backups
Read more

Deploying the ForgeRock platform on Kubernetes using Skaffold and Kustomize

Image
If you are following along with the  ForgeOps repository, you will see some significant changes in the way we deploy the ForgeRock IAM platform to Kubernetes.  These changes are aimed at dramatically simplifying the workflow to configure, test and deploy ForgeRock Access Manager, Identity Manager, Directory Services and the Identity Gateway. To understand the motivation for the change, let's recap the current deployment approach: The Kubernetes manifests are maintained in one git repository ( forgeops ), while the product configuration is another ( forgeops-init ). At runtime ,  Kubernetes init containers clone the configuration from git and make it  available to the component using a shared volume. The advantage of this approach is that the docker container for a product can be (relatively) stable. Usually it is the configuration that is changing, not the product binary. This approach seemed like a good idea at the time, but in retrospect it created a lot of c
Read more