Observability using Elasticsearch and Neo4j

10 min read Elasticsearch continues to add features at an astonishing rate, and people find really creative ways to use them and enhance it even more. What Neo4j can do is just way too cool to pass on. So we’ll look at how to ingest data with elasticsearch and analyze the data with neo4j. Combining the two helps us achieve some really powerful solutions.

I originally was intrigued by elasticsearch for log aggregation and its capability to instantly aggregate and search over millions of records. We could ship logs from all sorts of data sources like application logs, web server logs (Nginx, IIS). Then we can filter through those logs in Kibana’s Discover, choose the columns we wanted to see for particular use-cases and create saved searches. This immediately made it useful to us, the engineering team. We then use query-based filtering to add restrictions on documents people should access, and with field-level security, we can control which fields they even see inside each document. All of a sudden we have the ability to give our level 1 support real-time visibility into customer issues, without overloading them. On top of this, we add Windows event logs and Syslogs and create some alerts.

Monitoring DNS Lookups against your DNS Server using Elasticsearch

7 min read DNS is one of the most fundamental components of today’s internet, one of the simplest technologies at the same time, but it’s also one of the biggest risks, and can be one of the best indicators when something is at risk. There are many cyber attacks that either have their root in DNS, or just use DNS just as a part of the structure of the attack without even thinking about it. Understanding your organization’s DNS traffic is a very solid step in protecting the overall organization. Monitoring DNS lookups with Elasticsearch and analyzing with machine learning can significantly reduce risk around several types of attacks. Here are some of the threats based in DNS and how to know about them.