VMware: Log Insight – AIOps with vRealize (Part 2)
The challenge with the newspapers
If you consider the time you spend working with technology, reviewing log files has to be one of the most daunting tasks. It’s certainly for me, line after line, output messages, which may or may not be human readable, searching for the few words, which hopefully bring clarity to a failed deployment or to a production failure.
To successfully troubleshoot, you need to rely on a few characteristics of these logs:
Identifying the log level
Meaningful message body
There are also other pieces to the puzzle. I have ignored the most important so far, the log must exist for the system or device you are interested in. The parameters should be configured correctly, making sure that logging is enabled, that the output persists, restarts and spins.
Once all of this is in place. You are just waiting for the moment when you have to review the journal. Or rather, you hope you never do it, a sign that the systems are working fine.
When the inevitable happens, log into your system, find the log file output, and then start that long mouse wheel scroll, looking for that key information. If your outage is widespread, you may need to do it over and over again across all systems, gathering error snippets and clues to find the root cause. This is a manual effort, the clock is ticking as your solve time increases.
Quite simply, the scale and volume of the data generated by the machine is increasing exponentially and making sense of it is an overwhelming task.
But as you might have guessed from the title of this blog post, there are options, a smarter way to work. Deploy a log analysis tool and provide a highly scalable log management platform. Centralize all of these individual device and system outputs into one tool, which will help decode and analyze logs. Choose the right tool, and it will offer intuitive features such as actionable information, the ability to respond based on logs received; deep operational visibility and faster troubleshooting, perhaps even identifying anomalies for logs that are outside the observed baseline.
Introducing vRealize Log Insight
Part 1 of this series introduces the concept of operating an AIOps model to improve monitoring, alerting, and troubleshooting across your platforms and clouds. We have outlined the three pillars essential to every AIOps implementation (Observe, Engage, and Act), which we will continue to use as we deepen the capabilities of vRealize Log Insight.
vRealize Log Insight provides intelligent log management for infrastructure and applications in any environment. This highly scalable log management solution provides intuitive and actionable dashboards, sophisticated analytics, and broad third-party scalability in physical, virtual, and cloud environments.
In the observation pillar, we seek to put into context the large amount of data that you have to process on your IT platforms. In this case, his logs. We need to be able to view and query real-time and historical data. Leveraging machine learning, vRealize Log Insight provides the functionality of grouping similar events. Intelligent Grouping analyzes incoming data and quickly groups messages by problem type, enabling high-performance searches for faster troubleshooting and root cause analysis.
These groupings are displayed as event types, and each new type discovered is represented by a smart field. The types can be timestamps, strings, integers, hexadecimal, and others.
Using the Event Trends page, you can view the types of events and compare them to a baseline of the number of logs received over a specified time period.
In the example below, I have filtered the application logs for a VM deployed in our environment, set a custom time range. And now I can see three distinct types of events, two of which are decreasing and one is increasing.
From there, you can select any event or log that interests you, and either highlight other similar logs returned as you browse the output, or just colorize all of the events for you. Visually, you start to see patterns and target the areas that interest you, creating a view of logs from disparate sources together in the same query.
vRealize Log Insight allows you to flexibly ingest logs from all devices and deployments on your IT platform. Giving you complete visibility of your platform with the ability to analyze unstructured or structured log data. vRealize Log Insight accepts data from the following sources: Syslog, Log Insight Agent, REST API, Existing Archives, and vSphere Log Analyzer.
Now that you can effectively visualize and query your logs through this observation pillar, we want to start embedding value further across the platform and across the enterprise. This means that we have to integrate with other technologies. Log Insight achieves this in several ways:
We have already started to cover some UI features to clarify unstructured log data. To continue, vRealize Log Insight is designed to be an API first, the same requests that you run in your browser can be made by API calls, allowing easy integration into your own systems.
Content packs. These are developed and available from VMware and our technology partners, allowing you to import predefined dashboards and queries to quickly get value from your log data sources.
Integration with vRealize Operations. This extends operational visibility and proactive management capabilities beyond logs, across infrastructure, applications and cloud services as well. The combination of the two technologies gives you visibility into in-context log events, as well as metrics and other areas of vRealize Operations such as troubleshooting, alerts, and reporting.
Now that we can effectively visualize the logs, with the context, and have started to embed technology more into the business, we need to be able to act quickly to resolve any issues we see, or ideally avoid any issues. in the first place .
vRealize Log Insight provides the ability to define alert criteria, based on matching queries and thresholds. When these are exceeded, triggers can include sending emails, webhook notifications, or triggering notification events in vRealize Operations. Alerts can also be imported through content packs, for example the vRealize Automation content pack shown below.
Log Insight is an intelligent platform that fits well into an AIOps approach, it provides intelligent log management in a highly scalable architecture, with a content rich marketplace and strong APIs to enable you to act quickly on them. problems and improve the overall uptime of your cloud platforms.
Check back soon for Part 3, where we’ll talk about using vRealize Operations in your AIOps approach, for increased productivity, cost savings, and business acceleration!
This article was co-authored with Dean Lewis, Senior Solutions Engineer, VMware UK. Please see Dean’s personal blog vEducate.co.uk or join him on twitter @saintdle.