ChaosSearch brings SQL support to log analysis
Hear from CIOs, CTOs, and other senior executives and leaders on data and AI strategies at the Future of Work Summit on January 12, 2022. Learn more
ChaosSearch today announced that it has added SQL support to a log analysis platform that uses an index engine to make data stored in an object-based cloud storage service available to them. tools without having to convert them to another format. This SQL capability complements existing support for an application programming interface (API) compatible with the open source ElasticSearch engine.
The goal is to make any type of log data available for any type of analysis tool, including analysis applications that use machine learning algorithms through the ChaosSearch data platform, said the CEO Ed Walsh at VentureBeat.
How it works
Rather than forcing organizations to acquire and move all of their log data to a cloud data warehouse, the ChaosSearch data platform indexes log data stored in Amazon Web Services (AWS) S3 and Google Cloud Platform (GCP) without asking an IT organization to standardize them. This capability allows organizations to analyze data using applications from, for example, Tableau, Grafana, or Looker, in addition to using an Elasticsearch engine. Delivered as a managed service, ChaosSearch Data Platform does not require an internal IT department to set up or maintain it.
This approach also eliminates the need to hire a data engineer to prepare a cloud data warehouse for a new use case or worry about how long data might need to be available through that cloud data warehouse, a said Walsh. “All end users need is to have access to it,” he added.
With the rise of new cloud-native platforms, the amount of log data that organizations may need to analyze has increased by orders of magnitude. The bulk of these platforms run in cloud computing environments that make it easy to store log data, for example, an S3 bucket. Much of this log data is generated by applications driving mission-critical business digital transformation initiatives. Information on everything from application performance to how those applications are used can be obtained by analyzing the log data.
Much of that data isn’t analyzed today, either because it’s too difficult to access or too expensive to move into a data warehouse, Walsh said. The ChaosSearch data platform makes it economical to store and analyze months or years of data, Walsh noted. The ability to access this data through artificial intelligence (AI) tools like TensorFlow will only make this log data more valuable, he added.
Businesses are unlikely to abandon cloud data warehouses, but they can be more selective about the type of data they choose to store there, rather than relying on directly accessible object storage systems. using a simple SQL query. These object storage systems can be accessed across multiple clouds without the need for a data engineer to move data from GCP to AWS so that it can be stored in a data warehouse. Object storage systems are for all intents and purposes a virtual data lake.
Less clear is the degree to which organizations may be able to use additional platforms to directly access data stored in an object storage system. However, it is becoming clear that there are several ways to centralize data management. The bottom line for most organizations is to start moving towards a solution as the volume of data to store and analyze continues to increase. Most of this data, including logs, may not need to be stored for more than a few days. The signal-to-noise ratio in terms of the amount of data generated is high, but the information that the data enables can be invaluable.
VentureBeat’s mission is to be a digital public place for technical decision-makers to learn about transformative technology and conduct transactions. Our site provides essential information on data technologies and strategies to guide you in managing your organizations. We invite you to become a member of our community, to access:
- up-to-date information on the topics that interest you
- our newsletters
- Closed thought leader content and discounted access to our popular events, such as Transform 2021: Learn more
- networking features, and more
Become a member