In the latest tech industry rivalry, the competition between Databricks and Snowflake in the cloud data and analytics space is getting a lot of attention. It joins the other famous marquee rivalries over the past 100 years, such as those between IBM and HP, SAP and Oracle, or AWS and Azure. To learn more about the similarities and differences between these two big data service providers and how to make better buying decisions when choosing between the two, read on.
What do Databricks and Snowflake do?
For the uninitiated, Databricks focuses on analyzing data at scale regardless of its location. It can broadly be considered a data and analytics platform that helps enterprises extract value from their data. Snowflake is a cloud-based data warehousing platform that positions itself as being a simple replacement to other complex offerings from traditional vendors such as Oracle and even cloud vendors such as AWS, Microsoft, and Google.
Both the platforms apply AI to data issues for enterprises. Therefore, they are Enterprise AI companies that plan to transform the usage of data in enterprises. It could be using AI to integrate data lakes and warehouses, crunching massive scale data to make decisions, or just being an intelligent analytics platform.
Where are the firms today?
Snowflake went public in 2020, making it the largest software IPO in history at a valuation of US$33 billion. Databricks, on the other hand, continues to be private and recently reached US$38 billion in valuation. While money is less of a problem, mindshare, being first to market, and the threat from cloud hyperscalers are bigger challenges. Both vendors struggle from the significant talent demand-supply mismatch, as we covered in our research earlier.
The management of both companies has a strong respect for each other. Databricks, for example, understands that Snowflake had a head start. On the other hand, Snowflake realizes some features of Databricks need to be built for its platform as well.
What is happening?
The two vendors are well covered in the public arena, and many have written almost with a romantic spin about their roots, success, and management background. Both firms have different management styles, with Snowflake run by a professional and Databricks by the founder. However, clients are least bothered about the internal operating model of vendors. They are more concerned about whether to bet on these firms, given cloud vendors have been reshaping the industry. In addition, these two companies are dependent on cloud vendors for their own platforms.
Both the vendors have taken potshots at each other with competing offerings with similar-sounding names such as Data Ocean from Snowflake and Data Lakehouse from Databricks. They also collaborate and have connectors to each other’s platforms while they keep developing their versions of these offerings. The sales and technical teams of these vendors bring out challenges in each other’s platforms to clients, such as how Databricks focuses on Snowflake’s proprietary model versus their open-source platform. Snowflake emphasizes how its compute scaling is faster and data compression is better.
What will happen?
Developers, operators, and data professionals have strong views on which platform(s) they plan to leverage. Given Snowflake’s view on building platforms from a warehousing perspective, enterprises find it easier to migrate. Coming from a data lakes perspective, Databricks has to fight a tougher battle. Moreover, Snowflake is perceived as simpler to adopt compared to Databricks. The bigger issue for both of these vendors is the threat from cloud providers. Not only do these vendors offer their platforms on cloud hyperscalers, but these hyperscalers have built their own suite of data-related offerings.
Both Snowflake and Databricks are losing money and running losses. Innovation will be needed to compete with cloud vendors, and innovation is costly. In addition to cloud, one other big challenge these two vendors face is the growing trend of decentralization of data. As data fabric and mesh concepts gain traction, building a lake or warehouse may lose relevance. Therefore, both of these vendors will need to meet data where it is generated or consumed. They need to make connectors to as many platforms as possible. Moreover, as more open-source data platforms see traction, the earlier powerhouse of Oracle, SAP, Microsoft, and IBM may decline, which will impact these two vendors as well unless they scale their offerings to these open-source databases, messaging, and event platforms.
What should enterprises do?
It’s a known fact that a large number of Databricks clients are customers of Snowflake as well. We recommend the following to enterprises:
The market is still divided on cloud’s role in data transformation, given the challenges around cost and latency. However, as these platforms bring down the total cost of ownership by segregating compute and storage, cloud data platforms will witness growing adoption.
The general questions on best sourcing methods will always persist irrespective of technology. Enterprises will need to answer some of these such as lock-in, security, risk management, spend control, and exit strategy in making their purchasing decisions.
What has your experience been in using Snowflake and Databricks? Please reach out to me at [email protected].
Microsoft has gone all in on DevOps. I’ve been talking with Microsoft’s engineering team that builds Azure, and there are several things we can learn from them.
Microsoft has structured its entire development of Azure along DevOps principles. They made the requisite investments in automating testing and automating provisioning. Of course, Azure is, in fact, a software-defined data center environment, so in some respects they are eating their own dog food.
What are the implications to Microsoft’s technical investments in the software-defined environment – the standard middleware, productivity tools and provisioning tools that go with that?
One of the most important implications is the labor model. Microsoft recognized the need for collaboration and speed and (a) built its teams to address those needs and (b) built the right working environment to enable the teams to do that.
Microsoft divided its Azure organization into small, homogenous teams – feature teams – with only two job functions: product manager and engineers. These teams range from 15 to 20 people who are collocated together. The product manager who has responsibility for the functionality a team is driving (the what and the when) sits in the team of engineers (responsible for the how) and works shoulder to shoulder with them. Microsoft also tore out all their offices and created an open environment of neighborhoods in which they put these working teams.
Microsoft continues to host a lot of development work in China and India. The strategy is to use this talent pool as independent feature teams delivering product capability into general Azure services and sometimes optimizing Azure services for the local market. The highest-level road maps for Azure operate out of Redmond yet leverage the day-to-day product management and engineering capabilities embedded in the feature teams deployed around the globe. This allows Azure to move to a true DevOps environment with continuous releases and automated testing.
The result? Microsoft Azure’s pace of innovation has skyrocketed, quality has improved and the number of errors has dropped dramatically. The offerings have improved and are more user friendly and easier to use. And Azure is now making tremendous headway in capturing market share.
Moving to a DevOps environment enabled Azure to keep pace with the likes of cloud competitors AWS and Google. What could it do for your company?
©2024 Everest Global, Inc. Privacy Notice Terms of Use Do Not Sell My Information Research Participation Terms
"*" indicates required fields
