You might have noticed that I’ve been pretty quiet as of late. We’re working on a super top secret internal project here at my company, and we’ve got the need to ingest a LOT of data around the clock for some analytics work. My preferred DBMS is, of course, Microsoft SQL Server, and like a lot of DBAs, we want to make this swiss army knife of a relational DB platform do everything we can dream up. Thankfully, it can perform most of the tasks we throw at it pretty well. But, the pragmatist in me asks – “Is this the best tool for the job?”. Because we’re just starting this project, we can step back a bit and look at all of our options.
For our project, we do not want to deal with a datacenter of our own. Yes, we’re known as on-prem virtualization enthusiasts, and there in certainly many reasons for keeping things on-prem for some time to come, but cloud is the right choice for us for this project. We’re working on the cloud platforms just as much as we are on-prem these days, and we’re seeing the shift occurring in the industry.
Take a look at the costs of SQL Server licensing in the cloud. To design a SQL Server that can consume upwards of a few million data points a minute, we’re likely to need to spend quite a bit of capital on this platform. It’s just overkill for a straightforward ingestion then export platform. Then, we need to accommodate high availability, disaster recovery, reporting, and analytics needs.
Hmm.
Cloud brings some differences that might be advantageous here. We’re partial to MS Azure as a cloud platform for our company internally, so what does Azure have that can help us?
Now, it’s not as simple as that. Cosmos DB is a collection of APIs for different database types under the hood.
Each one are used differently, and all of the options include many differences in operation and architecture. Of the five listed platform APIs, which should we use? That’s a good question. For this particular project, we want the ability to store tons of inbound data and then will be pulling it out for analysis. Azure Table API seems to work best for this purpose.
SO! Over the next few months, expect a number of blog posts from me here exploring Azure Table on Cosmos DB and the questions, challenges, and experiences we have on ramping up on this new platform.
