This article excerpt, by Jeffrey Schwartz, originally appeared here: http://bit.ly/1dN99on

Microsoft took a key step forward in its quest to bring “big data” to the cloud this week when it released the public preview of its Windows Azure HDInsight offering. The cloud-based service, first made available on a limited basis last fall, aims to let enterprise customers process huge volumes of structured and unstructured data using Microsoft’s SQL Server and the Hortonworks distribution of the Hadoop file store.

Organizations of all types–including health care providers–are rapidly gravitating to Apache Hadoop to store information gathered from sources such as social media, news feeds and user-generated content to determine trends to deliver insights and intelligence in near real time.

With scores of startups and established players all jumping on the Hadoop bandwagon, Microsoft has hitched its wagon with the Hortonworks distribution, which the company emphasizes is 100 percent Apache-compatible. The HDInsight service in Windows Azure lets organizations spin up Hadoop clusters in Windows Azure in a matter of minutes, noted Eron Kelly, general manager for Microsoft’s SQL Server group.

“These clusters are scaled to fit specific demands and integrate with simple Web-based tools and APIs to ensure customers can easily deploy, monitor and shut down their cloud-based cluster,” Kelly noted. “In addition, Windows Azure HDInsight Service integrates with our business intelligence tools including Excel, PowerPivot and Power View, allowing customers to easily analyze and interpret their data to garner valuable insights for their organization.”

Among the first to test HDInsight is Ascribe, a U.K.-based Microsoft partner that provides healthcare management systems for hospitals and large medical practices. Its solution handles the lifecycle of patient care using key new components of Microsoft’s portfolio including Windows 8-based tablets, SQL Server 2012 and HDInsight to perform trending analysis using anonymous patient data.

Microsoft is addressing growing demand for in-memory databases, brought to the mainstream last year by SAP with HANA. In-memory databases can perform queries much faster than those written to disk. Microsoft revealed its plans to add in-memory capabilities to the next release of SQL Server, code-named Hekaton, at the SQL Pass Summit.

But with a growing number of startups looking to re-invent the data repository, such as NuoDB, Hadapt and the new Pivotal initiative from EMC, Microsoft is in a more crowded field. While Microsoft has broadened its data management portfolio with SQL Azure and now HDInsight, the requirement to find, process and analyze new types of information is greater than ever. All eyes will be on Hekaton and Microsoft’s ability to deliver new levels of performance to SQL Server.