Google BigQuery update aims for enticing Hadoop users

Hoping to lure more Apache Hadoop users to its own data analysis services, Google has outfitted BigQuery with the ability to query multiple data tables.

"Joining terabyte-sized tables has traditionally been a challenging task for data analysts, requiring sophisticated MapReduce development skills, powerful hardware, or a lot of time -- often all three," wrote Ju-kay Kwek, Google BigQuery product manager, in a blog post announcing the update. "Today with BigQuery you can get directly to business insights using SQL-like queries, with far less effort and far greater speed than you could before."

[ Andrew C. Oliver answers the question on everyone's mind: Which freaking database should I use? | Keep up with the latest approaches to managing information overload and compliance in InfoWorld's Enterprise Data Explosion Digital Spotlight. ]

Google also argued that using BigQuery instead of a Hadoop deployment will save users money, because they only pay for the queries that are processed, rather than pay for the computational costs of running individual Hadoop supporting components.

Launched in 2010, BigQuery has been marketed by Google as an interactive service for parsing large amounts of data. With BigQuery, a user submits a data set to Google, then can query the data through the BigQuery API (application programming interface).

The new updates expand capabilities BigQuery already has in place. Most notably, a new JOIN clause that combines the results of a query across multiple data sources. Prior to this update, BigQuery's JOIN clause could only work with a data set less than 8MB in size. The new clause, JOIN EACH, has no limit on the size of the data.

As a result, the service can now be more effectively used as a replacement to Hadoop's MapReduce. Many Hadoop jobs are designed to bring together large amounts of data from two or more data sets. To do this however, developers must write MapReduce processes from scratch, which can be time consuming. JOIN EACH can produce a single result set from two large database tables that share a common key.

"With these capabilities, you will now be able to join and perform aggregate analysis on multi-terabyte datasets using SQL-like queries or integrated [third] party tools, instead of having to initiate complex coding projects," wrote Michael Manoochehri, Google's cloud platform developer programs engineer, in a technical blog post explaining the update.

BigQuery also now offers a better way to group query results as well. The GROUP BY EACH statement increases the number of distinct entities that can be grouped in a result set, though at a potential cost to processing performance.

The BigQuery update includes a couple of other new features as well. The service has more supports for timestamps: BigData can now import timestamps from other systems, as well as query timestamp data. Users can now add columns onto existing tables. Users can now also bookmark the specific datasets they have access to, as well as receive automated emails when they have been given access to a new dataset.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

View the original article here

Dataguise introduces field-level encryption for Apache Hadoop database

Dataguise says the latest version of its data-protection product enables users to encrypt sensitive data right down to specific fields within an open source Apache Hadoop database.

DG for Hadoop 4.3 also makes use of the traditional Dataguise "masking" capability across single or multiple Hadoop clusters to camouflage sensitive data.

[ Discover what's new in business applications with InfoWorld's Technology: Applications newsletter. | Get the latest insight on the tech news that matters from InfoWorld's Tech Watch blog. ]

[RELATED: How one retailer is migrating away from encryption to protect customer data]

As for the new capabilities, the product can be used to encrypt structured and unstructured data via the Advanced Encryption Standard.

Subra Ramesh, architect and technical director at Dataguise, says DG for Hadoop 4.3 has a way to conduct a context-sensitive search of unstructured data. You can establish an automated policy to discover specific kinds of data, such as credit-card numbers, for example, and encrypt them, he says, adding, "Decrypting it is based on someone's permission."

When a task is carried out or changes occur, DG for Hadoop can send out automated notifications to the security manager by e-mail or SMS, and reporting is designed to be included in compliance reports.

DG for Hadoop 4.3 starts at $25,000.

Ellen Messmer is senior editor at Network World, an IDG publication and website, where she covers news and technology trends related to information security. Twitter: MessmerE. E-mail: emessmer@nww.com.

Read more about wide area network in Network World's Wide Area Network section.

View the original article here

Hortonworks brings Hadoop to Windows

Hortonworks is bringing the popular open-source Apache Hadoop data processing platform to Microsoft shops.

The company has released a beta version of its Hortonworks Data Platform (HDP) Hadoop distribution for Windows and expects to release the final, enterprise-ready version in the months to come.

[ Also on InfoWorld: Hadoop will be in two-thirds of advanced analytics products by 2015. | Harness the power of Hadoop with InfoWorld's 7 top tools for taming big data. | Discover what's new in business applications with InfoWorld's Technology: Applications newsletter. ]

HDP is "the first and only distribution of Hadoop available on both Linux and Windows," said David McJannet, Hortonworks vice president of marketing.

According to McJannet, Hortonworks heard a lot of demand from potential customers for a Hadoop distribution that would run on the Microsoft platform.

"The real catalyst is, frankly, market demand. The significant majority of the servers running in the enterprise today are running Windows Server," McJannet said. "We've seen significant interest from our customers towards using Hadoop on the platform that they rely on for their critical applications."

Hortonworks and Microsoft have been porting the software to Windows over the past 18 months, as well as testing the software for enterprise use, McJannet said. The HDP distribution consists of a set of different software programs -- including HDFS, MapReduce, Hive, Pig and others. Like the Linux version, the Windows HDP will be available as open source "so others can benefit and extend the work that we have done," McJannet said.

Going forward, Hortonworks will release new versions of the HDP in both Linux and Windows. This first Windows beta version is based on the HDP 1.1 codebase.

Initially, the Windows beta does not have feature parity with the Linux version, though it does have all the "core components" to run Hadoop, McJannet said. But it does not include the Ambari set of management tools. Over time, however, Hortonworks does plan to duplicate all the features on the Windows version.

Hortonworks expects that the kind of workloads run on the Windows platform will be similar to those run on Linux, in terms of size and scope. "We fully anticipate some of the largest deployments of Hadoop could well be on Windows," McJannet said.

The distribution does not support running a mixture of Windows nodes and Linux nodes in the same deployment. Deployments should be all in one OS or another. "In practice, we'd expect homogeneity across the infrastructure, though we'd have to wait and see how that pattern emerges," McJannet said.

Over time, Microsoft will provide more support in other software products, most notably System Center, for organizations that want to move Windows Hadoop workloads in between their own data centers and a Microsoft Azure cloud service, said Herain Oberoi, Microsoft director of product marketing in the company's server and tools division.

As of press time, Hortonworks hasn't finalized the versions of Windows Servers upon which HDP will run, though the beta will run on Windows Server 2008 and Windows Server 2012. The product will not run on Windows desktop versions.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's email address is Joab_Jackson@idg.com.

View the original article here