Google's BigQuery

We know you have a lot of data to work with within your organization, which can present big challenges. Your data can be large in volume and complex in structure. For example, large-scale web applications have millions of users, documents and events to manage. As a result, many engineering teams choose highly scalable NoSQL databases over relational databases. Though this approach is effective in storing and retrieving data, it poses challenges for interactive data analysis.

With Google BigQuery, you can run ad hoc, SQL-like queries against datasets with billions of rows. This can be your own data, or data that someone has shared with you. BigQuery works best for interactive analysis of multi-terabyte datasets, to which you can append fresh data.

Google BigQuery offers these features:

Scalability

• Data storage that scales seamlessly to hundreds of terabytes, with no management required

Speed and flexibility

• Ad hoc queries on multi-terabyte datasets
• Familiar SQL-like query syntax and intuitive web UI
• Ability to JOIN enormous fact tables to most lookup tables

Integration and accessibility

• Integration with Google spreadsheets, letting data analysts drive massive datasets in BigQuery directly from a spreadsheets interface
• Interactive dashboards easily built with Google AppEngine, and smooth data export to Google Cloud Storage
• HTTP REST API, a web UI for interactive querying, and command-line interface

Today Google enhanced BigQuery with several new features:

• Support for JSON: JSON is used to power most modern websites, is a native format for many NoSQL databases hosting large scale web applications, and is used as the primary data format in many REST APIs. With this update, it’s now possible to import data formatted in JSON directly to BigQuery without the hassle of writing extra code to convert the data format.

• Nested and Repeated Fields: If you’re using App Engine Datastore or other NoSQL databases, it’s likely you’re taking advantage of nested and repeated data in your data model. For example, a customer data entity might have multiple accounts, each storing a list of invoices. Now, instead of having to flatten that data, you can keep your data in a hierarchical format when you import to BigQuery.

• Additional improvements:
• Increased import quotas from 1000 jobs per day to 1000 jobs per table per day, and boosted the file size limit from 4GB to 100GB
• Faster data exports from BigQuery to Google Cloud Storage, by enabling large tables to be exported as multiple files in parallel
• Permanently save common queries in the BigQuery interface

If you need help with Extract - Translate - and Load (ETL), BigQuery design, or generating BigQuery reports send us an email to cloud@mmyconsulting.com