Posted:
Implementing high-availability for MySQL can easily become complicated with so many options, including some that go a step further in combining high-availability with multi-master replication in a cluster format and more. Not to mention that besides MySQL’s own replication functionality, separate open source projects also include Tungsten Replicator and Galera, amongst others. It’s clear that each option has pros and cons, and selecting the best option for your specific use case can require a lot of research and testing.

That's why today we’re going to try making this process easier by introducing Click to Deploy Percona XtraDB Cluster to help you simply launch a cluster preconfigured and ready-to-use in just a few clicks. Percona is an open source company supporting MySQL that provides a convenient package called Percona XtraDB Cluster, which combines Percona Server with Galera replication software. Using Percona XtraDB Cluster, developers can set up and configure a multi-master MySQL cluster with additional performance, scalability and diagnostic improvements over standard MySQL. Plus, Percona software is open source and compatible with existing MySQL environments.

Along with Percona XtraDB Cluster, each server node includes Percona Toolkit, a set of command-line tools for administering MySQL, giving developers more flexibility in their options for MySQL clustering. Our goal with this Click to Deploy application is to provide an environment to evaluate Percona XtraDB Cluster as a solid option in high-availability MySQL. We want to help illustrate how Google Compute Engine can be used on a team already using MySQL, so you can build your applications however way you like.

Learn more about running Percona on Google Compute Engine and deploy a Percona MySQL cluster today with our free trial credit. Please let us know what you think about this feature and the challenges with scaling MySQL in the cloud. You can also contact Percona for Consulting, Support, Managed Services, and Training. Deploy away!

-Posted by Chris Pomeroy, Program Manager

Posted:
Today’s guest blog comes from Steve MacPherson, chief technology officer for Framestore, a visual effects production firm headquartered in London. Framestore’s visual effects work has been seen in films like “Avatar,” “Gravity,” and “Guardians of the Galaxy,” winning the company numerous Academy Awards, BAFTAs, and Cannes Lions awards.

If you saw “Gravity,” hopefully you enjoyed the movie’s depiction of space travel and weightlessness. It was a brilliant experience to take on such a daunting project, and very rewarding to be part of the collective effort.
A lot of planning and effort goes into every visual effect, as well as a fairly large amount of computing power. At peak times – when we’re rendering images on several projects at once for advertisers and film studios – we’ll consume the processing power of up to 15,000 Intel cores. Managing peak provisioning and matching resources to projects is central to keeping the production pipeline moving toward various deliveries. The challenge that returns regularly is when demand for resources conflicts with capacity – usually during periods when the stress of delivery is at its highest, and the focus is on realizing the creative goals of our clients in time for the immovable object that is a major release date.

Historically, this boiled down to a simple scenario: purchase additional equipment. A design maxim I've long held is that we've never built a machine room that doesn't eventually run out of space, cooling or power. This “computational load bubble” is a result of both the scale of modern studio films, and the fact that we run multiple films through the facility in parallel. This is the peak provisioning problem that we were looking to address for a number of years. Once films are delivered, the demands recede, and we have an excess capacity for some period of time until we reach the next set of deadlines.

For the past few years, we’ve kept a close eye on the potential for using external resources as an overflow valve. Google, through its Google Compute Engine, is the first company we've worked with that was able to combine raw resources with a team that understood our requirements in detail and a business model that helped us manage the economics. Within a day of firing up the network, we built our image inside the Compute Engine container.

At Framestore, we’ve developed a sophisticated in-house job submission system based on our render queue manager, fQ – it’s extremely efficient at juggling various job types to match them accurately to rendering nodes available at any given time. This workflow is central to the sustained high levels of efficiency for our render farm, giving us up to 95% of overall capacity for weeks on end.

Having Google Compute Engine on the back end opened a number of opportunities for us to siphon off a certain class of work during a period of peak production. The load reduction on our farm allowed us to be much more specific in how we prioritized our deliverables, ultimately leading to a much more focused and predictable delivery schedule – great for production and great for maintaining confidence with the studios.

Google gives us breathing room during periods of peak capacity, allowing our artists more flexibility around creative refinement. We do many iterations of an image or a visual effect, making minor technical tweaks and submitting it to the render farm to see if it works. During periods of high load, if all of our rendering is in-house, the creative team might have to wait more than a day to see results when the in-house farm is at capacity. This introduces stress and management overhead around which shots get priority.

By adding Compute Engine to our workflow and allowing our in-house capacity to focus on the studio work, everyone’s project gets computing time – and the creative team can get as imaginative as they want to, with fast views of new iterations.

The results: fewer bottlenecks, more creativity and more predictability, not to mention saving about £200,000 (more than $300,000 USD) on the cores we didn’t need to buy. We can now confidently move into final stages of production on our biggest projects, knowing we have a reserve of computational ability on tap. When you check out new movies this year and next like “Dracula Untold” and “Jupiter Ascending,” you’ll be looking at our visual effects work, created using all the computing power at our fingertips.

Posted:
The true power of cloud computing is unlocked when developers can build resilient and cost efficient applications that use just the right amount of resources necessary at any given time. So the same team that designed the scaling infrastructure for products like Google Search and Gmail have brought a highly anticipated feature to Google Compute Engine - intelligent horizontal Autoscaling. Today we are releasing the service into Beta, which means it is now available for everyone to start using.

Autoscaling allows customers to build more cost effective and resilient applications. Using Compute Engine Autoscaling, you can ensure that exactly the right number of Compute Engine instances are available at any given time to handle your application’s workload. This saves you money when your application’s usage is low, and ensures your application is responsive when utilization is high.

The Compute Engine Autoscaler is able to intelligently and dynamically scale the number of instances in response to different load conditions by defining the ideal utilization level of a group of Compute Engine instances. This means that when the actual utilization of your service increases or decreases, Autoscaler will detect the change and adjust the number of running instances to match. Autoscaler can respond to a number of different metrics such as CPU load, QPS on a HTTP Load Balancer and metrics defined using the Cloud Monitoring service.

One early customer of Compute Engine’s Autoscaler was Wix.com, the popular website-building service. Golan Parashi, Wix.com's Infrastructure Team Lead, commented how Google uses heuristics to determine how many instances to add at one time to hit demand, “reducing [our] expenses, while giving us confidence that Google will manage the appropriate number of machines, even when a spike occurs."

Autoscaler not only chooses the right number of instances but also adapts automatically based on how far the current state is from the desired target. This means Autoscaler performs well even in unexpected scenarios such as sudden traffic spikes. At Google Cloud Platform Live, we demonstrated how an application could scale from zero to handling over 1.5 million requests per second using Autoscaler.

Here are some additional resources to get you up to speed on Compute Engine’s Autoscaler:

We can’t wait to see what you build - and scale - next on our platform.

-Posted by Filip Balejko, Software Engineer

Posted:
Today’s guest blog comes from Vincent Heuschling, founder and CEO of Affini-Tech, a creator of data platforms that help businesses make data-driven decisions. Affini-Tech is based in Meudon, France and was founded in 2003.

Affini-Tech’s primary goal is helping our customers make data-driven decisions, regardless of the industry they work in. This means giving them easy workflows and web-based interfaces for analyzing and managing Big Data – all at a cost that makes sense for their businesses. Google Cloud Platform has become the foundation for everything we do.

When we launched Affini-Tech, we knew we needed a scalable solution for building applications and analyzing information. We explored a number of Cloud vendors - including doing an initial storage deployment on another large public cloud. However, after trying Cloud Platform and speaking with the Cloud Platform team, we decided to go all-in with Google. Google's pricing was the most competitive, but we also found it to be the best platform for our developers. Google App Engine, Google Compute Engine and Google BigQuery provided us with an integrated technology stack that worked better than anything else on the market, making it easier for us to build complex applications.

We use App Engine to build applications that help our customers to control their data collections, model data sets and filter and group data. We sell these applications to marketing companies that want to run their software on top of our platform. App Engine is flexible enough to allow developers at marketing companies to customize our stack for their own needs.



Cloud Platform also helps us generate data findings at a faster rate. We’re storing data in Google Cloud Storage, creating ephemeral Hadoop and Apache Spark clusters, then pushing the data into BigQuery for analysis. Ephemeral clusters provide a more efficient, flexible and cost-effective processing model than old-fashioned static clusters and take full advantage of the Cloud model of computing. The key enablers to using the ephemeral clusters are the Google Cloud Storage connector, which lets us directly access data on Cloud Storage using standard Hadoop interfaces, and bdutil, which helps us automate cluster deployment. Our customers only have to “pay as they process,” which saves them money. Not to mention, setup takes less time. Traditional clusters can take days to install, whereas we can get ephemeral clusters up and running in just minutes.

We can pass these cost savings onto our customers, which makes our products and services more competitive. Many of our users are used to spending more than $250,000 to build a data analytics platform. We can often provide the same service for $2,000 per month. This saves our customers money and allows them to go deeper with their data analytics. This access allows our customers to create things like micro segments in their customer base so they can do better targeting for their marketing campaigns.

In a way, Google is helping to democratize data, since more businesses can afford to study it. If a customer is already using Google Apps – and many of them are – we can integrate our data platforms into Google Apps, making these tools even easier to use and understand.

As a small company, the support we receive from the Cloud Platform team is helping us think bigger. It enables us to build new tools and platforms that take advantage of big data. We plan to make a push for business beyond France and the retail sector – and we’re confident about our expansion, with Cloud Platform doing the heavy lifting.

- Contributed by Vincent Heuschling, founder and CEO of Affini-Tech

Posted:
The Internet is quickly running out of IPv4 addresses (the traditional IP addresses, e.g. 173.194.251.158), which has led to rationing strategies such as having to pay for reserved addresses. The industry response to this issue has been to develop a new, much larger, address space: IPv6 offers an abundance of available IPs (of the form 2001:4860:4864:1:329a:211f:1d19:a258). Google has been at the forefront of IPv6 adoption, and we are now assigning an immutable IPv6 address to each and every Cloud SQL instance and making these addresses available for free.

Everything is ready for you to benefit from IPv6: you can see the IPv6 address of your instance using the Cloud SDK command-line tool. With the current version of the tool, the command is:

gcloud sql instances describe

The response will be of this form (we’ve bolded the IPv6 address and shortened the output for readability).

currentDiskSize: 82.4 MB
databaseVersion: MYSQL_5_5
instance: testinstance1
instanceType: CLOUDSQL_INSTANCE
ipAddresses:
[
ipAddress: 173.194.251.158
]
ipv6Address: 2001:4860:4864:1:329a:211f:1d19:a258
maxDiskSize: 250.0 GB
(...)


Alternatively, you can see it in the Google Developers Console (in the Properties section on the instance overview page):

All new and existing Cloud SQL, instances will be assigned an immutable IPv6 address. Please note however that in order to use it you need to explicitly authorize your Cloud SQL instances to receive IPv6 traffic. You can do this using the same access control mechanism you use to control IPv4 traffic, by explicitly authorizing external IPv6 addresses that are allowed to connect to your instance. For details, see how you can configuring access control for IP connections.

If you currently connect to your instance over IPv4, you can continue to do so, or you can switch to connect over IPv6. If you make the change, you can then choose to release your IPv4 address and stop paying for it. To do so, follow the instructions to edit the properties of a Cloud SQL instance.

We understand how frustrating it is to optimize for artificial constraints, like a limited address space, which is why we’re bringing IPv6 to Google Cloud SQL so you can focus on your applications instead. For any question please join us on Stack Overflow using the google-cloud-sql tag.

-- Posted by Easwar Swaminathan, Software Engineer

Posted:
Last week at Google Cloud Platform Live, we demonstrated how Firebase could be used to build mobile apps quickly and also gave a sneak preview of some new features the team has been hard at work on.

One highlight onstage was showing how you could build a fully functional collaborative office planning application in about 15 minutes. Building this app so quickly was possible because Firebase provided the common backend infrastructure needed to get up and running, allowing us to focus on the application’s front-end code and user experience.

We’ve been busy
We’re proud of what Firebase already can do to speed up your development time, but we’re working harder than ever to improve the platform. At last week's event, we announced a new Firebase feature called enhanced query support. You can now query data by any child key and have that query update in realtime as your data changes. These improvements make it easier for developers to sort and filter their data and will make many common use cases much easier to implement.

Last week, we also demoed a new type of integration with server-side code that makes it especially easy to connect Firebase with Google App Engine. This new feature, called Triggers, is simple to implement: in your Firebase’s rules, you define a Trigger specification which includes criteria for when the trigger should fire. When the criteria are met, an outbound HTTP request is sent to an external service you specify along with any data and headers you provide. Use triggers to submit a payment, send an SMS, ping a server for logging and analytics or invoke some code on an external server -- all without needing to write your own server-side code.

We used Triggers to connect the office planning application to an analytics backend running on App Engine, and then used BigQuery to generate a report about how the furniture moved around the room -- all in just a few minutes with a couple lines of code. Our Triggers feature hasn’t launched yet, so stay tuned for our beta announcement soon.

And are going to get even busier...
We’re continuing to improve Firebase along every dimension, and a big part of that effort will go towards improved integration with the rest of Google Cloud Platform. Although developers building Firebase applications love the simplicity of focusing on client-side development, about half of our developers also run their own server-side code. These developers need to do computationally intensive tasks like video or image processing, perform complex analysis on their data, and keep proprietary business logic on trusted servers. With Firebase joining Google Cloud Platform, developers can power their entire product using a single platform.

This is just the beginning. We’re focused on finding all of the ways Firebase can work seamlessly with other Google Cloud Platform products to bring you the best developer experience possible. To get these updates first, please follow us on Twitter.

As always, we look forward to seeing what you build.

-Posted by Andrew Lee, Product Manager

Posted:
Last Tuesday, we announced the availability of the Google Container Engine Alpha, our new service offering based on Kubernetes, the open source project we announced in June. One of the advantages of using Kubernetes and Docker containers as the underpinnings of Google Container Engine is the level of portability they offer our customers, as both are designed to run in multiple clouds.

We listened to our customers explain their needs for a multi-cloud strategy, with either mixed public and private deployments or in multiple public clouds, so we decided to focus on mobility as a design goal for our next generation computing service. We also wanted to make sure these advantages would benefit both developers needing to run their workloads in multiple clouds indefinitely, as well as those just getting started and looking to move to the cloud. That’s why Google Cloud Platform is an ideal environment for customers who are in the process of moving to the cloud, want to run only part of an application in the cloud, or need to run an application in multiple clouds. Here are some common hybrid cloud use cases we hear from our customers:

  • Develop and perform scale out testing in the cloud, but deploy to an on-premises production data center. A huge benefit to many of our customers is being able to do high throughput scale out testing of an application on resources that are paid for by the minute because it reduces iteration time and improves team productivity. Because Google Compute Engine is billed in minute quanta, the incremental cost of accelerated scale out testing is low. You pay what you would for sequential testing-- it just happens on more cores and finishes much more quickly. This only works if the framework that runs your application is available in both your production and test environment. It also helps to have a management framework that makes it easy to deploy, orchestrate and wire together individual tests. Google Container Engine with Docker containers provide a framework that supports the easy deployment and management of an app, and also lets you easily integrate test management and orchestration frameworks.
  • Migrate a new piece of an application to the cloud, but have parts of it stay on-premises. With Google Cloud Platform’s newly announced direct peering and carrier interconnect network features, it’s now easier to connect a part of an application deployed in the cloud to Google’s data centers with the on-premises parts. With 70+ peering locations in 33 countries, it’s possible to get unprecedented levels of throughput and low latency access to your cloud resources. Many of our customers also highly value a common toolchain and management paradigm, as it makes sense to build modern applications using the same tools, packaging format and management services, but let the pieces that need to stay on-premises remain there.
  • Burst to the cloud during peak load. The cloud offers the ability to quickly and easily spin up a large number of VM instances that are charged on a per minute basis. Compute Engine instances tend to boot in around 30 seconds, giving our customers the ability to react quickly to unexpected demand spikes.

Kubernetes and Container Engine were designed from the ground up to meet the needs of those looking to benefit from application mobility. The following properties ensure our customers receive high levels of portability:

  • Container based. Docker has created a highly portable application container framework and is committed to the vision of making it run everywhere. The natural decoupling of application pieces from the OS and infrastructure environment is a really important ingredient in achieving high levels of portability.
  • Modular to the core. To become broadly adopted, it was important to allow providers to adapt and extend pieces of the stack without invalidating the core API. We focused on rigorous and principled modularity in the design, and pretty much everything in Kubernetes can be unplugged and replaced by other technologies.
  • Decoupled services. A key insight into what allows portability and mobility is the idea that different pieces of an application may be moved to a different cloud at different times. Kubernetes is built with a focus on micro-services based architecture and ensuring that the pieces of an application are not tied together. The beauty of Kubernetes is that its naturally decoupled model creates the feeling like the pieces are co-deployed. You don’t have to jump through hoops to get a decoupled deployment.

To achieve unprecedented levels of portability of applications, the community has pulled together to support integration from the start. Some of the biggest names in technology have stepped up to help bring Kubernetes to their technology stacks, including Microsoft, IBM, VMware, and HP. Beyond basic integration, a set of our partners have been working hand-in-glove with us on the core product to strengthen the platform, and add new capabilities and abstractions that offer even higher levels of portability. For example, Red Hat has contributed tirelessly to almost every component of the stack and has been instrumental in shaping and improving the overall production readiness of Kubernetes.

In addition, we have relied on CoreOS technologies in Kubernetes for some time, such as using etcd for distributed state management. Looking forward, they are working to deliver new technology to achieve high levels of portability for Kubernetes and have also started developing new capabilities for the platform, most prominently Flannel. Because Kubernetes relies on virtualized networking capabilities, some of our earlier customers indicated that it was challenging to move to environments that were not running on the same virtualized network technologies that Google offers (Andromeda based). With Flannel, we now have a more portable network layer for Kubernetes.

CoreOS also just contributed code to ensure that Kubernetes works well on Amazon Web Services and have signed up to qualify our binary releases and ensure high levels of mobility between Google and Amazon. Alex Polvi, CEO of CoreOS said, “We really respect the architecture behind Kubernetes. CoreOS stands behind the project and is working to provide support across cloud and on-premises environments to encourage interoperability. You can run Kubernetes in any environment CoreOS supports, which includes AWS and all other major cloud providers.”

We openly invite others to join in our journey with this project. Our IRC channel is here, and the open source project is hosted on GitHub. You can take Container Engine for a free test drive and get all the details you need to get started with our technical docs.

-Posted by Craig McLuckie, Product Manager