Google’s corporate IT department emancipates itself from the corporate practice of purchasing proprietary technology from other companies by instead building their management software through in-house means or adopting an open source software package. While they are not completely straying away from proprietary software, it is usually considered as a last option.
According to Google software engineer Justin McWilliams, building services in house is ultimately cheaper in the long run compared to buying proprietary ones, stating that it’s company policy to try and build technology from scratch first before even thinking of checking out open source alternatives, with proprietary purchases being the least prioritized option.
Google already employs a number of in house or modified open source technology for their IT management, which includes applications for full disk encryption, remote computer management, compliance management, VPNs, teleconferencing, and even for SS).
In the past, IT departments at large organizations have become accustomed to purchasing commercial, off the shelf solutions for their infrastructure, because it is less expensive and time consuming than building and maintaining the software in house, but the same approach doesn’t work with Google, notes McWilliams.
Google’s infrastructure is so tightly interconnected that even if they buy something off the shelf, they still have to build on top of it so that it can function effectively from within the company’s infrastructure. The company doesn’t want to nurture data silos. In the end, the cost of employing engineers to build and maintain applications is still cheaper than maintaining expensive support contracts with third party software vendors, and then hiring an engineer to build on top of said software in order to make it fit within the company’s infrastructure.
The key reason behind Google’s build philosophy is the speed in which they grow as a company. With over 32,000 employees, the company’s IT staff (which is not growing at the same rate) has to prioritize scalability when setting up operations. Commercial software is inherently more difficult to scale, especially at speeds that Google requires, compared to in-house technology.
One of Google’s hurdles is machine management, which is true for any organization these days. This is partly because Google engineers are allowed to choose their own operating systems for their work machines, so an engineer can opt for Apple’s OS X, Google’s own Chrome, or even Linux and Windows. The Apple machines, in particular, are very hard to maintain for the company due to the limited tools available for the OS and Apple’s strict policies for their code. This is especially made difficult by the fact that Google currently has one of the largest IT deployments of Macs in the world, which is estimated at 30,000 units.
Google used to employ an open source configuration management tool called Puppet for pushing patches and software updates to the Mac workstations, but the company soon ran into problems related to scaling. When it looked at commercial third party solutions, they realized it’s not cost efficient as the ones off the shelf charge around a hundred dollars per machine every year, added to the cost of additional file servers and load balancers that need to be deployed.
The real solution for Google’s predicament came in the form of the open source software called Munki, which came from an engineer at Walt Disney Animation Studios. Since Munki was deployed on Google’s own App Engine, there is no need for Google to deploy and manage additional physical servers to run the software.
Another Mac-related hurdle that Google managed to tackle without reosrting to commercial software is the encryption of disks. While OS X Lion 10.7 has a built in FDE with a program called FileVault 2, it has issues for corporate users, such as its lack of ability to force users to encrypt disks. There is also no means of storing keys except for the one provided by Apple itself. To solve this predicament, Google went ahead and developed their own software called Cauliflower Vest, which oddly enough, is an anagram of “FileVault Escrow”, which perfectly describes what it does.
The Cauliflower Vest allows an admin to get a recovery key, unlock a hard drive, and reset the password in case a Googler forgets his or her password. Like Munki, Cauliflower Vest also runs on Google’s App Engine.
What’s more interesting about Google’s approach to scalability is that it’s not exclusive to IT, since the company also employs California’s biggest private fleet of shuttles and buses that employees can use to get to and from work. Each vehicle comes equipped with Wi-Fi, so that the employees can be productive even during the commute.