SPEC (Standard Performance Evaluation Corporation) plans to use its testing methodology to benchmark cloud deployments. The company is well known in the industry because of its benchmarks on computer systems. It maintains and develops benchmarks which are standardized to evaluate enterprise computing systems. A not-for-profit organization, SPEC recently formed a working group to create standards for measuring cloud deployments. A subgroup of the Open Systems Group, the new group will invite participants from VMware, Red Hat, Oracle, Intel, IBM, Dell, and AMD. Yun Chao, SPEC’s long-time benchmark developer will help the group together with SPEC Research Cloud group members.
Because cloud computing doesn’t make use of a single computer at any given time, new standards must be created in order to quantify cloud computing operating concepts. The SPEC benchmarks, once it’s done, can be used by cloud services and software providers in order to publicize and measure their products and services. The new benchmarks are expected to specify the one-of-a-kind attributes of cloud computing. Variability, response time, throughput, and elasticity can be included in the new benchmark together with power consumption, response time, and server throughput.
Elasticity measures the time it takes for a service to change in order to meet clients’ needs. Benchmarking elasticity can include the time it takes to spin a specific workload, or the time it takes to add additional resources like memory and CPUs to a present workload. A test of agility can be incorporated to test how closely a cloud computing provider can match the customer’s needs in terms of needed resources for a specific project.
The new benchmarks to be developed must be able to apply to various services. Such benchmarks are seen to span two cloud services types: black box and white box. Black box services are those services which do not detail its software and hardware offerings while white box services are those services which offer particular details about software and hardware the services use.
Initially, SPEC will develop benchmarks for Infrastructure-as-a-Service but it will also create benchmarks for Software-as-a-Service and Platform-as-a-Service. The initial benchmarks are set to be released in a year’s time.
