How Computers Automatically Adjust Tasks to Run Faster in Data Centers

A method for cloud computers to monitor their own performance while processing massive data tasks and automatically changing their settings or resource levels to stay efficient.

What it covers

This patent describes a system that acts like a smart thermostat for large-scale computing tasks, specifically MapReduce jobs. As a job runs, the system constantly builds a usage profile that tracks how much memory and processing power the cluster is consuming. If the system detects inefficiencies, it triggers a control loop to reconfigure parameters like input size, resource allocation, or the number of concurrent tasks. It can also dynamically add or remove physical computing resources from the cluster to ensure the job finishes as quickly as possible without wasting capacity.

What it doesn't cover

• —Does not cover static job configurations that are set once before the job begins.
• —Does not cover manual intervention or human-triggered adjustments to job parameters.
• —Does not cover non-distributed computing environments where a single machine processes the entire task.
• —Does not cover general load balancing that does not involve reconfiguring specific job parameters like input size or concurrent component counts.

The clever bit

The innovation is the closed-loop feedback system that ties the job's internal configuration parameters directly to the external cluster's real-time resource availability.

Why it matters

In data centers, running massive data analysis jobs is expensive and time-consuming. Before this, engineers often had to guess the best settings for a job before it started. This patent provides a way to make those systems self-optimizing, which is essential for cloud providers like IBM, AWS, and Google to maximize their hardware utilization and reduce costs.

Real-world examples

• 1.Large-scale data processing on Apache Hadoop clusters
• 2.Automated resource scaling in cloud-based big data analytics
• 3.IBM Cloud data processing services