Transit Note #105
Global Cooperative Computing

Andre DeHon

Jeremy Brown

Ian Eslick

Original Issue: April, 1994

Last Updated: Tue May 10 22:06:44 EDT 1994

Abstract:

Global cooperative computing exploits the National Information Infrastructure / World-Wide Web to deploy new software and hardware technology, leverage scarce human expertise, and acquire feedback information on deployed software use. Global cooperative computing links deployed software back to standard, home repositories on the NII/WWW. Throughout each program's lifetime, the computing system uses this rendezvous point to communicate usage experience, program improvements, and program variations.

Cooperative Sharing

The rich interconnection between machines fostered by the Internet has allowed programs to be distributed easily and used widely. More recently, the network resources have provided an environment which allowed free software ( e.g. Free Software Foundation) to develop and thrive. The availability of free software in source form has been beneficial to many research and commercial efforts.

Nonetheless, the current support for program sharing is primitive and ad hoc.

The large burden of distribution and maintenance on a code's author is a disincentive to potential providers.
The lack of an organized sharing mechanism makes location of relevant code difficult.
The lack of integrated mechanisms for quality feedback makes it difficult for consumers to communicate and share their experience with shared code.
Consumer verification of the utility of shared code is tedious, providing further disincentive for use.

Further, current techniques primarily support coarse-grained sharing at the program level. With current techniques and limited software tools it is impractical to share common solutions and building blocks at sub-program levels.

We envision a framework which supports cooperative code optimization and sharing over the NII, built on top of the WWW. The key elements of our approach are to:

Exploit the World-Wide-Web as a global knowledge base for sharing code, optimizations, and quality information.
Exploit human expertise to determine when errors occur; then provide automated assistance for error localization, correlation, and reporting.
Leverage the massive human expertise available on the NII to rapidly advance software technologies.
Support the automated maintenance of employed use and feedback information as provided by both human and computer consumers of code and optimizations in the global knowledge base.
Employ quasistatic feedback and specialization to compile high-quality code adapted to the end-user's application.
Exploit ``Idle'' workstation cycles for high-performance, automated tuning of compiled code.

Our vision of cooperative computing is closely tied to exploiting the World-Wide Web and providing programmers with a flexible and powerful code management environment -- HyperCode. HyperCode brings HyperText technology to program source code. Program source code is richly annotated with direct links to relevant information including definition sites, program genealogy, quality annotations, and documentation. The base functionality HyperCode provides is explicit function Call Callee-source links. These links provide a number of key benefits. They allow programs to precisely indicate their dependencies making it easier for people to share code. Further, the links identify a canonical home site for each piece of code. This home site serves as a rendezvous location for information about the code, including performance, functional quality, usage statistics, and version information. The canonical HyperCode version of a piece of code collects this information in an automated manner. New versions are linked to the original, making innovations or fixes rapidly and automatically available to all users. The support structure we envision automates the tedious aspects of shared code maintenance, usage reporting, verification, and updates. Human expertise is still vital, but the effort required on the part of the programmers and user is drastically reduced by sophisticated support tools.

We envision this sharing paradigm eventually applying to all programming and scripting languages and hardware description languages. Global cooperative sharing provides an opportunity for programs to rapidly and automatically ride the software technology curve. As soon as a new version of a routine is posted, a cooperative-computing sensitive quasistatic compiler ([DE94] [DEMK93]) locates and evaluates the new version. If the new routine allows an existing program to run faster, the compiler replaces the older routine with the new and faster one.

We can take this software technology curve one step further by sharing basic compiler transforms. As new transforms are shared, the quasistatic compiler experiments with the new transforms on existing code. In this way, programs rapidly and automatically take advantage of new compilation techniques.

To date, the WWW has been used primarily to publish information. Cooperative sharing represents a new class of applications for the WWW and NII wherein distributed computers globally cooperate on common computing problems. In the cooperative-sharing paradigm, the computers cooperate towards the common goals of achieving highly optimized application code and collecting broad-based quality feedback information.

Hardware Design Library

As hardware designs rely more and more heavily on CAD tools, hardware design language descriptions, and common interchange formats, basic hardware designs have come to exhibit many of the same properties as software. Publicly known circuits and architectures can be transferred in electronic description form at no cost. Consequently, non-proprietary hardware designs could be shared as widely as free software. The community would benefit from the free availability of stock hardware design solutions. Rather than replicating the work of describing standard hardware elements in electronic form and testing the results, designers could acquire the electronic description free, or at nominal costs, from publicly maintained libraries.

We envision the WWW/NII providing a framework for such a public hardware design library. Hardware designs at all levels from VLSI masks to high-level architectural descriptions are collected. Hardware designers pick parameterized cells from stock library solutions and specialize them to their needs. Such a library builds on the same technology used for the global cooperative sharing of software code. Automatic annotation maintenance allows distributed feedback on the utility of publicly available designs. Automated submissions and genealogy allow distributed maintenance of design alternatives. In a similar manner to quasistatic-code sharing, CAD tools notify designers of potentially superior alternatives to stock solutions as they are made publicly known.

The analogy with software is especially relevant when considering architectures suitable for reconfigurable logic ( e.g. (tn100)). The selection among alternative implementations of a specialized, configurable logic design is identical to selection among alternative implementations of a software subroutine. In fact, we can combine these technologies to make innovations in hardware architecture immediately accessible to applications running on configurable computing machines. We often accelerate programs by replacing a software subroutine with a specialized, configurable logic implementation. In this case, the quasistatic compiler evaluates the feasibility and merit of such a migration of functionality from fixed processing resources to configurable resources. In the short-term, this technique of providing configurable hardware implementation alternatives for key subroutines is likely to be quite popular. In a quasistatic, global cooperative computing scenario, such alternatives will allow any program using such routines to automatically adapt to exploit the newly provided alternative whenever suitable configurable hardware is available.

Current State-of-the-Art

The state-of-the-art in software and hardware design sharing is poor and ad hoc. Contributors make whole programs available via FTP. When potential consumers find the code, they can manually pick it up and install it. Such programs often implicitly depend on other programs and the consumer is burdened with understanding this relationship and making sure all the implicit dependencies are supported. Consequently, a huge cognitive burden is placed on potential consumers to properly install the software and prepare it for use. At the same time, complicated dependencies within the code make it impractical to reuse portions of contributed software without undertaking significant effort to understand the context required for the code to operate. Coordinating bugs and improvements is also a huge logistical problem. Consequently, maintaining and evolving shared code is an enormous undertaking. Despite the cognitive undertaking, many programmers do come to understand the shared code and can provide improvements. Nonetheless, most of their improvements are not available for public use due to the enormous human effort currently required to verify and coordinate the evolution of shared code. Global cooperative computing will automate much of the coordination and dependency tracking, lowering the transaction cost of using shared code while simultaneously lowering the effort required to evolve shared code.

Impact of Cooperative Sharing

Our research efforts to establish a global cooperative computing framework and Hardware Design Libraries will provide novel mechanisms for rapid technology transfer. Both efforts facilitate dissemination of new developments on drastically shorter time scales than current technologies. Further these technologies dramatically reduce the amount of effort required to effectively transfer new results both on the part of the developers and on the part of the consumers.

Rapid Dissemination and Incorporation

The technologies suggested here will allow new software technologies to be distributed and incorporated into existing code in days. Thus, end-users can benefit from technology developments far faster than possible with today's ad hoc techniques for technology transfer. In contrast, many software and compilation techniques take years to get from academia to industry. Even then, the techniques often end up only in large volume applications. Smaller or older applications may never benefit from the new technologies.

Quasistatic and global cooperative computing make possible this unprecedented, rapid deployment of new technology. Quasistatic computing allows new software technologies to be evaluated and applied to existing applications. This incorporation occurs without intervention by the end users, with the exception of routine safety checks. Global cooperative computing exploits the NII to distribute the new technology and provide a substrate for computational cooperation.

Reduced Effort: Contributors

The tools proposed to support cooperative computing take a large burden off of potential contributors, thereby encouraging more people to contribute, encouraging people to contribute often, and encouraging contribution at a finer granularity than is currently practical. Notably, the automatic collection of computer generated feedback information and the automatic maintenance of consumer supplied feedback information frees the contributor from manual collection and maintenance of this information. Further, the automatic support for cooperative submission of alternatives removes the burden of coordinating low-level improvements. Individuals who wish to augment a work need only link their contribution as an alternative to the original without involving the original author.

Reduced Effort: Consumers

Consumers can benefit from the new technology with almost no effort and often without needing to understand the nature of the development. For security reasons, one or more people at each consuming site will need to make a final inspection of any new software. Beyond this safety check, no human intervention is required to benefit from new technology. The combination of quasistatic feedback and explicit code links to designated repositories allow cooperative computing software to automatically locate results which are relevant to the user. Once identified, the quasistatic compiler can experiment with the new software routines or transforms. In this process the compiler will weed out developments which do not accelerate the end-user's application or do not function properly. Only when the new software is found to be beneficial and functional is any user even bothered for inspection. Incorporation of software proceeds automatically once accepted.

Routine Safety Checks

We are not proposing to automatically recognize software which may have undesirable side effects ( e.g. hidden viruses), in this formulation. Nonetheless, we expect any such side effects will be noticed by some human reviewer and quickly reported. Automatic maintenance of such human feedback information will quickly notify all other potential consumers to the dangers. All software will thus have a pedigree of adoption, use, and complaints. For many users, a pedigree showing widespread use and no complaints may be sufficient.

In some cases a consumer may employ a vendor or remote maintenance site to provide pre-approved code. The consumer may then give the system blanket authority to automatically incorporate any code or transforms provided by these select sites. Cryptographic authentication and checksums can serve to guarantee the uncorrupted and unforged delivery of code and its associated approval.

Increased Scope

This reduced effort, in turn, leads to a tremendous increase in the potential scope of adoption for each new development. Each program, or hardware design, provides explicit links to the home locations of its shared, constituent parts. This allows every program to identify relevant new developments as soon as they are made available. Further, since the recognition, trial, and incorporation of new developments is automatic, every program which benefits from a new development can be updated with minimal human effort. The aforementioned routine safety checks are the only point where humans may need to be involved in this process. This kind of sweeping update is possible with our global cooperative computing model and is not even conceivable with current computing practices.

Transit Note #105
Global Cooperative Computing

Abstract:

Cooperative Sharing

Hardware Design Library

Current State-of-the-Art

Impact of Cooperative Sharing

Rapid Dissemination and Incorporation

Reduced Effort: Contributors

Reduced Effort: Consumers

Routine Safety Checks

Increased Scope

See Also...

References

Transit Note #105 Global Cooperative Computing

Abstract:

Cooperative Sharing

Hardware Design Library

Current State-of-the-Art

Impact of Cooperative Sharing

Rapid Dissemination and Incorporation

Reduced Effort: Contributors

Reduced Effort: Consumers

Routine Safety Checks

Increased Scope

See Also...

References

Transit Note #105
Global Cooperative Computing