Ref: 052202
Oxford University is a highly devolved institution, with nearly 100 departments, 40+ colleges and many other academic and administrative units. Its form of governance allows considerable autonomy to these various entities (indeed, the colleges are separate legal and financial organisations). Furthermore, the University is housed in numerous buildings distributed throughout the city.
This environment has always presented special challenges to the orderly and effective development, acceptance and implementation of IT strategy. For instance, the University has invested in a substantial FDDI backbone network which now reaches all these units, connecting over 200 Ethernets, and with about 15,000 computers attached. Computing facilities are thus highly distributed, and also highly diverse. All major brands of Unix workstation are represented, and personal computers include DOS, Windows3.1, Windows95, WindowsNT, OS/2 and MacOS. As befits such an environment, IT support is also well-distributed. Central support, meanwhile, is responsible for supporting core services like the network and providing backup to the distributed support staff.
The IT Strategy of the University (accessible at http://info.ox.ac.uk/it/strategy) seeks to ensure that this diversity works harmoniously, that there is central provision of services of strategic importance and where it makes sound economic sense to do so, and that sensible standards are promoted throughout the University. The result at Oxford is a rich computing environment, but which avoids the excesses of anarchy and duplication.
2. The Problem of Electronic Assets
Against the setting described above, the University, in common with all others, is faced with a staggering increase in the quantity and variety of electronic information. Some of this is ephemeral, so its storage, management and preservation is of no great concern. A very substantial proportion of it is, however, of immense value, not just to its creator/owner, nor just to those local or international colleagues involved in related work, but often to posterity.
The problem of making important university information accessible to others, in time and space, has been solved in the past by a combination of formal publications (meeting minutes, research reports, proceedings, journals, books, etc), and formal storage arrangements (departmental and central filing systems, libraries, archives, etc). This system itself is straining under the load of the exploding amount of information being produced and published, especially in the scientific world.
To this we are now adding, at an enormous rate, information in electronic form. In many cases, this is duplicating the conventional printed or visual form, and in others it is substituting for it. In other cases again, however, it is information not only in a new form, but sometimes of a totally new kind.
Not only do we have few paradigms for managing this electronic information (since we are able to carry so few across from our experience with conventional media - which themselves took centuries to develop), but we have been slow to recognise the existence, importance and vulnerability of some of this information.
There are several dimensions to the storage of this information. There is its durability; its appeal (how widespread is the interest in it?); its size; its vulnerability; its popularity (the frequency with which it is accessed); and its rarity.
These dimensions create a vast array of situations and policies which must be implemented, including ensuring an effective backup regime is operating; preservation of electronic information beyond the current media and our lifetimes; building suitable metadata files; and dealing with the vexing question of formats which will survive current software and hardware platforms.
Among the many types of electronic data we need to safeguard are scientific research results which are often extremely expensive, sometimes impossible, to recreate; electronic images of rare and sensitive documents; even the Web pages we are creating, into which much intellectual effort is being poured.
3. Elements of the Solution
The solution to at least some of these problems at Oxford was perceived to be a very large central file store to which all users had access across the network. Issues of economics dictated that this must be a hierarchical store, ie one with a hierarchy of media. Given the environment at Oxford, it was also clear that this must be a client-server system, and capable of supporting thousands of clients, from a wide range of vendors. It also had to provide a range of functions, including backup, archive, ftp storage, and file migration.
Designing an effective backup regime is not as simple as it sounds. We define backup to be the process of making a duplicate copy of a file for storage in a different place, so that in the event of damage to the original, it is possible to revert to the backup copy. One needs to recognise that some files undergo constant revision, so it may be necessary to keep copies of a multiplicity of revisions. It is not cost-effective to keep copies of all revisions for ever, and some arrangement must be made to delete backup copies when the original really is no longer required. The system therefore needs to have the capability of setting appropriate parameters to cope with different regimens (eg numbers of versions to keep, amount of time to keep them after the original dies, etc).
An archived file is distinguished from a backup by the fact that its retention is totally independent of the original. Indeed, the operation of archiving is more like a "move" than a copy, and the whole point of making an archived copy is that it should be retained indefinitely (ie regardless of the fate of the original).
File migration is the process whereby local file stores "overflow" onto a much larger file store. Migrated files would not stay on the local system, but would remain in the local directory so that they could still be found. Ideally, migration should occur only when space on the local system reaches a critical threshold, and normally the least-used and/or largest files should be migrated. Recovery should be transparent (apart from a tolerable delay). Once again, it must be possible to implement a variety of policies through options and parameter settings.
Ftp storage is a manually-operated case of file migration, in which the process is handled by direct action of the user. The lack of automation gives greater control over file location, but does require extra effort. It also requires direct user access to the server.
A further option which can be considered is NFS-mounting of files. We decided not to seek this capability in its present form, since without (say) a Kerberos environment it could be quite insecure, and file space management at the central store could be difficult, and unless due care is taken use of files held in this way could put a heavy load on the network.
Other very important factors required of the system are performance, reliability, availability and data integrity. The server needs to have adequate performance to cope with substantial numbers of clients, to transfer data to and from the server without delay, and to locate and deliver files from its hierarchy of media in a rapid fashion. Reliability of course is an important requirement, as is availability, in such a populous and diverse environment. Data integrity is, of course, the most vital requirement of all, with provision for ensuring data is free from attack, is moved and stored without error, and safe from a catastrophic failure of the whole system.
It goes without saying that capacity, ease of use, expansibility, conformance with standards, etc were also important considerations.
Funds were allocated in early 1995, and a 2-stage tendering process initiated. The first stage used a Request for Information as a means of testing the feasibility (in functional and in economic terms) of the project, and of informing vendors of our needs, and in turn of shaping the requirements according to what was reasonable. Formal tenders were then called, in line with European Procurement Guidelines.
4. The System in Practice
A system from IBM was chosen, and delivered during 1995. Itfirst went into productive use in mid-1995. It is based on dual RS/6000 computers, and employs an integrated suite of software, known as Adstar Data Storage Management (ADSM), and has a large automated tape silo. One unit acts as the storage manager, providing backup, archive and ftp services direct to clients. The other acts as a file server, migrating files between media as needed.
Overall, we have been very pleased with the system, which has now been in production for two years, especially its functionality. It has been especially gratifying how well the system has appeared and functioned to end users on a wide range of platforms. We have to say, however, that the solution did not work out as we had anticipated, in a number of ways.
Our first priority had been to provide some form of "unlimited disk space" service. In ADSM this would be implemented through file migration; however, we found that the migration software was not available at the time of installation, and in any case implementing such a scheme would have presented us with a number of difficult policy decisions. Archiving, like migration, also needed careful consideration, preferably based upon some non-operational experience. On the other hand, the backup facilities of ADSM very attractive - they were fully-integrated into the system, and clients were available for all major platforms in use at Oxford. The ftp service was also perfectly satisfactory. Accordingly, we decided to introduce the backup service and the ftp service first.
Secondly, we quickly found that the load placed on the system by so many backup clients was so great that the system could barely cope. Typically, personal systems are backed up weekly, to a set schedule, and on an incremental basis, while departmental servers are backed up more frequently. Traffic volumes now exceed 250 GBytes/week. Only constant attention, administrative effort, pressure upon IBM, and additional resources has enabled the system to keep pace with this demand.
Thirdly, we had expected a turn-key solution, which would function with very little attention, apart from some ongoing tuning and once options had been selected and initial parameters set. However, we have also found, possibly because we have been pushing the technology to its limits, that a high level of system errors has been encountered (most in software, but we have also encountered a surprising number of tape drive faults). This has put considerable pressure on the 2.5 staff responsible for caring for the HFS, and has meant delays in rolling out new services because of the preoccupation at times with keeping it operational.
The fewest difficulties have been encountered with the ftp service, which has consistently provided satisfactory service, though the relatively small number of users has meant that we could anticipate and control demand to some extent, and adjust accordingly.
We are now able to offer an archive service. At this stage we have only accommodated projects of importance to the whole University. And we are now also gradually rolling out the migration service.
We have been very conscious of the limitations of previous archiving arrangements in use at Oxford (and probably at most universities), whereby users could offload files from their local systems with no limit on time and space. Our current "archive" holdings are therefore likely to be very largely of zero value, but we must faithfully preserve them. Accordingly, we have redefined the term "archive" to reflect more accurately its use by archivists, and established a university committee to develop suitable policies about who can archive material, how much they can archive, and for how long (see http://info.ox.ac.uk/oucs/services/archiving/archive-policy.html). For people who wish to store material which has been deemed to be not of general interest, we will charge for the privilege of retaining it in perpetuity at the rate of about £20/GByte pa (covering regeneration of the data and replacement of the system when required), and we also provide a CD-ROM writing service.
5. Future Developments and Requirements
We are hopeful that both the level of system errors and performance deficiencies will be eliminated before long, and some progress is being made on that front. Regardless of the impact of these expected performance improvements, it is clear that we will not be able to roll out the backup service to all 15,000 clients using only the existing system. In reality, this was never likely to have been possible. Instead, ADSM is intended to support a hierarchy of collaborating servers. "Regional" or departmental ADSM servers will provide first-line file services, which will then pass on their data, in consolidated fashion, to the main system.
Arrangements have just been put into place to make a third copy of the data held in the system which will be stored in fire-proof safes at a location some 10 miles away. A reciprocal agreement is also in place (informal at this stage) to have access to a similar system installed at De Montfort University, should any disaster result in the whole system becoming inoperable.
Of course, the current system will only have a limited lifetime, but we expect to migrate to future developments of the system as they emerge and as the existing system needs replacing or upgrading (whether from the same supplier or another), and the University is expected formally to commit to that shortly. The ramifications of being able to make meaningful use of this data in the very long term which are being considered by the Computer Archive Group, including metadata, data format and hardware and software obsolescence. Clearly, current standards, like SGML and JPEG, should be employed wherever possible, and a regimen for updating the data held in these formats when they are upgraded will need to be developed.
We have made little progress on implementing a system to provide metadata, and have been disappointed at the lack of offerings from suppliers in this area. However, we are aware of national and international efforts to explore this area, which are being pursued in connection with major digitisation projects being conducted within the UK as part of the eLib project, as well in the USA, Australia and elsewhere. Indeed, we are collaborating with the University Library and external partners in several small projects in this area.
We are also exploring further implications and implementation strategies of additional "migration" services, believing that this is a service which may well only be offered to departmental servers.
In the longer term, we envisage (rather, we hope!) that an environment will gradually emerge in which there would be a uniform view of files right across the University, with a range of storage and access options, and which will support all the functions and more that we are seeking in this present system. Perhaps DFS/DCE will bring this, and we await such developments with interest.
Copyright EUNIS 1997 Y.E.