SMT Backup Scheme for Data Files and System Disks
	    Bill Peters				    18 Dec 96



A) Description

    Separate command procedures for backing up data files and system disks have
been developed.  The tapes are given special names (labels) to keep them
separate and in sequence.  Data tapes have labels 

	    nDATA	n=1-9
	    nnDATA	nn=10-99
and	    nnnDAT	nnn=100-999

Disk backups from Mt Graham computers are stored on tapes labeled

	SMTn		n = 1-9
	SMTnn		nn = 10-99

and Tucson (Gaffel) disks are put on tapes labeled

	TUCn		n = 1-9
	TUCnn		nn = 10-99

Log books at the telescope and in Tucson list the disk backup tapes written at 
the corresponding site.  (The raw data tapes are NOT logged in this book.)

    Each tape may have several BACKUP savesets stored on it.  Since it is
inconvenient to deal with savesets that span more than one tape, the intention
is to estimate how much more a tape will hold and start a new one when the
files to be written (probably) won't fit.  (The actual capacity of a tape
depends upon the effectiveness of the compression algorithm on the data written
on it.  Should the scheme fail, BACKUP will encounter the end of tape and ask
for a second one.  One should then abort the procedure and begin again using an
empty tape.  If needed, the partial saveset written on the first tape can be
manually erased.) 

    To indicate what is written on the tapes and to aid the estimate of tape
space remaining, a "directory" file is maintained for each tape in [DAT_TAPES]
on the Kronen system disk, KDISK, (for Mt Graham tapes) and on Gaffel's disk,
GSYS, (for Tucson tapes).  The "directory" file simply lists the names of the
savesets stored on the tape, the size of each (in tape blocks, usually 16
MBytes), and the date it was written. Directory files are named
<tape_name>_DIR.LIS. 

    Savesets for data files are named according to the dates spanned by the
data. (for example, 21APR_9MAY95.BCK).  The command procedure adds the scan
number range contained in the saveset to its line in the "directory" listing.
Savesets for disks are /IMAGE savesets.  I.e. they are suitable for completely
restoring the disk from the tape (as well as recovering specific files).  These
savesets are named <disk_name>n.BCK where n is either null or a number chosen
to make the saveset name unique for that tape.  (The system disks are named
GAFFEL, KRONEN, and FRANKENHEIM.)  A listing of all the files in each disk
saveset on a tape is stored in [DAT_TAPES]<tape_name>.LIS.  (No such listing is
made for the data file tapes.) 


B) Backing up raw data

    The raw data should be archived after about 500 MBytes (1,000,000 blocks)
have been accumulated.  This allows one to save 4-5 BACKUP savesets per tape
and will not result in hours of wasted time when a data set won't fit on the
tape.  (And this way it is faster to retrieve files from the end of the tape 
since one can just skip to the desired saveset without having to read all the 
files from the beginning of the tape.)

    To backup raw data, login as SMTSYS and set default to the directory
containing the data (usually TEL$NEW_DATA which is the same as
KRAW:[SMT.DATA]).  (Actually, SMTSYS is not required if one uses another
account with privileges to write in the KRAW:[SMT] directory tree.)  A backup
of TEL$NEW_DATA should only be started while no observations are being taken.
Once the RENAME (below) is done, it is safe to start a new observation.  To
begin the procedure, type 

	    SAVE_DATA [max_MBytes]

where max_MBytes is the maximum number of MBytes that will fit on the tape to
be written (default = 2700).  The procedure will then type further instructions.
One of the first things it does is to check if the current directory is 
TEL$NEW_DATA (i.e. KRAW:[SMT.DATA]).  If so, it will select a new name and 
prompt you to accept that name or type a new one.  The default directory names 
are KRAW:[SMT.OLD_DATAn] where n is the next number in the sequence.  After
the rename is done, an empty KRAW:[SMT.DATA] directory is created.

    The procedure expects to write the data on the end of the most recent SMT
data tape (determined by examining the directory listings in
KDISK:[DAT_TAPES]).  But one may also write on an arbitrary tape.  To force a
write on the next tape in the sequence, specify 0 for max_MBytes.  To write on
an arbitrary tape, insert that tape when requested and type OVERWRITE when
BACKUP complains that it is the wrong tape.  (OVERWRITE is poor wording.  It 
will actually append the new data to the end of the tape.)  After the tape is
written, the tape's saveset directory listing in [DAT_TAPES] is updated: 
<tape name>_DIR.LIS.  If the data may not fit on the current data tape, a new
tape can be begun.  If a new tape is required, the procedure will initialize it
after some confirming questions.  (This erases all data previously written on
the tape!) 

    The DAT tapes are written in "compressed mode".  But since the data
compression done by the tape drive varies in effectiveness, it's hard to tell
from the amount of data already on the tape, how much more data will fit. 
About the only way to determine this is to write data until the tape is full. 

    If there is not enough room on the DAT tape to fit the data, BACKUP will
eventually prompt for "volume 2".  As mentioned above, rather than do this, 
given that the data will surely fit on one tape, it is preferable to abort the
procedure and start it over with a new tape.  Otherwise, one would have to use
two tapes when one wanted to access the archived files.  

    Another option is to move some of the data files back to TEL$NEW_DATA until
the remaining files would just fit on the current tape.  When the current DAT
tape is less than 80% full (< 1900 MBytes), this is the usual choice. The
SAVE_DATA procedure will do this automatically if it is interrupted by a CTRL-C
when BACKUP is asking for the "Volume 2".  SAVE_DATA will then try move the
data that didn't fit, back to the (newly created, empty) TEL$NEW_DATA directory,
but it first needs some guidance.  Since it can't know at what scan the
tape filled, it then will first ask you to type the scan number of the last
scan completely written (which is also the second-to-last one successfully
compared).  After moving the files, it will erase the Save Set that overflowed
the tape so that it can be replaced with a smaller one.  Finally, it will start
a new SAVE_DATA on the remaining files.  When this is complete, one might also
check the amount of data now stored in TEL$NEW_DATA.  If a lot of files were
moved back to it, it may be necessary to run SAVE_DATA again soon.

    Note that the BACKUP program first writes the files then reads back what it
has written to compare it with the original.  When it finds that it is going to
need more than one tape, it first compares what it has written on the filled
tape before continuing with the next one.  It types the name of each file after
it is compared.  When the tape is full, the last file name typed will likely be
one that is only partially on the tape. Therefore, the script asks you for the
second-to-last scan that BACKUP compared.  That one ought to be last complete
scan that will fit on the current tape.  If you happen to notice when BACKUP
first starts comparing files, you'll then know what was the last scan
completely written.  If you see that it was not really the last scan, you can
issue the CTRL-C immediately.  It is really a waste of time to be comparing
these files since they are going to be erased and re-written, anyway. 

    Alternatively, if one decides not to try to try to fill the current tape,
but start over with a new one, he should use the following procedure:  Give a
second CTRL-C when SAVE_DATA asks for the last scan number written.  This will
terminate the SAVE_DATA script.  Since the original directory will have already
be renamed and the SAVE_DATA will have that already set that directory as the
default directory, one can simply type 

	    SAVE_DATA 0

to force a write on the next tape in the sequence.
    
    Eventually, the raw data should be erased from the disk.  This may take 
a long time, so the best approach is to submit a batch job to do this.  (DO
NOT do this while SAVE_DATA is writing a tape, as the RENAME eventually done
by the batch job script will cause it to crash.)  From the SMTSYS (or
privileged) account, type: 

	    DELETE_RAW_DATA <directory name(s)>

where <directory name(s)> is a comma separated list of directories w/o square
brackets or blanks.  For example, DELETE_RAW_DATA OLD_DATA3,OLD_DATA4 will
submit a batch job that will delete the files in KRAW:[SMT.OLD_DATA3] and
KRAW:[SMT.OLD_DATA4].  Before submitting the batch job, the procedure will
type the name of the first file and creation date in each directory to be
deleted.  This is to give one a chance to verify that the directory name and
scan number is appropriate for the deletion.  After the files are deleted, the
directory itself will also be deleted.  Finally, the batch job will rename the
remaining OLD_DATAn directories if there are now gaps in the numbering.  (It
uses @SYS$SMT:[SMT]SHUFFLE_DATA_DIR to do this.) 

C) Disk Backups:

    To backup an entire disk, that disk cannot have any files open on it.  This 
implies that it not currently be the boot disk for any CPU.  (If it is, 
shutdown that CPU and re-boot it from its alternate disk:  Franke -> DKA100,
Kronen -> DUA2, Gaffel -> DKA100.)  Next, login to a privileged account and do

	    @SYS$SMT:[SMT]SYS_BACKUP <disk_device> [max_MBytes]

where <disk_device> is the disk to be backed up.  (FDISK is the Frankenheim
system disk, KDISK is the Kronen system disk, GSYS is the Gaffel system disk,
FDATA is the Frankenheim user disk, KDATA is the Kronen user disk, GDISK is the
Gaffel user disk.  The procedure will then give further instructions.  It will
first re-mount the disk as a private disk so that no changes can be made during
the backup procedure.  Normally, the procedure will write the data on the end
of the most recent system backup tape, based on the directory listings in
KDISK:[DAT_TAPES] on Kronen and GSYS:[DAT_TAPES] on Gaffel.  But one may also
write on an arbitrary tape.  The optional second parameter is the maximum
number of megabytes that the tape can hold (default 4000).  To force a write on
the next tape in the sequence, specify 0.  (To write on an arbitrary tape,
insert that tape when requested and type OVERWRITE when BACKUP complains that
it is the wrong tape.)  If the data may not fit on the current data tape, a new
tape can be begun. 

    After the tape is written, the disk will again be mounted as a public disk,
and a saveset listing of the files written will be added to <tape_name>.LIS in
the [DAT_TAPES] directory.  Also, the list of the saveset names on the tape
will be updated in [DAT_TAPES]<tape_name>_DIR.LIS.  If a new tape is required,
the procedure will initialize it after some confirming questions.  (This erases
all data previously written on the tape!)