Multi-Site DIVIMP Development:
Some Organizational Suggestions
University of Toronto
Institute for Aerospace Studies
October 13th, 1998
The DIVIMP code is now being independently worked on at JET in the UK
(combined with the EDGE1D, EDGE2D and NIMBUS codes), Juelich in Germany
(KFA) (combined with B2 and EIRENE), Asdex Upgrade at Garching (Germany),
DIIID in San Diego (USA), Tokamak de Varennes in Quebec, CMOD at MIT in
Boston (USA) and at UTIAS in Toronto. The number of users and sites where
this code is used may continue to grow, and as this occurs, the logistical
problems in maintaining a base and reference version of the code will also
This document is intended as a draft, proposing methods and simple
procedures to make the development and maintenance of DIVIMP at multiple
sites by multiple persons both easier and more efficient. With three or
four numerical modeling codes which are becoming interoperable and whose
source code is being modified in an independent fashion at a number of
different sites, it would be valuable if consistency and compatibility
between the codes could be maintained. The following are suggestions
intended to reduce or eliminate difficulties that may arise in this
regard. Further recommendations and suggestions on how to make the
independent development and subsequent reconvergence of the resultant
DIVIMP versions as quick and painless as possible are happily welcomed.
- To suggest or develop procedures to try to ensure that the different
code versions maintain some level of consistency. There are two aspects
to this. First of all, keeping track of the different sites where the
code is in use so that when and if errors with the code need to be
corrected, it will be possible to contact the person locally responsible
and supply them with a fix. This will help to prevent two different
locations reporting differing conclusions based on similar input to
DIVIMP. A second aspect of consistency is code divergence. As the codes
develop independently, different sites will add functionality in
accordance with their priorities. These additions can be and have been
useful to other sites at a later time. The difficulty lies in
reconverging the differing code versions from separate sites. In order
to maintain an overall consistency of the DIVIMP code it would advisable
to develop some guidelines to make the reconvergence process as simple
and easy as possible.
- To try to ensure that DIVIMP can run and will continue to run on the
broadest array of hardware possible. This facilitates the transfer of
DIVIMP to many sites with differing hardware and software.
3) Software Engineering Issues
- To address some aspects of programming style, technique or practice
that will make the process of developing, maintaining and using DIVIMP
as straightforward and easy as possible. This means looking at
additional developmental procedures that have a minimum of additional
effort involved but which will substantially decrease the difficulties
of later developers or users of the code. In addition, with the growing
number of interfaces between DIVIMP and other codes, it may be desirable
to discuss a simple common interface for passing data between these
pairs of codes.
There are a number of strategies which can be employed to meet these
objectives. The single drawback is that the time required for a specific
code addition will be increased by a finite amount. However, in the long
run, the information recorded about any code addition will increase the
efficiency of later additions. Furthermore, such information will also
make the codes much easier to understand for new people using and
developing them. These advantages will overcome the cost of the additional
time required to maintain the code.
A. Designate Primary Site
- The primary site should have what could be called the most recent "official"
version of the code. This would make it the preferable source for new
users of the code. It would also be the source of primary documentation
for the code. The primary site for the DIVIMP code is currently Toronto.
This is simply a useful administrative tool since it provides a
reference for anyone searching for information about the code as well as
providing a useful base about which DIVIMP developmental efforts can be
B. Keep track of Users
- In order to keep track of the dispersion of the code and how many
variants may be available, it would be valuable to inform the primary
site when copies of the code are distributed to new users by any of the
C. Periodic Updates from the Primary Site
- If the primary site can incorporate the important changes made by a
local site then the primary site can redistribute updated versions
containing the results of continued development at the primary site, in
addition to developments from the other local sites. This would allow
all the users to benefit from additions at the various development
locations. In order for this to be feasible, there would need to be a
concerted effort on a periodic basis to reconverge all of the code
versions. This is necessary because a local site will likely find the
new version of the code unusable unless the majority of their local
changes have been incorporated. The only alternative to this is to
distribute updates in the form of small patches that can be applied to
the code version at the local site. For this alternative to be feasible,
however, all the sites would have to adhere to fairly strict guidelines
of code development, so that it would in fact be possible to extract the
changes in small parcels.
DIVIMP is now known to run on a variety of platforms. These include an
IBM 3090 mainframe, a Cray YMP and XMP, as well as IBM RS6000, Sun, and
SGI workstations, and even a test version on an IBM PC (Pentium 90). The
reason this is possible is because system dependent code has always been
as isolated as possible in the development process.
A. Bundling of Necessary Subroutines
- Source code for as many subroutines as possible should be included
with the main code. This point is aimed at such things as mathematical
subroutines, fitting routines, interpolation routines and others which
might be found in a "standard" library. It would be preferable
to include source code since this allows sites without access to the "standard"
library to utilize the code. Second, it can allow the algorithms used by
the routines to be modified by the local site if that becomes desirable.
If using library routines is unavoidable, then it should be clearly
stated that access to a specific standard or local library is necessary
for running the code and references to the specific routines required
and the functions they perform should be made in the documentation.
B. System Specific Routines
- The system dependent subroutines should be bundled into one source
code module so that references throughout the code do not need to be
changed when moving from one platform to another. Thus, for example,
instead of calling the system function TIME from many locations in the
code, one should reference a subroutine that in turn calls the system
specific function to return the time. The extra level of indirection
will not be a performance problem unless the call is located inside a
heavily utilized loop. In this case one should still implement it as a
call to another routine, if at all possible, and use procedure in-lining
optimizations in the compiler to make it efficient. If neither route is
usable then the developer should include the calls directly in the code,
preferably with suitable documentation.
C. Language Standardization
- It would be desirable to continue to code DIVIMP using language
constructs that are defined in the FORTRAN 77 standard. This should
ensure a minimum of difficulties in moving the code from one platform to
another since most platforms have a FORTRAN 77 compliant compiler. In
addition, all compilers certified for use by the US government must have
a flagging facility for identifying all constructs in a program that do
not conform to the FORTRAN 77 standard. This FIPS option can usually be
turned on by a compiler option and can be useful in ensuring the
portability of the code. If and when FORTRAN 90 standard compilers are
commonly available it may be worthwhile shifting the desired level of
conformance because FORTRAN 90 supports a richer and more diverse family
of language constructs.
A. Records of Changes
- Each site using the code could keep a version exactly as it was
received. Using any number of system utilities (this would vary
depending on the system involved), the current version of the code at
the local site could be compared to the original. All the differences
could be stored in a file and this file sent back to the primary site
along with any documentation. This would make re-converging codes that
had been developing independently much easier. Furthermore, it would
make the reasons for variations in results between differing versions
much easier to determine since the specific changes in the code would be
available for comparison. These records of changes could then either be
incorporated into the current master code version or archived for later
- The additional benefit of this is that the work of many sites could
then be incorporated into one cohesive whole.
- Place comments in the code giving a rough idea of what the code is
attempting to do and the algorithm it is using. It has been found that
even very simple one-line comments can be instrumental in making complex
sections of code more easily understood.
- Create external documentation for the subroutines that are added.
This can vary from photocopies of hand-written notes to word-processed
documents. The contents of the notes can be anything from a detailed
description and code listing to a paragraph describing what the routine
should do and how it should do it. Any information can be valuable.
- Any information on sections added to a code can be crucial in
deciding whether or not to add it to the other versions of the same
code. This situation has been dealt with previously when co-development
of LIM and DIVIMP was being done at both JET and UTIAS (University of
Toronto - Institute for Aerospace Studies). Code added at JET was later
integrated into the master code version at UTIAS.
- The designated primary site for the specific code should be
responsible for any serious errors in the base or "official"
code version. It should also be responsible for informing all users of
such errors and supplying either instruction or a code fragment to fix
the error. In addition, the primary site should be informed as soon as
possible about any serious errors found. Errors found this way will be
corrected in the master version so that later versions will lack the
specific problem. Serious errors would be defined as program code, logic
or data problems that result in the code producing what would be
considered either incorrect or inaccurate results, or modeling a process
in a way that seems unrealistic and at odds with current understanding.
Serious errors would not include program modifications required to
support new features being added at the local site, or problems
introduced by code additions at a local site.
D. Programming Practices
- In order to maintain compatibility with older versions of the DIVIMP
code and to be able to verify that the code is still behaving correctly
relative to results given when run at an earlier date, it is essential
to add new code as options. This strategy can be seen in the DIVIMP
input guide, where several of the inputs have a great number of options,
some of them dating back to the inception of the code. This approach
makes it much easier to examine the behaviour of the code as newer more
complex options are added. Furthermore, some of the options that may be
added will only work at one site or with a particular grid style or
configuration. Integrating changes of this type into the base version at
the development site, rather than including them through the use of an
input switch, may make the code unusable by another of the development
sites. This may make it impossible to include new and potentially useful
code additions into the "official" or master version of the
code. An example of such a change might be integrating the ability to
deal with non-orthogonal grids in such a way that depends on the
characteristics of one particular grid set, Asdex Upgrade for example.
The code would then rely on characteristics of the Asdex grid that the
others might not share, thus making the code effectively unusable by the
other sites. If, on the other hand, code of this type was added as an
option, then not only could the other sites benefit from the model code
solution for the specific grid type, they would also be able to use
Asdex grids in the new mode as well as whatever grids they would
normally use in the old mode.
E. Programming Techniques
- Variable naming and commenting. Whenever possible, the names of
variables, subroutines, functions, and common blocks added to the code
should be descriptive of the function they are performing. In addition,
if it is convenient, the local site could consider using a naming
convention so that most variables and functions added by them could be
readily identified. One possibility is starting all variable names added
at the site with a letter or letter combination representative of that
site. Also, blocks of code added by an individual or a site could be
identified by surrounding comments. e.g. Inserting a comment line "cldh"
before and after an added code segment indicates that it was added by
Lorne D.Horton at JET. A similar strategy could be adopted elsewhere.
- Use of subroutines and functions. If it is reasonable, the developer
should always add new code in a subroutine or function as opposed to
adding a branch statement within the code and inserting it in-line. The
reason for this is encapsulation of the new code will make the whole
easier to maintain and will make the new code much easier to integrate
into the base version.
- Use structured programming practices wherever possible, avoid the
use of GOTO statements or others that interrupt the expected flow of the
code, this will facilitate the understanding of one of the other
developers looking at code added potentially several years previously by
some-one no longer associated with the project.
- Where the DIVIMP code needs to interfaced with another numerical
code, it is a good idea to discuss with the developer/owner of that
other code what would be a good and convenient style and type of
interface between the two codes for the type of data to be transferred.
Information about this interface specification should be transferred to
the primary site so that it would be available to other developers
working with, or considering working with, the same ancillary codes.
These are just a few suggestions, any others would be appreciated. This
document is intended to encourage organized cooperation without unduly
interfering in any group's DIVIMP development activities. The ideas
presented are aimed at making the development of this modeling code,
especially when used in conjunction with other codes as consistent and
unified as possible. If one considers that the current generation of
Tokamaks will be operating for at least another two years and that ITER
modeling may well be required for another 10, 15 or 20 years then keeping
these codes in a form that can be made into a useful tool as our knowledge
base increases is of paramount importance. This can only be done
successfully if the people who work on the codes tomorrow understand what
is being done today. This can be best accomplished by producing
well-written modeling routines with in-line documentation that describes
the processes and procedures utilized in the model and by structuring
development in such a way that little is lost or wasted. The suggestions
contained here are aimed at this goal.