Vector Length

Most computations in Radioss, like element or contact forces, are performed by packets of MVSIZ. This parameter is adjustable to match so-called vector length. This parameter is also important for cache locality. It is optimized by parallel development team according to hardware characteristics

New treatments need to respect this programming model which is to split the loop over number of elements or nodes by packets of MVSIZ. This will ensure optimal vector length and cache size as well as minimal local storage (local arrays of size MVSIZ instead of number of elements or nodes)

Loop Control

Loop Size

Inside a loop it is recommended to keep the number of instructions reasonable. 20 instructions or less is good. Very long loops should be split to keep cache efficiency

Most compilers will be able to fuse short loops, while they will probably fail to vectorize long complex loops

Data Dependency

The loop below is not vectorized due to possible dependence (same value of INDEX(I) for different I): 

DO I = 1, N
         K = INDEX(I)
         A(K) = B(K)
         B(K) = 2*A(K)
END DO

In case of no true dependence, vectorization needs to be forced by adding a compiler directive

To keep portability across different platforms and compilers, an architecture specific include file exists named vectorize.inc that manages vectorization directives. The programmer just needs to add this include file just before the DO loop:

#include "vectorize.inc"
       DO I = 1, N
            K = INDEX(I)
            A(K) = B(K)
            B(K) = 2*A(K)
       END DO

Notice there is another include file named simd.inc which makes unconditional vectorization, even if a true dependence is detected by the compiler. It is recommended to only use vectorise.inc which is more conservative regarding correctness

For Intel compiler:

• vectorize.inc corresponds to !DIR$ IVDEP

• simd.inc corresponds to !DIR$ SIMD

Procedure CALL

Calling a procedure inside a loop inhibits vectorization therefore it is not authorized

Inside Radioss, there are vectorized versions of procedures, basically the loop is put inside the procedure rather than outside:

• VALPVEC to replace JACOBIEW

• VINTER to replace FINTER

Nested Loops

In practice, only the inner most loop will be vectorized. So the inner most loop needs to be the largest one.

For fixed size loops it is possible to unroll them by hand or to use Fortran90 enhancement. Then the compiler is able to vectorize the outer loop

Example Nested Loop with NEL >> DIM

DO I = 1, NEL
    DO J=1, DIM
      A(I,J) = B(I,J) + C(I,J)
    END DO
END DO 

To be transformed to:

DO J=1, DIM
    DO I = 1, NEL
     A(I,J) = B(I,J) + C(I,J)
    END DO
END DO 

Or using Fortran90 notation:

DO I=1,NEL
    A(I,:DIM)=B(I,:DIM) + C(I,:DIM)
END DO

Or for this simple case:

A(:NEL,:DIM)=B(:NEL,:DIM) + C(:NEL,:DIM)

Power

Never use real variable for integer power because of the cost of real power arithmetic. Take

care to not use real variable defined in constant.inc when integer is enough

Div

For invariant, it is advised to multiply by invert instead of doing a division by a constant inside a loop

Fortran90 Array Operations

Use of Fortran90 array operations is encouraged as long as code readability is kept, by always specifying array bounds to avoid confusion between variable and array arithmetic.

 Example:

INTEGER, DIMENSION(NUMNOD) :: A, B, C
A = B + C

    ! confusion between variable and array operation

A(:NUMNOD) = B(:NUMNOD) + C(:NUMNOD)

   ! default lower bound:1 

Multidimensional Arrays

Data Locality

Large arrays over a number of nodes or elements are defined to maximize data locality and have therefore the smallest dimension first, like in the example below:

X(3,NUMNOD), V(3,NUMNOD), A(3,NUMNOD)

Leading Dimension for Vectorization

For vectorization on Xeon, it is better to have leading dimension first. So, depending on array size and access pattern a compromise needs to be found:

• For large arrays like X, V, A, it is better to keep locality

• For data structure of few times MVSIZ, like new element buffer arrays or temporary arrays of size x times MVSIZ, having largest dimension first is better:    

According to test with recent Intel compiler, Fortran90 array notation can also improve code generated:

Structure Of Arrays

Use structure of arrays (left example) rather than arrays of structure (right example)

TYPE GRID_STRUCT_
 my_real ::  x(100),y(100),z(100)
END TYPE GRID_STRUCT_

TYPE POINT_STRUCT_
  my_real ::  x, y, z
END TYPE POINT_STRUCT_

TYPE(GRID_STRUCT_) MESH;
! a grid of 3 arrays x, y, z of size 100

TYPE(POINT_STRUCT_) P(100);
! an array of size 100 of 3 scalars x, y, z

Warning On Large Array Initialization

Especially inside Radioss Starter it is common to find code using large array flag over NUMNOD initialized to zero many times which is time consuming

Here is a method to avoid such an issue: left is original poorly performing code, right is optimized version

DO I=1, IX_TIMES
!IX_TIMES set to 0 
         TAG(1:NUMNOD)=0
         DO J=1,FEW_ITEMS
           N = INDEX(I,J)
           IF(TAG(N) == 0)THEN
! but few values change
             TAG(N)=1
! additional treatments…  
           END IF   
         END DO
       END DO

! single init to 0
TAG(1:NUMNOD)=0
DO I=1, IX_TIMES
  DO J=1,FEW_ITEMS
    N = INDEX(I,J)
    IF(TAG(N) == 0)THEN
      TAG(N)=1
! additional treatments…
    END IF
  END DO
! set to 0 only if needed
  DO J=1 , FEW_ITEMS
    N = INDEX(I,J)
    TAG(N) = 0
  END DO
END DO

Additional Fortran90 Restrictions Regarding Efficiency

Operator Overloading

Usage of object oriented feature like operator overloading, while it is efficient in code writing and clarity is not recommended due to lack of performance efficiency