13.5 Processing large files

When processing large files it is not always convenient to run in parallel but process each parallel partition in serial, for example when interpolating one field to another. To do this we can use the –nprocs and –procid options. For example the following option will interpolate partition 2 of a decomposition into 10 partitions of fiile2.xml from file1.fld

FieldConvert --nprocs 10 --procid 2 \ 
        -m interpfield:fromxml=file1.xml:fromfld=file1.fld \ 
        file2.xml file2.fld

This call will only provide part of the overall partition and so to create the full interpolated field you need to call a loop of such commands. For example in a bash shell you can run

for n in ‘seq 0 1 9‘ 
    FieldConvert --nprocs 10 --procid $n \ 
            -m interpfield:fromxml=file1.xml:fromfld=file1.fld \ 
            file2.xml file2.fld 

This will create a directory called file2.fld and put the different parallel partitions into files with names P0000000.fld, P0000001.fld, ?., P0000009.fld. This is nearly a complete parallel field file but the Info.xml file which contains the information about which elements are in each partitioned file P000000X.fld. So to generate this Info.xml file you need to call

FieldConvert --nprocs 10 file2.xml file2.fld/Info.xml:info

Note the final :info extension on the last argument is necessary to tell FieldConvert that you wish to generate an info file but the extension to this file is .xml. This syntax allows the routine not to get confused with the input/output xml files.