Configure Fusion performance¶

You can tune the performance of Fusion by setting up task rules and by modifying the sizes of the asset caches.

Configure task rules

You can use task rules to change how Fusion builds data. As soon as you implement a task rule, it customizes how specific processes in Fusion operate. In other words, you cannot apply task rules to individual assets or projects.

• Set the number of CPUs assigned to Fusion processing
• Task rule types
• Implement task rules

Set the number of CPUs assigned to Fusion processing

As some of the task rules described here rely on setting the number of CPUs assigned to Fusion processing, it is useful for you to know how many CPUs are available on your machine and how many are assigned for Fusion processing. The getop output lists the available CPUs on your machine as the maxjobs entry and corresponds to how many concurrent jobs Fusion may spawn at any one time.

You can use the tool geselectassetroot --assetroot path --numcpus to assign more CPUs. For example:

geselectassetroot --assetroot /gevol/assets --numcpus 7

For information about assigning CPUs to Fusion processing, see Running Fusion on a Machine with Multiple CPUs.

Task rules impact Fusion processing in two different ways: by enabling/disabling multithreading, and by specifying output locations (KRP/KRMP), as shown in the following table.

Task rule types

You can add one or more of these task rules to your asset root’s .config directory.

Implement task rules

To implement a task rule:

1. Make sure you have superuser privileges, i.e., root or sudo access.

2. Create the task rule using the XML sample code shown below for each rule.

3. Depending on the rule you want to use, name the task rule:

   • ImageryPacketLevel.taskrule
   • TerrainPacketLevel.taskrule
   • CombinedTerrain.taskrule
   • MapLayerLevel.taskrule
   • KRP.taskrule
   • KRMP.taskrule

   Note

   Fusion uses these names when the tasks run, so do not change them.

4. For the KRP and KRMP task rules, define the volumes you want Fusion to write files to. See Defining volumes.

5. Set file permissions to read-only for all (chmod 444).

6. Set file ownership to the gefusionuser user account and gegroup group (chown gefusionuser:gegroup).

7. Place the task rule files in the asset root’s .config directory.

8. Restart the system manager using the /etc/init.d/gefusion ``stop`` and ``start`` commands.

ImageryPacketLevel task rule

During a project build, Fusion will have multiple concurrent imagery project tasks. The watchpoint is balancing the total number of CPUs assigned for Fusion, and the number of CPUs assigned for each imagery project process. For example, with eight CPUs assigned for Fusion, with two CPUs per imagery project task, you can have four concurrent imagery project tasks.

Sample: ImageryPacketLevel.taskrule

TerrainPacketLevel task rule

During a project build, Fusion will have multiple concurrent terrain project tasks. The watchpoint is balancing the total number of CPUs assigned for Fusion and the number of CPUs assigned for each imagery project process. For example, with eight CPUs assigned for Fusion, with two CPUs per terrain project task, you can have four concurrent terrain project tasks.

By default, Fusion assigns one CPU to work on each terrain project task. You can use this task rule to assign up to six CPUs instead. Set the minimum CPU value to a number from 1 to 6, and set the maximum to a number between 2 and 6.

Sample: TerrainPacketLevel.taskrule

CombinedTerrain task rule

Since gecombineterrain is a serial process and must be completed before the other database-level building activities can start, you can use the CombinedTerrain task rule to assign all available CPUs to the build. For example, with eight CPUs assigned for Fusion, assuming that one CPU is reserved for the system, you can assign seven CPUs to the CombinedTerrain serial process.

Sample: CombinedTerrain.taskrule

MapLayerLevel task rule

By default, Fusion assigns one CPU to work on map tile generation. As maptilegen can be run concurrently, you can use this task rule to assign multiple CPUs instead. Set the rule to min/max 4 on a 2 dual-core CPU system (3 may work well enough). On a 2 quad-core CPU system, set the rule to min/max 3.

Sample: MapLayerLevel.taskrule

KRP task rule

When Fusion creates an asset, it writes large imagery files to the raster.kip directory in asset root by default. You can manage your disk space by using the KRP task rule to write these files to a different volume. See Example of File Locations.

In the sample below, the volume you have created is products. To learn how to create a new volume, see Defining volumes.

Sample: KRP.taskrule

You can also enter a second requiredvolume tag in your task rule to specify the next available volume in case the first volume fills up. However, you must to restart the system manager to trigger Fusion to wri.. code-block:: xml.. code-block:: xmlte files to the next volume.

KRMP task rule

When Fusion creates an asset, it writes the mask files to the mask.kmp directory in asset root by default. You can manage your disk space by using the KRMP task rule to write these files to a different volume. See Example of File Locations. In the sample below, the volume you have created is products. To learn how to create a new volume, see Defining Volumes.

Sample: KRMP.taskrule

You can also enter a second requiredvolume tag in your task rule to specify the next available volume in case the first volume fills up. However, you must restart the system manager to trigger Fusion to write files to the next volume.

Defining Volumes

The KRP and KRMP task rules include a requiredvolume tag that indicates where to write files. You need to make sure that this volume is defined and has enough space to hold the files. To define the volume, use the geconfigureassetroot -- editvolumes command, then edit the XML as shown in the sample below. In this sample, the volume you are creating is called products.

Sample: volumes.xml

Example of File Locations

The table below shows an example of the default file locations and the locations after applying the KMP and KRMP task rules. In this example, the asset root is /gevol/assets, the volume you have created is products, and you have built an asset called TestImage in the imagery subdirectory.

Cache configuration

gesystemmanager caches information about the assets and asset versions it is working with. The default sizes for these caches should work fine for most builds; however, increasing the cache sizes may speed up very large builds at the cost of increased memory usage. The cache sizes are controlled by the file misc.xml which can be found at /gevol/assets/.config/misc.xml (assuming your asset root is /gevol/assets; if not, modify the path accordingly). The asset cache size is controlled by AssetCacheSize and the asset version cache size is controlled by VersionCacheSize. Note that the asset version cache is generally larger than the asset cache because it may contain multiple versions of the same asset.

Within misc.xml, the following options are deprecated and are treated as if they are permanently enabled: DisablePacketLevelVersionCachePurge and ConsolidateListenerNotifications.

Also included within misc.xml is an experimental option that allows the user to limit the caches by memory usage instead of number of objects. This option is controlled by LimitMemoryUtilization and its default value is 0 (false). The maximum memory usage for each cache can be set with MaxAssetCacheMemorySize and MaxVersionCacheMemorySize. This size is measured in bytes and the default setting is 1GB. While this option is enabled, the caches will not be limited by the number of objects in the cache. It should be noted that as of now the reported memory usage is very low and that this option is not recommended for production environments.

The Xerces library maintains a separate cache for dealing with XML files. Users can modify aspects of this cache through a configuration file at /etc/opt/google/XMLparams. This file is not created automatically as part of the Open GEE Fusion install, so users that want to modify the parameters below must first create an XMLparams file. The XMLparams file has the following format:

# comments are preceded by '#'
# options are case-insensitive, and all need not be present

# the initial size of the XML heap in bytes
# default is 16384
INIT_HEAP_SIZE=<value>

# the max size of the XML heap in bytes
# default is 131072
MAX_HEAP_SIZE=<value>

# the size of XML heap allocation blocks in bytes
# default is 4096
BLOCK_SIZE=<value>

PackGen analysis thread configuration

Within misc.xml, there is an experimental feature for setting the number of threads to use while analyzing and creating packgen tasks. The number of threads is controlled by PackGenAnalysisThreads. This option is experimental and may contain bugs.

Status request timeout

Status requests to the system manager from clients like getop must wait to be fulfilled if the system manager is already engaged in another task. The system manager has a timeout that specifies the maximum amount of time it should spend waiting to fulfill a status request before it times out. This timeout can be configured manually by setting MutexTimedWaitSec in /gevol/assets/.config/misc.xml

<MutexTimedWaitSec>60</MutexTimedWaitSec>

Improving Performance

Included in /gevol/assets/.config/misc.xml is a GraphOperations option that enables new versions of certain operations. These updates are experimental and may contain bugs, and in some cases may reduce performance compared to previous versions of Fusion. GraphOperations can be set to the following values:

• NO_GRAPH_OPS or 0: use the legacy version of all operations
• FAST_GRAPH_OPS or 1: use new operations that are likely to improve performance
• ALL_GRAPH_OPS or 2: use all new operations regardless of performance impact

Learn more

• Benchmarking outcomes for applying task rules