Config Dumps and Periodically Saving Information about Cache / Rendering Statistics

There were a couple of new pipeline processors added in Sitecore 8 update 3 which periodically save the Cache and Rendering statistics to files in the /data folder. I haven’t seen much mention of them and speaking to a few people it seems to have been largely missed/ignored

Sitecore supportability improvements

• A Sitecore dumping configuration feature on every Sitecore restart has been implemented. The DumpConfigurationFiles processor has been added to the Sitecore pipeline. The processor is defined in the Sitecore.Diagnostics.config configuration include file and supports various configuration settings, such as the location for storing configuration files, support for compressing, and files to be tracked. By default, dump files are stored in /diagnostics/configuration_history in the Data folder. (373420)
• Periodic dumping data from /sitecore/admin/stats.aspx and /sitecore/admin/cache.aspx pages to the file system has been implemented. New processors have been added to the pipeline. The location and format of the log file can be configured as part of the processor configuration. By default, dump files are stored in /diagnostics/health_monitor in the Data folder. (415409, 415411)

Sitecore 8, Update 3 release notes

As mentioned, the above happens using 2 new pipelines processors in a new config file, Sitecore.Diagnostics.config

<healthMonitor>
    <!-- Dumps the information that the /sitecore/admin/cache.aspx page contains -->
    <processor type="Sitecore.Pipelines.HealthMonitor.HealthMonitor, Sitecore.Kernel" method="DumpAllCacheStatus">
      <dumpFile>$(dataFolder)/diagnostics/health_monitor/CacheStatus.{date}.{time}.html</dumpFile>
    </processor>
    <!-- Dumps the information that the /sitecore/admin/stats.aspx page contains -->
    <processor type="Sitecore.Pipelines.HealthMonitor.HealthMonitor, Sitecore.Kernel" method="DumpRenderingsStatistics">
      <dumpFile>$(dataFolder)/diagnostics/health_monitor/RenderingsStatistics.{date}.{time}.html</dumpFile>
    </processor>
</healthMonitor>

The processors simply read the caches and rendering statistics, create an HTML table of the values and then dump them in the folder location specified:

As you can see the output is very similar to what you see when you visit /sitecore/admin/cache.aspx or /sitecore/admin/stats.aspx.

The healthMonitor pipeline is invoked using a hook:

<!-- HOOKS -->
<hooks>
  <hook type="Sitecore.Diagnostics.HealthMonitorHook, Sitecore.Kernel"/>
</hooks>

Which sets an alarm to periodically call the pipeline. By default the stats are dumped to disk every 10 mins, as specified by the following setting:

<!--  HEALTH MONITOR INTERVAL
        Specifies the interval between running the healthMonitor pipeline.
        Default value: 00:10:00 (10 minutes)
  -->
<setting name="HealthMonitorInterval" value="00:10:00"/>

Along with the above, a new DumpConfigurationFiles processor in the initialize pipelines was also introduced:

<initialize>
<!-- DUMP CONFIGURATION FILES
Dumps the specified configuration files and allows you to monitor all the changes that are made to the configuration files.
  Supported child nodes:
    DumpFolder: The path to the root folder where the config file dump is stored. Each config file dump is stored in a folder or
                a zip archive. The name of the folder or zip archive has the following format: {date}.{time}.
                Default value: $(dataFolder)/diagnostics/configuration_history
    Zip:        Boolean value that determines whether each dump should be zipped. Default value: true
    files:      Contains <file> nodes where the "path" attribute value is the path to the configuration file or the folder that
                should be dumped.
-->
<processor type="Sitecore.Pipelines.Loader.DumpConfigurationFiles, Sitecore.Kernel">
  <dumpFolder>$(dataFolder)/diagnostics/configuration_history</dumpFolder>
  <zip>true</zip>
  <files hint="raw:AddPath">
    <file path="/App_Config" />
    <file path="/Web.config" />
    <file path="/Global.asax" />
    <file path="/sitecore/shell/sitecore.version.xml" />
  </files>
</processor>
</initialize>

The above will zip your configs on each startup.

In order to clean up after itself, any config dumps older than 30 days and any cache/rendering files older than 7 days are deleted periodically:

<scheduling>
  <agent type="Sitecore.Tasks.CleanupAgent">
    <files>
      <remove folder="$(dataFolder)/diagnostics/configuration_history" pattern="*" maxAge="30.00:00:00" recursive="false" />
      <remove folder="$(dataFolder)/diagnostics/health_monitor" pattern="*.*" maxAge="07.00:00:00" recursive="false" />
    </files>
  </agent>
</scheduling>

All the above was primarily introduced to help Sitecore Support with diagnostics information when trying to resolve issues on websites. Personally, it’s just a few more processors which you really shouldn’t care about on a CD environment. I believe @kamsar removed the DumpConfigurationFiles processor in his latest Developer Performance config, but I can’t seem to find it at the moment. EDIT: Find them in this gist.

In any case, the Cache and Rendering Statistics can be really useful when tuning a system for performance. You may even want to turn down to the dumping of the caches from 10mins to 30 seconds or 1 min while you run performance test scripts to allow you to better see what happens to the caches over time. It’s also very useful on a hardened CD server where access to the /sitecore folder has been removed.

Running an earlier version of Sitecore? The code is pretty simple, just reflect the classes with dotPeek and copy/paste the relevant bits of code and patch in yourself using the default Sitecore.Diagnostics.config as a starting point.