How much memory does my ColdFusion variable really use? – Part III

by kai on 10/03/2010



Finally, nearly the last part (there’s one more coming…). In part II I talked about the different problems we’d run into using the instrumentation code out-of-the-box without modifying it for the special scenario of using it to size ColdFusion variables. We declared those issues solved (sic!) 🙂 – so let’s have a look at some code and how to use it. All I’ve done was to grab the example from Heinz’s JavaSpecialists newsletter and changed some of the signatures (from private to protected). I then wrote a class CFMemoryCounterAgent that simply extends MemoryCounterAgent and overrides some of the methods with modified versions to cater for ColdFusion (well, technically it’s probably overloading because I modified the signature…).

Also while I was at it – I added a second argument to sizeOf() and deepSizeOf() named ignoreFlyweights. That refers back to the last section of part II – it enables you to define if the (what we have defined as well-known) flyweights are going to be counted for the total memory usage or not.

This is the modified internalSizeOf() method, that’s being used to “filter” for the the two well-known “troublemakers” in CF as discussed in part II. When I was doing some further testing, I came across a few more scenarios that could cause the memory sizing to drift off, those have been added as well:

 private static long internalSizeOf(Object obj, Stack stack, Map visited, 
   boolean ignoreFlyweights) 
  {
    if (skipObject(obj, visited, ignoreFlyweights)) 
    {	
      return 0;
    }
   	
    Class clazz = obj.getClass();  
   
    if (clazz.isArray()) 
    {
      addArrayElementsToStack(clazz, obj, stack);
    } 
    else 
    {
      // add all non-primitive fields, non-CF memory tracker objects and non-SessionContext objects to the stack
      while (clazz != null) 
      {
        Field[] fields = clazz.getDeclaredFields();
        for (Field field : fields) 
        {
          if (!Modifier.isStatic(field.getModifiers()) && !field.getType().isPrimitive() && 
            field.getType().getName() != "coldfusion.runtime.NeoPageContext" && 
            field.getType().getName() != "coldfusion.runtime.CfJspPage" && 
            field.getType().getName() != "coldfusion.monitor.memory.MemoryTrackable" && 
            field.getType().getName() != "coldfusion.monitor.sql.QueryStat" && 
            field.getType().getName() != "coldfusion.monitor.memory.MemoryTrackerProxy" && 
            field.getType().getName() != "javax.servlet.ServletContext") 
          {
            field.setAccessible(true);
            try 
          	{
              stack.add(field.get(obj));        
            } 
            catch (IllegalAccessException ex) 
            {
              throw new RuntimeException(ex);
            }
          }
        }
        clazz = clazz.getSuperclass();
      }
    }
    visited.put(obj, null);

    return sizeOf(obj, ignoreFlyweights);
  }

I’ve provided a .zip file (version 0.1.1) containing the source. If you don’t want to compile it yourself, just grab the CFMemoryCounterAgent.jar and apply it to your CF instance as described in part I. The .zip also contains a memorytest.cfm that should be self-explanatory and will give you an idea how it’s supposed to be used. If you want to compile it, the build file for Ant should do the trick – please note that there is a copy task in there that copies the file into the location of my CF installation’s lib folder. You pretty much would want to change that to copy the .jar file to a different location OR remove that copy task in the first place. Also – I’ve build and tested this on CF 8.0.1 on OS X. You need Java 5 or newer to run this in the first place and YMMV with other CF versions. If you try it and get weird results, let me know.

Edit: The .jar file is compiled with Java 6 and the Ant file targets Java 6 as well. Make sure that you potentially recompile the source and change the Ant file if you try to run this on Java 5.

Please keep in mind that all results you’re getting from this are JVM-based estimations and sort-of right. Also, I would strongly discourage you to build totally messed-up code for the sake of saving 250 bytes or similar. Don’t forget that the JVM has a reasonably well-working garbage collection and the GC will collect most variables that you create on a page very quickly in the first place. The techniques shown here are more for information and to raise the behind-the-scenes awareness. Besides that, it’s just interesting to see how CF deals with the different variable types. The next (and final) post of this series will have a look at some results and try to explain those. Stay tuned.

{ 9 comments… read them below or add one }

Hendrik Kramer March 10, 2010 at 10:48 am

Very interesting article. I’ll try it within the next days on some parts of our codebase. But the link to the zip is not working (yet)? Link returns “not found” here.

Reply

kai March 10, 2010 at 11:56 am

@Hendrik Sorry for the broken link, had a typo in the file name. It works now.

Reply

Jay April 22, 2010 at 11:03 am

I keep getting an error: “Instrumentation environment not initialised”

I realize that this is because the -javaagent switch is not configured properly. I have tried to put it in the jvm.config under
java.args=-server -javaagent:[path on my server]/MemoryCounterAgent.jar [more args follow…]
but I keep getting th error.

any ideas?

tia,

Jay

Reply

Jay April 22, 2010 at 11:15 am

cleanup suggestion for the CF object: add a static array of strings for the types-to-ignore, then use the Arrays.asList( String[] ).contains( ) ) method to test if the type is in the list. this will allow to easily add more items to the list in the future:

public static String[] ignoreTypes;

static {

ignoreTypes = new String[] {

“coldfusion.runtime.NeoPageContext”,
“coldfusion.runtime.CfJspPage”,
“coldfusion.monitor.memory.MemoryTrackable”,
“coldfusion.monitor.sql.QueryStat”,
“coldfusion.monitor.memory.MemoryTrackerProxy”,
“javax.servlet.ServletContext”
};
}

then in the internalSizeOf() method, replace all the type tests with

!Arrays.asList( ignoreTypes ).contains( field.getType().getName() )

Reply

kai April 22, 2010 at 8:59 pm

Jay, have you figured out the javaagent issue? Are you on Windows or Linux/Mac – I wonder if it’s an issue how to specify the path in Windows. Anything in logs and can you provide a bit more info on your setup? The suggestions for the Java code changes are good – the solution discussed here is really just a proof of concept at this stage – I’ll work those changes in a further release, thx!

Reply

Jay April 23, 2010 at 10:46 am

Kai, thank you for replying. it actually works now. I added it again to the args in order to test it before I send you the information here and now it works. perhaps I had a typo yesterday and that’s why it didn’t work.

while I was still trying to get it to work yesterday, I found the jVisualVM monitor that comes with the JDK. you need to start the JVM with the switches below (which might not be a good idea for a production server), but then you can generate thread and heap dumps and look deep into the results to find out memory allocations etc. pretty cool tool. I thought that your readers might find it useful.

Jay

java.args: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9500 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

then launch jVisualVM from the JDK\bin folder and connect to localhost:9500 (or whatever port you used in the switches)

Reply

kai April 25, 2010 at 2:43 pm

Hi Jay, thx for pointing jVisualVM out. There are a few tools around that do stuff like that and also look deeper into things like garbage collection etc.

I agree – those settings are not something you constantly would want to run in production.

Reply

Joel Spriggs November 20, 2010 at 1:51 am

Did anyone ever figure out the jvm arguments to correctly initialise the env? I’ve tried the suggestions and keep getting java.lang.IllegalStateException: Instrumentation environment not initialised.

Reply

Joel Spriggs November 20, 2010 at 2:00 am

I spoke to soon, problem is I run CF standalone. So the jar goes to /Applications/Coldfusion9/runtime/lib instead of /Applications/Coldfusion9/lib and the jvm argument to add into jvm.config was -javaagent:{application.home}/lib/CFMemoryAgent.jar

Thanks!

Reply

Cancel reply

Leave a Comment

Previous post:

Next post: