2
Vote

Eventually get OutOfMemoryException when VS is started with extension enabled

description

Unfortunately, if (and only if) I have the RazorGenerator extension enabled in VS2012, I eventually get a yellow screen of death (see below) when running my app in the Azure SDK 2.0 emulator (using IIS Express).

To avoid getting the OOM condition, I can disable the extension and the precompiled views continue to work fine, but of course I'm not able to edit/recompile any of the views because the tool is disabled!

Anyone else have this issue? David, have you seen this and do you know what could be causing this problem?

I have used VS2013 Ultimate preview and procdump generated managed memory dumps of iisexpress to verify there is no memory leak in my app.

[Aside: I also have the issue described here -- https://razorgenerator.codeplex.com/workitem/72. This is NOT the cause of the OOM condition, but is a little aggravating since the dlls disappear regularly, requiring the workaround in the post each time it happens.]

Detailed exception info follows.....

Server Error in '/' Application.

Insufficient memory to continue the execution of the program.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.OutOfMemoryException: Insufficient memory to continue the execution of the program.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[OutOfMemoryException: Insufficient memory to continue the execution of the program.]

System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine) +3103

System.CodeDom.Compiler.Executor.ExecWaitWithCapture(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine) +240

Microsoft.CSharp.CSharpCodeGenerator.Compile(CompilerParameters options, String compilerDirectory, String compilerExe, String arguments, String& outputFile, Int32& nativeReturnValue, String trueArgs) +347

Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames) +808

Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromFileBatch(CompilerParameters options, String[] fileNames) +203

System.Web.Compilation.AssemblyBuilder.Compile() +2695

System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath virtualPath) +566

System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) +565

System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) +201

System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath virtualPath, HttpContext context, Boolean allowCrossApp, Boolean throwIfNotFound) +177

System.Web.Compilation.BuildManager.CreateInstanceFromVirtualPath(VirtualPath virtualPath, Type requiredBaseType, HttpContext context, Boolean allowCrossApp) +56

RazorGenerator.Mvc.PrecompiledMvcEngine.CreateInstance(String virtualPath) +235

System.Web.WebPages.VirtualPathFactoryExtensions.CreateInstance(IVirtualPathFactory factory, String virtualPath) +219

System.Web.WebPages.VirtualPathFactoryManager.CreateInstanceOfType(String virtualPath) +201

System.Web.WebPages.StartPage.GetStartPage(WebPageRenderingBase page, IVirtualPathFactory virtualPathFactory, String appDomainAppVirtualPath, String fileName, IEnumerable1 supportedExtensions) +273

System.Web.WebPages.StartPage.GetStartPage(WebPageRenderingBase page, String fileName, IEnumerable
1 supportedExtensions) +236

RazorGenerator.Mvc.PrecompiledMvcView.Render(ViewContext viewContext, TextWriter writer) +782
System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context) +380

System.Web.Mvc.<>c__DisplayClass1a.<InvokeActionResultWithFilters>b__17() +33

System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func1 continuation) +613

System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList
1 filters, ActionResult actionResult) +263

System.Web.Mvc.Async.<>c__DisplayClass25.<BeginInvokeAction>b__22(IAsyncResult asyncResult) +240

System.Web.Mvc.<>c__DisplayClass1d.<BeginExecuteCore>b__18(IAsyncResult asyncResult) +28

System.Web.Mvc.Async.<>c__DisplayClass4.<MakeVoidDelegate>b__3(IAsyncResult ar) +15

System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) +53

System.Web.Mvc.Async.<>c__DisplayClass4.<MakeVoidDelegate>b__3(IAsyncResult ar) +15

System.Web.Mvc.<>c__DisplayClass8.<BeginProcessRequest>b__3(IAsyncResult asyncResult) +42

System.Web.Mvc.Async.<>c__DisplayClass4.<MakeVoidDelegate>b__3(IAsyncResult ar) +15

System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +640

System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +288

comments

davidebbo wrote Jul 18, 2013 at 10:22 PM

This is strange because the generator runs in the VS process (devenv), while the site runs in the web server process (e.g. w3wp, though there could be others). Can you confirm that? Also, in Task Manager, are any processes taking an abnormal amount of memory? Is the system low on memory?

sethb wrote Jul 19, 2013 at 9:05 PM

Yes, that's right. devenv loads the tooling and core dlls of RG, and iisexpress loads my dlls and the Mvc and templating dlls of RG. (WaIISHost and WaWorkerHost also load my dlls.)

And yes, if my site is running fine (no OOM yet), I can close devenv and the site continues to run.

But no, my system is not low on memory. It usually has at least 700MB of physical RAM free, and I even saw this OOM today with 1.5GB of RAM free. No processes are taking up an abnormal amount of memory, virtual private bytes used by iisexpress and devenv generally stay below the OOM range of 1GB. (In fact, the OOM often happens when both have private bytes under 500MB.)

Thinking that the virtual address space might be getting fragmented, I even switched all .NET processes to x64 so fragmentation of the virtual address space could not be a cause. (I assume I'll need to switch my projects back to targeting "Any" rather than x64 before deploying to Azure, isn't that right? I assume I can't count on having x64 on Azure...)

Unfortunately, that didn't solve the problem either. The OOM usually occurs after running the RazorGenerator custom tool on a view and rebuilding the solution, although if I'm coding for a while and rebuilding a number of times, I have even seen the OOM immediately after launching the emulator and iisexpress.

With all the free RAM and the OOM, it looks like fragmentation (e.g., http://blogs.msdn.com/b/david.wang/archive/2006/02/14/more-on-virtual-memory-memory-fragmentation-and-leaks-and-wow64.aspx), but it seems to be related to the running of the RG extension in devenv.

Indeed, with the RG extension running, I have seen nearly 100 threads in the devenv process with start address clr.dll!InstallCustomModule+0x1b2. Even though the threads use very little CPU (the max was 0.48%), the threads are constantly started and stopped.

Without the RG extension running, I see just a few threads with that start address, and many few total threads (e.g, about 25 rather than 100) in devenv.

I just came across this reference for fragmentation:
http://blogs.msdn.com/b/tom/archive/2008/02/18/high-memory-part-5-fragmentation.aspx

One of the two scenarios described there may be the cause. I can do further memory dump analysis as described in that blog post, but I'd rather send the results to you privately.

gbanczak wrote Oct 7, 2013 at 8:54 AM

Hi,
I'm having a similar issue (same stacktrace) For me it appears randomly on a MVC 3 app when it's hosted in a single Application Pool together with other sites. (w3p process takes 1.6 GB of 4 GB available memory)

Could you post what was the cause of your issue/how have you solved it ?

sethb wrote Oct 9, 2013 at 7:14 PM

No, but actually the problem persists after removing the RazorGenerator extension if I leave all the precompiled views in the solution. Only after removing all or most of the precompiled views (which dramatically speeds up the time for VS to compile the remaining C# code) does the problem disappear. Of course, I still run into trouble if I navigate the site and let enough views be dynamically compiled. When this occurs, the way I work around it during development is to shut down processes with .NET memory and reopen them, including the web server. This, of course, is a completely unsatisfactory workaround for a deployed site.

Are you able to upload a demo that illustrates this problem? I was in touch with folks at Microsoft a few weeks ago and could forward your demo to them. Or if you feel more comfortable speaking with them directly, I can ask them to get in touch with you.

davidebbo wrote Oct 9, 2013 at 8:16 PM

It's best to keep the discussion public here, so if you have a simple repro, please share it.

And please do note that while the main contributors work at Microsoft, this is an OSS project with no ties to Microsoft.