Posts tagged ‘MSBuild’

July 14th, 2010

A MSBuild convention proposal – Targets per assembly

MSBuild is a very powerful build tool. Unfortunately, in my experience, I have seen it utilized very little. Instead, developers rely entirely on Visual Studio for the build, which is a mistake. I think there are a couple reasons for this:

  • Modifying the build feels like you are mucking in the internals of Visual Studio
  • There is no convention based approaches for how to handle custom bulid targets

Now, you could go the way of NAnt, and have complete independence from Visual Studio. I think this is a mistake as well. Visual Studio, while being a royal pain much of the time, is still a very very very powerful tool. I believe we should embrace it, while using MSBuild plus conventions to achieve ultimate flexibility.

The proposal

It’s actually quite simple. Each assembly/project already has its own *.csproj file, which is in essence a partial MSBuild file. The problem is editing it is weird and scary at times, because it is auto-generated by Visual Studio.

Just add the following to any or all your csproj files.

<Project>
  ... All the visual studio muck ...
  <Import Project="$(MSBuildProjectName).targets" Condition="Exists('$(MSBuildProjectName).targets')" />
</Project>

Then add a file with that name. If your projects name is Reference.Web, then add a file named Reference.Web.targets and include it in the project. The content could contain something like the following.

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 
  <ItemGroup>
    <Files Include="$(SolutionDir)\configuration\$(Configuration)\$(AssemblyName).*.config" />
    <Files Include="$(SolutionDir)\configuration\$(Configuration)\Common.*"/>
  </ItemGroup>
 
  <Target Name="InstallConfiguration">    
    <Copy SourceFiles="@(Files)" DestinationFolder="$(OutputPath)" />
  </Target>
 
  <!-- Override the AfterBuild target -->
  <Target Name="AfterBuild" DependsOnTargets="InstallConfiguration" />
 
</Project>

Now you have a build file separate from the project that is really easy setup and use, straight for Visual Studio.

The only gotcha is Visual Studio won’t automatically notice the build file changes, so you still have to relad the assembly after you make a build file change.

UPDATE – More robust Import statement

Via @sayedihashimi

Even better would be a conditional import, and you should use the MSBuildProjectName property instead of AssemblyName.

Post to Twitter Post to Delicious Post to Digg Post to Facebook Post to Reddit

April 25th, 2010

MSBuild performance tips

A while back, I posted about a Powershell script I wrote to help analyze build performance. Today I spent a good amount of time looking at the output and tweaking things. I came across a couple BIG performance enhancements that I thought I would share.

Tip #1 – Don’t start external processes

Starting another process is a time consuming activity, and if you are doing it on a post/pre build event, that is going to slow your build down tremendously. I was looking through my project, and found this post build event.

<PropertyGroup>
  <PostBuildEvent>"$(MSBuildBinPath)\msbuild.exe" "$(SolutionDir)ApplyConfiguration.msbuild" /p:Configuration=$(ConfigurationName)</PostBuildEvent>
</PropertyGroup>

In the diagnostic output, I saw the Exec call took 649ms.

Improving performance is simple, just import the MSBuild file and call the target directly. The refactored build looks like the following.

  <Import Project="$(SolutionDir)ApplyConfiguration.msbuild" />
  <Target Name="AfterBuild" DependsOnTargets="InstallConfiguration" />

And the MSBuild timing is pretty amazing, 5ms. That’s half a second in savings.

Tip #2 – Understand Inputs/Outputs

In MSBuild, targets can have a list of input and output files. Before building the target, it checks to make sure there is a need to build based on the timestamps of the inputs and outputs, also known as building incrementally.

Lets look at an example target for compressing Javascript.

<Target Name="CompressJavascript" 
        DependsOnTargets="_CreateScriptFolder;MergeJavascript">
  <Exec Command="..\Bin\jsmin.exe &lt; $(MergedJavascript) &gt; $(CompressedJavascript)" />
</Target>

This executes in about 104 ms. Not a long time, but if you are doing it over and over, it could end up adding up quickly. Refatoring this to execute quickly is extremely simple.

<Target Name="CompressJavascript" 
        Inputs="$(MergedJavascript)"
        Outputs="$(CompressedJavascript)" 
        DependsOnTargets="_CreateScriptFolder;MergeJavascript">
  <Exec Command="..\Bin\jsmin.exe &lt; $(MergedJavascript) &gt; $(CompressedJavascript)" />
</Target>

The first build, it takes the same time. But all subsequent builds execute in 0ms if the $(MergedJavascript) file hasn’t changed.

Tip #3 – Reduce dependencies

After you have optimized everything else, you will see a couple targets that take the majority of your build time, ResolveAssemblyReferences and ResolveProjectReferences. Unfortunately, there isn’t a quick fix for this one. It comes back to your application architecture and dependency management.

Even compiling isn’t that big a deal, since compiling is done incrementally if files have changed. Dependencies have to be resolved each build though.

       78 ms  CoreCompile                                1 calls
      149 ms  ResolveProjectReferences                   1 calls
      188 ms  ResolveAssemblyReferences                  1 calls

Post to Twitter Post to Delicious Post to Digg Post to Facebook Post to Reddit

Tags:
December 22nd, 2009

Analyzing Visual Studio Build Performance

Every now and again, your build time gets so long you have to investigate what the cause is. Since Visual Studio uses MSBuild, it is possible to get very detailed build information.

The first step is getting into diagnostic mode. This will print out way too much information, but it is the only mode that prints out execution summaries for each project. To get into diagnostics mode, simply go to Tools -> Options -> Projects and Solutions -> Build and Run. From there, you can change the MSBuild verbosity to diagnostic.

12-22-2009 7-44-06 PM
12-22-2009 7-45-08 PM

Now just build (or rebuild). Then copy the build output to a txt file and save it somewhere.

12-22-2009 7-50-52 PM

The only problem with the output is the amount of information (300K lines on a rebuild of a medium size project). It has ton’s of valuable information, but an easy way to process it is needed. The biggest value, in my opinion, is the section at the end of every project where the Summary information about projects, targets, and tasks are displayed. To help retrieve this info, I wrote a quick little Powershell script.

 
param
(
    [string] $file = ""
)
 
if(!(test-path $file)){
    throw "$file not found"
}
 
function accumulate_and_print_project_info(){
    begin{
        $curr_project = ""
        $should_print = $false
    }
    process{
 
 
        if($_ -match "----- (re)?build (all )?started: project: (.*) -----"){
            $curr_project = $matches[3]
            "----- $curr_project -----"
            $should_print = $false
        }elseif($curr_project -and !$should_print -and $_.EndsWith("Summary:")){
            $_
            $should_print = $true
        }elseif($curr_project -and $should_print -and $_ -eq ""){
            $should_print = $false
        }elseif($should_print){
            $_
        }
    }
}
 
gc $file | accumulate_and_print_project_info

The output is going to look something like the following.

----- ControlPanel, Configuration: Development Any CPU -----
Project Performance Summary:
     2672 ms  C:\Users\brian.hartsock\Documents\Visual Studio 2008\Projects\Api\cp3\ControlPanel\ControlPanel.csproj   1 calls
Target Performance Summary:
        0 ms  CreateSatelliteAssemblies                  1 calls
        0 ms  _ComputeNonExistentFileProperty            1 calls
        0 ms  Clean                                      1 calls
        0 ms  _AfterCompileWinFXInternal                 1 calls
        0 ms  BeforeCompile                              1 calls
        0 ms  BeforeClean                                1 calls
        0 ms  AfterBuild                                 1 calls
        0 ms  GetTargetPath                              1 calls
        0 ms  _CheckForCompileOutputs                    1 calls
        0 ms  EntityDeploy                               1 calls
        0 ms  ResolveVCProjectReferences                 1 calls
        0 ms  _SetEmbeddedWin32ManifestProperties        1 calls
        0 ms  StyleCopForceFullAnalysis                  1 calls
        0 ms  CleanPublishFolder                         1 calls
        0 ms  BeforeBuild                                1 calls
        0 ms  CoreResGen                                 1 calls
        0 ms  BeforeRebuild                              1 calls
        0 ms  _CopySourceItemsToOutputDirectory          1 calls
        0 ms  AfterResGen                                1 calls
        0 ms  Build                                      1 calls
        0 ms  SetWin32ManifestProperties                 1 calls
        0 ms  AssignTargetPaths                          1 calls
        0 ms  CoreBuild                                  1 calls
        0 ms  AfterResolveReferences                     1 calls
        0 ms  Compile                                    1 calls
        0 ms  ResGen                                     1 calls
        0 ms  AfterClean                                 1 calls
        0 ms  Rebuild                                    1 calls
        0 ms  PrepareResourceNames                       1 calls
        0 ms  BuildOnlySettings                          1 calls
        0 ms  PrepareResources                           1 calls
        0 ms  CleanReferencedProjects                    1 calls
        0 ms  AfterMarkupCompilePass1                    1 calls
        0 ms  FileClassification                         1 calls
        0 ms  AfterCompile                               1 calls
        0 ms  CompileRdlFiles                            1 calls
        0 ms  BeforeResolveReferences                    1 calls
        0 ms  CreateCustomManifestResourceNames          1 calls
        0 ms  ResolveReferences                          1 calls
        0 ms  DesignTimeMarkupCompilation                1 calls
        0 ms  AfterRebuild                               1 calls
        0 ms  GetReferenceAssemblyPaths                  1 calls
        0 ms  PrepareForRun                              1 calls
        0 ms  AfterCompileWinFX                          1 calls
        0 ms  BeforeResGen                               1 calls
        1 ms  PrepareRdlFiles                            1 calls
        1 ms  GetCopyToOutputDirectoryItems              1 calls
        1 ms  SplitResourcesByCulture                    1 calls
        1 ms  _SplitProjectReferencesByFileExistence     1 calls
        1 ms  _GenerateCompileInputs                     1 calls
        1 ms  _GenerateSatelliteAssemblyInputs           1 calls
        1 ms  GetWinFXPath                               1 calls
        1 ms  EntityClean                                1 calls
        1 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
        1 ms  ResolveProjectReferences                   1 calls
        1 ms  PrepareForBuild                            1 calls
        3 ms  GetFrameworkPaths                          1 calls
        8 ms  CoreClean                                  1 calls
       14 ms  IncrementalClean                           1 calls
       17 ms  CopyFilesToOutputDirectory                 1 calls
       17 ms  _CleanGetCurrentAndPriorFileWrites         1 calls
       66 ms  ResolveAssemblyReferences                  1 calls
      399 ms  _CopyFilesMarkedCopyLocal                  1 calls
      519 ms  StyleCop                                   1 calls
      636 ms  CoreCompile                                1 calls
      974 ms  PostBuildEvent                             1 calls
Task Performance Summary:
        0 ms  Delete                                     2 calls
        0 ms  Message                                    6 calls
        0 ms  EntityDeploy                               1 calls
        0 ms  GetFrameworkSdkPath                        1 calls
        0 ms  CreateProperty                             1 calls
        0 ms  AssignTargetPath                           5 calls
        0 ms  MakeDir                                    2 calls
        1 ms  ConvertToAbsolutePath                      1 calls
        1 ms  AssignCulture                              1 calls
        1 ms  GetWinFXPath                               1 calls
        1 ms  FindAppConfigFile                          1 calls
        1 ms  ReadLinesFromFile                          2 calls
        1 ms  EntityClean                                1 calls
        2 ms  CreateItem                                 3 calls
        2 ms  WriteLinesToFile                           2 calls
        3 ms  GetFrameworkPath                           2 calls
       13 ms  RemoveDuplicates                           3 calls
       20 ms  FindUnderPath                              7 calls
       66 ms  ResolveAssemblyReference                   1 calls
      415 ms  Copy                                       6 calls
      515 ms  StyleCopTask                               1 calls
      635 ms  Csc                                        1 calls
      973 ms  Exec                                       1 calls

Now you should have all the data you need to figure out what is taking so long to build your project.

Post to Twitter Post to Delicious Post to Digg Post to Facebook Post to Reddit

October 20th, 2009

Using Powershell scripts from MSBuild, Scheduled Tasks, etc.

There are a few different ways to use Powershell from the legacy cmd shell. The most common way is to call it like the following.

> powershell write-host "hello world"

As you can see, the powershell.exe is called with Powershell commands as the parameters. I started noticing some odd behavior though. I have the following script, TestScript.ps1. It has code as follows:

param
(
  $str
)
 
write-host $str

Very simple right. Well guess what happens when I call it from powershell.exe like above?

> powershell .\TestScript.ps1 "hello world"
hello

A little odd. I have a very cordial script but it is a little tongue tied. I didn’t spend a lot of time trying to figure out why this was occurring, I instead used powershell -? to help me find an alternate method, and probably better way to call Powershell scripts from the legacy cmd shell.

> powershell -Command "& { .\TestScript.ps1 'hello world' }"
hello world

This worked like a charm. Note the quotes, as script blocks aren’t interpreted from the cmd shell properly, and will cause odd behavior. And from MSBuild, a little bit of XML escapage and you can easily use Powershell.

  <Target Name="AfterBuild">
    <Exec Command="powershell.exe -command &quot;&amp; {.\Register-EmailApiSnapIn.ps1 &apos;$(TargetPath)&apos;}&quot;" />
  </Target>

Post to Twitter Post to Delicious Post to Digg Post to Facebook Post to Reddit

March 13th, 2009

New Home – Mosso and Cloud Files

Yesterday, I took the plunge, changed my DNS, and am now on Mosso. Dreamhost was getting slow, and having intermittent outages, so I needed the switch. Not to mention Mosso is another division of Rackspace, just like Mailtrust.

I am super happy with Mosso so far. The blog is faster than ever, and I have had almost no problems. What’s even crazier is I am using CDN Tools to host a lot of my static content on the Cloud Files CDN. I could have done this while on Dreamhost, but it was easier since I already have a Mosso account.

Automation

Setting up Mosso is quite different than Dreamhost, since SSH access isn’t permitted. FTP or SFTP is really the only means to get your code uploaded, so automation is a must. Luckily I love automation, so I got to work on an MSBuild script.

Just to give some background on how my WordPress blog works. I have a SVN checkout of the latest WordPress tag. I also have a few custom files that I want uploaded, in addition to to the WordPress code. Instead of interminginly the custom code with WordPress, I created a custom directory.

My idea for the build was to follow a fairly simple workflow.

  • Update WordPress
  • Export WordPress checkout to a deployment directory
  • Copy custom files to deployment directory
  • SFTP deployment directory

Here is how the majority of the build script came out. You will need MSBuild Community tasks as well as SVN installed somewhere.

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Upload">
  <Import Project="lib\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
  <PropertyGroup>
    <!-- Fill this in with all your info -->
    <CloudSiteFtp></CloudSiteFtp>
    <CloudSiteDbHost></CloudSiteDbHost>
    <CloudSiteDb></CloudSiteDb>
    <CloudSiteDbUser></CloudSiteDbUser>
    <CloudSiteDbPass></CloudSiteDbPass>
    <CloudSiteFtpUser></CloudSiteFtpUser>
    <CloudSiteFtpPass></CloudSiteFtpPass>
    <SvnSource></SvnSource>
    <SvnToolPath></SvnToolPath>
    <DeploymentDir></DeploymentDir>
    <CustomSourceDir></CustomSourceDir>
    <CloudSiteDir></CloudSiteDir>
  </PropertyGroup>
 
  <Target Name="DeleteDeploymentDir">
    <Message Importance="normal" Text="Deleting directory $(DeploymentDir)" />
    <RemoveDir Directories="$(DeploymentDir)" />
    <Message Text="Done." />
    <Message Text="" />
  </Target>
 
  <Target Name="UpdateSource">
    <Message Text="Updating $(SvnSource)..." />
    <SvnUpdate LocalPath="$(SvnSource)" ToolPath="$(SvnToolPath)" />
    <Message Text="Done." />
    <Message Text="" />
  </Target>
 
  <Target Name="ExportSource" DependsOnTargets="DeleteDeploymentDir">
    <Message Text="Exporting $(SvnSource) to $(DeploymentDir)..." />
    <SvnExport LocalPath="$(DeploymentDir)" RepositoryPath="$(SvnSource)" ToolPath="$(SvnToolPath)" />
    <Message Text="Done." />
    <Message Text="" />
  </Target>
 
  <Target Name="MergeCustomSourceCode">
    <Message Text="Copying custom files..." />
    <Exec Command="xcopy $(CustomSourceDir) $(DeploymentDir) /E /Y" />
    <Message Text="Done." />
    <Message Text="" />
  </Target>
  <!-- Upload, BackupDb, and ImportDB targets should go here -->
</Project>

Uploading the data from a Windows machine required a tool for SFTP or FTP. MSBuild Community Tasks have FTP tasks, but I wanted to make sure my data was transfered securely, so I chose SFTP. pscp was the obvious tool because it is the simplest solution to SFTP file transfer on Windows, in my opinion.

<Target Name="Upload" DependsOnTargets="UpdateSource; ExportSource; MergeCustomSourceCode">
  <Exec Command="lib\pscp -r -pw $(CloudSiteFtpPass) $(DeploymentDir)\* $(CloudSiteFtpUser)@$(CloudSiteFtp):$(CloudSiteDir)" />
</Target>

The next step was getting the database working. The database and database user have to be created from the web interface before anything can be done from the client. Once it is created, you can execute commands remotely, so automation is pretty easy. Word of warning, I am one of those crazy’s that run MySQL on my laptop, so access to mysql and mysqldump is required for these tasks to work.

<Target Name="ImportDb">
  <Error Condition="'$(ImportFileName)' == ''" Text="No ImportFileName specificed (msbuild /p:ImportFileName ...)" />
 
  <Exec Command="mysql -h $(CloudSiteDbHost) -u $(CloudSiteDbUser) -p$(CloudSiteDbPass) $(CloudSiteDb) &lt; $(ImportFileName)" />
</Target>
 
<Target Name="BackupDb">
 
  <Error Condition="'$(BackupFile)' == ''" Text="No backupfile specified" />
 
  <Exec Command="mysqldump -h $(CloudSiteDbHost) -u $(CloudSiteDbUser) -p$(CloudSiteDbPass) $(CloudSiteDb) > $(BackupFile)" />
  <Zip Files="$(BackupFile)" ZipFileName="$(BackupFile).zip" />
  <Delete Files="$(BackupFile)" />
</Target>

What next…?

I am still somewhat unhappy that my upload process. It uploads everything, each time. rsync, or something similar, would be awesome if SSH access was allowed. I am pretty sure I can accomplish a dumb rsync over SFTP, but I haven’t devoted enough time to it. The other downside to this is wordpress’s automatic upgrade feature will get overwritten with the next upload. So be wary about automatically upgrading through the wordpress interface.

After that, I want to incorporate data backups before the upload and allow for easy rollback if something fails. After that, I should be set.

Post to Twitter Post to Delicious Post to Digg Post to Facebook Post to Reddit

Tags: