A better Start-Job cmdlet

With Powershell 2, the Start-Job command was added, which allows statements to be executed in the background. There are a few oddities regarding script blocks that need to be understood first though.

  • Powershell doesn’t support closures
  • The current working directory isn’t preserved

I confronted these issues first hand today while working with a Rails app. I wanted to execute ruby script\server in a job, but it wasn’t working.

PS C:\Users\Brian\workspace\myapp> start-job { ruby script\server }

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
3               Job3            Running    True            localhost             ruby script\server

PS C:\Users\Brian\workspace\myapp> receive-job 3
C:\Ruby19\bin\ruby.exe: No such file or directory -- script/server (LoadError)
    + CategoryInfo          : NotSpecified: (C:\Ruby19\bin\r...ver (LoadError):String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

For some reason, when executing the job, it is executing in a different working directory. It is pretty easy to confirm.

PS C:\Users\Brian\workspace\myapp> start-job { get-location }

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
3               Job3            Running    True            localhost             get-location

PS C:\Users\Brian\workspace\myapp> receive-job 3

Path
----
C:\Users\Brian\Documents

Knowing that the working directory was not preserved, I figured it would be easy enough to write my own function that added this functionality. This was my first stab at it.

 
function start-jobhere([scriptblock]$block){
  start-job -argumentlist (get-location),$block { set-location $args[0]; . $args[1] }
}

But, it didn’t work.

PS C:\Users\Brian\workspace\myapp> start-jobhere { ruby script\server }

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Running    True            localhost             set-location $args[0]...

PS C:\Users\Brian\workspace\myapp> receive-job 1
The term ' ruby script\server ' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
    + CategoryInfo          : ObjectNotFound: ( ruby script\server :String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

I had a sneaking suspicsion that Powershell was serializing the scriptblock to a string, which I confirmed.

PS C:\Users\Brian\workspace\myapp> $sb = { "hello world "}
PS C:\Users\Brian\workspace\myapp> start-job -ArgumentList $sb { $args[0] | gm }

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
15              Job15           Running    True            localhost             $args[0] | gm

PS C:\Users\Brian\workspace\myapp> receive-job 15

   TypeName: System.String

Name             MemberType            Definition
----             ----------            ----------
Clone            Method                System.Object Clone()
CompareTo        Method                int CompareTo(System.Object value), int CompareTo(string strB)
Contains         Method                bool Contains(string value)
CopyTo           Method                System.Void CopyTo(int sourceIndex, char[] destination, int destinationIndex,...
EndsWith         Method                bool EndsWith(string value), bool EndsWith(string value, System.StringCompari...
Equals           Method                bool Equals(System.Object obj), bool Equals(string value), bool Equals(string...
GetEnumerator    Method                System.CharEnumerator GetEnumerator()
GetHashCode      Method                int GetHashCode()
GetType          Method                type GetType()
GetTypeCode      Method                System.TypeCode GetTypeCode()
IndexOf          Method                int IndexOf(char value), int IndexOf(char value, int startIndex), int IndexOf...
IndexOfAny       Method                int IndexOfAny(char[] anyOf), int IndexOfAny(char[] anyOf, int startIndex), i...
Insert           Method                string Insert(int startIndex, string value)
IsNormalized     Method                bool IsNormalized(), bool IsNormalized(System.Text.NormalizationForm normaliz...
LastIndexOf      Method                int LastIndexOf(char value), int LastIndexOf(char value, int startIndex), int...
LastIndexOfAny   Method                int LastIndexOfAny(char[] anyOf), int LastIndexOfAny(char[] anyOf, int startI...
Normalize        Method                string Normalize(), string Normalize(System.Text.NormalizationForm normalizat...
PadLeft          Method                string PadLeft(int totalWidth), string PadLeft(int totalWidth, char paddingChar)
PadRight         Method                string PadRight(int totalWidth), string PadRight(int totalWidth, char padding...
Remove           Method                string Remove(int startIndex, int count), string Remove(int startIndex)
Replace          Method                string Replace(char oldChar, char newChar), string Replace(string oldValue, s...
Split            Method                string[] Split(Params char[] separator), string[] Split(char[] separator, int...
StartsWith       Method                bool StartsWith(string value), bool StartsWith(string value, System.StringCom...
Substring        Method                string Substring(int startIndex), string Substring(int startIndex, int length)
ToCharArray      Method                char[] ToCharArray(), char[] ToCharArray(int startIndex, int length)
ToLower          Method                string ToLower(), string ToLower(System.Globalization.CultureInfo culture)
ToLowerInvariant Method                string ToLowerInvariant()
ToString         Method                string ToString(), string ToString(System.IFormatProvider provider)
ToUpper          Method                string ToUpper(), string ToUpper(System.Globalization.CultureInfo culture)
ToUpperInvariant Method                string ToUpperInvariant()
Trim             Method                string Trim(Params char[] trimChars), string Trim()
TrimEnd          Method                string TrimEnd(Params char[] trimChars)
TrimStart        Method                string TrimStart(Params char[] trimChars)
Chars            ParameterizedProperty char Chars(int index) {get;}
Length           Property              System.Int32 Length {get;}

So, instead of invoking the scriptblock like your normally would, you need to call Invoke-Expression instead. My function ended up looking like the this.

function start-jobhere([scriptblock]$block){
  start-job -argumentlist (get-location),$block { set-location $args[0]; invoke-expression $args[1] }
}

Now it works like a charm!

Get file extension information with Powershell

I wanted to get a list of different file extensions in a project I was looking at so I knew the makeup of the code and technology. Pretty cool little command.

gci -recurse | select -expand extension | group | sort count -Descending

Which returned the following.


Count Name                      Group
----- ----                      -----
  667 .asp                      {.asp, .asp, .asp, .asp...}
  530 .gif                      {.gif, .gif, .gif, .gif...}
  345                           {, , , ...}
  264 .js                       {.js, .js, .js, .js...}
   47 .txt                      {.txt, .txt, .txt, .txt...}
   46 .htm                      {.htm, .htm, .htm, .htm...}
   45 .ai                       {.ai, .ai, .ai, .ai...}
   42 .css                      {.css, .css, .css, .css...}
   31 .pdf                      {.pdf, .pdf, .pdf, .pdf...}
   21 .doc                      {.doc, .doc, .doc, .doc...}
   16 .xml                      {.xml, .xml, .xml, .xml...}
   16 .xsl                      {.xsl, .xsl, .xsl, .xsl...}
   15 .jpg                      {.jpg, .jpg, .jpg, .jpg...}
   14 .png                      {.png, .png, .png, .png...}
   11 .html                     {.html, .html, .html, .html...}
    9 .psd                      {.psd, .psd, .psd, .psd...}
    8 .wsml                     {.wsml, .wsml, .wsml, .wsml...}
    5 .bak                      {.bak, .bak, .bak, .bak...}
    4 .WSDL                     {.WSDL, .WSDL, .WSDL, .WSDL}
    3 .bat                      {.bat, .bat, .bat}
    3 .WGen                     {.WGen, .WGen, .WGen}
    2 .harbinger                {.harbinger, .harbinger}
    2 .aspWORKING               {.aspWORKING, .aspWORKING}
    2 .concord                  {.concord, .concord}
    1 .aspUPLOAD                {.aspUPLOAD}
    1 .ini                      {.ini}
    1 .mdb                      {.mdb}
    1 .tmp                      {.tmp}
    1 .LOG                      {.LOG}
    1 .asa                      {.asa}
    1 .xls                      {.xls}
    1 .xml~                     {.xml~}

Analyzing Visual Studio Build Performance

Every now and again, your build time gets so long you have to investigate what the cause is. Since Visual Studio uses MSBuild, it is possible to get very detailed build information.

The first step is getting into diagnostic mode. This will print out way too much information, but it is the only mode that prints out execution summaries for each project. To get into diagnostics mode, simply go to Tools -> Options -> Projects and Solutions -> Build and Run. From there, you can change the MSBuild verbosity to diagnostic.

12-22-2009 7-44-06 PM
12-22-2009 7-45-08 PM

Now just build (or rebuild). Then copy the build output to a txt file and save it somewhere.

12-22-2009 7-50-52 PM

The only problem with the output is the amount of information (300K lines on a rebuild of a medium size project). It has ton’s of valuable information, but an easy way to process it is needed. The biggest value, in my opinion, is the section at the end of every project where the Summary information about projects, targets, and tasks are displayed. To help retrieve this info, I wrote a quick little Powershell script.

 
param
(
    [string] $file = ""
)
 
if(!(test-path $file)){
    throw "$file not found"
}
 
function accumulate_and_print_project_info(){
    begin{
        $curr_project = ""
        $should_print = $false
    }
    process{
 
 
        if($_ -match "----- (re)?build (all )?started: project: (.*) -----"){
            $curr_project = $matches[3]
            "----- $curr_project -----"
            $should_print = $false
        }elseif($curr_project -and !$should_print -and $_.EndsWith("Summary:")){
            $_
            $should_print = $true
        }elseif($curr_project -and $should_print -and $_ -eq ""){
            $should_print = $false
        }elseif($should_print){
            $_
        }
    }
}
 
gc $file | accumulate_and_print_project_info

The output is going to look something like the following.

----- ControlPanel, Configuration: Development Any CPU -----
Project Performance Summary:
     2672 ms  C:\Users\brian.hartsock\Documents\Visual Studio 2008\Projects\Api\cp3\ControlPanel\ControlPanel.csproj   1 calls
Target Performance Summary:
        0 ms  CreateSatelliteAssemblies                  1 calls
        0 ms  _ComputeNonExistentFileProperty            1 calls
        0 ms  Clean                                      1 calls
        0 ms  _AfterCompileWinFXInternal                 1 calls
        0 ms  BeforeCompile                              1 calls
        0 ms  BeforeClean                                1 calls
        0 ms  AfterBuild                                 1 calls
        0 ms  GetTargetPath                              1 calls
        0 ms  _CheckForCompileOutputs                    1 calls
        0 ms  EntityDeploy                               1 calls
        0 ms  ResolveVCProjectReferences                 1 calls
        0 ms  _SetEmbeddedWin32ManifestProperties        1 calls
        0 ms  StyleCopForceFullAnalysis                  1 calls
        0 ms  CleanPublishFolder                         1 calls
        0 ms  BeforeBuild                                1 calls
        0 ms  CoreResGen                                 1 calls
        0 ms  BeforeRebuild                              1 calls
        0 ms  _CopySourceItemsToOutputDirectory          1 calls
        0 ms  AfterResGen                                1 calls
        0 ms  Build                                      1 calls
        0 ms  SetWin32ManifestProperties                 1 calls
        0 ms  AssignTargetPaths                          1 calls
        0 ms  CoreBuild                                  1 calls
        0 ms  AfterResolveReferences                     1 calls
        0 ms  Compile                                    1 calls
        0 ms  ResGen                                     1 calls
        0 ms  AfterClean                                 1 calls
        0 ms  Rebuild                                    1 calls
        0 ms  PrepareResourceNames                       1 calls
        0 ms  BuildOnlySettings                          1 calls
        0 ms  PrepareResources                           1 calls
        0 ms  CleanReferencedProjects                    1 calls
        0 ms  AfterMarkupCompilePass1                    1 calls
        0 ms  FileClassification                         1 calls
        0 ms  AfterCompile                               1 calls
        0 ms  CompileRdlFiles                            1 calls
        0 ms  BeforeResolveReferences                    1 calls
        0 ms  CreateCustomManifestResourceNames          1 calls
        0 ms  ResolveReferences                          1 calls
        0 ms  DesignTimeMarkupCompilation                1 calls
        0 ms  AfterRebuild                               1 calls
        0 ms  GetReferenceAssemblyPaths                  1 calls
        0 ms  PrepareForRun                              1 calls
        0 ms  AfterCompileWinFX                          1 calls
        0 ms  BeforeResGen                               1 calls
        1 ms  PrepareRdlFiles                            1 calls
        1 ms  GetCopyToOutputDirectoryItems              1 calls
        1 ms  SplitResourcesByCulture                    1 calls
        1 ms  _SplitProjectReferencesByFileExistence     1 calls
        1 ms  _GenerateCompileInputs                     1 calls
        1 ms  _GenerateSatelliteAssemblyInputs           1 calls
        1 ms  GetWinFXPath                               1 calls
        1 ms  EntityClean                                1 calls
        1 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
        1 ms  ResolveProjectReferences                   1 calls
        1 ms  PrepareForBuild                            1 calls
        3 ms  GetFrameworkPaths                          1 calls
        8 ms  CoreClean                                  1 calls
       14 ms  IncrementalClean                           1 calls
       17 ms  CopyFilesToOutputDirectory                 1 calls
       17 ms  _CleanGetCurrentAndPriorFileWrites         1 calls
       66 ms  ResolveAssemblyReferences                  1 calls
      399 ms  _CopyFilesMarkedCopyLocal                  1 calls
      519 ms  StyleCop                                   1 calls
      636 ms  CoreCompile                                1 calls
      974 ms  PostBuildEvent                             1 calls
Task Performance Summary:
        0 ms  Delete                                     2 calls
        0 ms  Message                                    6 calls
        0 ms  EntityDeploy                               1 calls
        0 ms  GetFrameworkSdkPath                        1 calls
        0 ms  CreateProperty                             1 calls
        0 ms  AssignTargetPath                           5 calls
        0 ms  MakeDir                                    2 calls
        1 ms  ConvertToAbsolutePath                      1 calls
        1 ms  AssignCulture                              1 calls
        1 ms  GetWinFXPath                               1 calls
        1 ms  FindAppConfigFile                          1 calls
        1 ms  ReadLinesFromFile                          2 calls
        1 ms  EntityClean                                1 calls
        2 ms  CreateItem                                 3 calls
        2 ms  WriteLinesToFile                           2 calls
        3 ms  GetFrameworkPath                           2 calls
       13 ms  RemoveDuplicates                           3 calls
       20 ms  FindUnderPath                              7 calls
       66 ms  ResolveAssemblyReference                   1 calls
      415 ms  Copy                                       6 calls
      515 ms  StyleCopTask                               1 calls
      635 ms  Csc                                        1 calls
      973 ms  Exec                                       1 calls

Now you should have all the data you need to figure out what is taking so long to build your project.

Find Orphaned Solution Files

Each day, I am becoming more and more OCD. Tonight I wrote a script that will analyze a Visual Studio csproj file and compare it to the files in the directory. The main goal being an easy way to find orphaned files that might still be in version control. It probably only works on very vanilla projects, but I figured I would share anyways.

# find-orphanedFiles.ps1
#
# Compare files in a Visual Studio project file to the actual files in the directory
#
# Usage:
# .\find-orphanedFiles.ps1 Project\FooBar.csproj
# .\find-orphanedFiles.ps1 Project\FooBar.csproj -exclude "*.csproj","*.csproj.user","svn.ignore"
 
param
(
    $csProjFile,
    $exclude = ("*.csproj*")
)
 
function custom-gci($path=(get-location)){
    gci $path -exclude $exclude | 
    %{
        if($_.PSIsContainer -and ($_.name -ne "bin" -and $_.name -ne "obj")){
            custom-gci $_
        }elseif(!($_.PSIsContainer)){
            $_
        }
    } 
}
 
if(!(test-path $csProjFile)){
    throw "File $csprojFile does not exist"
}
 
$dir = split-path $csProjFile
$csProjfile = split-path $csProjFile -leaf
 
push-location $dir
 
$x = [xml] (gc $csProjFile)
 
$filesInProject = $x.Project.ItemGroup | 
    foreach {
        if($_.Compile) { 
            $_.Compile
        }elseif($_.None){
            $_.None
        }elseif($_.Content){
            $_.Content
        }elseif($_.EmbeddedResource){
            $_.EmbeddedResource
        }
    } | 
    select -expand include | 
    resolve-path | 
    gi | 
    select -expand fullname
 
$filesInDir = custom-gci |
    select -expand fullname
 
compare-object $filesInProject $filesInDir
 
pop-location

Restoring Transaction Logs in Powershell

Using the SQL Server UI to restore a bunch of transaction logs is the most painful thing I have every done. I created this script, and then tested restoring around 10,000 transaction logs and burnt through them pretty fast.

Create the script restore-transactionLog.ps1 with the following code. (I am using integrated auth and running the script on localhost. It wouldn’t be too hard to use different connection strings in the code if you wanted)

param
(
    [string] $dbName
)
 
process {
 
    if(test-path $_){
        $sql = "RESTORE LOG [$dbName] FROM  DISK = N'$($_.fullName)' 
                WITH  FILE = 1,  NORECOVERY,  NOUNLOAD,  STATS = 10"
 
        write-verbose $sql
 
        sqlcmd  -Q "$sql" -E
 
    }else{
        write-error "Can't find file $_"
    }
}

Then you can use it pretty easily in the following scenario.

gci c:\MyTransactionLogDir | 
    sort LastWriteTime | 
    .\restore-transactionLog.ps1 "myDb"