In this blog post I'm going to describe how you can use a Setup and Deployment project in Visual Studio 2005, along with a few other tools, to create a single, self-extracting executable to install your managed software and prerequisites. I'm also going to provide a solution to one of the problems that I encountered when I recently attempted to create a self-extracting installer for one of my open source projects.

Introduction

Using a Setup and Deployment Project in Visual Studio, you can easily build a single Windows Installer file (.msi) that allows you to redistribute your custom software to your end-users, providing an automated and familiar setup experience. If your software depends on other software such as the Microsoft .NET Framework 2.0, as it probably will, then you can install them as prerequisites by having your Setup and Deployment project create a Generic Bootstrapper executable - Setup.exe. End-users should execute the bootstrapper instead of the .msi so that it will discover which dependencies are missing on the target machine, download and install them. The bootstrapper will then automatically run the .msi to install your software as long as the prerequisite installations were successful.

You can also develop custom Generic Bootstrapper packages and have them show up in Visual Studio's Prerequisites dialog, along with predefined packages such as the Microsoft .NET Framework 2.0 and Windows Installer 3.1. Custom packages can be configured to download the prerequisites from a website or they can be deployed along with the bootstrapper executable and your project's .msi file in their own sub-directories, which is extremely useful if the prerequisites are not available for download from some third-party website and if you don't have the means or desire to host them on your own web server.

User Experience

Having a bootstrapper to automatically install your software's dependencies is obviously useful, but your end-users will probably expect a single file to install your software and might be surprised to discover that they must download a Setup.exe file and an .msi file. You may also be tempted to deploy a Readme.txt file along with the installers to instruct users to run the Setup.exe program instead of the .msi. If you have added custom prerequisites to your bootstrapper then you may have to deploy them along with the installers as well if they aren't available for download from a web server. But since your end-users aren't going to be happy having to download several different files and then manually create the appropriate folder structure expected by Setup.exe, you're forced to zip all of the files and sub-directories together. An end-user must unzip the files, browse to the target directory in Windows Explorer and then double-click the Setup.exe file to start the installation.

All-In-One Deployment

So how about deploying a single, self-extracting executable that can dump all of the files into a temporary directory and then run the Setup.exe program automatically? There are several commercial products that can produce self-extracting zip files, but if you're like me you'd prefer a no-cost solution for something as simple, practical, and as seemingly common as having a single executable to install custom software.

Enter IExpress. Pre-installed on Windows systems, this program can create a self-extracting installer by compressing the files of your choice and may be configured to execute your Setup.exe bootstrapper upon extraction. The result can be distributed as a single, stand-alone file that contains the bootstrapper executable, any prerequisites that must be deployed along with your installer, and the .msi installer itself. That's right, pre-installed on Windows!

Start > Run > iexpress

The Directory Structure Problem

However, this amazing tool doesn't come without some pain. IExpress 2.0, the current version that ships with Windows Vista, doesn't preserve the folder structure of the files that it compresses. But the Setup.exe bootstrapper program is automatically built to look for each prerequisite that is not already being downloaded from a web server, in a local folder of the same name as the prerequisite itself. This obviously presents a problem when using IExpress since it will extract each prerequisite installer into the same temporary directory as the Setup.exe bootstrapper. Running a simple test reveals that the Setup.exe bootstrapper complains that it cannot locate one or more of the prerequisite installers.

But have no fear, there is a way. In order to better explain the process of using IExpress to extract and run the Setup.exe bootstrapper program with local file dependencies I'm going to use an example.

DocProject

DocProject is software that I've written to integrate Microsoft's managed API documenter, Sandcastle, into Visual Studio 2005. Up until the 1.6.0 Release Candidate of DocProject its installer shipped as a stand-alone Windows Installer file (.msi). But as of 1.6.0, a few of Microsoft's Primary Interop Assemblies for Visual Studio have become a prerequisite (namely, Microsoft.mshtml and stdole), which means that vs_piaredist.exe must be deployed along with the .msi. In order to ensure that end-users have Microsoft's redistributable assemblies installed on their systems before the DocProject installer is executed, I've create a Generic Bootstrapper package for vs_piaredist.exe that is required by DocProject's Setup and Deployment project.

As described in the introduction, DocProject's Setup and Deployment project builds a Setup.exe bootstrapper that checks for the prerequisites on the target machine and then, if the prerequisites aren't found, executes the vs_piaredist.exe file to install them. After the prerequisites are installed or skipped, the Setup.exe program then runs the .msi to install DocProject on the user's system.

When researching DocProject's prerequisites I discovered that Microsoft does not provide vs_piaredist.exe over the web, which means that I could have either zipped it with DocProject's installer, hosted it on my own web server, or figured out a way to include it into IExpress. I chose the latter since it requires the least amount of effort on mine and the user's part, but wasn't sure how it could work or if it was even possible at all.

Solution to the Directory Structure Problem

The Setup.exe bootstrapper is actually a generic executable that has a few null resources. When you build a Setup and Deployment project that has prerequisites, the MSBuild GenericBootstrapper Task is used to make a copy of the generic Setup.exe program. The copy is then loaded with a resource file that tells the bootstrapper how to install your program's prerequisites. Unfortunately, the MSBuild task doesn't provide the option to have the configuration resource use prerequisite installers found in the target directory, so you must manually update the appropriate resource file to remove the hard-coded path that looks for prerequisites in a sub-directory of the same name.

Sounds scary, but it's actually quite easy to do. Here are the instructions to create DocProject's installer:

1. Build the Setup and Deployment project.
2. Open the Setup.exe program in Visual Studio's resource editor:
   1. File > Open > File (or Ctrl+O)
   2. Browse to the bin directory of the Setup and Deployment project and you'll find the Setup.exe file in the folder for the current configuration (e.g., Debug or Release), along with the other installer output.
3. Double-click the resource named, SETUPCFG in the 41 folder
4. Search for vs_piaredist\, exactly like that - with the trailing slash. Two (2) occurrences should be found. For each, make sure the entire vs_piaredist\ string is highlighted and press the Delete key to remove it from the resource.
5. Save the resource file and the Setup.exe executable will be updated automatically.
6. Start > Run > iexpress
7. Create a new package by following the IExpress wizard's steps and make sure to include the following files:
   1. The Windows Installer file (.msi).
   2. The Setup.exe bootstrapper file.
   3. The vs_piaredist.exe file.
8. Note: You should select the Store files using Long File Name inside Package option in one of the last wizard steps.

Step #4 is how to reconfigure the Setup.exe bootstrapper to look in the current directory for vs_piaredist.exe instead of in a subdirectory of the same name. We do this by simply removing the folder from the path in two different locations within the same resource file.

You should repeat step #4 for each of the local prerequisites that your Setup.exe bootstrapper will install, but instead of vs_piaredist\ use the name of the folder as it appears in your Setup and Deployment project's bin directory.

Conclusion

Visual Studio's Setup and Deployment project, the Generic Bootstrapper and IExpress are powerful tools that you can use to create installers for your custom software.  Using IExpress to build a self-extracting installer that automatically runs your software's Setup.exe bootstrapper requires additional steps if the bootstrapper depends on prerequisite installers that are deployed with your software. By using Visual Studio's resource editor you can modify the Setup.exe file to look for prerequisites in the current directory.  This allows you to build a self-extracting installer that contains all of your software's depencencies, providing a nice installation experience for your end-users.

In this blog post you'll see and read about some of the new features of DocProject that will be released with the 1.6.0 Release Candidate. But first, I'd just like to say thanks to everyone who downloaded DocProject and provided feedback. I appreciate it. Please keep the comments and feedback coming!

If you're only interested in screen shots – scroll down :)

1.6.0 Changes

This is one of the biggest releases yet, IMO. Much of the code-base has been refactored to support new features and to improve existing ones. Changes were made not only to the source code, but also to the solution and project files, and to almost every aspect of DocProject. There were also several new additions to the code-base in the form of source code and new projects. But don't worry; the end-user experience has not changed other than to include new features and improve existing ones. And the source code, for the most part, is still basically the same. Your existing knowledge of how to use DocProject will still apply, but it will no longer be the whole story.

Please note that existing DocProjects and DocSites may no longer work as expected. You are encouraged (if not required) to create new documentation projects again by removing existing ones and using the New Project Wizard to recreate them. I have not tested older projects with DocProject 1.6.0, but I suspect that they will not be able to complete a help-build.

Solution and Projects

The first change to the solution was to start using solution folders to categorize the projects. The Sandcastle code-base was then pulled out of the Add-In project (which has been renamed) and included in a new project named, DaveSexton.DocProject.Sandcastle, which appears under the Plug-Ins solution folder along with the Sandcastle/Deployment project. There is also a new project named, DaveSexton.Controls.HtmlEditor. I'll leave that one alone for now and let you figure out its purpose for yourself ;) There have also been several changes to the InstallPrep and Installer projects, although existing functionality has not really been removed or altered, for the most part. The preparation process is still the same as it was before, but with some additional files and requirements.

Building the Installer

If you're planning on building the Installer project then the only concern you should have is that the HtmlEditor project requires the Microsoft.mshtml PIA to be installed on the target machine, so the installer requires a bootstrapper to be built (Setup.exe) that includes the redistributable vs_piaredist.exe file, which installs the required dependencies. In order to include MSHtml in the bootstrapper I created a prerequisite bootstrapper package that is included in the InstallPrep project under the Bootstrapper folder, the contents of which should be copied to Visual Studio's bootstrapper directory manually. The Readme.txt file that appears in the same folder explains its use and also provides import legal information that you must agree to before using DocProject's source code. The agreement is also presented in a dialog when DocProject is installed, but only if the required dependencies must be installed on your system. You should read this file before attempting to build and redistribute the installer and the source code.

Although I haven't tested yet whether building the installer will fail without having the MSHtml bootstrapper package present on your system, I assume that it will. I'll test this to make sure and update the appropriate wikis on CodePlex to indicate end-user requirements to build the installer using the source code, but for now just assume that you must follow the instructions in the Readme.txt file.

1.6.0 Features

Here's a preview of some of the new features that you can expect in DocProject 1.6.0 RC. I'm still testing some of them on Vista, XP, and Windows Server 2003, but things are going well. I expect to deploy by Monday June 4, 2007.

Installation

The installer will now detect the following versions of Visual Studio on your system and automatically install the templates and Add-In, where appropriate, for both Just Me and Everyone installations. Having more than one version of Visual Studio on your system at the same time is acceptable since they can all share the same templates or have their own copies if you install for everyone:

• Visual Studio 2005 Standard+
  • The Add-In and all templates are installed.
• Visual C# 2005 Express
  • Only the C# Project Documentation template is installed.
• Visual Basic 2005 Express
  • Only the VB.NET Project Documentation template is installed.

Note: Currently, the DocSite templates will not work in Visual Web Developer 2005 Express.

The DocProject installation directory will always have the same contents regardless of what versions of Visual Studio are on your system when you install DocProject.  The only variance based on the different versions of Visual Studio are updates to the registry and the location where the Add-In and/or templates are copied.

There is only limited support for express editions, which I'll detail in the release notes for the 1.6.0 RC. Basically, express editions get the New Project Wizard and use the MSBuild task, but projects in express editions do not have Add-In-specific features such as the Include Project Output Dialog and the Tools > Options > DocProject pages. Users can still configure DocProject by editing the project file manually. However, Sandcastle is still configurable in the same way as the full version of DocProject; i.e., by editing files in the Presentation folder inside Visual Studio. Projects are also built in the same way, with support for a build process component as well.

Visual C++ Source Projects

DocProject now provides support for Visual C++ projects as sources for DocProjects and DocSites. Reference Visual C++ projects as you would any other project reference and their APIs will be included in the compiled help.

Html Files Names: GUID or Friendly

By default, the Sandcastle Build Assembler utility, which generates the HTML topic files, is now configured to use GUID file names instead of friendly names that corresponded to the names of the API topics, as used previously. This allows projects that have long, nested directory structures to build without throwing a System.IO.PathTooLongException (special thanks to the people who pointed this out to me :). You can change a setting to use the friendly names if you want: Tools > Options > DocProject > Active Projects > Build > Use friendly HTML file names (disabled, by default).

MSBuild Task and Command-Line Support

The template files for DocProjects and DocSites have been refactored to invoke a custom target named, BuildHelp, which references the DaveSexton.DocProject.targets file.  The targets file, located in DocProject's bin folder, starts the assembly-build process and then, afterwards, starts the help-buld process, which is executed by the Add-In in much the same way that it has in previous versions of DocProject.

To support this feature the installer creates an environment variable named, DocProjectBuildPath, which points to DocProject's bin directory. The environment variable is created with user-scope if you select Just Me in the installer and system-scope if you select Everyone.

The MSBuild task simply delegates the help-build process to the DocProject Add-In, which has been refactored so that it can execute outside of the Visual Studio IDE as well as inside. In other words, MSBuild is now used to start the build process and DocProject's Add-In takes over, using the appropriate method depending upon whether it has been initialized as a Visual Studio Add-In or simply invoked out-of-process by the MSBuild engine. This means that the Add-In will build your project files when they are used as Visual Studio projects, as they always have been, but also on build servers that use MSBuild - without running Visual Studio. No changes to the DocProject or DocSite project files are required for them to be buildable in any environment.

You can build DocProjects and DocSites on the command-line by simply running MSBuild and specifying the project file as an argument. For example, open the Visual Studio Command Prompt and enter the following to build a C# DocProject named, "your project":

msbuild.exe "c:\your project.csproj"

Note: Visual C++ projects must be built by the solution file. MSBuild will warn you about this if you attempt to build DocProjects or DocSites that reference a Visual C++ project. In that case, just pass the solution file to msbuild.exe.

Your DocProjects and DocSites can be specified as targets in other MSBuild project files on a build server as long as DocProject has been installed on the server (or at least with the DocProject assemblies and appropriate configuration present, even if DocProject is not actually installed). Make sure that the relative references to other projects in the DocProject or DocSite project file are valid.

New Project Wizard Header

The first change that you'll notice when creating a new DocProject or DocSite is in the New Project Wizard's header, modified slightly just to keep things interesting ;) (See Figure 1 below)

New Project Wizard – Choose a Sandcastle Presentation

The first useful change that you'll notice in the New Project Wizard is that the Choose a Sandcastle Presentation step provides readable names and a short description of the presentations that are available. (See Figure 1 below)

The data is located in DocProject's configuration file (commonly found at, C:\Program Files\Dave Sexton\DocProject\bin\DaveSexton.DocProject.dll.config) in the new presentation configuration section. This section describes the presentations that are installed with Sandcastle, indicating a human-readable name and description, and providing the locations to all of the tools and directories that DocProject expects. There is also a place to add custom Regular Expression and XPath expressions that will be used by DocProject to automatically transform the imported Build Assembler configuration file (sandcastle.config) when creating a new DocProject or DocSite. You can modify, remove or add new transformations to customize the default configuration files as they are imported into new projects.

Note: You can use the new presentation configuration section to add your own custom presentations too!

Figure 1: New Project Wizard - Choose a Sandcastle Presentation

As you can see from the screen-shot, the wizard looks a little bit different now. Other than the slight change to appearance and the new pre-load behavior, it functions identically to earlier versions of DocProject.

New Project Wizard – Create Shared Content

The next feature is the ability to create a header and footer using an Html Editor. The header and footer content are shared by all API topics and can be found in your project's Presentation\Content\shared_content.xml file, which you may edit manually as well. I'm also considering whether to add more sections that would correspond with other items in the shared_content.xml file, for a future version of DocProject.

Figure 2: New Project Wizard - Create Shared Content
Figure 2.1: Html Editor Context Menu

Notice in Figure 2 that the Header and Footer sections may be toggled between visible and hidden states by clicking their respective title bars. The box next to the title will contain a plus sign (+) when a section is collapsed and a minus sign (-) when a section is expanded.

The header and footer may be edited at a later time using the API Topic Designer, which is located at Tools > Options > DocProject > Active Projects > Content > API Topic Designer. Locate this setting for your project and click the ellipses button to open the editor, which is identical to the editor in the Create Shared Content wizard step, but without the wizard's chrome.

The Html Editor that is used here may be the beginning of a topic editor control as well; although I'd much rather use Visual Studio's HTML editor instead of reinventing the wheel, which would not be easy. One of the reasons that I wanted to use Sandcastle inside Visual Studio in the first place is so I could use its HTML and XML editors, but more on this discussion another time :)

API Topic Management Dialog

The API Topic Management dialog provides some new options for filtering API topics based on categories and regular expressions, such as the ability to find the next match only and to specify whether the dialog should automatically show matched topics in the TOC (i.e., if a matched node is hidden then its parent nodes will be expanded until it becomes visible, and then it will be scrolled into view).

Figure 3: API Topic Management Dialog – Topic Filters

The TOC is loaded on-demand now, by initially loading only the root topics and then loading the children of each individual topic as they are manually expanded for the first time. This saves some initialization time, but may increase the time that it takes for the filters to work. For this reason, the filters now execute asynchronously and the find next match only option is selected by default.

The API Topic Management dialog also provides support for editing project and namespace summaries in text mode (the contents of which must be valid xml just like with code comments) or using an Html Editor, as in Figure 4 below. You can freely switch back and forth between text mode and html mode and your changes will be persisted.

The Summary Text and Summary HTML tabs are only activated when the project node is selected, as in Figure 4, or when one of the namespace nodes is selected. For example, the Contoso namespace that appears in Figure 4 can be selected in order to edit its summary. If the AppConfigElement node is selected, however, the summary editor will be disabled.

Figure 4: API Topic Management Dialog – Summary HTML Editor

You can open this dialog by locating Tools > Options > DocProject > Active Projects > Configuration > API Topic Management and clicking the ellipses button. It's in the same place that it was in 1.5.0 :)

And More…

There are other features that I haven't discussed here, but I think the ones that I've mentioned are probably the most important.

I hope you're excited to use these new features, since I worked hard to create them. I'd love to know what you think about them and your ideas for other features, so please use the contact form here or on CodePlex, or leave a comment on this blog post and let me know what's on your mind.

Thanks!

Recently, a few people have mentioned that they'd like support in DocProject for generating custom topics that maintain the look and feel of the selected presentation. While designing DocProject I had thought about this and tried to provide an easy way for developers to create custom topics, which is why the partial build functionality was developed. But until now I haven't mentioned much about this feature or how it works.

I just added a tutorial to DocProject on CodePlex, which explains how to create custom topics using the partial build feature:

Tutorial: Creating Custom Topics
http://www.codeplex.com/DocProject/Wiki/View.aspx?title=Creating+Custom+Topics

The tutorial explains how to create an HTML topic template from one of the files that Sandcastle generates (namely, R_Project.htm). It also describes how you can use Visual Studio's HTML designer to generate your own topic files based on the template and how to include them in the Html Help Workshop table of contents (TOC). Code is also provided that extends the default build process component to automate some of the steps in the tutorial.

Enjoy, and let me know what you think :)

In this blog entry I'm going to show a preview of one of the new features that I've included in DocProject for the next release. I'll also write a little about my open-source development habits, how I test DocProject and my plans for DocProject in the future.

But before I begin I'd like to thank everyone that has downloaded DocProject and provided feedback. It's been helpful. Please keep adding work items and starting discussions so that I can continue to be informed of the community's needs.

Those of us that use Visual Studio 2005 do so for different purposes and in different manners, and there's no way that I could test DocProject in every possible situation on my own, so I'm relying on the community to let me know when I've missed a bug in testing and when there's room for improvement. And I'm not just looking for constructive criticism either, so if you have something nice to say about DocProject then please don't be shy :) It's also useful for me to know that people are actually using DocProject and that it's working to meet their needs.

My Multi-Project Open Source Development Life-cycle

I work weeknights and weekends on the planning and development of my open source projects and my proprietary software. I normally don't work from beginning to end on any particular release, but instead jump back and forth between projects. Usually, a context switch occurs when an idea pops into my head or if I get bored with what I'm doing. That's the beauty of working alone and on personal software. I'm currently working full-time on a desktop application for a client so I don't do much coding on personal projects during daylight hours; however, I do try to answer questions and provide help to the community in a timely manner and, from time-to-time, I'll do some wiki-work mid-day. Working from home has its perks ;)

Recently, I published AIP 1.0.0 RTW to CodePlex and now I'm switching gears to finish work on DocProject for the April 30^th deployment. I've fixed all of the bugs and added all of the features that have associated work items, including a few that don't, and now I'm on to stabilization.

Stabilizing DocProject

Stabilizing DocProject isn't so easy. Because it's an Add-In I have to make sure that every feature works in Visual Studio 2005, in many different contexts. Of course, I must also ensure that regression issues don't spring up.

I didn't write unit tests when DocProject was first conceived, but I'm actually glad now. Unit tests aren't going to work for a very complex Add-In that is hosted by Visual Studio 2005 because I won't be able to properly emulate VS in any mock object, IMO. And even if I could it wouldn't give me peace of mind. Having a script to automate Visual Studio might be useful, but I don't think it's worth the effort to create one.

I test DocProject by installing it on my Vista box, which is also my development box, and then I create new DocProjects and DocSites from scratch, testing each of the features as I go. I also test DocProject on a Windows Server 2003 VM. To be perfectly honest though, I don't test all of the features each time. There are just too many features to test; although, I usually don't make code changes that will affect every bit. I guess that's another benefit of working alone: I know what I did and usually have a good idea of what components it may have affected. The components that aren't touched at all, for which I'm sure, I'll skip testing.

Thankfully, I haven't run into many regression issues. I think that's due, in part, to the amount of refactoring that I did in previous versions before I added several of the major features that are in DocProject now. There's still certainly a lot of room for refactoring and once I publish the first RTW I think I may revisit those classes for which I made a mental note. Actually, I'll probably review the entire code-base from scratch and see if I can re-architect DocProject for some of my future plans (more on them below).

But for now, I'm going to continue stabilizing DocProject until I'm ready to release 1.5.0, by April 30^th. (BTW, I try to schedule deployments on a Monday so that I have an entire weekend prior to really get down and dirty with testing, last-minute tweaks and writing wikis.)

API Topic Management Dialog - Preview

Here's a preview of a new feature that I've added to DocProject. It's a dialog that allows a user to configure the apiFilter configuration element in Sandcastle's MRefBuilder configuration file, which is now included as a project item in the Presentation\Configuration folder. You can also edit the configuration file manually without tripping up the dialog, as long as your changes conform to the xml schema that Sandcastle requires, of course.

An API topic is included by checking its check box node in the tree view control and excluded by unchecking it. All topics are included (checked) by default if the apiFilter configuration section hasn't been used.

The dialog is accessed in the Tools Options Page for Active Projects by clicking the ellipses button of the Configuration > API Topic Management property. Notice in the image below that another new property has been added as well, MRefBuilder Configuration File Name, which specifies the name of the config file in the Presentation\Configuration directory to be used with the Sandcastle MRefBuilder program.

Figure 1: Finding the API Topic Management Dialog

Figure 2: The API Topic Management Dialog

Regular Expression Filter

Use the regular expression filter to include, exclude or locate topics by name using a managed-style regular expression. A link to an MSDN help reference on regular expressions is provided next to a link that pops-up a regular expression quick-help guide, which includes a few examples. You can also choose the regular expression's options.

The Find button will locate all matching topics by highlighting them.

The Apply button will locate all matching topics, highlight each of them, and ensure that the state of each check box matches the state of the Include matching topics check box.

An icon will appear next to the Find button after either button is clicked. The icon's tooltip displays the number of matching topics. The tooltip is automatically displayed for one and a half seconds after the search has completed. Clicking the icon repeatedly will cycle through all of the matching topics, selecting the current node and expanding nodes when necessary. The highlighted color and the icon's appearance differ depending upon whether you have clicked Find or Apply. Find was clicked in Figure 2 and the blue highlights in the tree view are the matching topics.

Clicking the icon to cycle through the matching topics can be useful. Use the spacebar to toggle (check or uncheck) a matching topic and click the icon to cycle to the next match. Using the icon and the Find button instead of the Apply button will allow you to manually choose from all of the matched topics the ones that should be toggled.

Category Filter

Use the category filter to include or exclude topics by API category. For instance, to include all interfaces you would first check the Interfaces checkbox and ensure that no other category is checked. Then make sure that the Include matching topics checkbox is checked to include matching topics (unchecking it will exclude matching topics). Press the Apply button to apply the changes.

Like the regular expression filter, an icon will appear next to the Apply button after it's clicked. The icon's tooltip displays the number of matching topics. The tooltip is automatically displayed for one and a half seconds after the search has completed. Clicking the icon repeatedly will cycle through all of the matching topics, selecting the current node and expanding nodes when necessary.

How It All Works

When you open the dialog it takes some time to fill the tree view, but the dialog remains responsive to user interaction and displays a marquee progress bar in the mean time. In the background, DocProject is actually using the Sandcastle build engine to run a build of the TOC only (the engine was refactored a bit to provide this new functionality).

Basically, all of the steps up to the Build Assembler are executed in the background.  The toc.xml and reflection.xml files that are created by the the Sandcastle build steps are then parsed into the tree view nodes.  The toc.xml file is used for the layout of the tree view and the reflection.xml file provides most of the information used to construct each topic node.  And for each topic node a look-up into the existing apiFilter configuration section of the Presentation\Configuration\MRefBuilder.config file is performed to discover whether the topic should be included or excluded (checked or unchecked, respectively), by default.

A temporary copy of each node is made at the time they are created.  When you modify the tree view, the copy of each node is preserved for the OK and Cancel functionality of the dialog.

Clicking OK on the dialog doesn't apply the changes, however. You must click OK on Visual Studio's Options dialog to commit the changes. This allows you to edit the API Topic Management property multiple times before you finally save your changes. If you click OK on the property's dialog and then reopen it, modify something, and click Cancel, then only the changes that you made since the last time the dialog was opened will be canceled. The original changes for which you clicked OK last will be retained.

When you finally save the changes by clicking OK on the Options dialog, they are written directly to the MRefBuilder.config file, overwriting any existing configuration of the apiFilter element. Other existing elements and content in the configuration file is left unmodified.

One stipulation to use this feature is that you must build all of the source projects once before the dialog will display their topics. If you only build some of the source projects then the dialog will only display those API topics. If you clean the solution or otherwise delete any of the output assemblies from the source projects then the corresponding API topics will not be displayed. In other words, the source assemblies have to be present in order for the dialog to display their topics. Showing the dialog does not automatically build all of the source projects so you must build the projects before using the dialog. If you close Visual Studio after building a source project and then reopen it, the assembly will still be there so you won't need to build the project again to use the API Topic Management dialog for a referencing DocProject or DocSite.

Performance

The MRefBuilder utility is very quick, even for the 13-assembly library that I use to test DocProject's performance. But the dialog, for the same library, does take a considerable amount of time to fill the tree view, and that's what the dialog is doing for the majority of the time that you wait for it to be ready. In testing it took about 5 minutes for the dialog to be ready for 2715 API topics. However, that number only represents API  topics, not other topics such as All Members, so the number presented as the API Topic Management property value is somewhat lower than the total number of topics that end up in the compiled help.

I already see room for improvement in the code base to increase the performance of the dialog, and I was even thinking about possibly adding nodes-on-demand functionality, but I'm going to save those changes for a subsequent release. The idea behind the on-demand behavior, BTW, is that expanding a node for the first time will cause its children to be loaded at that time instead of all nodes being loaded when the dialog opens. However, I have some doubts if that would be useful since the filters will require all nodes to be loaded before they will function properly. Obviously, I still have some things to work out for that feature (which is why I'm not adding it now).

To improve the performance I plan on using an XmlReader to parse the topics instead of an XmlDocument and I plan on using a lot more custom iterators in C# too (i.e., the yield statement).

The performance of the tree view and the filters after all of the topics have been loaded is actually quite good; as is the process that writes the changes to disc when you click OK on the Options dialog.

DocProject's Past, Present and Future

DocProject hasn't been around that long but it has accumulated over 800 downloads between all of its versions, up to the 1.4.0 RC. Currently, DocProject is downloaded about 10 times per day, on average.

I have some plans for the future of DocProject, feature-wise, but nothing is set in stone. Feedback is a must for me. Please let me know what features you want to see in DocProject and I will consider them.

Why doesn't DocProject provide any content-based features?

Currently, DocProject is built for automating Sandcastle from within Visual Studio 2005, with extensibility built-in to provide an open-ended environment for developers to control the help-build process. All of the Sandcastle presentation files (xml transformations and configuration files) are included in DocProjects and DocSites so that users can configure Sandcastle the way they want without being restricted to GUIs. In other words, DocProject is not focused on content-based features. DocProject relies on Sandcastle and you, the user, to develop and configure your own documentation. This has always been the most important guideline that I followed since I started developing the first beta.

Recently I received a request from a user to include some GUI support for things such as adding namespace summaries and customizing the default copyright, header and footer of the HTML topic files generated by Sandcastle (see work item #9717 and vote if you're interested). Although I have considered adding these features in past releases, I always ended up siding with the idea that DocProject is better off without content-related features, for now. What I mean is that DocProject has always been geared toward making it easier to build documentation using Sandcastle and for users to be able to develop their documentation using Sandcastle, directly. I've purposefully avoided creating interfaces for users to manage the HTML topic files that Sandcastle generates in favor of simply including the xml transformations and configuration files as project items so that users could configure Sandcastle themselves. The reasons are simple:

• Sandcastle is not RTW yet and there are changes between CTP releases, naturally.
• I consider adding content-based features a major version change from not having any content-based features and I'd rather wait for the first DocProject RTW to correspond with the first Sandcastle RTW before any major changes are implemented in DocProject.
• I didn't want to be bogged down with presentation-related issues while I was developing and stabilizing DocProject's feature-base for building documentation, which I consider to be more important now since all Sandcastle files are visible to users and are not modified by DocProject in any way. So users are completely free to configure Sandcastle however they want and I don't have to worry about supporting end-user/DocProject configuration mistakes or issues with the appearance of HTML files that Sandcastle generates.
• It seems a bit ridiculous to me for DocProject to offer a textbox so users could edit an HTML header, for example, inside of the Visual Studio 2005 IDE when they can use the XML editor to edit the entire shared content XML file.
• I can't help but think that it should be all or nothing. A topic designer must be included or else DocProject should simply rely on Sandcastle and DocProject users to produce the documentation, as it currently does. Without a topic designer, any UI support for presentation would just be a hack, IMO.

Planned Features

I plan to research some content-based features, including the ability to add namespace summaries and maybe even a basic topic designer for the next major release of DocProject.

In preparation for the next release of Sandcastle, which I read will include a new presentation design (A. Raman, MSDN Forums, Apr 20, 2007; Last Post), I've already begun to find places in the code base that I want to refactor. I'm also going to start testing DocProject in the Orcas CTPs so that it will be ready for Orcas documentation, as long as Sandcastle supports it.

I'd also like to add a command-line build feature (work item #9788) and an MSBuild task (work item #9787). I've already thought about how that will work and started refactoring the code base in 1.5.0 in preparation for the changes required to support these features. Some of the changes were included in 1.5.0 simply because they were easy enough to do.

I've begun to create a command-line executable named, docop (DocProject Operator) that will eventually automate the build process of a DocProject or DocSite outside of the Visual Studio 2005 IDE. The plan is to run the MSBuild task using the MSBuild API, which I started playing around with last weekend.

If you are interested in any of the features that I mentioned then please follow the corresponding link and vote on the work item in CodePlex. Thanks!

I've just published 2 open source deployments in one month, one of which is a brand new project: Auto-Input Protection (AIP). Both deployments are available on CodePlex.

DocProject 1.4.0 Release Candidate Is Now Available

DocProject drives the Sandcastle help generation tools using the power of Visual Studio 2005. Choose from various project templates that build compiled help version 1.* or 2.* for all project references. DocProject facilitates the administration and development of project documentation with Sandcastle, allowing you to use the integrated tools of Visual Studio 2005 to customize Sandcastle's output.

Download DocProject from CodePlex, which comes with the complete source code (C#). Try it out and let me know what you think.

Here are some of the new features in 1.4.0:

• DocProject's Sandcastle build engine now uses the Sandcastle March CTP transformations.
• DocProjects and DocSites can now build Html Help 1.*, Help 2.*, neither or both via a single build.
• The New Project wizard has been extended for the Sandcastle build engine, allowing you to choose the type of help that your project will build (see previous feature).
• DocSites use AJAX to improve client-side performance.
• DocSite's index can be filtered, client-side.
• The tools options page, "Dave Sexton's Tools", has been renamed to "DocProject" and was split into separate sub categories.
• A status notification and progress bar now appear during help builds.
• Improved error handling, including logging to the Application event log.
• Several bug fixes.

Use a DocSite template to build an AJAX-enabled ASP.NET Web Application for your compiled help, which includes an interactive TOC, filterable index, breadcrumbs, an auto-generated header and footer, and a link to download the compiled help file. The auto-generated website has been tested for compatibility with the IE7, Firefox and Opera web browsers only.

Figure 1: Example DocSite shown in Internet Explorer 7 on Windows Vista; Sandcastle vs2005 presentation.

Auto-Input Protection Beta 1 for ASP.NET Is Now Available

AIP is an extensible ASP.NET web control that provides CAPTCHA protection for your blogs, forums, wikis and websites, greatly reducing the likelihood of unwanted form submission from automated spam and hacks.

Want to see it in action? Look no further than in the comments section of my own blog posts. :)

Download AIP from CodePlex, which comes with the complete source code (C#). Try it out and let me know what you think.

Add the AIP web control to a web page with full designer support and only a few modifications to your web.config file:

Modify the AIP web control's presentation with a custom template and modify the CAPTCHA image that it generates by changing the bitmap and filter providers' settings in your web.config file:

Easily implement custom CAPTCHA algorithms that produce randomized patterns with a custom bitmap provider and filter providers:

This post isn't exactly related to programming; nevertheless it does offer something that may be of interest to programmers: science, video and a break from work.

NOVA, The Elegant Universe

I just stumbled across some videos on the web that I really enjoyed and I highly recommend taking some time out of your busy schedule to view them. The videos are part of a short series on PBS called, The Elegant Universe [1], which is available on the web for free. The series is really entertaining and quite informative, with beautifully rendered 3D visuals to guide you through some complicated theories and facts relating to an ultimate theory of everything (TOE), such as the general theory of relativity, quantum mechanics and string theory. String theory, a TOE for which some scientists are hopeful will tie together Einstein's explanation of gravity with the chaotic world of quantum physics, is explained with a level of detail that I haven't seen elsewhere (due to the cool visual effects and the fact that I haven't really looked anywhere else ;). The videos are peppered with historical facts about scientists such as Einstein and Newton, some of which are junior-high level, but most are interesting anyway.

The entire series is hosted by Brian Greene [2]. Several physicists also appear in the videos as interviewees to offer some insightful information and opinions. One thing I like about the videos is that competing ideas about string theory are presented, so you can decide for yourself based on evidence from both sides whether string theory seems like a viable candidate for a TOE; although, the video does seem to lean towards string theory being the most plausible option at the moment, whether that's true or not. One interesting fact mentioned is that we have no way, currently, of physically experimenting with string theory – it's purely theoretical at the moment since it cannot be observed using the technology that we have.

A Note about Viewing

The order that the videos are listed in [1] is a bit unintuitive, IMO. The series is divided into 3 hours worth of programming with each hour represented by a single column. There are, therefore, 3 columns. Each column has the same number of chapters, aptly labeled, making it appear that you can view any given chapter by watching each video in the same row from left to right, but that's not the case. Each video, in its entirety, represents one whole chapter that is distinct from chapters with the same label in any of the other two hours.

To watch the entire series in order you should start at the top-left and watch each video downward in the same column until you reach the bottom. Then, start again at the top of the next column to the right. I recommend viewing them in order or else you may end up doing what I did, which is to choose the videos that seem the most interesting to you at first and then end up watching the whole series anyway once you're turned on to it. If you do it that way and don't mind watching some of the same episodes twice then just ignore me :)

Off-topic Rant: Blogging and Downtime

I mentioned in a previous blog entry that I was upgrading to a new version of Community Server. Well, I finally got my blog updated. This post, and all of my subsequent posts, will be written and published using MS Word 2007. Unfortunately, I lost some of the site statistics such as referrers and post view totals along with the upgrade, but I still think that the change was worth it. Blogging in Word is much easier than the way I was doing it previously: a combination of notepad, Visual Studio and the Community Server interface.

For over a month I wasn't able to access my blog due to an issue that my ISP blamed on my website host, who blamed the issue on AT&T. I was recently able to access my website again after calling each company a few times over the course of about a month and a half to 2 months, while having only limited and random access to my own website and email (limited, meaning that I had a total access time of only about 3 or 4 days in the last month and a half to check my email and view my blog). This is why I haven't been posting anything :( Calling the companies seemed to produce absolutely no results, BTW. I think, ultimately, my ISP changed my dynamic IP address, which they claim was causing the issue with my website host, who claimed no responsibility and didn't even acknowledge that there was a problem even after my ISP created an official statement that relinquished fault to my website host, by name. Obviously I've been quite bitter about this whole experience, but since I'm able to access my website and email again and since I've completed the upgrade to Community Server I think I'm happy now :)

References

[1] NOVA, The Elegant Universe
http://www.pbs.org/wgbh/nova/elegant/program_t.html
Full Screen:
http://www.pbs.org/wgbh/nova/elegant/program_d_t.html

[2] NOVA, The Elegant Universe, Credits
http://www.pbs.org/wgbh/nova/elegant/credits.html

It seems possible now to blog in Community Server from within Microsoft Office Word 2007, and that's awesome.

I was considering writing a program to make blogging easier; to get away from the tiny little text box that Community Server provides for me, the really annoying problems associated with undo/redo and browser editing in general.  I've been writing my blog posts in Visual Studio and pasting them into CS.  It's such a pain that I've actually avoided blogging about some topics that I wanted to blog about due to procrastination.  Thankfully, blogging will be much easier in the future, maybe...

The Whirl I Gave It

So, I tried to connect to CS from Word and it didn't work.  I think it's because I'm using an older version of CS.  Supposedly, there should be a metablog.ashx  file on the root, or somewhere on my site, but I just can't find it anywhere.

Downtime

The entire website, davesexton.com, including my blog, will be down temporarily.  Probably within the next day or so.

I'm going to try installing the latest version of CS and see if that doesn't fix the problem.  The blog will be down for however long it takes to uninstall and reinstall CS, configure it, repost all of my existing posts and re-upload all of my C# code snippets and software (I don't have access to the DB or file system so I'll have to redo everything from scratch).  Needless to say, it might be down for a day or two if I start on a weekday since I still have a lot of work to do for my client.  It might be down even longer if I run into some unforeseen issues, but I'll try my best to mitigate problems and get this thing running.  If any of the aforementioned procedures fail, I just may roll back to the earlier version of CS (if possible), which I know works.

I just installed Windows Vista on my main PC and I was wondering if there was an easy way to have the Windows Sidebar pop-up over any maximized windows, just in case I need to quickly find out the temperature or play that amazing puzzle game ;).

There is a Bring Gadgets to Front context menu item on the Sidebar icon in the tasktray, but is there a keyboard shortcut?

I found the answer here: Windows Help and How-to, Keyboard Shortcuts.

Awesome link. It shows some of the well-known shortcuts and a bunch of new ones. BTW, the keyboard shortcut to bring the Gadgets to the front is Windows logo key + SPACEBAR, which also focuses the Sidebar.

The Windows logo key has some other, new functionality in Vista. For instance, Windows logo key + G cycles through the Gadgets (TAB, alone, cycles through the controls in a Gadget once it is focused), Windows logo key + TAB cycles through taskbar programs using Windows Flip 3-D, and CTRL + Windows logo key + TAB cycles through taskbar programs with their live thumbnails, using the arrow keys.

As always, I'd love to hear from anyone that has something to say about this post.  Drop me a comment.

I've made some more modifications to the DocProject software. Several classes have been refactored again and some new classes have been added. A few work items have been addressed and now there is support for Visual Basic.NET and ASP.NET Web Application projects (DocSites). You can also cancel the build in the middle of a step.

Download the latest release, version 1.2.0, at DocProject on CodePlex.

Collaboration

Steve Bokser, a programmer and a friend of mine, helped me to test the new release by installing it on his machine before I made it available to the public. We identified a few issues that didn't surface during my own testing since we have different versions of Visual Studio installed. I took care of the issues that I could fix before the CodePlex deployment and I adjusted the release notes and wikis to include information regarding the setup and configuration issues. Thanks Steve!

Custom Templates

An organization may want to create a custom template that their developers can use to create DocProjects or DocSites for their Visual Studio projects. A custom DocProject or DocSite template can provide a standardized means of building compiled help documentation for projects across an entire organization.

A new tutorial has been added to provide guidance for creating new DocProject and DocSite templates:

Creating New Templates
http://www.codeplex.com/DocProject/Wiki/View.aspx?title=Creating%20New%20Templates

Wiki Updates

Many of the How To... wikis have been modified to reflect changes in the new release.

A new diagram that illustrates the different components of DocProject and their relationships appears on the home page wiki now:

Figure 1: DocProject Components

As always, I'd love to hear from anyone that has something to say about this post.  Drop me a comment.

Version 1.1.0 of DocProject, a release candidate, is now available at CodePlex. Download the DocProject installer, which also installs the complete source code, Visual Studio solution file and project files, here.

Updates

The latest build includes dramatic changes to the class foundation, refactored to address some of the previous work items. One reason why I made such a drastic overhaul was to provide more robust support for extensibility and the ability to add plug-ins for custom build engines other than just Sandcastle, which is still included as the only build engine that's built-in to DocProject.

Another reason why I did a lot of refactoring was because I saw room for improvement. The code is more flexible now, which was required to support the new plug-in framework, but also makes it easier to navigate the code and to extend it at a later time, IMO.

A description of the changes that were made can be found here.

A few new help wikis

1. How To Use The Source Code
2. Tutorial: Creating a DocProject for a new solution

There have also been a number of modifications to some of the previous wikis. The complete list of How To... wikis for DocProject can be found here.

I'm planning to add some more help topics soon, so check CodePlex for updates. One thing I'd like to add is a simple tutorial that explains how to create your own DocProject templates (it's really easy). Another thing I'd like to write about is how to create a custom build engine and plug it into DocProject. It's not quite as easy as creating a new template since it requires writing some code, but it's not too difficult either.

Feedback!

I'd really like your comments and feedback on this software, so don't be shy :). I've already received some positive feedback. Nothing negative yet - not even constructive criticism. I'm sure it's out there, and I can take it, so bring it on!

Please feel free to comment here on my blog or in the DocProject forums at CodePlex.

As always, I'd love to hear from anyone that has something to say about this post.  Drop me a comment.