I've decided not to release DocProject 2008 Beta 2 with the 1.8.0 feature set.  Instead, I plan to reduce the number of conversion requirements and release Beta 2 with the next VS2005-compatible release, 1.9.0 RC.  Sorry for any inconvenience this may have caused.

DocProject 1.8.0 RC is now available for download on CodePlex.  It contains bug fixes as well as several new features:

• Compatibility with the Sandcastle September CTP.

• DocProject Properties context menu item for DocProjects and DocSites (on the project node in Solution Explorer) quickly opens a property page in Visual Studio Standard and higher.

• DocSite Templates now support MSIE 6.0 (as well as IE 7.0, Firefox and Opera) and are themed, localizable, configurable and componentized.
• DocSite Templates have a full-text search that supports complex query operations and uses an in-memory search index provider (based on .NET 2.0 Providers) that can be configured on the server, even after deployment.
• DocSite Templates have a full-text index browse page that makes finding topics very easy.
• The selected project Configuration and Platform work in MSBuild, Team Build and the DocProject External UI, which now displays drop-down lists for changing these values before building (like in Visual Studio).
• The DaveSexton.DocProject.targets file has been refactored to make it more flexible.
• Build output and Sandcastle project items are now located in a root folder named, Help.
• After a build, compiled help is automatically copied to the project's output directory (for example, Debug\bin); however, for the DocSite templates the target is Help\bin. The DocSite TOC and index files are now generated in the App_Data folder.
• DocProject External UI now supports the Include Project Output Dialog, which is automatically displayed after a build unless it's disabled by the user.
• DocProject External UI provides a tools options page for configuring build engine providers just like the Engines tools options page that's available in Visual Studio Standard and higher.
• Start Menu shortcuts to the DocProject External UI and various resources, including the Source folder in DocProject's installation directory, are now installed.
• User configuration data is now stored in the system's isolated storage so that one user's changes do not overwrite another's; e.g., the default build engine provider and Sandcastle presentation style.

You can read more about the DocSite features in my previous blog post: DocSites 1.8.0 Preview

In the upcoming DocProject 1.8.0 release candidate I've made some changes and added several substantial features to the DocSite templates that I'd like to share with you in this blog post. Changes to the DocSite templates include:

• ASP.NET Theme presentation:
  • Ships with one default theme named, "BasicBlue", which has the same appearance as the current DocSite templates.
  • Subtle improvements to appearance, mostly for Firefox and Opera.
  • Retained div-only + CSS layout with a sticky footer; i.e., no tables.
  • Includes some useful .SKIN files.
  • Full support for IE6 as well as continued support for IE7, Firefox and Opera.
• Componentized Master page, with user controls for the header, footer, breadcrumbs, sidebar, TOC and index.
• ASP.NET Localization with resource-based text in all pages and controls.
• Customizable and highly extensible full-text search built from the ground up:
  • Designed for general-purpose indexing but customized specifically for indexing HTML help topics.
  • Has a default in-memory search index provider (I'll also build an SQL Server search provider for a subsequent release).
  • Complex query support with operators such as AND, OR, - (not) and grouping with parenthesis.
  • Customizable, factor-based weight and rank calculations.
  • URL query string supports bookmarking searches.
• Keyword browse page for live browsing of the search index (also supports a URL query string for bookmarking).
• Administration page for configuring DocSite options, search factors and for viewing simple search statistics.
• New Project Wizard – Create DocSite Credentials step allows a user to set the administrative DocSite credentials when a DocSite is first created.
• The DocSite templates reference a new class library that encapsulates common DocSite functions.

Breakdown of Features

The rest of this article will provide more information on some of the features listed above. If you have any questions or comments please let me know!

New Project Wizard – Create DocSite Credentials

The first change that you'll notice to the DocSite templates is a new wizard step:

Figure 1: New Project Wizard - Create DocSite Credentials

This step allows you to enter an administrative user name and password for accessing the DocSite administration page (more on that later). The credentials are stored in the web.config file in the standard ASP.NET forms authentication section.

For custom DocSite templates authors can easily enable this feature by adding a single user to the authentication section with "$admin$" as the name (the password will be ignored). The mode of encryption that is chosen for the authentication section (e.g., Clear, SHA1, MD5) is reflected in the dialog's help paragraph automatically, and will also be applied to the user's password before it's stored in the web.config file.

Themes and Localization

The DocSite templates now use a default theme named, BasicBlue. Its appearance is almost identical to the appearance of the current DocSite templates, but now it has full support for IE6 as well as a sticky footer for all .aspx pages that do not use an IFRAME to display page content.

Supporting IE6 without a table layout was difficult, but I succeeded with the help of CSS expressions, which are specific to IE (but that means they're supported by IE7 too). The hard part was trying to support IE7's more standardized rendering engine without breaking IE6, and then supporting IE6 without IE7 falling back to things like CSS expressions. But after some flipping out and loss of hair, I finally got it working :)

Customizing the appearance of DocSites should be much easier now that it's fully themed. And with Sandcastle's high level of flexibility, anyone with some previous experience working with XSL and CSS should be able to come up with a custom Sandcastle presentation style and DocSite theme that work together for a completely customized appearance without affecting the DocSite's functionality, or even better, to improve on it. BTW, I'm certainly interested in hearing new ideas for themes if you have any.

Text that appears in all of the DocSite pages and user controls have been extracted into ASP.NET global and local resource files for easy localization.

Indexed Search and Browse

Finding help content in DocSites will be much easier now with the addition of the new search and browse features.

Note: The screenshots depict the Admin link as a different color than other hyperlinks, but that's simply because I clicked it.  What you see is the "followed" hyperlink color.  Before Admin is clicked it's the same color as the other links in the header: blue.

Search Page

Figure 2: DocSite Search Page

The abstract, locale, namespace and library information are all taken from each topic's MSHelp data found in an XML data island generated by Sandcastle for Help 2.x. Although, due to a possible limitation in the next Sandcastle release, I might have to remove this feature temporarily until I can create a custom solution for a subsequent release.

DocSite search supports complex queries with operators that are defined by the search provider itself. By default, your DocSite will support the Boolean AND and OR operators, - (the negation operator) and parenthesized grouping:

Figure 3: DocSite Search Help

The search help page is accessible by simply leaving the search box empty and clicking the search button (the one with the large magnifying glass).

NOTE: In the future I plan on adding an advanced search page that contains HTML topic-specific search functions, such as searching by the culture in which a topic was written, the topic's title, etc.

Browse Index Page

The button with the tiny magnifying glass, next to the search button, is the browse button. Clicking this button will open the browse page using the keywords entered into the search box, although all operators and grouping will be ignored. Clicking the button with an empty search box simply opens the browse page without any selected keywords.

Figure 4: Browse Index Page

The browse page allows you to drill down through a query by removing or adding one keyword at a time until you find a result set that contains the topic in which you're interested. The browse page is also useful for administrative purposes as it displays some statistical information about individual keywords as well.

The letter bar at the top of the page provides a list of keywords that start with the selected letter. Clicking a letter will display the keyword list. Then, clicking a keyword will add it to the filter and the new filtered results will be shown. You can continue to browse the index by the first-letter of keywords or view and modify the active filter at any time. The letter bar itself is encapsulated in a user control and is completely customizable using the administration page (discussed later).

Toggle between Search and Browse

The button next to the search query on the search results page and the button next to the keywords on the browse page allow you to easily toggle between these two distinct modes of locating help topics so that you can choose the one that is better suited for your particular query.

The Index

Both the search and browse features work with a full-text index of the HTML help topics generated by Sandcastle. The index is stored in-memory and generated on-the-fly by the website itself, not DocProject, which means that you don't have to rebuild the project to rebuild the index. It also means that you can tweak factors for weight and rank calculations and then rebuild the index on-demand through the web interface, allowing fine-tuning of the search results for common keywords that you might expect your end-users to search against. You can manage the search index and factors through the administration page, which I'll discuss later.

Your DocSite will automatically rebuild the in-memory index after the application has started upon the first request for the search page or a particular keyword on the browse page. In the future I plan on adding an SQL Server search provider that you can use instead of the default in-memory provider so that the index does not have to be rebuilt each time the ASP.NET worker process is recycled or when changes to the application are published that will cause it to restart (like certain modifications to the web.config file).

Site Administration Page

The DocSite administration page provides a set of configurable options with application-scope.

Figure 5: DocSite Administration Page

To reach this page simply click the Admin link in the top-right corner of your DocSite. If you haven't logged in then you will be brought to the login page first. Enter the credentials that you configured when the project was first created and then click the Login button to continue to the DocSite Administration page.

Settings

Settings are stored in the DocSite.config file found at the root of the project. Changes made directly to this file will not take effect until the application is restarted; therefore, I highly recommend using the web interface to configure settings whenever possible so that you do not have to restart the application (and lose the entire in-memory search index).

There are two main categories in the administration page: General and Search. Changes to any options in either category are applied immediately.

The Create Index link allows you to regenerate the index on-demand. This is useful for ensuring that the index is generated immediately after the DocSite has been published and for updating the index after modifying keyword, weight or rank settings.

Client Settings

The Search category has a few subcategories for viewing statistics and managing the DocSite Search and Browse Index pages.

Index Statistics

This category provides read-only information:

Settings

Provides general index-related settings:

Keyword Settings

Settings related to indexed keywords and keywords in search queries.

Weight Factors

Weight factors are integers that affect the weight of indexed keywords. Note that weight is calculated while the index is being created and rank is calculated when a keyword matches a search query. The values in this section do not affect rank directly; however, the weight of a keyword does add additional rank to matches.

Title Rank Factors

Additional rank values for keyword matches that are found in titles.

Conclusion

In this blog post I described new features to come for the DocSite templates in DocProject 1.8.0 RC. These new features will surely be appreciated by your end-users since they make it much easier than before to locate and browse help topics. As a site administrator you now have a much easier way to configure some of the DocSite features and will have an easier time modifying the appearance and localizing text to meet your particular needs and the needs of your clients.

A couple more public DocSites have come to my attention and I thought I'd share them with my readership. You can read my original DocSites Online post here.

If you haven't been paying attention and you're not sure what a DocSite is, it's an AJAX-enabled ASP.NET Web Application project in Visual Studio 2005/2008 that provides a web interface for API documentation. The documentation can be built using Microsoft's Sandcastle utilities from managed assemblies and optional XML documentation files that contain your triple-slash code comments. DocSite templates come in two flavors: C# and VB.NET (although either template can build help for projects written in other languages, including managed C++).

DocSite templates are available as part of DocProject.

MbUnit 2.4

Ben Hall provided some great feedback on DocProject and has done an excellent job creating his DocSite at http://docs.mbunit.com/, which includes a custom TOC, topics and appearance.

According to the docs…

MbUnit is an advanced, extensible unit testing framework originally developed by Jonathon 'Peli' de Halleux and Jamie Cansdale.

Ben posted about the documentation's release at http://blog.benhall.me.uk/2007/08/mbunit-new-website-and-documentation.html.

You can read about the MbUnit project at: http://www.mbunit.com/.

Interactive Brokers C# API

Dinosaur Technology and Trading has ported a Java socket interface to C# for the Interactive Brokers API.

Their DocSite can be found at http://ibhelp.dinosaurtech.com/default.aspx and you can read about the utility at http://www.dinosaurtech.com/utilities/.

Nice work guys.

Style

The API topics for each DocSite were built using Sandcastle's Hana presentation style, which adds a number of improvements over MSDN's style; however, it seems that the final VS 2008 (Orcas) documentation will probably still use the VS 2005 style according to a post at Sandcastle's blog.

You can read about the Hana style here.

In case anyone's interested, this post contains my notes for converting the code base of DocProject 1.7.0 to support Visual Studio 2008 Beta 2. After that I've included DocProject's backlog of tasks, which may contain some ideas that you'd be interested in seeing me implement in a subsequent release.

Notes on the Conversion from 1.7.0 to 2008

It was no easy task, though I don't think my experience is any indication of what must be done for the conversion of general 2.0 Framework applications since DocProject relies heavily on Visual Studio itself at runtime, which is uncommon (at least until future apps start using the new Visual Studio Shell).

While logging this information I was also running into various issues building the source code and running the program. In order to ease maintenance I made whatever changes I could to both the 1.7.0 and 2008 code bases to fix the bugs while keeping them in sync. As a result the code base of DocProject 2008 is 99.99% the same as DocProject 1.7.0, with a few special adjustments such as a different hard-coded installation path under the root installation directory, for example. Many of the updates that I logged were simply changes to references of version numbers but there were also many modifications to the InstallPrep and Installer projects. I don't think any bug fixes are logged here, however.

This is not an exhaustive list of what was required for the conversion. If the time comes again that I must convert DocProject's source I'll certainly follow these notes as a guideline, but I'm sure the process will require my attention.

1. After installing DocProject on the build machine, copy the Source folder to "My Documents\Visual Studio 2008\Projects\DaveSexton.DocProject08\" and uninstall DocProject.
2. Update the Installer project to search for "VSORCAS OR VCSORCASEXPRESS OR VBORCASEXPRESS", but rename the search and conditions to use 2008 instead of ORCAS; e.g., VS2008 OR ...
   NOTE: In 1.7.0 RC for VS 2005 the VSORCAS searches were removed to support side-by-side installation. The VS2005 searches should actually be replaced with VS2008 searches instead.
3. Update the DaveSexton.DocProject.InstallPrep.CustomActions.Contents property to return ONLY the Orcas content.
4. Update all of the projects, including the templates, to use the 3.5 framework. Note that the DocSite templates require a temporary modification to the web.config file before hand unless the modification is done manually. To automate it, use the properties dialog - Application tab for C# projects and Compile tab > Advanced Compile Options button for VB projects.
5. Replace the 2.0 framework prereq with the 3.5 framework in the Installer project.
6. Update the DaveSexton.DocProject VCProjectEngine reference to 9.0.0.0
7. Update DaveSexton.DocProject.InstallPrep\PostBuildEvents\DaveSexton.DocProject.bat, and the Sandcastle, Sandcastle/Deployment and HtmlEditor projects' post build events:
       "%programfiles%\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" ...
8. Update DaveSexton.DocProject.InstallPrep:
   1. BootStrapper\vs_piaredist should be changed to vs90_piaredist.
   2. Copy in the vs90_piaredist files and update the Readme.txt file.
   3. Update the Readme.txt file in the project's root to reference vs90_piaredist instead of vs_piaredist.
   4. Contents\Installer\DocProject.AddIn file should be updated to version 9.0 in the HostApplication element.
   5. Replace all occurances of <ContentVersion>1.0</ContentVersion> with <ContentVersion>2.0</ContentVersion> in Contents\Installer\DocProject.vscontent
9. Delete the DaveSexton.DocProject.DocProjectAddIn.GetMSBuildBinPath methods since they are no longer used (see next step)
10. Delete all the code in the DaveSexton.DocProject.MSBuild.MSBuildAnyProject(string) class constructor above and including the local variable named, "vsVersion" and use the parameterless Engine() constructor instead. (There is a warning about this in the Error List when building the DocProject Add-In assembly).
11. Delete Microsoft.Build... references from DaveSexton.DocProject and DaveSexton.DocProject.ExternalUI and replace them with references to the new 3.5 versions.
12. Remove the DaveSexton.DocProject.MSBuild.BuildDocProject.MSBuildBinPath property and all remaining related code, including code that constructs and MSBuild Engine class using a custom bin path. (There will be warnings in the Error List after building the DocProject Add-In assembly).
13. Update the DaveSexton.DocProject.InstallPrep\DaveSexton.DocProject.targets file by removing the MSBuildBinPath attribute from the BuildDocProject and CleanDocProject tasks.
14. Update DocProject and DocSite project templates (found in the InstallPrep's Contents\[template name] folders):
    1. Change version to <ProductVersion>9.0.20706</ProductVersion>
    2. Append <TargetFrameworkVersion>v3.5</TargetFrameworkVersion> to the first PropertyGroup
    3. Change targets reference to <Import Project="$(DocProjectPath)\VS2008\bin\DaveSexton.DocProject.targets" />
       (NOTE: DocProjectPath environment variable must be defined - this is indicated below)
15. Update DocSite templates:
    1. Change type guids to (actually, I don't think this step is necessary):

       C#: <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{fae04ec0-301f-11d3-bf4b-00c04f79efbc}</ProjectTypeGuids>
       VB: <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

    2. Change web app target to <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\WebApplications\Microsoft.WebApplication.targets" />
16. Global search for "8.0" just in case any more updates are required for "9.0" (NOTE: the binding redirect in the External UI project's app.config file should NOT be updated to 9.0.0.0)
17. In the InstallPrep project delete the WiRunSQL.vbs file and the UpdateMSI.bat file, and then remove the Installer project's post build event.
18. In the Installer project:
    1. In the File System Editor, change the Application Folder's DefaultLocation to "[ProgramFilesFolder][Manufacturer]\DocProject"
    2. Create a folder under the Application Folder and name it "VS2008"
    3. Drag each root folder into the VS2008 folder and then drag the files located in the root into VS2008 as well.
    4. Changeg the Version to 1.0.0
    5. Change the ProductCode and the UpgradeCode.
    6. Change the Subject to "DocProject 2008 (Full)"
    7. Change the ProductName to "DocProject 2008"
    8. Open the Registry Editor. Hard-code "DocProject" instead of using [ProductName] used in the key for the InstallPath value (for both the HKCU and HKLM).
    9. In the Registry Editor, change the value of each SafeImports key (DocProject value) to [TARGETDIR]VS2008\bin\DaveSexton.DocProject.targets
    10. In the Registry Editor, change the "8.0" key to "9.0" in each of the paths to the SafeImports values.
    11. For each SafeImports key (DocProject value) also udpate the condition to use 2008 instead of ORCAS; e.g., VS2008, VCS2008EXPRESS and VB2008EXPRESS
    12. Change the VS2008\bin\DaveSexton.DocProject.dll.config file's TargetName property to DaveSexton.DocProject08.dll.config
19. Change the assembly and file version for each project to "0.0.1.0"
20. Append "08" to each project's assembly name (Applications tab in the project properties dialog).
21. Do a global search in the entire solution for "DaveSexton.DocProject (with a single quote prefix). Where appropriate, append "08" to the assembly name of references and upate the version to "0.0.1.0" as well, for all projects - not just DaveSexton.DocProject.dll.
22. Rename the solution to "DaveSexton.DocProject08". Then locate the .sln file in the Installer project's File System Editor under the Source folder. Delete the existing file and add the new 08 .sln file.
23. Rename the DocProject.AddIn and DocProject.vscontent files in the InstallPrep project to DocProject08.AddIn and DocProject08.vscontent. They're located in the Contents\Installer folder. Also update DocProject08.vscontent so that it references the renamed .AddIn file.
24. Append "08" to the name of each template folder and update the Installer\DocProject08.vscontent file to point to the renamed .zip files.
25. Update the InstallPrep project's Install.CustomActions class:
    1. Repeat the previous step on the static fields that reference the template zip files and the .AddIn file.
    2. Update the InstallEnvironmentVariables method to use "targetEnvironmentVariable, Target"
    3. Update the UninstallEnvironmentVariables to use "targetEnvironmentVariable, null"
    4. Delete the TargetBin property
    5. Change the buildPathEnvironmentVariable field to: private const string targetEnvironmentVariable = "DocProjectPath";
    6. Add a new property (after the Target property) named TargetFlavor that returns "Path.Combine(Target, @"VS2008\");"
    7. Update the CreateInstaller method to use TargetFlavor instead of Target.
26. Update the DocProjectAddIn class's InstallPath property to return Path.Combine(installPath, @"VS2008\");
27. Update the BuildOutput.bat file in the root of InstallPrep:
    1. Update the template names (only the second argument must be updated). For example: CALL :ZIP_TEMPLATE "%TemplateDirRoot%\VBDocSiteTemplate" VBDocSiteTemplate08
    2. SET InstallerName=DocProjectInstaller08
28. Build the InstallPrep project once. Then open the Installer project's File System Editor and navigate to Application Folder\VS2008 in the tree:
    1. Delete the DocProjectInstaller.vsi file and replace it with the new DocProjectInstaller08.vsi file found in the InstallPrep project directory.
    2. Open the Setup\AddIn folder and delete DocProject.AddIn. Add DocProject08.AddIn from the Contents\Installer folder.
    3. Open the Setup\Templates folder and delete all of them. Add them back from the Contents\Templates folder (hidden in Solution Explorer), but only choose the 08 versions (previous versions can be deleted since this is a temporary folder and they are no longer in use).
29. Run Code Analysis on each project, including InstallPrep but not the templates or the Installer. In some projects there may be certain rules that should be disabled. For example:
    1. Naming: CA1703 (resource spell-checker)
    2. Naming: CA1704 (identifier spell-checker)
    3. Naming: CA1716 (indentifiers that are keywords, such as "step")
30. Make sure to exclude all detected dependencies in the Installer project before building it.

Make sure that the solution builds without any errors. Then build the InstallPrep project and make sure that it completes without any errors.
(Check the build output window for DSZip errors). Build the Installer project and install DocProject on the build machine.

• Install DSZip on the build machine.
• Make sure to copy the "vs90_piaredist" bootstrapper folder into the "C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bootstrapper\Packages\" folder on the build machine.

Backlog

Here's some stuff that's been on the backburner for quite some time now.

• Auto-merge changes to partial-build items. Concrete engines are responsible, but the BuildEngine class could copy all partial items to a temporary folder before a build. Then after the build completes, call the derived-implementation of protected abstract void MergePartialBuildItem(DocProjectItem oldItem, DocProjectItem newItem) for each file in the temp folder. Finally, each temp file would be deleted. This method should be added to the IBuildEngine interface as well. If an implementation can't or shouldn't merge the changes then it doesn't have to do anything with the temp file, or even open it.
• Create an item template for Sandcastle's build component (the New Project Wizard may have to be responsible for referencing the BuildAssemblerLibrary assembly unless the item template supports some sort of wizard that can do the job – but this needs to be researched)
• Components dialog that served as an interface for the .config files. It would be a list of components found in the Sandcastle assemblies, the project's assembly, and any registered assemblies. The order and parameters could be specified, somehow.
  • Create a wizard step that asks the user to choose from various build components to be included in the project's {root}\Components\ folder. For example, XAMLSyntaxGenerator.cs, which would automatically be registered in the .config file.
  • Allow users to add cusotm build components to a DocProject or DocSite via an item template.
  • List of "registered" assemblies on left (registered assemblies cannot be removed from the list if there are stack components that reference them)
  • Build component stack displayed on right ("New" drop-down button, with "External Component", like a browse button, and "Internal Component", that searches the project's assembly (also with "Add New" as an option, somewhere)
  • Up down arrows that control the order that components are executed in the stack
  • Reset button that will import the configuration file for the project's presentation, replacing all changes (Are you sure? dialog)
• Have a start debug button ">" in the Sandcastle toolbar that will put Visual Studio into debug mode and attach to an external "msbuild.exe /project" process for debugging custom build componenents and the build process component. For DocProjects, this may be possible without a custom button by setting the startup program on the Debug tag, if it's possible to do that from the templates.
• ApiFilter Issues:
  • reference method signatures?
  • discover generic type name? (currently, the id is parsed by searching for .name` where name is the value of the name attribute of the corresponding apidata element)
• API Topic Management Dialog:
  • Add/Save/Load feature: Allow users to add a filter (regex or category) to a list that can be ordered using up/down arrows. Saving the list would create an XML doc. Loading the XML doc would recreate the list in the dialog, which could be applied by clicking the list's Apply button (each individual filter would be processed one at a time, top-down).
• Attribute Filter Dialog for Syntax Generator ("Syntax Attribute Filter")
• Automatic token replacement in code comments:
  • Have a dialog that lists token values that can expand to full HTML, which can be edited by clicking an ellipses button that opens an Html Editor dialog, for each individual token.
  • Users can import and export the list to and from an xml file.
  • Make the token replacement start and end strings (both "$" in the examples below) configurable in the tools options page in case the default values conflict with some other device. The values are required and cannot contain ANY white-space. Token replacement will only occur if the following regular expression is matched: \$[^\s\$]+\$
  • When customized, for example, if the start token is % and the end token is ^ then the expression would be: \%[^\s\%]+\^

• Examples:

  Custom Tokens
  
  $isnull$      "is a <strong>null</strong> reference (<strong>Nothing</strong> in Visual Basic)"
                (this could be built-in, but editable)
  $my-token$    "Hello <strong>World</strong>!"
  more custom...
  

  
  Uneditable Tokens (cannot be modified, imported or exported):

  $time$        DateTime.Now.ToLongTimeString(), in the current culture
  $date$        DateTime.Now.ToLongDateString(), in the current culture
  $datetime$    DateTime.Now.ToString(), in the current culture
  $user$        Expands to the current user's name.  e.g., Administrator
  $name$        Expands to the name of the member or type to which the comment applies.  
                e.g., ToString, IsEnabled, BuildEngineProvider
  $fullname$    Expands to the full $name$.  e.g., MyNamesapce.MyObject.ToString, 
                MyNamesapce.MyObject.IsEnabled, MyNamesapce.BuildEngineProvider
  $type$        Expands to the containing type of the member, or the type itself if the 
                comment applies directory to a type.
  $-type$       Exapnds $type$ to a link.  Same as: <see cref="$type$"/>
  $-x|y$        Expands to a hyperlink: <a href="y" mce_href="y">x</a>
                or <a href="x" mce_href="x">x</a> if the vertical bar is not specified.
  $project$     Expands to the name of the project: {project name}
  $-project$    Expands to the project link:  <a href="R_Project.htm" >{project name}</a>
                (or the GUID form)
  $home$        Expands to a hyperlink with its url designated by the user.  By default, this token 
                will be ignored unless the user enters a valid Uri and name.  (custom editor, I guess)
  $art$         Expands to the virtual artwork path; currently, /Art
  

Several of these ideas are already work items that you can vote on. Please let me know if any of them seem interesting to you. I have several other ideas as well that I haven't written down yet but I will when the time comes.

The only reason why I haven't started on the token replacement feature is because I think Microsoft already has an internal implementation that they use, but I'm just itching to create it myself. I'm waiting now to see if they'll make theirs public first before I attempt to reinvent the wheel, if in fact their implementation supports the feature set that I'm after.

The latest version of DocProject has been deployed.  Also, there is a new version called DocProject 2008, currently in beta, that provides support for Visual Studio 2008 Beta 2, Visual C# 2008 Express Beta and Visual Basic 2008 Express Beta.

You can find DocProject at http://www.codeplex.com/DocProject.

It's about that time again…

Just FYI, DocProject 1.6.2 RC had more downloads in the last month than any other version to date. Thanks to everyone who tried it and provided valuable feedback!

And BTW, I wrote a new article called How to Diagnose and Resolve Issues that provides guidance for gathering diagnostic information if you're having issues with DocProject and would like to try to solve the problem yourself or if you would like to ask for assistance.  You can find other How To... articles here.

In the next version, 1.7.0, there are a number of really cool features that I think people will like, as well as some bug fixes that should solve the problem with Team Build and provide a better experience using the HTML Editor. In this blog post I'll show a preview of some of the new features and describe my plans for them in the future.

Like usual, please let me know what you think so that with your support I can continue to make DocProject a useful community tool.

New Features

• XML Documentation Editor
• API Topic (Shared Content) Designer Improvements
• Missing Dependencies List
• External UI (DocProject.exe)

I'm still testing and finishing up some other tasks, features and bugs. I'm running a bit late since I was hoping for a mid-July release, but a few of the features that I will describe below somehow snuck into this release (that seems to happen a lot with me ;). I'm also quite busy with work but I think I might be able to manage a deployment next week, without any showstoppers popping up. So far, things are looking good though.

XML Documentation Editor

The API Topic Management Dialog now supports authoring XML documentation (summary, remarks and example) for all API members, not just the project and namespace summaries. See Figure 1 below.

I've created a new control named, ContentEditor and another control named, ContentListEditor, which is used for the new and improved content dialogs: API Topic Management (Figure 1) and the API Topic Designer (Figure 4). As you can see in both figures, there are now two tabs at the bottom for switching between design-mode and source-mode and a list of items that can be edited. Only one item can be edited at a time by selecting it in the list. After editing an item its name becomes bolded in the list until you save your changes.

Note: The content editor control actually provides another tab named, Preview, which raises an event that controls can handle to provide read-only HTML based on the changes made in the editor. Because this feature has not been implemented yet the Preview tab is currently hidden at runtime.

Figure 1: API Topic Management – Xml Documentation Editor

Xml Documentation

Remember that this is an xml documentation editor, which means that tags such as see, c and code may be used in Source mode even though they won't be formatted in Design mode. Sandcastle uses these tags when it builds your documentation to apply special formatting or to link to other topics.

One feature that I plan on adding in the future is the ability to drag nodes from the tree onto the designer and have it serialize <see cref="{id}">name</see> automatically. I'd also like to show see links as hyperlinks in the designer, but maybe as green or using double-underline to distinguish them from normal hyperlinks. Another possibility is to use the stylesheets from your DocProject's or DocSite's selected presentation style and then format all of the xml documentation tags in the designer as they will appear in the compiled documentation.

Another feature that I plan on adding is API type-specific sections in the item list. In other words, if you select a method node to edit then the editor should automatically list each of the method's parameters next to summary, remarks and example. Tags such as permissions and seealso should be listed as well, although I'd like to open a custom editor interface for these tags.

Currently, you can add custom tags to the list using DocProject's configuration file (DaveSexton.DocProject.dll.config, normally found in C:\Program Files\Dave Sexton\DocProject\bin), but they'll apply to every API element. I'd like to implement a feature that allows you to specify which sections apply to which API elements though (probably using DocProject's configuration file, partially shown in Figure 2 below).

Note: Do not confuse the DaveSexton.DocProject.dll.config file with the DocProject.exe.config file (I'll explain more on that below) located in the same directory.

Figure 2: DaveSexton.DocProject.dll.config xmlDocumentationTags attribute

HTML to XML Conversion

I've made a few improvements to my HTML Editor control and the way HTML is converted to XML (a few bug fixes).

The HTML Editor control that you see in Figure 1, which you may already be familiar with if you've used DocProject 1.6.0, is based on the WebBrowser control. The WebBrowser control is basically a managed wrapper around Internet Explorer. For the HTML Editor it's a wrapper around IE's design mode, which only produces HTML markup (in some cases, very poor HTML markup.) Sandcastle requires valid XML, so DocProject converts your HTML to XML automatically. The results of the conversion can be viewed in Source mode. The markup that you enter in Source mode is also converted automatically, so don't be surprised if when you forget to add a closing p tag, DocProject adds it for you when switching views or saving your changes :)

The HTML Editor is a work-in-progress, so I expect issues with HTML to XML conversions to spring up every now and then. Please let me know if you experience an error. If you can provide me with the exact steps to reproduce the problem and the markup that you used, I'll do my best to fix parser bugs and improve the editor's usefulness.

Persistence

Your comments are saved in individual xml files in your DocProject's or DocSite's Comments folder, at the root of the project. An {assembly name}.xml file is created for each assembly and one project.xml file is created to contain the project and namespace summaries. If you prefer you can edit these files using Visual Studio's XML editor. If you haven't edited any API elements for a particular assembly then its XML file will not be present in the Comments folder, but you can create it yourself. The name of an assembly's xml documentation file must be the display name of the assembly with .xml appended to the end or else DocProject will not be able to merge your documentation with imported documentation.

Merging

XML documentation that appears in your code is not shown in this dialog. By default, comments created in the dialog overwrite code comments, on a section by section basis; e.g., summary, remarks or example. But you can change this setting to give code comments precedence instead of being overwritten by selecting KeepSource for the Merge xml documentation setting found in Tools > Options > DocProject > Active Projects > Content.

Figure 3: Merge xml documentation options for documentation written in the API Topic Management dialog

As you can see in the figure above, the API Topic Management dialog can be accessed from the option immediately below the Merge xml documentation option.

Xml Documentation Clean-up

If you create documentation for a type that is later removed from your project then the documentation will still remain in the XML file for the assembly in which the API element is defined, however it will no longer be displayed in the API Topic Management dialog so you must remove it from the file manually. In the future I may add a feature that will allow you to clean up your xml documentation files at the click of a button.

API Topic (Shared Content) Designer Improvements

The API Topic Designer uses the new content list editor control and provides a list of content items that you can edit, such as the header, footer and various titles.   The items are loaded from the Presentation\Style\Content\shared_content.xml file in your DocProject or DocSite.  Changes to the items using the dialog are written directly to this file as well when they are saved.  Deleting an item in the file will remove it from the list and adding an item to the file will add it to the list the next time the dialog is opened.

Figure 4: API Topic Designer in 1.7.0

Just like in 1.6.0, the API Topic Designer appears as a step in the New Project Wizard with all of the new features.  The header section is the default item selected.

As of the June 2007 CTP release of Sandcastle, presentations have a hard-coded header and footer that will probably be inappropriate for your projects.  I've decided not to handle this in DocProject so you must delete the header and footer manually if you do not want them to appear in your documentation.  Although the New Project Wizard is a good place to do that, you can still update the header and footer at a later time using the API Topic Designer.

FYI, I've already requested in the forums that the Sandcastle team set these values to empty in the default presentations to save effort on behalf of end-users.

Text-only Content Items

Not all of the items are XML-editable using the API Topic Designer. Most are text-only since there will probably not be a need to use HTML formatting for them. Text-only items only show the Source tab and will escape any XML markup that you enter. To configure which items use the HTML editor on a per-presentation basis you must edit DocProject's configuration file (DaveSexton.DocProject.dll.config, normally found in C:\Program Files\Dave Sexton\DocProject\bin):

Figure 5: DaveSexton.DocProject.dll.config sharedContentXmlItems attribute

Missing Dependencies List

A while back, around version 1.3.0, I attempted to build documentation for DocProject using DocProject itself but I quickly found that a bug in Sandcastle's MRefBuilder program (actually, CCI) relating to binding redirection prevented me from doing so. In the last version, 1.6.2, I decided to try again but this time I found that DocProject's ReferenceResolver class wasn't able to resolve the Interop.MSHelpCompiler.dll reference (provides the Help 2.x compiler services) for the Sandcastle project (DaveSexton.DocProject.Sandcastle.dll). The problem is that DocProject's Sandcastle assembly is being resolved in the GAC, but the MSHelpCompiler interoperability assembly is not installed in the GAC, so it cannot be found. I do plan to fix this bug in a future release, but since I've expierenced this issue myself I assume that there may be other scenarios that cannot be handled automatically by the reference resolver.  This features was added so that users can manually provide the path to dependencies that cannot be resolved automatically.

FYI, nobody has actually mentioned any problems with DocProject's automatic discovery of source dependencies so I assume that this feature will be used quite rarely, if at all. But then again, it must be used by DocProject itself just to build its own documentation ;)

I suggest that you do not use this feature unless you get an error when building. The error will be produced by Sandcastle's MRefBuilder utility and will state something about one or more unresolved dependencies, mentioning the full assembly name as well. From the information in the build output window you should be able to locate and add the missing dependency yourself using the new dialog:

Figure 6: Missing Dependencies List

This dialog is similar to the External Sources dialog that was added to DocProject 1.6.2 RC. You can find both dialogs in Tools > Options > DocProject > Active Projects > Build.

To add an assembly (.dll or .exe that cannot be resolved automatically by DocProject) to the list you must enter the full path and name into the text box in the new row (the row with the *) or click the button with the ellipses on the right side to select a file. You can also edit existing rows or delete rows by clicking the row header and pressing the Delete key.

The button on the left side of each row will map an absolute path to a path that is relative from the project's root directory, like the example path in Figure 6. This is useful for team and build-server scenarios.

External UI

DocProject 1.6.0 provided preliminary support for Visual C# Express and Visual Basic Express editions of Visual Studio. Unfortunately, because add-ins are only supported by Visual Studio Standard and higher, Express users had to manually edit the project files of DocProjects and DocSites to configure them after using the New Project Wizard since all of DocProject's options and dialogs are hosted by the add-in.

DocProject 1.7.0 RC has an external UI that uses a property grid, just like DocProject's Active Projects tools options page, so Express users can configure their DocProjects and DocSites using the same tools as Visual Studio Standard+ users: 

Figure 7: DocProject.exe

For Standard+ users, though this feature may be handy at times, I still recommend using Visual Studio in most situations to configure your DocProjects (via the tools options pages) and to build them in Visual Studio since it will automatically build dependency projects for you. Also, the Include Project Output Dialog does not work in the external UI and if you add an image to the XML documentation editor or the API Topic Designer while using the external UI, the image will not be included as a project item in Visual Studio (although it will still be imported into the Art folder - you'll have to Show All Files and include it manually).

Automatically Open the Active DocProject

You can add a link to the program in Visual Studio for quick access by going to Tools > External Tools... and clicking Add (this can be done in Visual Studio Standard and higher as well if desired).

By specifying the arguments in Figure 8 below, when you run the External UI from Visual Studio's Tools menu the program will automatically open the active DocProject or DocSite in Solution Explorer (that is, the project that contains the current file that is opened or the current file/folder that is selected in Solution Explorer).

The DocProjectBuildPath environment variable used in Figure 8 is registered on your system when you install DocProject.  It points to DocProject's bin directory, commonly found at C:\Program Files\Dave Sexton\DocProject\bin.

Figure 8: DocProject.exe - Visual Studio Tools Registration

Building DocProjects and DocSites

While developing this program I realized that it would be quite easy to build a DocProject from this interface using DocProject's BuildController class, so I gave it a shot. It actually worked so well that I now build my test DocProjects and DocSites using the GUI for debugging purposes. Breakpoints in code being hosted by Visual Studio's process tend to time-out after a few seconds of inactivity, due to COM issues, but since DocProject.exe is fully-managed it makes it much easier to attach, debug and continue, without having to restart Visual Studio and load up an entire solution.

Figure 9: DocProject.exe build trace

DocProject.exe will not only build your DocProject's or DocSite's help, but will also build the project's assembly output using MSBuild first (remember, DocProjects and DocSites are MSBuild projects). During a build you'll see descriptive status information describing the current step and the progress bar, just like you see in Visual Studio. You can even cancel builds at any time.

The program will also remember some settings for when you restart, such as the last position, size and window state, with multi-monitor support, the position of the toolbars and the height of the build trace window (Figure 9).

Currently, this program does not support DocProjects and DocSites that have managed C++ projects as sources since they can only be built using a solution file, but I plan to look into implementing that feature in a future version of DocProject.

DocProject.exe Extensibility

The code that is used to configure DocProjects and DocSites in Visual Studio Standard+ is the same code that is executed by the external UI. This means that if you have created a custom build engine provider for your organization it will automatically work with the external UI. Not only does this include the options in your DocProjectOptions implementation, which will appear as new properties in the grid, but also any Visual Studio toolbars that are created by your provider will appear as well. For example, in Figure 7 above you can see the Sandcastle toolbar that appears in Visual Studio Standard and higher in the DocProject UI (the right-most toolbar).

Sandcastle and Sandcastle/Deployment are examples of build engine providers that are registered in DocProject's configuration file (DaveSexton.DocProject.dll.config, commonly found in C:\Program Files\Dave Sexton\DocProject\bin).  Several of the options that appear in the figures above, including the API Topic Designer and Management Dialogs are provided by the Sandcastle build engine provider plug-in.

The next major How To... article on my to-do list is guidance for creating custom providers that can be used across projects in an organization, but if you feel like testing the waters yourself check out the DaveSexton.DocProject.Sandcastle and DaveSexton.DocProject.DeploymentSandcastle projects in the source code and try creating your own plug-in.

Conclusion

With the 1.7.0 RC, Visual Studio Express users will be able to do much more with DocProject than they could before with the new VS-like external UI. As a documentation author, you'll be able to write long examples and remarks sections using an HTML Editor and store all of the comments in your DocProject's Comments folder, which is merged with your xml code comments automatically before each build. You'll also be able to use the HTML editor to design topics' shared content, such as the header and footer – like before – but with the addition of every other item that appears in Sandcastle's shared_content.xml file as well. And now you can configure your DocProjects and DocSites to reference dependencies that could not be resolved automatically, however rare that might be.

I hope you like the new features and I look forward to receiving your feedback.

Recently I've been receiving more feedback than usual about the addition of non-API topics when generating compiled help with DocProject (also referred to as custom, additional or conceptual topics). In this blog post I'm going to describe DocProject's current support for conceptual content and my goals for better integration in the future.

Community Feedback

First, I want to say thanks to the users that have emailed me, added feature requests to the issue tracker or started discussions. It's been very helpful.

Special thanks goes to Ben Hall, a computer science student in the UK and also a Microsoft Student Partner (MSP), for his great blog post on how to extend my custom topics tutorial (discussed below in Creating Custom Topics with Partial Builds) to make it work in a more automated fashion.

Ben has also written another blog post on DocProject in which he discusses his experience with DocProject, and for which I have a few responses (a bit-off topic for this blog post though, sorry):

• I know that developers can give up easily sometimes, myself included, when an open source tool doesn't work correctly the first time around. But as you've discovered there can be problems with Visual Studio that I just can't account for in DocProject so patience and a bit of effort is sometimes required to get things going. So thanks for trying DocProject again :)
• You mention that all of the settings are saved in the project.xml file, but actually the settings are saved in the project file; e.g., .csproj or .vbproj. The project.xml file is simply used for the project and namespace summaries (but that will be changing in a subsequent release - I'm going to give each namespace summary its own file.)
• Technically speaking, DocSites don't have a search – they have an index filter. I've already submitted a feature request in the issue tracker for a DocSite search, which I'm currently working on and have been planning for a while now. The indexer and search engine are being written from the ground-up, but are nowhere near completion yet. I also plan to provide an administration screen for managing the index and viewing statistics. The search will be in addition to the filter and will appear in the top-right corner of the DocSite template, just below the link to download the compiled help file.
• The TOC icons, as you mentioned, can be changed easily since it's just an ASP.NET TreeView control. To change the icons open the DocSiteContents.ascx file and change the ImageSet attribute's value from WindowsHelp to Msdn, for example. Maybe I'll change the default in the next release :)

Thanks again Ben! And now back to our scheduled programming…

Custom Topics - But other tools are doing it!

DocProject supports the addition of conceptual content through its partial-build functionality, which I'll explain below. I'm aware that this may not be ideal for most authors.

If you've used other community projects that automate Sandcastle you may have noticed one important difference between them and DocProject: Other tools provide extensions to Sandcastle, such as Eric Woodruff's Sandcastle Help File Builder (SHFB), which apparently has support for additional topics, custom build components such as one that formats code examples, and even patches for some of Sandcastle's presentation bugs. I think it's really great that Eric has done all of that work, but my philosophy for DocProject is different (and no, it's not to slack off ;).

First and foremost, DocProject's purpose is the integration of Sandcastle into Visual Studio 2005 and to expose its core functionality through easy-to-use dialogs and the New Project Wizard. DocProject does not provide any custom build components or bug fixes for Sandcastle, and does not add any additional functionality to Sandcastle's standard build process. DocProject exposes Sandcastle's configuration and transformation files to end-users with minimal changes so that they can customize Sandcastle without being restricted to GUIs, the same as on the command-line without using DocProject or any other automation tools.

A feature that allows users to add additional non-API topics may require a custom build component, custom configuration file or custom transformation files to be shipped with DocProject, possibly breaking DocProject's open-Sandcastle-source policy. In addition, or alternatively, DocProject could make changes to Sandcastle's intermediary files during the help-build process to support the addition of conceptual content, but IMO that would be a hack. Sandcastle already supports conceptual content for Microsoft's internal use according to this blog post – it just hasn't been exposed to the community yet.

As soon as Sandcastle provides a standardized and supported way to include conceptual content in the canonical build process, I'll begin work so that DocProject will support it too.

If Sandcastle goes RTW before providing this functionality, which to be perfectly honest is what I've expected for a while now, then I will attempt to extend DocProject so that it supports the addition of conceptual topics without Sandcastle's support. But let's hope Microsoft realizes how important it is that Sandcastle provides this functionality before it is RTW.

I also plan to offer a Custom Topic Management Dialog that will allow users to manage their conceptual content and even create new topics based on an HTML template, specific for the Sandcastle presentation chosen for DocProjects and DocSites.

Once DocProject fully supports the addition and management of conceptual content it will be a practical solution for generating compiled help in Visual Studio 2005, with designer support, and even without the need to build API documentation.

If you haven't already then hopefully now you see why DocProject does not support the addition of custom topics in the same way as the other community tools.

Creating Custom Topics with Partial Builds

When I first started getting some feedback from the community about conceptual content, not too long ago, I decided to write a tutorial that would help users take advantage of the partial-build functionality offered by DocProject. The tutorial is meant for anyone that uses DocProject, but is especially useful for help content authors that are already familiar with Html Help Workshop's file formats and manually editing them to add additional content. However, this tutorial doesn't describe the end of DocProject's support for custom content, but currently it's the easiest way without having to modify DocProject's code-base.

Partial Builds

To see a partial build in action, load up a DocProject or DocSite and build it as you normally would, but make sure that it will output compiled help (configure your project to produce Help 1.x, for example). After it builds successfully, delete the compiled help file (.chm, for example) and try building the project again. Notice that there is only 1 build step this time since DocProject detects that the file dependencies used to generate the compiled help are still in-tact and that none of the source projects have changed. Only the compiled help file is rebuilt, forgoing the entire Sandcastle build process.

This allows authors to modify the Workshop TOC file manually (.hhc) for the addition of conceptual content and DocProject will just rebuild the compiled help, which will include your changes. If you're using a DocSite template then the DocSiteContents.xml file will be synchronized automatically during a partial build, so your changes to the Workshop TOC will appear in the DocSite's TOC as well.

Conclusion

DocProject does support the addition of conceptual content, but not in the way that most end-users desire or expect since it's not entirely automated. The lack of complete automation is because Sandcastle does not provide a standardized means yet for adding conceptual content, but once it does DocProject will integrate the process into Visual Studio 2005, without hiding the implementation details from end-users. The process should still be flexible and customizable to a certain extent. DocProject, as with several other features of Sandcastle, will provide a management dialog for authors to add, edit, delete and organize their conceptual content. As such, DocProject may eventually be a productive environment for creating and managing help topics and building compiled help inside Visual Studio 2005, even without API topics to be documented.

I'd like to know your thoughts on this so please leave a comment here, start a discussion on CodePlex or contact me using my site's contact form. Thanks!

I was browsing the web the other day looking for community feedback on DocProject and I was pleased to come across two public websites that use DocSite templates to provide public API documentation over the web.

XML Library API

The first DocSite that I found is at http://www.xmllab.net/mvpxml. It was created by Microsoft MVPs in XML technologies for their XML Library API on CodePlex. Nice work guys :)

Full Motion Racing

I stumbled across another DocSite at http://lfs.fullmotionracing.com/ for a project called, LFSLib.NET - A Live For Speed InSim/OutSim/OutGauge API. More information about the project can be found at the author's blog: http://www.claassen.net/geek/blog/.  The latest post even mentions DocProject: LFSLib.NET 0.13b released.  Thanks!

Others?

If anyone else has a public DocSite that they'd like to share with me please let me know :)

DocProject 1.6.2 was released and is now available for download on CodePlex.

I want to thank all of DocProject's users that took the time to report bugs and provide diagnostic information to me. You've been helpful.

Recently, Anand blogged about DocProject and some of the other community projects that can automate Sandcastle to generate help for managed APIs. After his post, DocProject's rate of download increased by three times and then kept increasing. And since then I've certainly received more feedback than usual. Thanks Anand!

Updates

The latest release addresses some issues that users were reporting relating to the refresh version of the June 2007 CTP of Sandcastle, which was released a few days after its original release. I would have updated DocProject sooner if I knew :)

There is also support now for adding external sources (assemblies and corresponding xml documentation). This new feature also provides support for Web Site Projects since you must use a Web Deployment Project to generate a managed assembly, which can be added as an external source for your DocProject or DocSite.

In Progress

Currently, I'm looking at providing the ability to create and manage custom topics. I noticed that in Sandcastle's June 2007 CTP refresh there are some transformations and files related to conceptual topics. That sounds interesting to me and may be the feature that people have been asking me to add. I'll definitely research this more.