January 17, 2008

DocProject Roadmap

As you may have already heard, Sandcastle [1] was officially released to the web (RTW) through CodePlex on January 15, 2008.  The license that was chosen is the Microsoft Public License (Ms-PL) [2], but keep an eye on it because it may change [3].  There are still a few unanswered questions that I have [3] about how Microsoft's work on Sandcastle will continue and how it will affect DocProject, but things are looking very positive.

In this blog post I'd like to express a few of my goals and ideas for DocProject and maybe even try to establish some preliminary timeline.  And of course, if you have any thoughts your feedback will be welcomed :)

The End of Phase 1

With the help of community feedback and my propensity for being anti-social in favor of sitting in a dark room, programming, DocProject's long initial development phase is nearing an end.  The next release, which you can read about in my blog (DocProject 1.10.0 RC Preview [4]), brings together a few long-awaited features that will hopefully start to make DocProject into a useful help authoring tool (HAT) [5].  I'm going to try for Monday, January 21st, to release 1.10.0 RC, and I'm hoping that the worst-case scenario will be the following Friday.

From there I expect to deploy only one more Release Candidate that contains various outstanding bug fixes and feature requests and then I'll freeze DocProject's feature set for Visual Studio 2005.  The following deployment, 1.12.0, should therefore be the first DocProject Production release.  Expect that in March, 2008.

Visual Studio 2005 vs. Visual Studio 2008

I expect to have my own licensed copy of VS 2008 Standard edition sometime before the end of February, but until then I've decided that I will not be concentrating on any new deployments for DocProject 2008 Beta, although I'm very excited about using VS 2008 and I'll certainly jump on that bandwagon as soon as I can.

So for now, you can expect development to continue like normal for VS 2005.  But once I start concentrating on VS 2008, I'm probably going to freeze the feature set for VS 2005, distribute the first Production release of DocProject 1.x, and then switch gears for .NET 3.5 and, possibly, VS Package development.  I intended to eventually release DocProject as a true package, hopefully with its own editors and actual project types [6] (i.e., DocProject and DocSite Templates [7] that are recognized by Visual Studio as distinct project types, with their own property pages, etc.).

After DocProject 1.x goes into stabilization (the end of phase 1) I'll certainly provide support for it and help people whenever I can if they have questions about how to modify the source code or how to add additional features themselves.  Although, the only official deployments that you'll see will probably be major releases that fix a large number of bugs.

Well that's my plan anyway because in the last couple of DocProject 2008 releases I've realized that maintaining two versions of the project is almost like double the work; however, since Visual Studio 2008 will allow me to target the .NET 2.0 Framework I may be able to work on both versions of DocProject simultaneously, without much additional effort.  If that's true then hopefully I won't feel the need to freeze the DocProject 1.x feature set.  Although, technically I can't really freeze DocProject's feature set anyway since it's open source ;)

DocProject Documentation

I plan to distribute compiled help for DocProject 1.x that consists of both API reference and conceptual documentation after the first Production release.  I'd say that a week or two should be enough time to author useful, preliminary documentation.  After that I'll probably just add content here and there and deploy whenever I reach certain milestones that I define on the fly.

A .chm will be downloadable from CodePlex and the installer will present you with the option to have DocProject's help content (.HxS) merged with Visual Studio's help.  I'm even considering hosting a DocSite as an online reference and as a great working example of DocProject's and Sandcastle's capabilities.

Phase 2

The next phase will pickup with the development of DocProject 2008, targeting Visual Studio 2008 and the .NET 3.5 Framework.  The timeline will probably start sometime in March, 2008.  From there I hope to deploy on a monthly basis as I've done with DocProject for the past year+ (since December, 2006 as a matter of fact).

Planning

Although DocProject, in the next release [4], will provide first-class support for building mixed reference and conceptual documentation, unbounded filtering capabilities, much improved performance and a mostly finalized API, I still see lots of room for improvement.  Namely, in the area of content-based features such as for authoring topics and localization.  Also, here's a few of the outstanding tasks [8] that I'm going to look into for DocProject 2008: 

  • Running DocProject in areas of heightened security (e.g., Vista with UAC enabled).
  • SQL Server full-text search provider for the DocSite templates.
  • Microsoft Word and Adobe PDF output.
  • The ability to annotate comments with developer notes. 
  • Automatic token replacement.
  • Some new DocSite skins, maybe including one that makes judicious use of SilverLight.  (Documentation might start looking better than the software it documents, if I achieve my goal, that is ;)
  • Change control and better support for team scenarios; e.g., the ability to have a server that continuously builds live documentation to keep it up-to-date.
  • The development of a simple community wiki for the DocSite templates, which I guess we can start calling the "DocSite Community Wiki" feature (sure, some might say it's a rip-off of the MSDN Community Content Wiki idea, and I guess they'd be correct ;).  This should be very useful in team scenarios for large, volatile, hosted documentation sets.
  • The ability to register multiple Build Process Components [9] per project, with a management UI as well.  I may even turn the Sandcastle/Deployment [10] engine into a BPC that can be added to any DocProject or DocSite.
  • Other various user-defined tasks (bug fixes and feature requests).

I have even more ideas that I've been tossing around for a while, so I hope to aggregate them into a blog post and eventually add them as work items in CodePlex.  And if you have any ideas for DocProject features please let me know!

Development

Development will target Visual Studio 2008 and the .NET 3.5 Framework, and as I mentioned previously, I'm definitely considering rewriting areas of DocProject as a true VS Package with its own custom editors and project types.  I'm also considering using WPF for the interfaces.  There will be a learning curve but it's one that I'm going to have to accept eventually (and I really do want to learn).

Collaboration

I expect to continue collaborating with community members to find bugs and DocProject's potential for new features.  It seems to me that the experience of DocProject users has been positive, so I'd like to continue working on DocProject in the future in the same way as I've been for the last year.  But if you feel differently please let me know.  Even though it's open source I'd still like to provide an acceptable level of service and support in my free time since the time that I put into helping end-users is also time that I'm putting into DocProject (even if it's just thought), which helps to make a better product even for me to use.

Extensibility

It might also be worth noting that I'm going to maintain the "openness" of DocProject by continuing to introduce new public APIs and areas for extensibility whenever possible.  With the limited feedback that I've received, I'd say that DocProject's high level of extensibility was a success.  A few people have mentioned to me that their businesses have created custom build engine providers [11] and build process components, and I've even used BPCs to help people debug issues in the past.  Actually, my favorite feature of DocProject might actually be the build process component, which, incidentally, doesn't really have anything in particular to do with documentation.

Deployment

It's hard to predict when the first Production release for DocProject 2008 will be deployed, but if I can use the past year as a metric then I'd say another year from now is certainly reasonable, and possibly even longer than might be required.  To be honest, I think the summer of 2008 is a reasonable goal as long as I keep my eyes on the prize.

Phase 3

At the start of phase 3 I hope to have already achieved my major goals for DocProject, which includes much better integration into Visual Studio and additional features that will make authoring documentation extremely simple and, in many cases, automated.  Thanks to Sandcastle I believe that this goal is attainable.

For the future I think I'm aiming for much better external support so that non-Visual Studio users will be able to benefit from DocProject's features as well.  The team-based features that I mentioned previously might get pushed into this phase, but we'll see how it goes.  And finally, I'd like to hear your ideas for where you'd like to see DocProject go.  After I'm able to meet most of my own goals I'd like to brainstorm with the community to come up with some new and innovative ideas to be a part of DocProject in the future.

References

[1] Sandcastle on CodePlex, http://www.codeplex.com/Sandcastle

[2] Microsoft Public License (Ms-PL), http://opensource.org/licenses/ms-pl.html

[3] CTP, License and Source Code, http://www.codeplex.com/Sandcastle/Thread/View.aspx?ThreadId=20557

[4] Dave Sexton's Blog: DocProject 1.10.0 RC Preview, http://davesexton.com/blog/blogs/blog/archive/2008/01/15/docproject-1-10-0-rc-preview.aspx

[5] Help authoring tool. (2008, January 9). In Wikipedia, The Free Encyclopedia. Retrieved 11:22, January 17, 2008, from http://en.wikipedia.org/w/index.php?title=Help_authoring_tool&oldid=183180723

[6] DocProject Work Items: VS 2008 Shell and Extensibility, http://www.codeplex.com/DocProject/WorkItem/View.aspx?WorkItemId=14029

[7] DocProject Components, http://www.codeplex.com/DocProject/Wiki/View.aspx?title=DocProject+Components

[8] DocProject Work Items, Advanced List, http://www.codeplex.com/DocProject/WorkItem/AdvancedList.aspx

[9] DocProject's Build Process, Build Process Components, http://www.codeplex.com/DocProject/Wiki/View.aspx?title=Build+Process#BuildProcessComponent

[10] DocProject Sandcastle/Deployment Plug-In, http://www.codeplex.com/DocProject/Wiki/View.aspx?title=Sandcastle+Deployment+Plugin

[11] DocProject tutorial: Creating a Build Engine Provider, http://www.codeplex.com/DocProject/Wiki/View.aspx?title=Creating+a+Build+Engine+Provider

January 15, 2008

DocProject 1.10.0 RC Preview

In this post I'm going to provide details and screen shots of some of the new features in the DocProject 1.10.0 Release Candidate, which includes several bug fixes and new features such as first-class support for conceptual topics (additional content written in MAML markup), some new tool windows for Visual Studio Standard+, and increased performance of the Build Assembler step.

I plan to deploy after the next Sandcastle release, which will give me some more time for additional testing, to ensure compatibility and to finish a few incomplete features.  And BTW, I'm going to try to get back on my monthly deployment schedule again :)

Improved Performance

First, I'd like to start out by saying that performance is something I should have paid closer attention to throughout development, but at some point I just lost focus on it (probably when I started testing exclusively against smaller documentation sets).  So as Sandcastle's performance improved, DocProject's didn't.  I think I may have finally redeemed DocProject in this release though.

A new project option named, Build assembler options allows you to choose from various settings that have a direct affect on the performance of the Build Assembler step:

Build assembler options Description
None The build trace will not contain any output from Build Assembler and it cannot be canceled by a UI command, substantially improving performance.  Choosing this option will disable all other options.
Trace information Informational messages from Build Assembler will be included in the build trace.
Trace warnings Build Assembler warnings will be included in the build trace and the Error List.
Trace errors Build Assembler errors will be included in the build trace and the Error List.
Trace all All messages from Build Assembler will be included in the build trace; warnings and errors will also be appended to the Error List.
Cancelable Build Assembler will be executed asynchronously so that the UI will remain responsive and so that it may be canceled by a UI command, at the cost of performance.

The options that you have chosen will be used whenever Build Assembler is executed, for all types of builds; e.g., reference, conceptual, help 1.x and 2.x.

The default for new projects is Trace errors only, which should provide the same performance as None for a build that does not fail, while providing diagnostic information if it does.

In testing, a substantial speed increase of approximately 8 times is normal when compared to building in Visual Studio and the External UI in the 1.9.0 RC.  The speed is now similar to that of building a DocProject or DocSite on the command-line with MSBuild.  Although, with informational and warning messages disabled there is no log or trace of the tasks that Build Assembler performs and the UI is not responsive while the Build Assembler step is executing.

Enabling Trace all and Cancelable will provide the same behavior as in previous versions of DocProject; i.e., all information and warnings are traced and the build may be canceled at any time.  And although there's an obvious performance penalty with these options enabled, there will also be a noticeable improvement of approximately 5 times compared to 1.9.0 RC.  The speed is now similar to the speed increase garnered by opening a modal dialog during the Build Assembler step in 1.9.0 RC, as per the discovery by Cis in this discussion.  So if you want the Build Assembler step in 1.10.0 to function the same as in 1.9.0, enable these two options and you should still see a speed increase of up to 5 times.

Build assembler options is located under the Build category in the DocProject Properties window.  It's implemented as an enum type with FlagsAttribute, EnumFlagsConverter and related classes (link is to the 1.9.0 RC codebase).

DocProject Topic Explorer New Tool Windows

In preparation for first-class support of conceptual content I had previously written an XmlTreeView control that was included in DocProject 1.9.0 RC.  It extended the FCL TreeView control by adding support for internal and external drag & drop operations, an XML data source and a specialized binding manager.  In 1.10.0 RC it has been refactored and new features have been added so that it's now being used to its full extent in the Topic Management dialog (formerly known as the API Topic Management dialog) and the new Topic Explorer tool window (shown at right).

The new Topic Filters and Topic Editor tool windows were also extracted from the old API Topic Management dialog's Topic Filters and XML Comments tabs, respectively, so that they can be used without having to open a modal dialog (in VS Standard or higher).  Both windows are automatically synchronized to the selected topic in Topic Explorer.  They function as documents although since they're actually tool windows there can only be one instance of them opened at any given time, which means that they can only be used to manage one topic at a time.  Their appearance and functionality hasn't changed since DocProject 1.9.0 RC (preview post) so I'm not going to go into any more detail about them here.

Topic Explorer

The new Topic Explorer tool window is only available in Visual Studio Standard edition or higher.  It can be shown by pressing the Topic Management button on the Sandcastle toolbar and supports docking with other tool windows or floating on its own.  It functions the same as the tree view from the old API Topic Management dialog, with additional features.

Table of Contents

You can drag & drop topics to create the table of contents (TOC) that will be used in compiled help and the DocSite templates.  The TOC will be generated exactly as it appears in Topic Explorer.  Its layout is stored in a file named topics.xml, found in your project's Help\Settings folder.

When you save your changes by clicking Visual Studio's Save All command button or menu item, Topic Explorer will rewrite the topics.xml file based on the current organization of its tree.

Auto-Generated Reference Topics

The Namespaces topic, which is the root API reference topic generated by Sandcastle, starts out as a placeholder topic and loads in the background so that you can manage the TOC without having to wait for Sandcastle.  Once Sandcastle has generated the reflection XML document and the reference TOC, the placeholder is automatically replaced with the Namespaces node (for large documentation sets this may freeze the UI for a few seconds while the tree is being updated).

This node does not appear, however, if your project has no valid project references or external sources.

You can drag & drop the Namespaces topic or its placeholder topic at any time to place it within the TOC; however, drag & drop is not allowed for the individual reference topics.  Neither is dropping a conceptual topic anywhere within the hierarchy of the Namespaces topic.

When saved, the placement of the reference TOC is stored as a single <stoc /> element, which may appear anywhere within the Help\Settings\topics.xml file.

Editing Topics

To open the new Topic Editor window for any particular API reference topic, simply double-click the topic in Topic Explorer.  To edit a conceptual topic simply double-click the topic in Topic Explorer and it will be opened in Visual Studio's XML editor.

I really wanted to provide the same editing support for conceptual topics as is available currently for reference topics, however, I discovered after some experimentation that the MAML schemas are a bit too complex for my simple XmlContentEditor control, and not to mention that HTML is not currently passed through into the final HTML topic as of the Sandcastle Oct. CTP, which would make an HTML-based editor kind of pointless.

In the future I hope to create editors for some of the specific MAML schemas and make them as dynamic as possible.

Toolbar

Topic Explorer's toolbar provides the following functions (from left to right):

Button Description
ToggleExpansion Toggles expansion of all nodes.
Filter2HS Opens or activates the Topic Filters tool window for the selected topic, which provides a document-like UI for the regular expression and categorical filters that are also present in the Topic Management dialog.  Note that you can also filter topics simply by unchecking them in Topic Explorer.
Refresh Synchronizes the tree with the active project.  Topic Explorer is automatically synchronized with the selected DocProject or DocSite in Solution Explorer, although if the topics.xml file is manually updated then the refresh button must be clicked to update the display immediately.  (This should be automatic though in a subsequent release.)
NewFolderHS Creates a new container topic that does not have an associated file, although as of the Sandcastle Oct. 2007 CTP this doesn't appear to be supported so I'll probably just hide the button for now.
NewDocumentHS Creates a new conceptual topic node as a child to the current node and asks the user which conceptual template should be used to create the file on disc.  The file is then opened in Visual Studio's XML editor and the node is placed into edit mode so that the label can be set.  The label is used as the topic's title.
Open Imports existing XML topic files into the project.  Note that existing topics must use MAML to be supported by Sandcastle; however, if a topic does not use MAML then you're free to update it using Visual Studio's XML editor, manually, after it's imported.
Delete Deletes the selected topic after confirmation.  Reference topics cannot be deleted however.  If a container topic (with the folder icon) is currently selected then its entire hierarchy of topics, including itself, will be deleted from the TOC and disc.  If an individual topic is currently selected then only it will be deleted from the TOC and disc, and if it has child topics they will be promoted to children of the deleted topic's parent.

Sandcastle Build Component Editors

I've only actually attempted to create one editor at this time, but expect to see a few more by the time I deploy 1.10.0 RC.  The editor that I've created already is for ResolveReferenceLinksComponent2:

 image

To open this editor simply click the ellipses button (...) of a Reference Link Types component in the DocProject Properties window:

Resolve Reference Links Property

MSDN hyperlink options

These options apply to all hyperlinks that are generated for the MSDN link type.

The Target window option indicates the target for all hyperlinks created by the component for MSDN topics online.  It's implemented as a combo box that accepts the name of a target window (i.e., any string) or one of the values from the drop down list: _blank, _parent, _self or _top.  The default is _blank, which means that clicking the hyperlink to an MSDN topic will open up a new window.

MSDN Locale indicates the default locale to use when generating links to MSDN topics online.  The combo box accepts a custom locale string or you can select one from the drop down list, which is automatically populated with a long list of known locales, none of which are guaranteed to have content on MSDN.  The default value is en-US.

XML reflection file targets

The grid lists each of the individual <targets /> elements of the component, one per row.  You can delete existing rows or add your own.

Base Path is simply concatenated, with respect to file path conventions, to the specified File Pattern.  If a Base Path isn't specified then the current directory is used (most likely being your project's, buildhelp\assembler directory).

File Pattern indicates the XML reflection files from which hyperlinks may be generated.

Recursive indicates whether the Base Path should be searched recursively for files that match the specified File Pattern.

Link Type must be one of the values from the drop down list: None, Local, Index or MSDN.  You can read about the different link types on the Sandcastle blog in this blog post.

Current Tasks

I'm still working on creating some of the Sandcastle Build Component editors and implementing a few work items.

Currently, I've been refactoring the behavior of state persistence for DocProject options.  In previous releases, options are committed immediately when a change is made in the DocProject Properties window and changes are buffered in the Tools Options page.  However, in 1.10.0 RC, options are committed only when the project is saved.  This is done manually by clicking Visual Studio's Save All command button or menu item (and the standard configuration of VS will automatically save the project before each build).  To support this feature, options have been branched into two categories (internally): project and customProject options are those that read and write their state directly using the IAnyProject.Settings property, whereas custom options, for example, might instead persist state to some file.  If you are writing your own DocProjectOptions implementation then you must override the new Save method in order to save changes to custom options.  For your project options (the ones that use IAnyProject.Settings) you shouldn't have to change anything.  The Commit method was used previously for this purpose, but it also was used to commit buffered changes in other situations, without persisting them to disc.  This type of functionality is now only required by the Tools Options page; although, clicking OK on Visual Studio's Options dialog will no longer save the changes to disc - it will only cause Commit to be invoked.  When the project is actually saved, then Save will be called.  In the DocProject Properties window changes are committed immediately, so Commit is never actually called (although that may change - I haven't determined it yet).  To check whether your custom options should be commited immediately have your code read the DocProjectOptions.CommitChangesImmediately property, which is already implemented in 1.9.0 RC.

The purpose of all this is to make sure that the behavior when updating DocProject options is always the same, regardless of whether they are project options or custom options or whether they are being edited in the DocProject Properties window or the Tools Options page.  The difference from previous versions being that changes are committed immediately and only saved to disc when Visual Studio's Save All command is executed.

Conclusion

The next release of DocProject finally provides that long awaited, first-class support for conceptual content.  Now you'll be able to use Visual Studio to easily create, manage and automate the generation of Help 1.x and Help 2.x documentation, optionally in combination with auto-generated API reference documentation.  This feature substantially increases DocProject's potential as a HAT (help authoring tool) and provides a strong foundation for more content-based features in the future.

December 12, 2007

AIP 1.0.0 Bypassed

Somebody from websecurity.com.ua has contacted me regarding a vulnerability of Auto-Input Protection (AIP).  I was able to confirm that reusing an older web form to bypass the AIP web control, even if the CAPTCHA image is no longer valid, is in fact a vulnerability of AIP 1.0.0.

See the original article here:  http://www.websecurity.com.ua/1568/

Although I believe that Security Through Obscurity [1] is not really security at all, one of the purposes of this control is to prevent unwanted SPAM such as automated comments in blogs, and in my experience it works as intended even though this vulnerability has always existed.  In most cases I think the potential risk of the control being bypassed is outweighed by the fact that its use is not related to security.  I think it's probably used mostly to prevent the nuisance of SPAM in blogs (that's how I use it) and forums, and to prevent unwanted user accounts from being generated, which may still be an appropriate use even with its current vulnerability as long as it works for now.

However, I certainly don't recommend AIP 1.0.0 to be used for the authorization of banking transactions or the arming of a nuclear weapon :)

The Attacks

Other than the obvious uses, AIP should be able to counter the following threats to itself (probably not an exhaustive list):

  1. Attempts to sequester many auto-generated images for pattern analysis by the bot's owner.
  2. Attempts to validate the same challenge repeatedly to learn, in an automated way, about the particular CAPTCHA algorithm being used.
  3. Attempts to repeatedly use the same challenge to bypass security.

AIP 1.0.0 does not provide protection for all of the points above.  If you can think of anything else that should make this list please let me know.

The Problem

In an attempt to remedy the situation I realized that it doesn't seem possible to have a 100% secure image-based CAPTCHA implementation because, at the very least, an implementation must be able to verify the submitted text and therefore must keep track of it either on the client or the server:

Client: Raw Text
When included in a response as raw text, such as in a query string or POST data, there is an obvious security flaw since bots can just find the "answer" in the response itself.  (AIP doesn't do this!)

Client: Encrypted Text
When encrypted in POST data, such as part of ASP.NET Control State (AIP does this in 1.0.0), there is an added level of security however it's still fundamentally flawed without taking extra precautions because the same page can be cached by a client and used repeatedly to bypass CAPTCHA protection, even using the same bitmap text and even when the CAPTCHA image has already been invalidated.

Server: Raw Text With Client Identity
When stored on the server, AIP must be able to identify the user agent posting the request as the same agent that requested the CAPTCHA challenge to begin with.  Regardless of whether the text is stored in-proc or out-of-proc, the easiest way to identify the user agent in ASP.NET is to use session state; however, this adds an additional requirement for a client: it must now respond with a session ID each time that it makes a request.  Commonly, a session ID is stored in a cookie on the client's machine; alternatively, it can be part of the request's query string instead.  But in both cases this can make the CAPTCHA susceptible to a Cross-site Request Forgery (CSRF) attack [2].  Though overall I think this can be the most secure solution with the help of absolute expirations.

The Solution

So here's what I've come up with to make AIP more secure.  The improved implementation will be available soon as part of the 2.0.0 Beta release.  Please let me know if you have any thoughts of your own about how security can be beefed up and I'll definitely consider it.

The main security points are:

  1. Configurable timeouts for requesting the image and responding with an answer.
  2. The same "user" that makes the original request for a page that contains the AIP web control must also be the one to respond with an answer.
  3. Only one CAPTCHA test can be active at any given time, per "user".
  4. Once an answer is given, the test is invalidated and cannot be used again (i.e., the current in-memory test cannot be used again, regardless of whether the answer was correct or not, but the actual text on the bitmap can still be used again if it happens to be generated, randomly, for a different request altogether).

Here's an outline of the implementation details, with the securest settings available all being enabled:

  1. Store the CAPTCHA text on the server instead of in the response and associate it with a value that uniquely identifies the user agent that made the request.
  2. Replace the last challenge, if one exists, so that only one at a time may be active.
  3. Invalidate the CAPTCHA image once it's requested for the first time after being generated (may not be appropriate in all situations).
  4. Minimize the amount of time in which a client has to request the image from the cache after it's been generated.
  5. Minimize the amount of time in which a client has to respond to a CAPTCHA challenge (may not be appropriate in all situations).
  6. Remove the CAPTCHA text from the server when an answer is submitted, regardless of whether it was correct or not.

1. Store the CAPTCHA text on the server and identify it by the user agent's ID.
The purpose is to ensure that only the same user that requested the CAPTCHA challenge can subsequently validate the CAPTCHA text.  However, this is not a RESTful [3] solution and it may not be 100% secure either since a CSRF attack [2] may be still possible due to the way Session state is implemented in ASP.NET (not to imply that there's a better alternative ;).

2. Replace existing CAPTCHA text.
The purpose of replacing the last challenge is to ensure that each user can only have one active challenge at a time, minimizing the chances that a bot will be able to scoop up multiple images to evaluate them asynchronously, without at least responding with an answer first.  It also cleans up server memory, with no reason to keep track of old tests.

3. Invalidate the CAPTCHA image after it's been requested once.
AIP 1.0.0 uses a sliding expiration for the image.  For added security, the expiration should not be sliding though - once the image is requested for the first time it should be removed from the cache.  But instead of assuming that this is the best approach for every case I've added a boolean property named, BitmapCacheKeepAlive that specifies whether the bitmap should be kept alive or expire immediately upon the first request.  The default value is false, meaning that once the first request is made the image is invalidated.  Requesting the image will result in the same message that appears when the sliding timeout has expired.  When the value is true, the sliding expiration will be used.  Note that if the value is false and an image is never requested then it will be removed from the cache automatically in the timeout specified by the BitmapCacheTimeoutSeconds property.

4. Minimize challenge request time.
This is already implemented in AIP 1.0.0 with the use of a property named, BitmapCacheTimeoutSeconds on the AIP web control. 

When a page with the AIP web control is requested an image is generated on the server and the timeout is started.  When the browser reads the page it will discover an HTML image tag and attempt to download it, but it must request the image from the server within the configured timeout period or else the image's key will be invalid.

This feature can be verified by using the same image URL twice in a row, after waiting for the sliding timeout to expire (the default is 15 seconds).  The first time a page is requested that contains the web control a URL will be generated and the image will be visible in the browser.  Copy the URL from the source and browse to it directly after 15 seconds; the result is an image containing the message: * AIP image not found in cache *.  If you're frequently receiving this message or if users report that your web pages take a while to load then increase the value of the control's BitmapCacheTimeoutSeconds property to give browsers more time to request the image.

5. Minimize challenge response time.
This has been implemented in AIP 2.0.0 as a timeout just like the bitmap request timeout, configurable via a property on the AIP web control named, BitmapTextTimeoutSeconds.  The default value is 30 seconds.  Another property on the AIP web control named, BitmapTextKeepAlive enables or disables the timeout.  The default value is false, indicating that the timeout will be used.  When true the challenge doesn't expire until an attempt is made.

Minimizing the response time may be appropriate for sign-in screens and pages with little information to read, but it's inappropriate in long documents and large data-input forms.  If you're using the control on a page that has a lot of reading material, such as a blog, or with many data entry fields, such as a long user membership form, then you should probably set BitmapTextKeepAlive to true; otherwise, increase the value of BitmapTextTimeoutSeconds to something that is more realistic than 30 seconds so that users will have time to read or enter data before submitting the form.

Alternatively, for a more secure solution, you can put the control on a separate web page by itself and show it after the form is submitted, but before the posted data is saved to your database.  This way you can keep the timeout short; maybe even shorter than 30 seconds, which should help to protect your site against the Human Solvers [4] attack and services such as the one provided by www.captchakiller.com.

6. One shot only.
Regardless of whether the first answer submitted is correct or not, the challenge is invalidated and cannot be used again.  If the value is wrong then a new challenge must be requested.  This feature, along with the new server-side storage, should fix the vulnerability in AIP 1.0.0 discovered by websecurity.com.ua.

How To Exploit AIP 2.0.0 Beta

I don't believe that the solution outlined above is 100% secure.  However, I don't think that any solution really is, but this new implementation should be enough to prevent hackers and spammers from bypassing the control's security when it's configured properly, depending upon the context in which it's being used.

To exploit any vulnerabilities in AIP 2.0.0 a malicious client can't just cache the page anymore with the new security measures in place since the session ID that was used to make the request is required in the response as well, in the form of a cookie or as a query string (depending upon how session state is configured in the site).  But even if the cookie value is obtained, somehow, it will eventually expire due to the session timing out.  If it's used before the timeout, only one challenge may be active at any time, per user, so it's possible that the test is no longer valid anyway.  And once the first answer is provided the challenge is invalidated automatically, regardless of whether the answer is correct or not.

In order for the AIP 2.0.0 web control to be breached, assuming that there aren't more vulnerabilities of which I'm unaware, a malicious client would have to:

  1. request the CAPTCHA image within the configured timeout period after the initial request is made for the page that contains the AIP web control.
  2. get the answer to the challenge.
  3. obtain a valid session cookie (or query string if the website is configured to use cookieless session state).
    1. The session ID must belong to the owner of the session that made the initial request for the page with the AIP web control (not the request for the image).  The owner could be the bot itself or it could possibly use a spoofed or hijacked session.
  4. submit the session information with POST data containing the correct answer to the challenge:
    1. within the configured challenge timeout period.
    2. before the user that actually owns the session submits their answer (assuming that the bot is using a spoofed or hijacked session).
    3. and before the user that actually owns the session requests a new challenge (again, assuming misrepresentation).

And finally, once an answer has been submitted the test cannot be used again.

Conclusion

This is still a work in progress.  I'll provide more information, probably in the form of a WIKI on CodePlex, around the time of deployment.  Also, I'd appreciate feedback from anybody that runs tests against the AIP web control to verify its security.  If you find a flaw, please let me know and I'll try to correct it.

References

[1] Security through obscurity. (2007, November 29). In Wikipedia, The Free Encyclopedia. Retrieved 20:40, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=Security_through_obscurity&oldid=174670068

[2] Cross-site request forgery. (2007, December 11). In Wikipedia, The Free Encyclopedia. Retrieved 19:33, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=Cross-site_request_forgery&oldid=177107398

[3] Representational State Transfer. (2007, December 12). In Wikipedia, The Free Encyclopedia. Retrieved 19:34, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=Representational_State_Transfer&oldid=177439704

[4] CAPTCHA. (2007, December 11). In Wikipedia, The Free Encyclopedia. Retrieved 19:33, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=CAPTCHA&oldid=177256271#Human_solvers

November 20, 2007

DocProject 2008 Beta 2 Released

DocProject 2008 Beta 2 has been deployed to CodePlex.

  • Compatible with Visual Studio 2008 RTM (tested against Team Suite Trial).

  • Compatible with Visual C# 2008 Express and Visual Basic 2008 Express.

  • Supports side-by-side installation with the DocProject 1.9.0 RC for Visual Studio 2005.

  • Built using the same codebase as the 1.9.0 RC, with updates to support Visual Studio 2008, a few subtle changes that will also be included in the 1.10.0 RC, and some bug fixes.

Bug Fixes
  1. Topic Management Error: Value cannot be null
  2. Deleting a comment section does not mark the content item as dirty, so if it's the only change that's made it's not committed when the dialog is closed.
  3. The Cancel and Back buttons cause control validation on the Wizard Form, but they shouldn't.
  4. DocProjectEnvironment.InstallVSExtensions adds the same commands and tool windows to the commands and tool window collections even if they're already present.
  5. DocProject Properties window may be added to the wrong Project command bar, depending upon the environment.
Known Issue

The Build Component Stack configuration options display Load Error messages. This issue is related to a problem in Visual Studio 2008 that has yet to be resolved; however, if you manually add Sandcastle's assemblies to the GAC then you will not experience this issue.

See my blog post about this issue for more information.

Side-By-Side Installation

DocProject 2008 Beta 2 will work side-by-side with DocProject 1.9.0 RC, although you must update new and existing 1.9.0 DocProjects and DocSites so that they use a specific version of the DaveSexton.DocProject assembly.

To ensure that your 1.9.0 RC projects continue to work, follow these steps after installing DocProject 2008 Beta 2:

  1. Open an existing 1.9.0 RC DocProject or DocSite in Visual Studio 2005.
    1. If you need to create a new project instead then continue to the next step after completing DocProject's New Project Wizard.
  2. Expand the References folder in Solution Explorer.
  3. Right-mouse click the DaveSexton.DocProject assembly and select Properties.
  4. Set Specific Version to True. The version number, which is read-only, should automatically change from 2.0.0.0 to 1.9.0.0.
  5. Save the project.
If you do not follow these instructions then you will get an error when you attempt to build the project that states, The specified type is not a valid BuildProcessComponent implementation.
November 16, 2007

DocProject 2008 Beta 2 - AssemblyResolve Showstopper

April 27, 2008 Update: The problem is with Type.GetType forcing a complete JIT when executing in VS 2008 (it is used by InstantiateProviders).  I've submitted a new bug report that provides a much cleaner solution to reproduce the problem: https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=340664

Nov. 20, 2007 Update: I've narrowed the problem down to a call to System.Web.Configuration.ProvidersHelper.InstantiateProviders, which tries to resolve all dependencies at the time of invocation even if they are not being referenced in any executing code.  If assembly resolution must be deferred (as is the case in DocProject) then this presents a problem because there will not be an AppDomain.AssemblyResolve event handler registered to help the framework resolve the assembly.  However, this behavior only occurs when VS 2008 is hosting the executing code.  It does not occur in VS 2005 or in a console application built on the .NET 3.5 framework, for example.

I've submitted feedback to the Sandcastle team to have them GAC the required Sandcastle assemblies to prevent the issue in this blog post from occurring.  For now though, you can manually GAC the BuildComponents.dll, BuildAssemblerLibrary.dll and CommandLine.dll assemblies using: gacutil /i assembly_name.dll

I will be releasing DocProject 2008 Beta 2 shortly.

I've finished working on DocProject 2008 Beta 2, however, there is an issue with the new build component stack properties that has prevented me from releasing it: When the stack properties are expanded, all of their components are replaced with load error messages.

If you feel that this issue should not be preventing me from releasing the product then please let me know and I'll consider removing the expandable stack properties for now in 2008 Beta 2.

More Details

The problem is that an assembly cannot be delay-loaded in certain situations while a program is being hosted by VS 2008 since the AppDomain.AssemblyResolve event is not being raised.  The same code works in VS 2005 and using .NET 3.5 in a console app, but it just doesn't work when executed by a VS 2008 add-in.

It appears to be the same bug in VS 2008 that I reported back in August, but at the time I had discovered a work-around that was successful because the assembly was being loaded in another AppDomain.  Unfortunately, it won't work this time since the components must be loaded in the default AppDomain.

I've submitted new comments (starting at 11/16/07) in the feedback with a project that reproduces the bug attached, so I'm hoping that someone will post a workaround (I assume that it's way too late to get fixed in VS now).  Download the attachment from my website if you think you can help.  The instructions are in my feedback post :)

Here's the output that the example produces in the debug window.  Notice how the console version of the program appropriately raises the AppDomain.AssemblyResolve event, where as the VS add-in version does not.  The exception is actually not the problem - it's to be expected since I didn't add any code to actually resolve the assembly.  (The point is that VS 2008 doesn't even give DocProject a chance to resolve the assembly).  And BTW, both hosting scenarios in my example call into the exact same code.

The example was developed on the latest release of the Windows Server 2003 VPC for Visual Studio 2008 Team System Beta 2.

CONSOLE OUTPUT:

'ConsoleApp.vshost.exe' (Managed): Loaded 'C:\Documents and Settings\Administrator\My Documents\Visual Studio 2008\Projects\TestAddIn\ThirdPartyLibrary\bin\Debug\ThirdPartyLibrary.dll', Symbols loaded.
AssemblyLoad: ThirdPartyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8fd3dad2bd735269
*********** Loading type...
AssemblyResolve: ThirdPartyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8fd3dad2bd735269
A first chance exception of type 'System.IO.FileNotFoundException' occurred in PlugInLibrary.dll
The thread 0xbfc has exited with code 0 (0x0).
The thread 0xd08 has exited with code 0 (0x0).
The program '[1180] ConsoleApp.vshost.exe: Managed' has exited with code 0 (0x0).

VISUAL STUDIO OUTPUT:

'devenv.exe' (Managed): Loaded 'C:\Documents and Settings\Administrator\My Documents\Visual Studio 2008\Projects\TestAddIn\ThirdPartyLibrary\bin\Debug\ThirdPartyLibrary.dll', Symbols loaded.
AssemblyLoad: ThirdPartyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8fd3dad2bd735269
*********** Loading type...
A first chance exception of type 'System.IO.FileNotFoundException' occurred in PlugInLibrary.dll
'devenv.exe' (Managed): Loaded 'C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.TeamFoundation.Build.dll'
AssemblyLoad: Microsoft.VisualStudio.TeamFoundation.Build, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
The program '[3292] devenv.exe: Managed' has exited with code 0 (0x0).

November 09, 2007

DocProject Tips and Tricks I

As you probably already know, DocProject is kind of weak in the documentation department (that is, in its own API documentation :)  I plan to address this issue for the first RTW, after I include first-class support for generating conceptual content and after Sandcastle's first RTW.  If you are new to DocProject, the articles on CodePlex should provide more than enough information to get you started.

In a series of blog posts, with this post as the leader, I'm going to provide some supplementary information that you may find useful.  Tips and tricks that will add value to DocProject and allow you to make the most of it.  Code examples can be found at the end of posts and will have names like, Example #.

Community Tricksters

As the author, I probably know more about DocProject than anyone.  But as an end-user, I haven't taken advantage of the full range of its capabilities yet - including things like using DocProject with Team Build, building AJAX documentation or customizing Help 2.x output.  Therefore, I'd be happy to accept tips and tricks from the community that I can include in this series.  If you have ideas please submit them as blog comments or email them to me and I'll consider including them in my posts.  Thanks!

General-Purpose Tips

Here are some tips for DocProject that apply to all types of templates and build engines:

  1. If you do not want to include build output as project items, and you don't want to be prompted after you build a project for the first time, then set Include Project Output Dialog | Don't ask me this again to True in the DocProject Properties window before you build.  You can leave the default values for the two related options, which should both be False.  (Note that the Apply to all default option is not used when Don't ask me this again is True.)
  2. You can insert dynamic build steps using your project's Build Process Component (see Example 1 below).
  3. Use an external or third-party Build Process Component:
    1. Open the DocProject Properties window (right-mouse click your project in Solution Explorer and select, DocProject Properties).  Visual Studio Express users must use the DocProject External UI.
    2. Locate the Extensibility | Process component type name option.
    3. Enter an assembly-qualified type name (the assembly must be located in the GAC or your project's output folder).
    4. Alternatively, you can enter a full type name and an absolute or relative assembly path, separated by a vertical bar; e.g., MyNamespace.MyType|..\anotherproj\bin\Debug\anotherproj.dll
DocSites Tips

These tips apply to the DocSite Templates only:

  1. To change the DocSite logo simply replace the Images\Logo.jpg file with a file of your own.  To use a different file name, change the value of the ImageUrl attribute on the <asp:Image ... id="logoImage" ... /> element in the Controls\DocSiteHeader.ascx file.
  2. To show rank information in DocSite search results, for the purpose of fine-tuning the search index, open the App_Themes\BasicBlue\DocSiteSearch.skin file and set Visible="True" on <asp:Label ... SkinId="searchResultRank" ... />.  When performing a search the rank value will appear before the title of each matching topic.  If you already have search results in your browser just refresh the page.
  3. To change the DocSite title, company name and copyright, set the value of the AssemblyTitle, AssemblyCompany and AssemblyCopyright attributes, respectively, in the Properties\AssemblyInfo.(cs|vb) file and then restart the web application.  (The DocSite caches these values in memory upon the first request to the site.)
Sandcastle Tips

This section provides some useful tips for projects that use the Sandcastle or Sandcastle/Deployment build engine providers.

  1. Set the Help 1.x Configuration File Name and Help 2.x Configuration File Name options to the same values and DocProject will only run Build Assembler once to build both Help 1.x and Help 2.x output.  The cost of doing this is that reference link types in one of the help output formats might not be valid (this was the main reason for splitting the Sandcastle configuration file into two distinct files in the first place.)
  2. Use XML documentation files generated in the API Topic Management dialog for IntelliSense by adding a short code snippet in your DocProject's Build Process Component (see Example 2 below).
  3. In your Build Process Component, you can get access to all of the Sandcastle-specific project options and APIs.
    1. Add a reference in your DocProject or DocSite to the DaveSexton.DocProject.Sandcastle.dll assembly, commonly found at C:\Program Files\Dave Sexton\DocProject\bin\.
    2. Check out the code in Example 3 below, which illustrates how to access the custom Sandcastle APIs that are provided by DocProject.
  4. Insert images into your documentation using the XML documentation editor in the API Topic Management dialog:
    1. In design mode, place the text cursor where you want to insert an image.
    2. Click the InsertPictureHSbutton on the editor's tool strip or right-mouse click and select Insert > Picture.
    3. In the Picture dialog enter the full path and name of an existing image or click the Browse... button to locate one.
    4. You can enter a local file path or even a web address such as, http://example.com/someimage.jpg.  Click OK and the image is inserted.
    5. After you are done authoring your documentation, click OK on the API Topic Management dialog and DocProject will automatically import the image into your project's Help\Art folder.  It will also rebase the reference in the XML documentation to ..\Art\{image name}.
      1. Note: If you enter a web address then DocProject will copy the image from your web browser's cache; otherwise, the file is copied from the location that you specify.
    6. Note: The next time that you view the XML documentation in the API Topic Management dialog, the path will be rebased so that it's rooted, for display purposes only.  The path in your actual XML documentation on disc will remain relative, however.
  5. Filter out <Module> topics when the Documentation scope option is set to Complete by adding a dynamic regular expression filter:
    1. Open the API Topic Management dialog.
    2. Select the Topic Filters tab.
    3. Select the Regular Expression tab.
    4. Enter the following into the Pattern text box: \<Module\>
    5. Click the Save button.  The Saved Filters tab will be shown automatically and your filter will appear in the list.
    6. For the new filter, ensure that the Build column is checked and that the Include column is unchecked.
    7. Enter a short description of this filter into the Memo field (optional).
    8. Click OK on the API Topic Management dialog to commit your changes.
C# Code Examples

The following examples illustrate points made in the information above.  They are referenced by number.

Example 1:

The following example shows how to inject a dynamic build step immediately before the Help 1.x compiler step using your project's Build Process Component.  The step will be executed in full and partial builds and will run on a background thread.  It outputs some text in the build trace window:

Step # of ##: Hello

World!

To use this example, replace the BuildStarting method in your project's BuildProcess.(cs|vb) file.  (Keep the existing code though).

   1: protected override void BuildStarting(BuildContext context)
   2: {
   3:   // existing code goes here 
   4:  
   5:   base.InsertBeforeBuildStep("Compile Help 1.x", "Hello", true, false, true,
   6:     delegate(BuildContext stepContext)
   7:     {
   8:       stepContext.TraceLine("World!");
   9:     });
  10: }

Example 2:

The following example will overwrite the source XML documentation file (that contains your code comments) with the merged XML documentation file (that contains both your external documentation and the source code comments) after each help build completes.

To use this example, replace the BuildCompleted method in your project's BuildProcess.(cs|vb) file.  (The existing code should be placed at the end).  You must also add a using System.IO declaration at the top of the file.

   1: protected override void BuildCompleted(BuildContext context)
   2: {
   3:   TraceLine();
   4:   TraceLine("Overwriting XML documentation files in source projects..."); 
   5:  
   6:   IDocProject project = context.Engine.Project;
   7:   BuildSettings settings = context.Engine.Settings; 
   8:  
   9:   foreach (ISourceProject source in project.Sources)
  10:   {
  11:     string target = source.Output.XmlDocumentationFile; 
  12:  
  13:     FileInfo autoGenFile = new FileInfo(Path.Combine(
  14:       settings.WorkingXmlDocumentationPath, 
  15:       Path.GetFileName(target))); 
  16:  
  17:     if (autoGenFile.Exists)
  18:       autoGenFile.CopyTo(target, true);  // overwrite
  19:   } 
  20:  
  21:   TraceLine("Done."); 
  22:  
  23:   // existing code goes here
  24: }

Example 3:

You can access Sandcastle-specific project options and APIs in your Build Process Components by adding a reference to the DaveSexton.DocProject.Sandcastle.dll assembly.  The following example shows how to obtain the SandcastleBuildEngine instance, and from that retrieve the SandcastleSettings and SandcastleProjectOptions:

   1: SandcastleBuildEngine engine = (SandcastleBuildEngine) context.Engine;
   2: SandcastleSettings settings = engine.Settings;
   3: SandcastleProjectOptions options = engine.Options;

The context is an argument that is passed to all of the four main Build Process Component overrides: BuildStarting, BeforeExecuteStep, AfterExecuteStep and BuildCompleted.

November 07, 2007

DocProject 1.9.0 RC Deployed

The DocProject 1.9.0 Release Candidate is now available on CodePlex.  I'm still working on updating the wiki documentation though.

Thanks for your patience with this one.  I invested more time than I probably should have in additional features that weren't immediately necessary, but hey, this project is fun for me and some times I can't seem to come to a logical stop :)

New Features in DocProject 1.9.0 RC

  • Compatibility with the Sandcastle October 2007 CTP.
  • Preliminary support for conceptual content using Sandcastle's conceptual configuration file. (The conceptual build process has been integrated although it hasn't been tested and DocProject does not automatically generate a conceptual TOC yet.)
  • Sandcastle's build component stacks are now available as properties in the DocProject Properties window and the DocProject External UI, with support for hosting custom build component editors that can either show a modal dialog, a drop down editor or subproperties.
  • Framework and project reference link types are configurable as subproperties of the build component stacks and can be set by simply choosing one of the following values from a drop-down list: MSDN, Index, Local or None.
  • DocSite Templates include four new buttons under the Contents tab: Save, Bookmark, Email and Print. The latter two are supported in IE6, IE7, Firefox and Opera, but the former two are only visible in IE. Note that Save does not currently download supporting files such as JavaScript or style sheets though.
  • DocSite performance generating the full-text search index has been greatly improved.
  • Version Management Dialog: Create documentation that includes version information by referencing external assemblies and reflection files, and then simply enter custom version names (DocProject uses Sandcastle's VersionBuilder program in the background.)
  • ChmBuilder is used exclusively to build the supporting Help 1.x files (WARNING: The .HHC file format has been changed. Beware if you were modifying it to include custom topics.)
  • Both the Topic Designer and API Topic Management dialog now remember their last saved position, size, and the positions of splitters.
  • Edit all content item documents using the Topic Designer instead of just the shared_content.xml file. All of the available documents are listed in the Create Shared Content step of the New Project Wizard as well.
  • Several new API Topic Management dialog features:
    • Improved performance when applying filters.
    • Topic tree nodes now use icons that reflect access type, as do the tooltips.
    • Filter topics by API accessibility or in combination with API types; e.g., protected fields, private interfaces, public or protected methods and delegates, etc.
    • Save filters and load them later or have them applied automatically during builds (dynamic filters).
    • Author external XML documentation in the appearance of the selected presentation style, with special styles for XML documentation tags such as see, c, code, paramref, etc.
    • New XmlTreeView project and control, which supports binding to an IXPathNavigable data source and allows drag/drop internally and externally (although internal drag/drop support is not currently enabled. This feature is planned for conceptual content in a subsequent release).
    • Drag and drop API elements from the tree into the XML documentation editor to generate see links automatically (works in both source and design mode).
    • The appropriate XML documentation sections for each specific API type (.e.g, summary, remarks, return, value, etc.) are listed automatically in the XML documentation editor, including individual param and typeparam tags for methods and delegates.
    • Add additional documentation sections that support multiple instances (e.g., exception, permission) by selecting them from a drop-down list. The list only shows those sections that are appropriate for the selected topic.
    • Quickly create custom XML documentation sections in the editor by entering a name and pressing enter, or register them as built-in sections in DocProject's configuration file. See this blog post for information on how to use custom tags in Sandcastle's XSL transformations.
    • The XML documentation editor automatically cleans up empty sections from the external documentation when it's saved.
  • Build Process Components can now take advantage of more of DocProject's API and now have the ability to inject dynamic build steps, pass delegates to DocProject's API and register event handlers.
  • Documentation now includes a root Namespaces topic to which the XML documentation's project summary is applied.
October 27, 2007

DocProject 1.9.0 RC Preview

The DocProject 1.9.0 Release Candidate will contain several bug fixes and some exciting new features (well, exciting to me anyway :). In this blog post I'm going to describe some of the new features and show a few screen shots, like usual.

Miscellaneous Features and Bug Fixes

For the sake of time and keeping this blog post short (well, shorter than I could make it), I'm going to first list a few of the important features that I feel deserve to be mentioned, even if I'm not going to elaborate:

  1. The time it takes to generate the DocSite's full-text search index has been greatly reduced - to the point where it should actually be useable now.  In my testing, no more than 10 minutes for over 10 000 topics on a regular desktop computer.  I still plan to improve performance even more though and will develop that SqlSearchProvider very soon.
  2. The DaveSexton.DocProject.DocSites library no longer requires full trust.
  3. DocSites now have Save, Print, Email and Bookmark buttons in the Contents bar to act on the selected topic.  The Print and Email buttons work in IE6, IE7, Firefox and Opera.  The Save and Bookmark buttons only work (and are only visible) in IE6 and IE7.
  4. Build Process Components can now use more of DocProject's API, including the use of cross-AppDomain delegates and event handlers and they are even able to insert dynamic build steps that are actually build steps.   I'd like to blog about this separately though.
  5. The Sandcastle plug-in now uses the ChmBuilder program to generate the supporting Help 1.x files.  Localization has not actually been implemented yet, though I plan to address that soon.
  6. The Topic Designer (shared content) dialog now displays all of the files that have Content/<item> semantics, and they are listed as Content Documents (this includes documents in Shared\Content\ as well).  Selecting a document will automatically list all of its items underneath, and from their the dialog works the same as usual when an item is selected.  By default, shared_content.xml and header are selected, like usual.
  7. Bug fixes galore.  (Though, it was more bugs due to major refactoring in this release than existing bugs from previously releases :)

And now for some elaboration...

API Topic Management Dialog

Lots of new stuff and refactored code here.  I'm not really sure where to begin, so I guess I'll just start off by saying that the API Topic Management dialog now saves its last location and size, and the position of the splitters (and so does the Topic Designer).  Now, let's continue on with some of the more obvious and important changes.

image

Figure 1: API Topic Management Dialog - Tree View and Regular Expression Topic Filter

Tree View

As you can see from the image above, the tree now shows general topics that are defined by the presentation style's doc model (e.g., All Members, Methods, Fields, Properties, etc.).  Theses types of topics cannot be filtered with the checkbox; however, uncheck them and all child nodes will be unchecked automatically.

You may also notice in the image that the icons are now synchronized to the API access levels of each topic.  For example, in the image above, AI.Cells.Enzyme.acids is a private field, which has an icon with a lock.  The tool tip will also indicate that the topic is a "Private Field".

The +/- button above the tree will, when clicked, collapse all nodes if any one is currently expanded, and will expand all nodes if they are all collapsed.  There are other buttons that are currently hidden because they have to do with conceptual topics.  More on that later...

It's probably also worth noting that the performance of the tree when loading data and finding/applying topic filters has been improved.

Topic Filters

Figure 1 also depicts the improved Topic Filters tab and some of its new features.  But let's just start with the basics...

Regular Expression Filter

The options for the Regex filter are now displayed as checkboxes, simply for convenience since there is now more room in the GUI due to the filter types having their own individual tabs.  Also, the pattern can now be entered as multi-line, although since new lines will not match any topic names, entering multiple lines (to improve readability of your patterns) will cause the Ignore Pattern White Space option to be checked automatically.

Categories Filter

The categorical filter has also been improved, for performance and functionality.  It now supports filtering by API access levels, in addition to API category types.  For example, you can now filter by protected fields, or private interfaces and structures.

image

Figure 2: API Topic Management Dialog - Categories Filter

If nothing is checked under Accessibility then it is simply not taken into account when matching.  In other words, any access type is a match - the same as if all access types are checked.

You can also choose an access level without choosing an API type and it will match all APIs with the selected access level.  Choosing more than one access level is an OR operation, just like with API types.

To match protected internal members you must check protected and internal since it's treated as a distinct access type.  In other words, DocProject does not treat protected internal members as just being protected, or internal.  One caveat is that searching for protected internal members will also explicitly match protected-only and internal-only members as well.

Saved Filters

Saved filters are a new addition in 1.9.0.  It allows you to save Regex and Category filters so they can be loaded and applied quickly in the future or, more importantly, applied automatically during each build.

image

Figure 3: API Topic Management Dialog - Saved Filters

You can get filters into the list by clicking the Save button that appears on each of the filter tabs (see Figures 1 and 2).  Alternatively, you can directly edit the XML file in which they are saved, located at Help\dynamicFilters.xml.  The file does not exist until you add a filter using the dialog and then click OK, but you can create the file yourself if you want (see Figure 4 below).

You can also import filters from a Dynamic Filters XML file using the Import... button (see Figure 4 below).  Imported filters are appended to the list and do not replace any existing filters.

Memo is a place to comment on filters.  You can edit the memo directly in the grid after you save a filter and it will be persisted along with the filter list when you save the changes to the API Topic Management dialog.

Clicking a Load button will simply load the corresponding filter into the appropriate filter tab (i.e., Regular Expression or Categories) and will then show the tab and apply the value of the Include column to the Include matching topics Search Option.  The filter will not be applied, however.

The Apply button will immediately apply the filter to the tree using the current Search Options and the value in the Include column, but will not load the filter first.

You can delete the selected filter by pressing Delete on the keyboard or by clicking the delete button (with the red X).

The arrows are used to order the filters, which is important because all filters that have a check mark in the Build column are applied in order, top-down, during a help build.  The Apply all filters during builds option will either check or uncheck the Build column for all filters so that they have the same value as the option and each other.  The Apply All button will apply all filters to the tree view immediately, top-down, but without regard to the Build column.

The Include column indicates whether matching topics are included or excluded, just like the Include matching topics Search Option.

These features together provide a powerful way to filter topics.  For example, in Figure 3 above the 3rd filter excludes all private fields, but then the 4th filter ensures that one private field in particular, acids is included again since the Include column is checked.  The 4th filter will be applied after the 3rd filter, since it appears after the 3rd filter in the list, so the acids field will end up being included after all filters have been processed on that topic.

The schema of the XML file that stores the filters is simple, so I'll just give an example file that was created from the filters in Figure 3:

<?xml version="1.0" encoding="utf-8"?>
<filters>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.RegexTopicFilter">
    <include>False</include>
    <pattern>Carbon$</pattern>
    <memo>Get rid of any topic that ends with "Carbon".</memo>
  </filter>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.RegexTopicFilter">
    <include>False</include>
    <options>IgnoreCase</options>
    <pattern>^Startup$</pattern>
    <memo>Get rid of the Startup classes in each assembly.</memo>
  </filter>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.CategoryTopicFilter">
    <include>False</include>
    <memo>No private fields.</memo>
    <types>Field</types>
    <access>Private</access>
  </filter>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.RegexTopicFilter">
    <include>True</include>
    <pattern>^acids$</pattern>
    <memo>Reinclude the "acids" field, even though it's private.</memo>
  </filter>
</filters>

Figure 4: Example Help\dynamicFilters.xml file

Note that the type attribute will accept assembly-qualified type names so that you can define your own custom filters; however, I've not actually tested custom filter types yet so they may not actually work, but I plan to support them in a subsequent release.

The element names for a filter's options are defined by the filter itself, although memo, include and build (which is not shown in the example because it defaults to True) are built-in to the base type because they have special meaning to DocProject.

XML Documentation

In 1.9.0 there are some new features to help make authoring documentation easier.  But before I begin on that, I'd like to lead in with some info about where DocProject currently stands on handling conceptual content since it has had a profound effect on 1.9.0.

Conceptual Content

The next release of Sandcastle might be available on Monday (the 29th of October), and you can probably expect DocProject's release to follow within a few days. One thing that I'm really excited about is that the Sandcastle team will be shipping their internal conceptual configuration file for the Build Assembler program, which should allow me to enable the long awaited first-class support for conceptual content in DocProject.  If I can include support in 1.9.0 without too much delay then I certainly will, but if that's not the case then expect it in 1.10.0.

The Tree View and XML Documentation

In preparation for conceptual content, I've refactored the code some more and made some assumptions about how the process will be integrated.  Also, I've included my new XmlTreeView control with DocProject 1.9.0 RC (it's being used in the API Topic Management dialog now).  It supports drag/drop to reorder nodes (to be used for conceptual content only, once it's supported) and to drag nodes onto the XML documentation editor, which automatically creates a <see> link (that is supported in 1.9.0).  Also, the XML documentation editor now applies the style sheets of the selected presentation and a special designer style sheet that you can edit yourself: Help\Styles\TopicDesigner.css.  The designer sheet formats XML documentation markup such as <c> and <see> tags.  All of the style sheets are used in both the Topic Management and Topic Designer (shared content) dialogs.

image

Figure 5: API Topic Management Dialog - Xml Comments

The list of XML documentation sections is now generated automatically from DocProject's configuration file based on the type of topic that is selected.  Then the list is also merged with the actual elements found in the external XML documentation files, which allows users to add custom elements to external XML documentation files outside of the API Topic Management dialog and then use the dialog later to edit them.  There are other ways to manage custom elements, which I'll discuss later.

image

Figure 6: API Topic Management - Xml Comments Sections

The parameter that is selected in the image above was added by DocProject based on the reflection information that was provided by Sandcastle.  The method in the example only has one parameter, but if it had multiple parameters then there would be one param name="value" item in the list for each.

BTW, I just noticed a regression: the GenericClass should not show the back tick in its name - I'll fix that :)

You cannot delete delegate/method parameters or generic type parameters from the list.  You also cannot delete sections that may only appear once, such as summary or remarks.  But you can delete custom sections and sections that may appear multiple times, such as exception and permission, by right-mouse clicking and selecting Delete from the context menu.

To add an instance of a built-in section that may appear multiple times, you can select one from the drop-down list at the bottom.  The list displays only those items that are related to the API type of the selected topic.  You can also add custom sections by simply entering the name into the combo box and pressing Enter on the keyboard.

When a custom item is selected the attribute grid, which is displayed above the XML documentation editor, is completely editable, except for the Type column.  (Although I may change that - I haven't decided yet.)

You can create more built-in sections easily by registering them in DocProject's configuration file (commonly found at, C:\Program Files\Dave Sexton\DocProject\DaveSexton.DocProject.dll.config) under the sections element:

image

Figure 7: documentationTags element in the DocProject configuration file

XML documentation tag usage data has been retrieved from the following sources:

  1. Recommended Tags for Documentation Comments (C# Programming Guide)
    http://msdn2.microsoft.com/en-us/library/5ast78ax(VS.80).aspx
  2. The "Tag Matrix" that is present in Dynicity LLC's XML Documentation Comments Guide:
    http://www.dynicity.com/Products/XMLDocComments.aspx
    (Note: There are differences between the config data and the published matrix due to bug fixes and/or differences in versioning and tag/aspect applicability.)

Conclusion

There's much more, but I have to go catch a train now...

October 21, 2007

C# Multi-line XML Documentation Comments

I was just checking out the documentation on MSDN about XML comments and I discovered that C# supports multi-line comments for XML documentation as well:

/**
 * <summary>Hello World!</summary>
 */
public void HelloWorld() { }

Interesting, but I wouldn't recommend using multi-line comments for a few reasons:

  • They do not support Smart Comment Editing, where by Visual Studio inserts multiple comment lines and an empty summary tag automatically after starting the comment, which in this case is done by typing /** instead of ///.
  • Intellisense is limited:
    • top-level tags are listed but context is ignored (e.g., the para tag is never listed, even if the cursor is within a summary element).
    • Entering a tag name and pressing enter will not produce a closing tag automatically.
  • The rules for processing markup may be problematic since the compiler will look for a prefix pattern consisting of white-space, an asterisk and more white-space; however, a slight change in formatting in the prefix of one or more lines (e.g., less leading white-space) may cause the compiler to include one or more asterisks as part of the comments.
  • They are certainly not as common as triple-slash comments and will probably be confusing to most people (I know if I were to have seen this format used, and I already may have, I would have doubted whether it was even valid.)

I'm curious to know if anyone has found this feature to be more useful than triple-slash comments, and why :)

October 08, 2007

The Help! Project on CodePlex

I've just published the Help! project on CodePlex, which is intended to provide a fully managed compiled help system to replace Help 1.x (.chm).  Please contribute to this project by using the discussions area to post comments and ideas or join the development team.  If you want to join the team then please see the Welcome! thread for information.

For now I'll be using my blog to post news and information about this project.