In this post, I'll introduce you to my new OSS project called White Tie and I'll describe how it relates to Rxx a bit.

I'll also introduce you to a new feature of another OSS project of mine, Visual Studio Settings Switcher.  This new feature allows you to easily share VS settings files with your solution, and OSS contributors to easily take advantage of it.

White Tie and VS Settings Switcher provide a set of features that I've specifically intended to be used by OSS developers, though anyone can benefit from them really.

I know that some of the ideas in these projects are probably not new, but perhaps the way in which I've implemented those ideas as features is new enough, and simple enough, to make them better than competing projects.

Your feedback is welcome.

White Tie

I've recently created the White Tie project on CodePlex.  I like to think of it as "formal buildware for use after 6PM".

White Tie is basically just a couple of unintrusive MSBuild files that are imported by your VS projects.  It's primarily intended for OSS developers who use Visual Studio (or MSBuild) to build their projects into NuGet packages, and optionally use Code Contracts, run static analysis and like to copy all build output into a local aggregate directory.  There are also work items currently under consideration to build reference documentation via Sandcastle and to enable remote deployment scenarios, for example.

White Tie makes building, packaging and deployment easier by offering sensible, out-of-the-box configuration, though it's also highly customizable by adding particular MSBuild properties and items to your project file.  (Read the Configuration documentation for details.)

White Tie is delivered as a NuGet package, so it's really easy to integrate into your projects.  Simply build your project in Release mode and a NuGet package will be generated automatically.  No configuration is necessary for existing projects, in general, though read the note at the bottom here for details about build errors caused by missing assembly-level attributes in new projects.

One of White Tie's key features is the NuGetFlavor item group.  By adding project references as NuGetFlavor items, rather than as ProjectReference items, White Tie will automatically include the referenced projects' output into the primary project's NuGet package.

For example, let's say that you have a primary class library project called Project A, which targets the full .NET Framework 4.5.1.  If you add a NuGet reference to White Tie, then whenever Project A is built in Release mode (by default), White Tie will generate a NuGet package for Project A.  Now let's say that you also have a portable class library project called Project B, which targets a portable profile that includes Windows 8 and Windows Phone 8.  Project B is intended to be the same as Project A, only targeting different platforms.  You might refer to Project B as a "flavor" of Project A.  Now you can simply edit the project file of Project A to add a NuGetFlavor item that points to the project file of Project B.  Whenever you build Project A in Release mode, White Tie ensures that Project B is built as well, and it also includes the portable library built by Project B into the NuGet package that it builds for Project A, and in the correct target framework subfolder.

White Tie offers several more features and configuration options.  Take a look at the documentation to get started, and let me know what you think.

White Tie's Origins and Future

White Tie is something that I've been working on for a few years, in one form or another.  Much of the work began as part of the Rxx project, such as its support for NuGet, though some features even predate Rxx.

Since Rxx it has evolved into something a bit more useful, though I've only been using it internally for my own OSS projects and business-related products (under the name DaveSexton.Build).  I finally decided recently that if I was ever going to "officially" release Rxx again, I'd have to bring its build system and project configurations up-to-date first.  White Tie is basically the best of DaveSexton.Build, formalized and revised (or "reimagined", if you prefer advertising jargon) to take advantage of NuGet 2.8 features, including complete support for portable libraries.  My transition to White Tie is not yet complete, and there are currently some road blocks, but I'm committed to getting it working fully.

Note that this isn't a guarantee that I'm going to build another NuGet package for Rxx, although it's certainly a large step in the right direction.  Not only have I checked in updates to Rxx's source code to include references to White Tie already, but I've also begun to think about and make changes to the portable library configurations that Rxx will target in the future.  But even after that, I'm still going to have to comb over all of its APIs before I'll release an "official" package again.  I have OCD and I haven't reviewed its source code or work items thoroughly since Rx 2.0 was released, so my confidence level in Rxx is (unreasonably, perhaps) lower than it should be.  :)

Visual Studio Settings Switcher

I've recently released version 1.2 of the Visual Studio Settings Switcher project on CodePlex, also available in the Visual Studio Gallery.

The latest release adds a new feature that I think OSS developers really want: Solution Settings Files

If you have an OSS project, VS Settings Switcher makes it really easy to generate a Visual Studio settings file that you can check into source control along with your solution, for your OSS contributors to use.  This settings file is in the typical VS settings file format.  It's also XML, which makes it easy to edit in Visual Studio, which you should definitely do before checking it in.  I highly recommend removing all of the extraneous settings that nobody wants to be shared in OSS projects, like the positions of your VS tool windows and other environment settings.  Typically, you'll want to reduce this file to only a small subset, such as your code formatting settings for your project's programming language.

This feature is also great for those of us that contribute to OSS projects.  It makes applying an OSS project's Visual Studio settings a breeze, without any manual intervention required.  If a solution has a .vssettings file with the same name and in the same directory, then VS Settings Switcher will automatically apply its settings whenever your open that solution.  And when you close that solution or open a different solution later, your original settings are automatically restored.

Spread the word: It's a good idea now to check in a partial .vssettings file with your solution!

I've touched on the topic of observable temperature while answering questions on the Rx forum and in my previous post on subjects.  I tried explaining it more deeply in this particular discussion, but now I'm finally ready to provide a deep and comprehensive¹ analysis in this blog post.

I'll answer the following questions: 

What really makes an observable hot or cold?
Why should I care?
How can I tell if an observable is hot or cold?
How can I change the temperature of an observable?

The analysis that follows is not intended for beginners, though you may find the following section and the conclusion to be informative.  You may also want to watch this first and look here for additional information.

Common Sense

We like to think of hot observables from two different perspectives as follows.

From the perspective of observers, there's the potential to miss notifications.  Hot observables are "live" sequences, meaning that they are happening whether we observe them or not.  The canonical example is a mouse, which of course is free to move regardless of whether you subscribe to an IObservable<MouseMoveEventArgs> or not.  In general, if I hand you an observable and tell you that it's hot, then as an observer you'd infer that you might have missed notifications that happened before you subscribed.  Pretty simple.

From the perspective of observables, hot observables broadcast notifications to all observers.  The canonical example is converting an event into an observable; e.g., FromEventPattern.  If three observers subscribe to IObservable<MouseMoveEventArgs>, then each observer will serially² observe the same notifications, as opposed to observing different notifications or concurrent notifications.

We like to think of cold observables from the same perspectives as follows.

From the perspective of observers, there's the potential for each observer to get different notifications.  Cold observables are "generated" sequences, meaning that they can generate different notifications for every observer.  Furthermore, observers may receive notifications asynchronously, with respect to each other.  The canonical example is Create, which can asynchronously generate notifications whenever an observer subscribes.

From the perspective of observables, cold observables won't generate notifications until an observer subscribes, and they generate notifications each time that an observer subscribes.  The canonical example is Range, which generates a range of numbers whenever an observer subscribes.

In summary, common sense tells us that:

Hot observables are always running and they broadcast notifications to all observers.

Cold observables generate notifications for each observer.

I intend to show that these ideas are more like symptoms rather than definitions.  I'll also identify a pattern to reduce them into a primitive concept: side effects.  Ultimately, I'd like to ensure that hot and cold are well-defined terms based primarily on the concept of side effects.³

Enough skipping rocks over the topic.  Let's dive into the watery refraction and expose the peculiarities of our naïve understanding of these concepts.

Uncommon Sense

We know that an observable can be either hot or cold, but can they mix?  Can an observable be generated while broadcasting?  Can it be generated  and yet always running?  Can it be always running without broadcasting?  Can the temperature change dynamically?

That last question is particularly interesting.  Can an observable change its temperature after an observer subscribes?  E.g., Observer E subscribes first, then F subscribes; E gets a hot observable while F gets a cold observable.

Can an observable change its temperature for the same observer after it subscribes?  E.g., Observer G subscribes to a cold observable that eventually transitions into a hot observable, for G, while it remains subscribed.

What about specialized subscription behaviors, such as one that causes an observable to behave like it's hot until all observers unsubscribe and then it becomes cold again for a subsequent subscription, which flips it back to being hot again -- does that behavior exist?

The answer to all of these questions is YES.  Well, it seems more like YES and SORT OF, based on our common sense definitions.  I'll try to explain.

Those behaviors might seem strange, but you've probably used them together before.  They're implemented by a couple of well-known Rx operators; e.g., Replay and RefCount.  Similarly, PublishLast and some particular overloads of Publish have strange behaviors with regard to temperature.

For example, keeping in mind our common sense definitions of hot and cold, let's assume that you have an observable O and you call O.Replay() to get an IConnectableObservable<T>, which you assign to C and then call C.Subscribe(J) and C.Subscribe(K).  I think we'd all consider C to be hot (or at least warm, whatever that means) because all of its notifications are broadcast to J and K, even though notifications won't arrive until Connect is called.  It's not entirely hot because it doesn't necessarily satisfy the always running factor, hence the introduction of the confusing term warm.

So far, Replay is identical to Publish.

Then you call C.Connect(), which may cause J and K to observe notifications from O.  If O is hot, then the always running factor was satisfied anyway, but if O is cold, then it's not satisfied; regardless, I think we'd all consider C to be hot at this point because of its broadcasting behavior alone.

Finally, you call C.Subscribe(L), and now there's a race condition on L with respect to our common sense understanding of observable temperature.  If O generated any notifications sometime between the calls to C.Connect() and C.Subscribe(L), then L will immediately observe replayed notifications, thus C was technically a cold observable when L subscribed because those particular notifications aren't rebroadcast to J and K; i.e., C generates specific notifications for L.  It means that while C is hot for J and K, it's cold for L; however, it's not permanently cold for L because after C replays all of the notifications that L missed, then L will be observing the same exact broadcast notifications as J and K, which means that C transitions from cold to hot for L.  Although, if O hadn't generated any notifications before you called C.Subscribe(L), then C would start out hot for L just like J and K.

Apparently, sometimes Replay is hot (though only if its source is hot, and only before Connect is called), sometimes it's warm (though only if its source is cold, and only before Connect is called), sometimes it transitions from warm to hot (though only for observers that subscribed before calling Connect) and sometimes it transitions from cold to hot, with a race condition (though only for observers that subscribe after Connect was called).

Part of the reason that this is confusing is because always running, broadcast and generate are actually relative concepts.  They implicitly refer to the time of subscription, though we tend to ignore this fact.  Always running refers to pre-subscription, generate refers to the time of subscription and broadcast is a post-subscription concept.  This explains how operators like Replay can offer strange combinations of these behaviors.

The common sense meanings of hot and cold are diluted by their differences in relation to the time of subscription and the connectability of operators like Replay, with a dependence on the temperature of the source and the presence of a race condition.

Technically it's possible for an observable to be always running without broadcasting; e.g., simply don't subscribe to a hot observable.  Our common sense definition tells us that it's hot even though it's not broadcasting.  Always running and generating are opposite behaviors, which makes broadcasting seem more like a symptom rather than a definition.

It's also possible for an observable to be generated while broadcasting; e.g., publishing a cold observable.  Again, common sense tells us that it's hot even though it's not always running, because we know that an underlying observable is actually responsible for generating notifications, which are broadcast to all observers.  This makes always running seem more like a symptom rather than a definition.

And finally, as shown by the Replay example above, an observable can be always running and yet generated at the same time.  They'll overlap until generation completes, like emptying a buffer, after which the observable seamlessly transitions into the always running sequence.  It's hard to say whether this kind of sequence is purely cold based on the common sense definition of generation alone.  We almost want to call it, "hot with some additional notifications".  This makes generation seem more like a symptom rather than a definition.

In general, our common sense definitions of hot and cold aren't precise enough to be useful to observers so we tend to ignore them and make assumptions based on what really matters: side effects.  All we really care about is answering one simple question:

If I subscribe to your observable, might it cause side effects?

It would be great if we could specify this behavior in a word.  The terms cold and hot should provide the answers yes and no, respectively.

Subscription Side Effects

Wikipedia defines side effect as:

Modifying some state or causing an observable interaction with calling functions of the outside world.

However, sometimes mutation is the primary effect; e.g., assignment.  Clearly the definition is relative.

So let's define side effect for our purposes as:

Any effect that is not the primary effect.

The primary effect of subscribing to an observable (i.e., calling Subscribe) is to register an observer for callbacks, which probably means that the observer will be added to the observable's internal list of observers and a disposable will be created and returned to represent unsubscription.

Likewise, the primary effect of .NET event registration is to add a handler to the list of delegates in the event's MulticastDelegate instance.  We can easily convert any event into an observable.  The primary effect of subscribing to an observable event is to convert an observer into an event handler, register it with the event, and return a new disposable that unregisters the observer from the event.

That pattern remains the same for other native asynchronous conversions as well, such as FromAsyncPattern and ToObservable (Task/Task<T>).

In addition, the compositional nature of Rx (more on that later in this post) allows operators to form subscription chains.  Subscribing to the outer-most operator's observable causes it to subscribe to the previous operator's observable, which subscribes to another observable, and so on.  The entire subscription chain is certainly part of the primary effect of subscribing.  Ultimately, the goal is to subscribe to the inner-most observable through as many intermediary subscriptions as necessary.

Subscription chains inherit side effects.  If any inner subscriptions cause side effects, then the outer-most subscription inadvertently causes them too; therefore, our definition of subscription side effects implicitly includes the sum of all inherited subscription side effects.

Based on the ideas above, a subscription side effect is:

Any side effect other than subscribing to another observable, adding an observer to an observable's "list of observers" and creating a new disposable for unsubscription.

where the meaning of "side effect" is defined by wikipedia and "list of observers" is defined by the necessary conversion to the underlying asynchronous model, if any.

Perhaps for the sake of simplicity I should reduce that definition.  Side-effect inheritance is implied, as is adding an observer to a "list of observers" and returning a disposable for unsubscription.  All of these behaviors are intuitively part of the subscription mechanism.

Therefore, a subscription side effect is:

Any effect beyond an observable's subscription mechanism.

Examples of subscription side effects include:  Calling Schedule, OnNext, OnError, OnCompleted, GetEnumerator or MoveNext, mutating a field, creating an object in memory, running a CPU-intensive computation, sending a web request, reading a file, ending a process, formatting your C drive, or really anything else that you can think of that isn't merely the subscription mechanism and may cause an observable effect on the outside world, including notifications.

Covering All Bases

The concept of subscription side effects completely covers our common sense understanding of temperature as follows.

Recall our definition of subscription side effects from above:

Any effect beyond an observable's subscription mechanism.

Recall our common sense definitions of temperature from above:

Hot observables are always running and they broadcast notifications to all observers.

Cold observables generate notifications for each observer.

Understanding how cold behavior relates to subscription side effects is easy:

Generate means calling OnNext, OnError or OnCompleted when Subscribe is called.  This can be done synchronously or asynchronously with an IScheduler.  Either way these actions are, by definition, subscription side effects.  That's pretty simple.

Now let's jump to the common sense definition of hot and see how it compares:

Always running means that as soon as an observer subscribes it begins listening in to notifications.  It implies that notifications will not be generated as a side effect of subscribing, but that the mere act of subscribing enables an observer to receive notifications that would have occurred anyway.  In other words, subscribing to an observable that is always running does not necessarily cause subscription side effects.  The primary effect of subscribing is all that is needed to begin receiving notifications asynchronously.

Broadcast means that when a notification is ready to be observed it is pushed to all currently subscribed observers, which does not seem to imply anything about subscription side effects.  Theoretically, an observable could asynchronously generate notifications upon subscription, pushing them to the original observer that caused these subscription side effects while also broadcasting them to any observers that are fast enough to subscribe before the notifications are generated.  That's not the same behavior as Replay, for example, because Replay doesn't broadcast the notifications that it generates for an individual observer.  Apparently this behavior doesn't exist in any Rx primitives.  When an observer subscribes to a cold observable, without publishing, it observes notifications that are generated for it alone; i.e., cold observables don't broadcast.

Broadcast in Rx means that an observable is not responsible for generating notifications itself, but instead broadcasts notifications on behalf of some underlying observable.  That's the difference between the common sense definitions of cold and hot, respectively; therefore, broadcasting does not cause subscription side effects, meaning simply that when you subscribe to a broadcasting observable there will be no subscription side effects.

Broadcasting a cold observable has to cause subscription side effects once, though it's not caused by an observer subscribing to the broadcasting observable, but the broadcasting observable subscribing to the cold observable; a.k.a., "connecting".  Connection is exposed as a public operation, either by a subject's ability to act as an observer or by an operator returning IConnectableObservable<T>.  Theoretically, connection isn't necessary if the underlying observable is always running because observers that haven't subscribed yet are going to miss notifications whether the broadcasting observable is connected or not; however, if the underlying observable isn't always running (i.e., it's cold), then connection is required to ensure that observers don't miss notifications before they've subscribed to the broadcasting observable.  As a result, exposing the connection behavior of broadcasting observables defers subscription side effects of the underlying observable.  This implies again that broadcasting doesn't cause subscription side effects.  Subscribing to a broadcasting observable is all that is needed to begin receiving notifications asynchronously from the underlying observable, although notifications may be deferred until connection if the observable is connectable.

An emerging pattern here is that subscription side effects aren't expected at all in hot observables, including the case where a hot observable broadcasts notifications for a cold observable, because subscription side effects are deferred until connection.  We'll call them connection side effects instead.

Connectability may be hidden from observers.  Subjects are observables, thus the AsObservable operator can be used to hide their connectability.  The same applies to IConnectableObservable<T>.  In either case, hot behavior is retained even when observers know nothing about the underlying connectability of observables; therefore, simply knowing that an observable broadcasts implies that it doesn't cause subscription side effects.  It may have caused or will cause connection side effects, but that's beyond an observer's control when connectability has been hidden.

In addition, some stateful behaviors in Rx use connection side effects to separate notifications from other kinds of subscription side effects.  This is perhaps the most confusing case as far as the common sense definitions of hot and cold are concerned.

Recall from a previous section, when we reviewed the strange behaviors of Replay and RefCount, we determined that our common sense definitions are diluted because of their differences in relation to the time of subscription and the connectability of operators, with a dependence on the temperature of the source and the presence of a race condition.

Replay, PublishLast, and overloads of Publish that have an initialValue parameter, broadcast all subscription side effects, including notifications, to any observer that subscribes before they are connected to the underlying observable.  They also broadcast all subscription side effects, including notifications, to any observer that subscribes after connection but before the underlying observable generates any notifications.  They broadcast all subscription side effects, excluding missed notifications but including future notifications, to any observer that has missed notifications.  The missed notifications aren't broadcast, they are generated specifically for the particular observer that missed them.

These operators show that hot and cold are relative terms with regard to different kinds of subscription side effects, of which notifications are just one example.  They separate missed notifications from other kinds of subscription side effects to return a relatively cold observable or a relatively hot observable, respectively.  This is possible because broadcasting is a post-subscription concept and the ability to miss notifications is a pre-subscription concept that can be remedied at the time of subscription through generation, thus allowing an observable to be hot for some observers and cold for others, simultaneously.

Furthermore, the ability of these kinds of observables to transition from the common sense definitions of cold to hot is of no importance now, given that all we really care about is subscription side effects.  In other words, the newly defined hot and cold terms are relative to the time of subscription and to different kinds of subscription side effects, so the ability to transition is irrelevant.  An observable either causes subscription side effects or it does not.

Thus we no longer need any confusing terms such as warm to describe the behavior of connectables.  A connectable observable is always hot before it's connected and is generally hot afterwards, though some operators apply special behavior to missed notifications to ensure that they aren't actually missed.  These operators return an observable that is cold with regard to missed notifications, for a particular observer, and hot with regard to any other kind of subscription side effect, including future notifications; however, because our new definitions of hot and cold make transitioning irrelevant, we can say absolutely that an observer that has missed notifications will get a cold observable from these operators.  (The race condition's effect on our new definition of cold will be addressed later in this post.)

In summary, our new understanding of temperature with regard to subscription side effects tells us that:

Hot and cold are relative terms with regard to the time of subscription and to different kinds of subscription side effects.

Hot observables either don't cause subscription side effects or defer them until connection, where they become connection side effects.

Cold observables rely on subscription side effects.

Perhaps we can distinguish between these terms further by defining whether subscription side effects may or may not occur:

Hot observables do not cause subscription side effects.

Cold observables do cause subscription side effects.  (Though keep reading because later on I'll relax this definition a bit.)

Finally, we now have a specific and useful definition of observable temperature:

Temperature indicates the propensity of an observable to cause subscription side effects.

Furthermore, we can largely ignore the relativity to particular kinds of subscription side effects since it only matters when discussing the special behaviors of Replay, Publish and PublishLast.  And those operators only distinguish missed notifications from other kinds of subscription side effects, so they can easily be described as generally hot yet cold with respect to missed notifications.

In general, the terms hot and cold cover all kinds of subscription side effects, including notifications. 

Concerning Replay, PublishLast, and Publish overloads with an initialValue parameter, they return observables that are cold for particular observers that have missed notifications, yet they are hot with respect to any other kind of subscription side effect.

Now let's test the reverse: We'll see if our original common sense definitions can be inferred from our new definitions of hot and cold.

Given an observable that is known to not cause any subscription side effects, one must infer that it's either always running, which never causes subscription side effects, or it must broadcast, which defers subscription side effects into connection side effects; therefore, we must refer to the observable as hot.

Our new definition seems to be holding up so far.  Let's see how it compares to the common sense definition of cold.

Given an observable that is known to cause subscription side effects, one can infer that it might generate notifications when an observer subscribes, thus we may refer to it as cold.

It seems like we've got a match; however, notice that I wrote "might generate notifications".  That's because, as described previously, subscription side effects are far more general than just generating notifications.  A cold observable, by our new definition, can cause any subscription side effect.  For example, it can send a web request or asynchronously read from a file, without ever generating any notifications.  Of course, generating a notification would be useful so it's quite common for these examples in particular to generate notifications asynchronously once the operation has completed.

There are other kinds of subscription side effects that do not generate any notifications - not even asynchronously.  For example, imagine tracking the number of observers subscribed to your observable; each time Subscribe is called an integer field is incremented by 1.  That's similar to the behavior of the RefCount operator, described above.  Another example is subscription logging, for diagnostic purposes, which may cause a file to be updated on disc every time an observer subscribes.  The common sense definition of cold doesn't cover these cases, but our new definition does!

It seems to me that our new definition of temperature entirely covers the common sense definition, and does so elegantly as diametrically opposed concepts.  Furthermore, our new definition of hot nicely ties together the two orthogonal concepts that make up the common sense definition of hot, while our new definition of cold identifies the important factor and generalizes it.

But why are subscription side effects the important factor?

Composition

LINQ operators enable the composition of observables into "queries".  Applying any operator to one or more observables means that it will subscribe to those observables when you subscribe to the query.

When defining a query, it's not uncommon to reference an observable multiple times, which means passing the same observable to different operators or to the same operator as multiple arguments.  Since we know that operators subscribe to the observables that you pass to them, referencing the same observable multiple times means that it will potentially have multiple subscriptions.

Some operators have the semantics of multiple subscriptions without multiple references.  For example, Retry only accepts a single observable and potentially subscribes to it multiple times.  In this case it's desirable to duplicate subscription side effects.  For example, if we pass to Retry an observable that makes a web request as its subscription side effect, then we want Retry to execute that subscription side effect every time it re-subscribes.  In other words, we want to retry the web request.

That's why subscription side effects are the important factor.  From the point of view of the Retry operator, we don't necessarily want to duplicate the observable's notifications (as indicated by our common sense definition of cold), we simply want to duplicate the subscription side effects, whatever they might be.  In other words, it's not about generating the notifications, it's about executing the side effects, which are often responsible for generating the notifications.

Sometimes an observable is referenced multiple times without the semantics of allowing duplicate subscription side effects.  For example, you could apply the Where operator twice to the same observable to create two different observables, then apply the Zip operator to merge them back into a single observable in a pairwise fashion.  The query's output semantics aren't important here; what's important is that you don't want to create two subscriptions to the original observable.  Instead, you want broadcast behavior.

Broadcasting is essentially the same thing as sharing subscription side effects among multiple observers.  In the previous example, imagine that the observable to which you are applying the Where operator is cold; e.g., each time that you subscribe it sends a web request to open a reactive socket connection to a server and then streams the responses as notifications.  The Zip operator is going to subscribe twice, but we don't want to send two web requests.  Our desire to broadcast the notifications is the same as our desire to broadcast the web request, because it's the web request (i.e., the subscription side effect) that is responsible for generating notifications in the first place.

And again, that's why subscription side effects are the important factor.  In the former case we wanted to duplicate them, while in the latter case we wanted to broadcast them.

Composition in Custom Operators

The previous cases are special because we had prior knowledge of an observable's temperature.  But what if we didn't?

Composite operators are queries themselves; however, they don't typically define contracts for the temperatures of their observable parameters.  In other words, when we define a custom operator, we must make assumptions as to the temperatures of observable parameters.

An obvious question is: Why can't we define temperature contracts on operators?

Well, technically you could, but then you'd be forcing callers in some cases to explicitly change the temperature of observables before calling your operator, which changes the declarative nature of some operators by bringing temperature into the foreground when perhaps it's unnecessary.

Furthermore, if a caller is unaware of the temperature, then an assumption has to be made anyway.  Unless the contract is propagated to a caller that can control the temperature - and now we've got a viral effect, as is common with contracts.

Alternatively, you'd probably want to have two overloads for every operator where temperature is of concern: one overload for cold observables, one for hot observables.  But how can you overload operators by temperature without defining new types?

Do we really need this complexity?  What are the disadvantages of making assumptions at the operator level?

If you were to assume that all observable parameters are hot, then it's fine if your operator relies on broadcast behavior, but it's wrong if your operator must duplicate subscription side effects; e.g., Retry.  Alternatively, if you were to assume that they are cold, then the reverse is true: It's fine if duplicating subscription side effects is what you want, but it's wrong if you need broadcast behavior.

Therefore, we need to consider the possibilities of conversion before we can determine which assumption is best in general.

Temperature Conversion

First, a quick note about language:

IObservable<T> is an immutable type; therefore, there's no way to mutate its temperature.  When I refer to "changing" an observable to a particular temperature or "making" an observable a particular temperature, I'm using those terms in the general sense of unary operator application.  For example:

var x = 10;
var y = -x;

You might say that the second line "negates x", but technically it doesn't because after the negation operation, x hasn't changed.  It's applying the negation operator to x and assigning the result to y.  The former is just a natural way of describing the essence of the operation.  Although it's ambiguous in this situation, it's not when dealing with IObservable<T> because it's immutable.  Therefore, "changing" an observable's temperature or "making" it have a different temperature simply refers to the application of some operator and the assignment of the result into a new variable, which holds a different observable with the desired temperature.

We can change a cold observable into a hot observable by applying the Publish operator, which has overloads that project the observable into a selector function as a hot observable (though the operator itself returns a cold observable) and overloads that return a connectable observable.  The latter defers subscription side effects until connection, effectively broadcasting them across multiple observers as connection side effects.

Publish makes a cold observable hot.⁴

Defer makes a hot observable cold.

The Defer operator allows us to add subscription side effects to a hot observable, thus making it cold.

Defer also allows us to convert invocation side effects into subscription side effects; e.g., given a function that returns a hot observable like Start or a C# 5.0 async method that returns a Task<T>, which you then convert into an observable with ToObservable, you can wrap either function in Defer to change its invocation side effects into subscription side effects, thus creating a cold observable from a hot function.

Alternatively, you can change the temperature of a hot observable by combining it in various ways with a cold observable.  For example, if you Concat a hot observable to a cold observable, then the result is a cold observable, by our new definition, because the subscription side effects of the cold observable are prepended onto the hot observable.  If you were to apply the Merge operator instead, then the result would also be a cold observable because, again, the new observable retains the cold observable's subscription side effects.  Remember, it's the subscription side effects that are important, which is why combining a hot observable with a cold observable in such a way that the subscription side effects of the cold observable are preserved yields a cold observable.

Notice, however, that converting from cold to hot simply adds broadcasting behavior, while converting from hot to cold relies on the addition of subscription side effects.  In other words, going from cold to hot is as simple as calling one operator, but going in reverse requires side effects to be added.

In the previous section, we left wondering which assumption is best for an observable with an unknown temperature: cold or hot.  Perhaps we can answer that now by examining the consequences of making wrong assumptions with respect to temperature conversions.

When we need a hot observable, assuming that it's already hot and being wrong means that subscription side effects are duplicated, which breaks the semantics of our query; therefore, let's assume that every observable is cold and apply Publish.  When we're wrong, we'll end up applying broadcast behavior to a hot observable, which would simply have no effect.  That's 1 point for assuming that observables are cold, because when we're wrong, applying Publish does no real harm.⁵

When we need a cold observable, assuming that it's already cold and being wrong means that subscription side effects won't be duplicated, which breaks the semantics of our query; therefore, we must assume that every observable is hot and... do what?  Remember, the semantics of our query aren't to introduce new subscription side effects, we want to duplicate the subscription side effects of the observable that was handed to us.  We can't make a hot observable cold without introducing subscription side effects; therefore, all we can do is assume that the observable is already cold and hope for the best!  That's 2 points for cold, 0 points for hot.

In summary:

We must assume that any observable with an unknown temperature is cold.

In other words, it's safest to assume that any given IObservable<T> may cause subscription side effects because it's easy to broadcast subscription side effects when necessary, but once broadcast it's impossible to reverse it.  There's no way to force a broadcasting observable's Subscribe method to cause subscription side effects that were already broadcast.

Note that there is a way to reverse the broadcast behavior of an IConnectableObservable<T>: simply dispose of the object that was returned by the Connect method.  In fact, that's exactly how RefCount works.  But this can't be done with any IObservable<T>.  Even trying to cast it to IConnectableObservable<T> won't help because casting doesn't give you a reference to the disposable that was returned by Connect.

Given that it's safest to assume that any unknown IObservable<T> is cold, it follows that sometimes we're going to be wrong.  We should probably relax our previous definition of cold to account for that fact.

Here are our latest definitions: 

Hot observables do not cause subscription side effects.

Cold observables may cause subscription side effects.

That should cover all possibilities.

Classification of Subscription Side Effects

A subscription side effect can be classified as any of the following.⁶

• Synchronous
• Notification
  E.g., directly calling OnNext, OnError or OnCompleted.
  (Technically, notifications are always Asynchronous in Rx due to its implicit use of the CurrentThreadScheduler to schedule subscriptions, though we'll ignore that fact and define Synchronous in relation to the outer-most subscription only, where synchronously calling OnNext, OnError or OnCompleted actually blocks Subscribe from returning.)
• Out-of-band
  E.g., directly assigning a field, creating a large object in memory, executing a CPU-intensive computation or (synchronously) deleting a file on disc.
• Asynchronous
• Notification
  E.g., using an async mechanism, such as Task or IScheduler, to schedule calls to OnNext, OnError or OnCompleted.
• Out-of-band
  E.g., making a web request or reading a file on disc.  Often these kinds of operations are used for Asynchronous Notification as well.
• Composition
  Subscribing to another cold observable that causes its own subscription side effects, either synchronously or asynchronously.
• Computation
  Executing costly or side-effecting code for each notification; e.g., a costly projection via Select or imperative side effects via Do.

Perhaps these classifications could be useful when documenting operators.

For example, the documentation for the Range overload without an IScheduler parameter could state that it returns an observable that causes Synchronous Notification subscription side effects.  The overload with an IScheduler parameter may cause Synchronous or Asynchronous Notification subscription side effects, depending upon the specified IScheduler; this is similar for all operators that are overloaded with an IScheduler parameter.

The documentation for the Interval overload without an IScheduler parameter could state that it returns an observable that causes Asynchronous Notification subscription side effects.

The documentation for the Defer operator could state that it returns an observable that may cause Synchronous Out-of-band subscription side effects, depending upon the specified function.  It could also state that it may cause Composition subscription side effects, which basically means that it inherits the subscription side effects of the observable that is returned by the specified function.

The documentation for the Select operator could state that it returns an observable that may cause Composition subscription side effects, which basically means that it inherits the subscription side effects of the source.  It could also state that it may cause significant Computation side effects, depending upon the specified selector function.

Most of these terms should be self-explanatory by now; however, Computation is particularly interesting since I haven't mentioned it yet.  I consider a computation subscription side effect to be:

Any effect beyond an observable's subscription and notification mechanisms that occurs for each notification.

The notification mechanism is the minimally required effects for notifying; i.e., directly calling OnNext, OnError or OnCompleted are the primary effects of notification.  Anything else is a computational side effect.

Although this kind of side effect is part of an observable's computation, it won't occur until subscription.  Subscribe is essentially the beginning of the computation, thus subscription side effects are technically computation side effects.  But since subscription side effects are the focus of temperature, that's why I refer to computation side effects as a kind of subscription side effect, in the sense that computation side effects don't occur until an observer subscribes.

Computation side effects are part of the definition of cold observables, though it basically means that applying any computational or side-effecting operator to a hot observable converts it into a cold observable; e.g., applying Select to a hot observable means that all observers will receive different notifications, generated specifically for them, which is the common sense definition of cold.  That's not a very useful definition of cold, so I propose to separate the notion of computation side effects from subscription side effects and to exclude them from the new definition of cold, introducing new terms: active and passive.  This separation allows a hot observable to be referred to as hot even if it causes computation side effects.  And you can still apply Publish to broadcast the computation side effects similar to how you'd publish a cold observable to make it hot.

Passive observables do not cause any significant computation side effects.

Active observables do cause significant computation side effects.  Every notification may be computed with some relative cost.

Select and Do are common examples of operators that return active observables.

These terms are context-sensitive, as indicated by use of the word "significant".  An observable that is passive in one usage can be considered active in another.  For example, a simple projection over an observable of integers, such as adding 5 to each value, probably isn't costly enough in terms of memory or performance to care if its computation side effects are duplicated.  In contrast, a projection that calculates the factorial of each value might be costly enough in certain scenarios to consider publishing.

Conclusion

Temperature indicates the propensity of an observable to cause subscription side effects.

A subscription side effect is:

Any effect beyond an observable's subscription mechanism.

Subscription mechanisms generally consist of storing an observer in a list or subscribing to other observables, and returning a disposable for unsubscription.

Examples of subscription side effects include:  Calling Schedule, OnNext, OnError, OnCompleted, GetEnumerator or MoveNext, mutating a field, creating an object in memory, running a CPU-intensive computation, sending a web request, reading a file, ending a process, formatting your C drive, or really anything else that you can think of that isn't merely the subscription mechanism.

Hot observables do not cause subscription side effects.

Cold observables do cause subscription side effects; however, we must assume that any observable with an unknown temperature is cold, and sometimes that assumption will be wrong; therefore, a more accurate definition is:

Cold observables may cause subscription side effects.

Again, we must assume that any observable with an unknown temperature is cold.

In the simplest case, an observable may be permanently hot or permanently cold, though some observables change their temperature over time so that different observers may get different temperatures; therefore:

The temperature of an observable is relative to the time of subscription.

Temperature is also relative to the kind of subscription side effect; however:

In general, the terms hot and cold cover all kinds of subscription side effects, including notifications.

Replay, PublishLast, and Publish overloads with an initialValue parameter return observables that, once connected, are cold for observers that have missed notifications yet hot with respect to any other kind of subscription side effect.

We must also consider an observable's computation side effects.  A computation side effect is:

Any effect beyond an observable's subscription and notification mechanisms that occurs for each notification.

Technically subscription side effects are initial computation side effects, although I've separated these concepts so that hot and cold only refer to subscription side effects, thus ensuring that hot is a more useful term.  As a result, the phrase computation side effects can be applied to either hot or cold observables.  I've assigned the following terms:

Passive observables do not cause any significant computation side effects.

Active observables do cause significant computation side effects.

The significance of computation side effects on performance is relative to the operation, query or application.  What is significant in one context may be insignificant in another.

Now I can finally answer the original questions from the top of the post with absolute precision and certainty.

What really makes an observable hot or cold?

Short answer:

An observable's temperature is its propensity to cause subscription side effects.

Long answer:

Hot observables do not cause subscription side effects.

Cold observables may cause subscription side effects.

In addition, there are two related concepts that may be applied to both cold and hot observables:

Passive observables do not cause any significant computation side effects.

Active observables do cause significant computation side effects.

Why should I care?

You need to be aware of whether subscribing to an observable multiple times is going to duplicate side effects; i.e., cold behavior.

When defining a query that references the same observable multiple times, you'll typically want to broadcast notifications among the operators to ensure that they observe the same exact notifications.  In other words, you want a hot observable.  A cold observable may generate different notifications for each observer.  You'll also want to broadcast other subscription side effects; e.g., given a cold observable that makes a web request each time an observer subscribes, you don't want two operators in the same query to make two separate web requests.

Alternatively, sometimes you want duplicate subscription side effects in your queries; e.g., given a cold observable that makes a web request each time an observer subscribes, applying the Retry operator means that you want to keep making web requests until one succeeds.  The very purpose of the Retry operator is to duplicate subscription side effects.  It's perhaps a fairly common mistake to pass a hot observable to Retry and then wonder why Fiddler isn't showing a second web request after the first attempt failed.

Furthermore, you must consider computation side effects to determine whether they may have a noticeable performance impact on your query.  If they do, then you'll probably want to broadcast them as well.

How can I tell if an observable is hot or cold?

There's no way to be sure from an observer's point of view.  Subscription side effects might be directly observable or they might not.

Furthermore, subscription side effects that generate notifications might be indistinguishable from the notifications that would be generated if you assumed that the observable is hot.

The only way to know for sure is to look at the documentation source code.  :)

When in doubt, it's safest to assume that an unknown observable is cold.

Temperature by Operator

(You may want to review Rx operators by category, although it's somewhat outdated as of Rx 2.0).

All of the Rx generator methods return observables that are of course cold by definition: Return, Range, Timer, Interval, Generate and Create.

There are 4 additional generators that are kind of edge cases: Empty and Throw are cold; Start is cold too, but we'll get back to it later; Never is technically hot, but it doesn't really matter.

In general, it's safe to assume that Rx's built-in conversion operators for .NET events return hot observables; e.g., FromEvent and FromEventPattern.  In reality, there's no guarantee.  For example, you can define a custom .NET event and implement an add method that synchronously invokes each handler that it receives.  That event would convert into a cold observable!  Luckily, events generally don't do strange things like that, so it's safe to assume that they are hot.

You can be sure that Replay, Publish and PublishLast overloads that return IConnectableObservable<T> are hot until you call Connect, though after Connect it's more complicated.  Publish overloads without an initialValue parameter are always hot.  Replay, PublishLast, and Publish overloads with an initialValue parameter, become cold when Connect is called since they introduce a race condition.

Each of those operators also have overloads that accept a selector function.  The selector overloads return cold observables, though the observable passed to the selector function is similar to above.  When the selector is called, the observable passed in is hot, but when the selector returns, the observable passed in is automatically connected and starts to behave as described previously.

FromAsyncPattern and ToAsync return functions that return observables similar to PublishLast, but connected when the function is invoked.  It's safest to assume that the observable is cold, just like PublishLast.  You can think of the function as a parameterized connectable since it acts like the Connect method of IConnectableObservable<T>.

Start is similar to ToAsync and PublishLast, though it's not connectable, thus it's safest to assume that it returns a cold observable.

Task and Task<T> are themselves hot, although the ToObservable conversion returns an observable similar to PublishLast, thus it's safest to assume that ToObservable returns a cold observable.

Defer returns a cold observable, though actually it depends on whether it causes subscription side effects or not.  Typically the only reason that you wouldn't know whether it causes subscription side effects is when you use it to defer an unknown function that returns an observable.  The unknown function might not cause any invocation side effects and may return an observable that doesn't cause any subscription side effects, thus Defer may simply be passing through a hot observable.  That behavior indicates either an edge case or improper usage of the operator.

StartWith, Sample and Using return cold observables.  They all cause subscription side effects.

All of the remaining operators inherit temperatures from their source observables.  Well, maybe not all remaining operators – I may have missed a couple when I reviewed them for this post ;)

Select, Do, Aggregate and Scan may cause significant computation side effects.  (See above for a description of computation side effects.)

Technically, any operator with a delegate parameter that's invoked for some or all notifications may cause significant computation side effects.  There are many more operators like this than what I've listed above.

Remotable causes significant computation side effects, assuming that you consider transmitting notifications through the .NET remoting infrastructure to be significant in terms of its performance implications.

How can I change the temperature of an observable?

Subscription side effects:

Simple Conversions

Publish is useful for broadcasting subscription side effects.  It makes a cold observable hot.

Defer is useful for adding subscription side effects and for converting invocation side effects into subscription side effects.  It makes a hot observable or observable-returning function cold.

Operators such as Merge and Concat, among many other combining/zipping/joining operators, can add subscription side effects to hot observables by combining them with cold observables, thus returning cold observables.

Complex Conversions

Using a Subject as an observer makes a cold observable hot, though it's not a good reason to use subjects.  See my previous post on subjects for details.

Replay, PublishLast, and Publish overloads with an initialValue parameter, are useful for broadcasting subscription side effects while also replaying notifications to observers that missed them.  These operators make cold observables hot in terms of all subscription side effects except for missed notifications; therefore, they remain cold in regard to missed notifications.

Computation side effects:

Publish is useful for broadcasting computation side effects.  Whether or not you want to broadcast them depends upon their significance with respect to resource usage and/or performance and how they compare to your operation, query or application.

Operators such as Select and Do, among many others, are useful for adding computation side effects.

1. Not in a mathematical or academically formal sense.  Posting the idea to my blog allows me to focus on rationalizing it and elicits proper feedback.  Though if anybody can explain it in a mathematically formal sense, then by all means please describe it in the comments.
2. Serial behavior is described in §§4.2, 6.7 of the Rx Design Guidelines document.  Note that FromEventPattern does not enforce this contract, so it's either the raiser's responsibility to ensure that it does not raise the event concurrently or it's an observer's responsibility to apply the Synchronize operator.
3. Obviously not in any kind of official capacity.  I just need a way to reference these new concepts now.  I'd be happy to adopt better or formal naming conventions if they already exist or will emerge in the future.
4. Publish, PublishLast and Replay also have convenient overloads accepting a selector function that acts as a scoped query in which the original observable is hot, thus allowing callers to avoid having to deal with connection themselves.  The temperature of an observable returned by these overloads is cold because it publishes and connects as a subscription side effect.
5. Perhaps it would be nice if Rx's implementation of Publish could somehow detect whether an observable is either Subject<T> or it's hidden by the AsObservable operator, in order to avoid introducing yet another subject into the query unnecessarily.  But that would be an internal optimization and it doesn't change the fact that publishing a hot observable probably has no noticeable performance impact in most programs.
6. I've made all of these terms up myself.  I'll gladly replace them with formal terms if someone provides them to me in the comments.

That is the question!  There appears to be some confusion on the web about whether or not Subject<T> should be used, ever.  Tis a question oft asked by developers new to Rx after reading expert recommendations to avoid subjects, due to common misuse, yet subjects seem to persist an air of mystery even for seasoned reactive developers.

First, a primer on Subjects.  Take a moment to read that.

Done?  Great.  Let's continue...

As you may already know, Erik Meijer doesn't like subjects.  End of discussion, right?  Wrong!  Erik doesn't like subjects to be used in place of purely functional alternatives.¹

As Erik puts it:

[Subjects] are the "mutable variables" of the Rx world and in most cases you do not need them.

So let's enumerate the cases where we should and shouldn't use subjects.

We'll start by examining two interesting properties of subjects:

1. They implement IObservable<T> and IObserver<T> simultaneously, which makes them suitable for both output and input, respectively.
2. They are hot.  (See also: Hot and Cold Observables)

Thus to answer the question of whether we should use a subject when defining an observable (output) we must consider:

1. the input; i.e., the source of the OnNext, OnCompleted and OnError notifications.
2. the desired temperature of the generated observable; e.g., hot or cold.

To determine whether or not to use a subject, start by asking yourself the following questions:

1. what is the source of the notifications?  I.e., where do the values of T come from?  Who is responsible for calling OnNext?
2. must observers share subscription side-effects or not?  I.e., does calling Subscribe execute some kind of initialization code each time it's called or does initialization occur once and its state is shared by every call to Subscribe?

Let's answer all possible combinations of these 2 questions.  I'll provide more details and examples later in this post to explain why these answers make sense.

Question 1:  What is the source of the notifications?

The source can be one of two kinds: Let's call them external and local.  An external source is any observable or event that already exists outside of your code.  A local source is when you generate an observable from your code, kind of like defining and raising a custom .NET event.  More details on these distinctions later in this post.

1. If the source is external, then do not use a subject.
2. If the source is local, then do use a subject.

Question 2:  Must observers share subscription side-effects or not?

1. If not, then do not use a subject.  (Define a cold observable.)
2. If yes, then do use a subject.  (Define a hot observable.)

Let's enumerate all possible combinations: 

1. The source is external and I want a cold observable.
2. The source is external and I want a hot observable.
3. The source is local and I want a cold observable.
4. The source is local and I want a hot observable. 

As you may have noticed, there are conflicts.

• What if you have an external source and you do want to share subscription side-effects?
• What if you have a local source and you do not want to share subscription side-effects?
• What if the temperature of an external source differs from the desired temperature?

I believe these conflicts cause additional confusion among developers.  They seem to blur the line when it's appropriate or inappropriate to use subjects, though in fact the line is actually quite clear, as we'll soon see.

Let's resolve these conflicts now.

The source is external and I want a cold observable.

1. If the source is cold, then use it as is.  You can also wrap it with Defer to add subscription side-effects.
2. If the source is hot, then wrap it with Defer to add subscription side-effects.

The source is external and I want a hot observable.

1. If the source is cold, then Publish it.  Alternatively, you can use PublishLast or Replay, if you need their additional behavior.
2. If the source is hot, then use it as is.

The source is local and I want a cold observable.

1. Define your observable with existing Rx generator methods; e.g., Return, Range, Timer, Interval, Generate or Create.

The source is local and I want a hot observable.

1. This scenario is a potential candidate for defining your observable as Subject<T>, or even BehaviorSubject, AsyncSubject or ReplaySubject, if you need their behavior.  More details on why it's merely a "potential candidate" later in this post.

Note that Publish, PublishLast and Replay use subjects internally (via Multicast).  This isn't a coincidence.  Subscribing a subject to a cold observable broadcasts its notifications to multiple observers, thus making it hot.

So it's clear there are only two scenarios where it's correct to use subjects:

1. The source is external and cold, and I want a hot observable.
2. The source is local and I want a hot observable.

Let's simplify those statements.  The two scenarios where it's correct to use subjects are:

1. Converting a cold observable into a hot observable.
2. Generating a hot observable imperatively and statefully, without any direct external source.

Although, only in the latter scenario is it correct to use subjects explicitly.  As mentioned previously, Rx defines various operators like Publish for use in the former scenario.  Thus, only 1 scenario remains.

And so now we've answered our original questions:

When should I use a subject?  To generate a hot observable imperatively and statefully, without any direct external source.
When shouldn't I use a subject?  Everywhere else.

Perhaps that answer is a bit vague.  What do I mean by imperative and stateful?  What about local vs. external?  And why did I mention before that our final scenario is only a "potential candidate" for using a subject?

If (true == interested_in_the_answers_to_these_questions) { continue; } else { goto Conclusion; }

External Sources

Let's redefine external as quite simply a source of notifications that you do not have to generate yourself.  It comes from outside the scope of the observable that you're defining, such as a dependent object or a parameter.  It could be any number of things; e.g., an event on another object, an event on a base class, a timer callback, a Task<T> that represents the asynchronous result of reading from a file or sending a web request, an IEnumerable<T>, etc.

All of these examples have something in common: they can easily be converted into IObservable<T> using various From* and To* operators in Rx.  For example, Task<T> is converted by the ToObservable operator.  So if your external source is not already an IObservable<T>, then simply covert it using one of Rx's existing conversion operators.

Once you have an IObservable<T>, you need to determine whether it's hot or cold and whether you need it to be hot or cold.  If there's a conflict, as defined above, then use either Defer or Publish to change the temperature accordingly.  For this, there's no need to use subjects in your code.²

Finally, use LINQ to query the observable.  Again, you shouldn't use subjects here either.

The result is that you'll have a succinct observable query, controlling side-effects and meeting your specification in a declarative manner.  Subjects are simply the wrong tool for this job.  It's like hammering in a nail with a sledgehammer.  Subjects do much more than what is needed.  (Though as you'll see later in this post, in some ways subjects actually do less than what you may think, potentially introducing nasty threading bugs.)

Local / Imperative / Stateful Sources

I'm defining local as a source where you imperatively push notifications toward the outside of a stateful scope.  Another way of looking at it: a local source is external to other scopes that can't access its implementation.

Sorry if that explanation is confusing.  Let's clear it up.  Consider the following abstract requirements:

1. You must define a hot observable.
2. The code that generates the observable's notifications must be encapsulated by some type that you define.
3. Your code generates notifications when callers invoke certain members of your type or when it performs certain operations.
4. The observable must be exposed as a public property.

Now let's apply these requirements to what we've already learned earlier: 

1. Since we're defining a hot observable, Subject<T> is certainly a candidate...
2. We've established a scope for what is external: any source defined by another type and referenced by ours is external to our type.
3. We must write code to push notifications (input); therefore, our source is local.  We have no external source to convert or to generate notifications for us.
4. Our code must be able to reference something in order to push notifications, thus our type must be stateful.  We'll store a reference to "something" in a field.
5. The field will be returned by a public property (output).
6. Thus we need a type that represents both input and output, and is hot.

How should we model these requirements?

Scroll down for the answer:
.
.
.
.
.
.
.
.

Answer: Define a .NET event and expose it as an observable public property using FromEventPattern!

Ha, I bet you didn't see that one coming, did you?

Yes, alternatively we could use Subject<T>.  The conditions are perfect.  If we don't actually need or want a .NET event, then we should use a subject; however, if we already have an event defined, then there's no need to duplicate state and "raise" two similar events.  The conversion approach is probably better in that case.

Let's compare the similarity of subjects to .NET events in general:

1. Events are also hot.
2. An event's delegate shares the same scope (encapsulation) within our type as our proposed observable.
3. We also write code to raise events (input).
4. Our code must be able to reference the delegate in order to raise events, thus our type must be stateful.  We store a reference to the delegate in a field.
5. The field will be returned by a public event (output).
6. A multicast delegate represents both input and output, and is hot.

Subject<T> is kind of like Rx's implementation of an observable "event".  It broadcasts notifications to multiple observers, like raising an event for multiple event handlers.  It is the stateful component of Rx as it can be stored in a field or a variable and mutated (invoked) imperatively.  It is hot, so subscribing does not cause any side-effects (and subscribers may have already missed notifications, which is a common symptom of events and hot observables alike.)

It's correct to use Subject<T> to replace local .NET events, though .NET events are still useful sometimes due to their metadata and first-class language/tooling support.  It all depends on context, so consider this carefully before replacing all of your .NET events with subjects.  You may want to examine each event on a case-by-case basis.  And keep in mind that you can always expose an event as an observable via FromEventPattern, or perhaps it's best to leave that decision up to the consumers of your class and simply expose a .NET event.

Subjects actually have an additional related use case that events cannot cover: If we shrink our scope from type to method and apply the same logic as before, then we'll discover that subjects may still apply though events do not.

Let's specify similar method-scoped requirements:

1. You must define a hot observable.
2. The code that generates the observable's notifications must be within a method that you define.
3. Your code generates notifications when it performs certain asynchronous operations within the method.
4. The observable must be exposed as the return value of the method.

And again, let's apply these requirements to what we've already learned earlier: 

1. Since we're defining a hot observable, Subject<T> is certainly a candidate...
2. We've established a scope for what is external: any source passed into our method and any fields on the method's defining type are external to our method.
3. We must write code to push notifications (input); therefore, our source is local.  We have no external source to convert or to generate notifications for us.
4. Our code must be able to reference something in order to push notifications, thus our method must be stateful.  We'll store a reference to "something" in a local variable.
5. The variable will be returned from the method (output).
6. Thus we need a type that represents both input and output, and is hot.

How should we model these requirements?

Scroll down for the answer:
.
.
.
.
.
.
.
.

Answer 1: Write an anonymous event (a.k.a., anonymous method, delegate or lambda expression) and convert it using FromEvent!

Ha, did I trick you again?

You thought for sure this time I'd recommend using Subject<T>.  You'd also be correct, we could use Subject<T>.  The conditions may be perfect, though as you'll see in a moment there are better alternatives, even better than this one.

Alright, that answer was actually a joke, I don't recommend that particular approach at all.  It was meant to get you thinking about alternatives to subjects in places where you may think they're absolutely necessary.

But out of curiosity, how would that have worked anyway?

An event is backed by a delegate; it's the delegate that has all of those important properties (e.g., input, output, hot) that we discussed earlier.  For example, we could start by declaring a delegate as our observer.  By assigning the delegate to a local variable we can invoke it imperatively like a method, which is similar to calling OnNext on an observer.  We can use Task.Factory.StartNew to execute an async computation that invokes the delegate.  Finally, we can convert our delegate into an observable via the FromEvent method, which gives us our return value.

NOTE:  I don't recommend this approach, as mentioned previously.  For one thing, it doesn't support OnError or OnCompleted.  It doesn't support cancellation either, but given that we're generating a hot observable perhaps it doesn't really matter.  It's also a bit strange anyway.

using System;
using System.Reactive.Linq;
using System.Threading.Tasks;

namespace Rx.Labs
{
    static class ToUseSubjectOrNotToUseSubject
    {
        public static void Main()
        {
            IObservable<string> xs = Generate();

            using (xs.Take(1).Subscribe(Console.WriteLine))
            using (xs.Take(2).Subscribe(Console.WriteLine))
            using (xs.Take(3).Subscribe(Console.WriteLine))
            using (xs.Take(4).Subscribe(Console.WriteLine))
            {
                Console.ReadLine();
            }
        }

        static IObservable<string> Generate()
        {
            Action<string> observer = _ => { };    /* We could use null, but then at every invocation 
                                                    * we'd have to copy to a local and check for null.
                                                    */

            Task.Factory.StartNew(() =>
                {
                    Console.WriteLine("Invocation side-effect.");

                    Task.Delay(TimeSpan.FromSeconds(1)).Wait();  // Simulate work

                    observer("Generating");
                    observer("a");
                    observer("hot");
                    observer("observable.");
                });

            return Observable.FromEvent<string>(
                eh => observer += eh,
                eh => observer -= eh);
        }
    }
}

Output:

Invocation side-effect.
Generating
Generating
Generating
Generating
a
a
a
hot
hot
observable.

Answer 2 (Recommended):  Define an observable using an existing Rx generator method, then apply Publish with RefCount.

Rx provides various generators that make it easy to create an observable from thin air; e.g., Return, Range, Timer, Interval, Generate, and also Create.  If none of the former generators apply, then fallback to Create.  Create is the Swiss Army knife of observable generation.  I suspect that it also has similar performance characteristics to Subject<T>.  (See the Thread Safety section below for details.)

A particularly useful approach is to write an async iterator using a special overload of Create:

Rx 2.0 (Preferred):    Create<TResult>(Func<IObserver<TResult>, Task>)
Rx 1.1 Experimental:   Create<TResult>(Func<IObserver<TResult>, IEnumerable<IObservable<object>>>)

Generators are better than subjects because they allow you to remain functional, though Create does seem quite similar to using a subject.  Create lets you take more of a declarative approach, which is hopefully easier to understand and maintain than code that uses a subject.  Note that a primary benefit of Create is that it supports cancelation and auto-detachment, though it's not very useful when you need a hot observable.

Generators aren't the whole story either.  They return a cold observable, but our specification was to return a hot observable.

That's where Publish is useful, as described earlier in this post.  Publish converts a cold observable into a hot observable; however, it returns IConnectableObservable<T>, which requires a call to Connect.  You can return IConnectableObservable<T> from your method, just make sure that you change the return type of your method to match so that callers have the opportunity to call Connect themselves.

Alternatively, you can call Publish then RefCount to get an IObservable<T> that is shared among multiple consecutive observers.  Note that this isn't truly a hot observable - it's more like a warm observable.  RefCount makes a single subscription to the underlying observable while there's at least one observer of your query.  When your query has no more observers, changing the reference count to 0, the underlying subscription is disposed.  If another observer subscribes to your query later, moving the reference count from 0 to 1 again, then RefCount makes a new subscription to the underlying observable, causing subscription side-effects to occur again.

Note that it's bad practice to call Connect yourself in this scenario.  If you're tempted to do so, perhaps because RefCount doesn't meet your needs, then you should strongly consider returning a cold observable instead and leave it up to callers to decide whether Publish should be used at all.

Also beware to avoid closures when using Create.³  Closing over a local variable that you intend to mutate may cause unexpected behavior since every observer will share the same variable.  This may seem fine when generating a hot observable, and perhaps it is, though RefCount generates a warm observable, as mentioned previously.

Here's a correct example that uses Create, Publish and RefCount instead of a subject: 

using System;
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Threading.Tasks;

namespace Rx.Labs
{
    static class ToUseSubjectOrNotToUseSubject
    {
        public static void Main()
        {
            IObservable<string> xs = Generate();

            using (xs.Take(1).Subscribe(Console.WriteLine))
            using (xs.Take(2).Subscribe(Console.WriteLine))
            using (xs.Take(3).Subscribe(Console.WriteLine))
            using (xs.Take(4).Subscribe(Console.WriteLine))
            {
                Console.ReadLine();
            }
        }

        static IObservable<string> Generate()
        {
            return Observable.Create<string>(observer =>
                TaskPoolScheduler.Default.Schedule(() =>
                {
                    Console.WriteLine("Subscription side-effect.");

                    Task.Delay(TimeSpan.FromSeconds(1)).Wait();  // Simulate work

                    observer.OnNext("Generating");
                    observer.OnNext("a");
                    observer.OnNext("hot");
                    observer.OnNext("observable.");

                    observer.OnCompleted();
                }))
                .Publish()
                .RefCount();
        }
    }
}

Output:

Subscription side-effect.
Generating
Generating
Generating
Generating
a
a
a
hot
hot
observable.

Thread Safety

Subject<T>, Observable.Create and ObserverBase<T> do not make the same guarantees that are made by Rx generator methods, for performance reasons.  They allow callers to violate some of Rx's contracts for IObserver<T>, though they do ensure some contracts automatically regardless.

The contracts to which I'm referring are defined in the Rx Design Guidelines document.

Section 4.2 basically states that notifications must be pushed serially for Rx operators to behave properly.  But more generally, this should be a safe assumption everywhere.  When you subscribe to an observable, you don't expect to observe concurrent notifications.  Imagine coding subscriptions against this assumption but actually receiving concurrent notifications at runtime.  Take a look at some of your existing code.  What could happen?  For example, if you subscribe on a UI thread then you may get a threading context exception.  If you subscribe on some other thread instead, then maybe you'll get some strange threading bugs or maybe you'll get an exception, if you're lucky.

All Rx generator methods, excluding Create, ensure that this contract is satisfied.  Subject<T> and ObserverBase<T> do not.  For instance, calling OnNext concurrently on a Subject<T> will generate concurrent notifications.⁴

Section 4.1 states the Rx grammar as follows:

OnNext* (OnCompleted | OnError)?

All Rx operators and generator methods ensure that this contract is satisfied.  ObservableBase<T>, ObserverBase<T>, Create and Subject<T> actually ensure this contract as well, but only for serial notifications.  If a caller doesn't satisfy the §4.2 contract, then it seems that these APIs do not automatically ensure the §4.1 contract is satisfied either, due to lock-free race conditions.  For instance, it seems to me that calling a subject's OnNext and OnError methods simultaneously on different threads may cause an observer to receive OnError followed by OnNext.  But if you call OnError followed by OnNext on the same thread, then the call to OnNext is correctly ignored.

Rx offers Synchronize to satisfy these contracts for a given subject.  It offers this Synchronize method to do the same for observers, in general.  And of course, just because you can violate the contracts doesn't mean that you necessarily will.  But you do have to be aware of them when using these APIs, especially considering the kinds of bugs that may arise.  Threading bugs originating from within Rx operators, due to the misuse of a subject somewhere within a query, can be very difficult to diagnose.

So when you find yourself invoking OnNext, OnError or OnCompleted on a subject or within Create, make sure that you're satisfying Rx's contracts.

The takeaway is that when a subject isn't necessary due to the presence of an external source or an appropriate generator method, it's best not to use a subject in the first place so you can avoid having to invoke it imperatively and risk introducing subtle threading bugs in your queries.

Correct Examples

As shown in this post, there's only 1 particular set of conditions in which it's absolutely appropriate to use subjects.  Here are some examples.

Reactive Property

A common example of when it's correct to use a subject is to represent a sequence of property changes for a particular property, exposing it with AsObservable to hide its identity.  It's similar to implementing the PropertyChanged event, but only raising it for a particular property.  Instead of raising an event from within the property's setter, you'd call subject.OnNext(value).

Subject<T> works fine, though more commonly BehaviorSubject<T> is used instead because it stores the latest value of the property and pushes it immediately to new observers.  If your program is highly reactive, then you may find that you don't even need to keep a backing field for the property since BehaviorSubject<T> encapsulates it.  To set the value, call subject.OnNext(value).  Unfortunately, there's no way to extract the value imperatively (at least not until Rx exposes a Current property), so you may find that a backing field is still necessary unless your program is highly reactive.  Subscribing to the subject is the natural way of extracting the current value and all subsequent changes.

using System;
using System.Reactive.Linq;
using System.Reactive.Subjects;

namespace Rx.Labs
{
    class SubjectAsProperty
    {
        public static void Main()
        {
            var foo = new Foo();

            Console.WriteLine("Normal property usage:");

            Console.WriteLine(foo.Number);
            foo.Number += 8;
            Console.WriteLine(foo.Number);

            Console.WriteLine("Reactive property usage:");

            using (foo.NumberObservable.Subscribe(n => Console.WriteLine("Observer A: {0}", n)))
            {
                foo.Number = 1;
                foo.Number = 2;

                using (foo.NumberObservable.Subscribe(n => Console.WriteLine("Observer B: {0}", n)))
                {
                    foo.Number = 3;
                }

                using (foo.NumberObservable.Subscribe(n => Console.WriteLine("Observer C: {0}", n)))
                {
                    foo.Number = 4;
                }
            }

            Console.WriteLine(foo.Number);
            Console.ReadKey();
        }
    }

    class Foo
    {
        public int Number
        {
            get
            {
                return number.FirstAsync().Wait();
            }
            set
            {
                number.OnNext(value);
            }
        }

        public IObservable<int> NumberObservable
        {
            get
            {
                return number.AsObservable();
            }
        }

        private readonly BehaviorSubject<int> number = new BehaviorSubject<int>(42);
    }
}

Output:

Normal property usage:
42
50
Reactive property usage:
Observer A: 50
Observer A: 1
Observer A: 2
Observer B: 2
Observer A: 3
Observer B: 3
Observer C: 3
Observer A: 4
Observer C: 4
4

Cross-Cutting APIs

It's common to use a subject to represent cross-cutting concerns in a reactive manner.  For example, imagine exposing a property on a purely memory-bound logging class, returning an observable that pushes a notification whenever something is logged, such as entering and exiting methods (possibly via AOP).  This could be useful for real-time performance analysis, reactive heuristic-based adjustments, post-mortem debugging and even real-time debugging.  You could have one observer writing notifications to disk and one writing them to a GUI console, while another adjusts algorithms at runtime based on performance heuristics.  ReplaySubject<T> could be used to buffer notifications in memory and replay them to new observers, if necessary.

Another example would be a sequence of alerts exposed as an observable property on an Alert class.  Various parts of the program could send alerts concurrently by calling Alert.Unimportant(...) or Alert.RequiresUserInteraction(...), for example.  One observer could send emails while another displays toast notifications.

using System;
using System.Reactive.Linq;
using System.Reactive.Subjects;

namespace Rx.Labs
{
    class AlertLab
    {
        public static void Main()
        {
            var toast = from alert in Alert.All.OfType<ToastAlert>()
                        select alert.Message;

            var email = Alert.All.OfType<EmailAlert>();

            using (toast.Subscribe(a => Console.WriteLine("Important: {0}", a)))
            using (email.Subscribe(a => Console.WriteLine("Email \"{0}\" with \"{1}\".", a.To, a.Message)))
            {
                new ToastAlert("Hello Console!").Send();
                new EmailAlert("Hello Fred!", "fred@example.com").Send();
                new ToastAlert("Goodbye!").Send();
            }
        }
    }

    public abstract class Alert
    {
        public static IObservable<Alert> All
        {
            get
            {
                return alerts.AsObservable();
            }
        }

        public string Message
        {
            get
            {
                return message;
            }
        }

        private static readonly ISubject<Alert, Alert> alerts = Subject.Synchronize(new Subject<Alert>());
        private readonly string message;

        protected Alert(string message)
        {
            this.message = message;
        }

        public void Send()
        {
            alerts.OnNext(this);
        }
    }

    public class ToastAlert : Alert
    {
        public ToastAlert(string message)
            : base(message)
        {
        }
    }

    public class EmailAlert : Alert
    {
        public string To { get; private set; }

        public EmailAlert(string message, string to)
            : base(message)
        {
            this.To = to;
        }
    }
}

Output:

Important: Hello Console!
Email "fred@example.com" with "Hello Fred!".
Important: Goodbye!

Conclusion

If you've got anything that can be converted into an observable, such as a Task or Task<T>, an event, or an enumerable, then convert it using a From* or To* operator such as these or this, or if you've already got an observable, then use it.  Now consider temperature conflicts, change the temperature if necessary via Defer or Publish, and finally query it.  Don't use subjects.

Else if you don't have an observable or anything that can be converted into one:

If you require a cold observable, then use an existing Rx generator method; e.g., Return, Range, Timer, Interval, Generate or Create.  Don't use subjects.

Else if you require a hot observable:

If the scope of your observable is a method (i.e., it's entirely encapsulated by a method), then use an existing Rx generator method.  You can also apply Publish and RefCount, but consider relaxing the conditions to simply return a cold observable.  Don't use subjects.

Else if the scope of your observable is a type (e.g., it's exposed as a public property and backed by a field):

If you need to define a similar event or a similar event already exists, then convert the event into an observable like the first case.  Don't use subjects.

Else if you don't need to define a similar event and no similar event already exists, then use a subject.

In summary:

When should I use a subject?

When all of the following are true:

• you don't have an observable or anything that can be converted into one.
• you require a hot observable.
• the scope of your observable is a type.
• you don't need to define a similar event and no similar event already exists.

Why should I use a subject in that case?

Because you've got no choice!

1. At least that's my interpretation of Erik's comments.  I find it to be really good advice in practice when working with Rx.
2. Perhaps in very advanced and rare scenarios you may find a custom ISubject<T> implementation to be useful, in which case it's safe to use Multicast rather than Publish.
3. Which happens to be an easy-to-miss bug when implementing custom operators in Rx.  If you need to return a cold observable, as generally operators do, and you must use a closure, then wrap the closure and the call to Create in a call to Defer to ensure that the closure isn't shared across multiple subscriptions.
4. In some advanced and rare scenarios you may want to elide this contract and permit concurrent notifications through a subject, for performance reasons, though you should do so only if you can guarantee that none of Rx's query operators will be applied to the subject and all subscribers are thread-safe.

(Edit: Rx is now open source!)

Start by bookmarking the Rx hub on MSDN:
 
http://msdn.microsoft.com/en-us/data/gg577609

Especially the beginners guide and the .NET Resources and Community pages, which together provide several links to official videos, blog posts and articles, all of which are worth reviewing.
 
http://msdn.microsoft.com/en-us/data/gg577611
http://msdn.microsoft.com/en-us/data/gg577612

Next, bookmark the Rx forum.  This is the official place to search for answers and to ask new questions:

(Edit: 10/3/2014 - There's still a lot of good stuff on the Rx Forum, though Stack Overflow appears to be more active now.)

Rx Forum
http://social.msdn.microsoft.com/Forums/en-US/rx/threads

Depending upon your current level of knowledge with Rx, you may want to continue by going through the official hands-on labs.  They are a bit out-dated although differences between the labs and the latest Rx class library should be minimal.  When you do come across differences, a quick search on the Rx forum will show you what changes need to be made.

Hands-On Labs
http://go.microsoft.com/fwlink/?LinkId=208528
 
For further reading, I recommend starting off with the official conceptual documentation:
 
http://msdn.microsoft.com/en-us/library/hh242985(v=VS.103).aspx     
 
followed by the recommended design guidelines.  This document is a bit outdated, although where there may be differences between the examples and the latest Rx class library, the purpose and reasons for the guidelines still apply.

Recommended Design Guidelines
http://go.microsoft.com/fwlink/?LinkID=205219

After that there's plenty of community resources to choose from.  A new book has been written about Rx recently:
 
http://social.msdn.microsoft.com/Forums/en-US/rx/thread/813e56db-e410-4f35-8f72-1be34b08ce8a

[Edit (8/14/2012)]

Here's another new book.  This one's written by Lee Campbell and is available online for free:

http://www.introtorx.com

[/Edit]

And here's a list of blogs and open source projects that may interest you:
 
http://social.msdn.microsoft.com/Forums/en-US/rx/thread/2cbd3b1c-d535-46ba-a9cf-3cd576a8e7c2

As for open source projects, I’m particularly fond of Rxx, and not just because I’m the co-founder and author along with James Miles.  ;)

If you’re not familiar with the Rxx project yet, it's something I’ve been working on for several months now along with James Miles.

Rxx is a library of unofficial reactive LINQ extensions supplementary to Microsoft's Reactive Extensions for .NET (Rx).  Rxx is developed entirely in C# and targets the .NET Framework 4.0, Silverlight 4.0 and Windows Phone 7 (WP7).

Why should you use Rx and Rxx?

Well, if you’re doing any kind of programming that involves asynchrony or concurrency, such as what is commonly found in UI layers, business layers, data access layers, middle tier services – basically everywhere, then you’ll definitely find Rx to be very useful.  Rxx adds a whole bunch of useful features on top of Rx and the .NET Framework Class Library (FCL), making it quite easy to introduce common patterns of asynchrony into any .NET program through the use of IObservable<T> and LINQ.

Rxx provides the following features.  See the Documentation for details.

1. Many IObservable<T> extension methods and IEnumerable<T> extension methods.
2. Many useful types such as CommandSubject, ListSubject, DictionarySubject, ViewModel, ObservableDynamicObject, Either<TLeft, TRight>, Maybe<T> and others.
3. Various interactive labs that illustrate the runtime behavior of the extensions in Rxx.  Individual labs' source code included.

The latest release of Rxx is now available:

• CodePlex
  http://rxx.codeplex.com/releases/view/69545
• NuGet
  http://nuget.org/List/Packages/Rxx
  http://nuget.org/List/Packages/Rxx_Experimental

New features for Rxx 1.2 include:

1. Compatible with Microsoft's Ix Experimental library.
2. UI extensions for WPF and Silverlight, including AnonymousCommand, CommandSubject, a Subscription XAML markup extension for binding UI elements to observables, EventSubscription trigger and a reactive view model infrastructure.  (Download the labs application for examples.)
3. N-ary Zip and CombineLatest combinators.
4. Several parser updates, including new operators, non-greedy (lazy) quantifiers and major performance and memory improvements, such as avoiding stack overflows due to recursion in quantifiers.
5. Cursor types and extensions (Rx and Ix).
6. ListSubject and DictionarySubject.
7. Consume extensions that generalize the producer/consumer pattern over observables.
8. ApplicationSettingsBase extensions.
9. ObservableSyndication for RSS 2.0 and Atom 1.0.
10. ObservableFile and additional ObservableDirectory extensions.
11. Stream, FileStream and TextReader extensions.

More details can be found in the latest release notes: http://rxx.codeplex.com/wikipage?title=Release%20Notes

We’d really appreciate your feedback.  Please let us know about your experiences with Rxx by starting a new discussion or submitting an issue.  Thanks!

Microsoft Dev Labs has released a new version of Code Contracts.  (For more information about Code Contracts in the .NET Framework 4.0, see System.Diagnostics.Contracts and this article).  If you haven't yet, I highly recommend installing the latest Code Contracts release.  If you're using Visual Studio 2008 then you'll also get a shiny new tab in your project's Properties window.

Figure 1: Code Contracts tab in Visual Studio 2008 project Properties window

Along with the latest features is a new tool that adds documentation for Code Contracts to an assembly's XML documentation file.  The latest release also includes a Sandcastle patch that allows the vs2005 style to include the code contract documentation within an expandable Contracts section, as shown in the following screenshot.

Figure 2: Example Sandcastle output with Contracts section

The following blog post describes how you can patch Sandcastle so that it uses Code Contracts in XML documentation files.  The author chose to use DocProject to automate the build process, but I suspect that it will work with or without any automation tools for Sandcastle.

http://www.leading-edge-dev.de/?p=447 

Note that I've tested this procedure with success in Visual Studio 2008 standard edition.

Just don't forget to create a new DocProject or DocSite project after applying the Sandcastle patch.  You can use the Import Topics and Settings step in the New Project Wizard to copy over the important stuff from an existing project if you'd like.

--

On a side note, I think this also about ruins my ContractN project ;).  I've abandoned it anyway, so I'll be doing the honorable thing and will close down the ContractN project on CodePlex at some point (to help keep CodePlex clean - just doin' my part).  I'll continue to host the code on my blog in case anybody's interested in learning how to use Context-Bound Objects, in some form or another.

Are you part of that herd of developers that is used to documenting applications by writing help topics in raw HTML?  The power of it is nice, being able to add a pinch of bold here, a splash of italics there, some CSS for different layouts, a floating image, several nested tables, an abundance of hyperlinks, embedded Flash and media players, and even some JavaScript to boot.  What more could we want?  Or maybe a better question is, what could possibly make us want to give any of that up?

XML Documentation Comments

Well, XML documentation comments that may be added to code modules (I'm assuming everyone's familiar with this stuff by now) was one thing that prompted .NET developers to start documenting their code without using HTML.  It's nice to be able to apply a bit of XML structure to our documentation, isn't it?

Commonly used semantics for describing an API may be expressed in a universal way with XML tags such as summary, remarks and example, and the compiler builds an XML documentation file that contains the code comments found in each module when we build our project.  If you take a look at the contents of this file you may see a repeating pattern - schema - that seems like it could be used by some other tools to do, well, other things with it...

Compare XML documentation to your legacy HTML help topics and what do you notice?  The XML comments that you add to APIs do not typically contain much layout or formatting, whereas your HTML topics are chock-full of <b>'s, <i>'s, and <u>'s, and a whole mess of other HTML to describe the document's layout and formatting.  Ok, ok, if you've done it correctly then you've made judicious use of CSS - applying class names to all of those <h*>'s, <div>'s, <span>'s, <p>'s, <a>'s, <td>'s, <tr>'s, <ol>'s, <ul>'s, <dl>'s, and certainly many other HTML tags that only add to the confusion when authoring topics (as opposed to designing them).

Now you might be thinking, "Dave, it's not entirely true that XML documentation is without formatting.  What about the para, code and c elements?".  And to that my reply would be, "Ok, so then what exactly do they look like?".  If you look in the XML documentation files that are produced by your compilers, you'll see the markup exactly as it appears in your code modules.  In other words, no HTML and no CSS - nothing more than semantic usage: paragraph, code block and in-line code.  (If you were thinking something more like, "leading white-space, use pre formatting, code coloring and a fixed font", then you're getting ahead of yourself, so slow down!)

My point is that the semantics for the aforementioned XML documentation tags are clear (i.e., what the tags represent), but their appearance is not yet defined (i.e., their style and format).  Take a look at the other Recommended Tags for Documentation Comments and you won't find anything out of the ordinary.  Each tag has an obvious reason for its existence - to mark up regions of text that serve a particular purpose in the documentation.  But how do they look?  Nobody knows!  ;)

Sandcastle

Now's probably a good time to introduce Sandcastle.  For those of you that aren't familiar with it yet, Sandcastle is Microsoft's tool set for producing HTML help topics dynamically by inspecting managed assemblies and incorporating the markup from XML documentation.  From the assemblies that you provide to Sandcastle, it automatically infers a table of contents (TOC), various pseudo-topics such as Properties and Methods, and also generates many individual topics to cover the entire API.  The documentation you've written within XML documentation tags, such as summary, remarks and example, is automatically added to the generated topics in the appropriate places.

The results of running Sandcastle on your assemblies and XML documentation is a set of files that are web-ready HTML help topics for your project.  This is typically referred to as reference documentation, since it provides a reference for developers that use your API.  These topic files can be used as input to a tool such as HTML Help Workshop (Help 1.x) to produce a stand-alone compiled help file (.chm) that may be distributed with your application as an external help module.  The .NET Framework even gets in on the action by providing helpful APIs for integrating context-sensitive help and Help 1.x navigation into your managed applications.  (See the Help class for more information.)

Presentation Styles

Sandcastle provides three presentation styles that it can produce for your documentation out-of-the-box.  Each one consists of a set of XSL transformation files that convert XML documentation into XML-based HTML (not XHTML, however).  They also contain resources such as icons and, of course, CSS style sheets.

For an example of a Sandcastle presentation style, look no further than the documentation for Visual Studio and the .NET Framework on MSDN.  The appearance that MSDN uses is similar to the VS2005 presentation style in Sandcastle.  I believe that Microsoft actually uses a customized version to build their internal documentation, even for Visual Studio 2008.  The other, experimental styles, that ship with Sandcastle are Prototype and Hana.

For more information about Sandcastle, see my Sandcastle Help article on CodePlex.

From XML Documentation Comments to Reference Documentation

So the process is actually quite simple.  As developers we can easily document our source code using XML documentation, which allows us to concentrate more on writing the content instead of having to worry about formatting it with HTML.  When we build our project, the compiler will produce an XML documentation file that can be passed to Sandcastle, which then inspects our assemblies and automatically generates reference documentation that includes the comments that we added to our source code, but in a pretty HTML/CSS-based style that looks very similar to MSDN.  Nice!

User Documentation

Sandcastle can automatically generate reference documentation that is useful to other developers, but what about user documentation?  I mean things like How To, Sample, Walk-through, Overview, etc. - stuff that an end-user would want to have.  Well don't expect Sandcastle to know what you're thinking - we still, unfortunately, have to get concepts out of our heads and into help topics manually.  (At least for the time being, until someone invents HAL ;)

Conceptual documentation (how Sandcastle refers to user documentation) is often much harder to write than XML documentation comments since it requires a more in-depth understanding of the application being documented.  It's easy to look at the source code, notice that an exception is being thrown and then add an exception element to the XML documentation comments for that API.  Or to notice that a particular algorithm is being used and to add a comment in the remarks element that mentions it.  But to understand and be able to express the purpose of different user interface (UI) elements, how to perform various UI-related tasks, and how the individual APIs and components fit into the designs of other high-level processes in an enterprise-level application, is certainly more difficult and typically requires an understanding of many different aspects of the application.  So the bigger the application the harder it is to write conceptual documentation, and not just because it's more time consuming but also because it's more complex.

So if writing conceptual documentation can be more time consuming and harder to accomplish than writing XML documentation comments, why do people still insist on writing conceptual documentation in HTML?  Maybe the advantages of XML documentation comments can be applied to conceptual documentation as well.

The Perils of Writing HTML Help

I started this post by pointing out one very common way of writing help: raw HTML.  We've all done it, and I know that each time I do I end up reinventing the wheel all over again.  A new HTML layout, CSS styles, some new and strange way of cross-referencing, JavaScript for collapsible sections, etc., must all be redeveloped.  (Yea, some companies are too cheap to buy a tool that does this automatically - and so am I. :)

Creating a new help topic starts with copying an existing HTML file that is used as a template, of which there's usually only one kind containing a header, with style sheet links and scripts, a body that's empty, and a footer.  Writing a help topic requires having to look through the other topics quite often to find out which HTML tags and CSS class names I should be using for various styles.  This is especially annoying when I have a good idea that I simply want to put down quickly and be done with it.  Uh oh, that hyperlink to an HTML topic that I've been copying and pasting throughout my documentation is actually misspelled - time to do a search and replace.  Hmm, I'm not sure that I like the format that I've been using for laying out tables - oh well, it's not worth the effort to fix it now.

Is There a Better Way?

Technical writers, I can only assume, take help authoring more seriously than that.  They get paid to worry about things such as structure, readability and maintenance, so it shouldn't surprise us to know that there's a much better way to write help than simply using raw HTML.  As developers, we could probably learn a thing or two from them when writing our own documentation, whether it's for an API or conceptual topics.

Lucky for us, Microsoft has a huge library of documentation and employs technical writers to write their "official" help, which is then published to the web on MSDN.  (Sorry about that horrible reference for technical writers, but I couldn't find anything better.  I know that I've seen someone from Microsoft, probably Anand, state that they don't use developer code comments internally and instead have professional authors write it.)  This means that over the years they've had to come up with a solution that makes authoring help manageable, which is a huge task for such a large documentation set.  They also needed a way to manage file names and links for cross-referencing help topics (think, See Also section).  Since the look and feel of MSDN changes from time to time, the ability to write documentation that is absolutely independent of any one style or format was imperative as well.

So we have an invaluable example to which we can aspire.  A whole plethora of documentation written with clarity and precision using standardized techniques.  If you take a look at the documentation on MSDN, you should see a crisp and clean style that, when compared to your raw HTML help topics, probably looks far more professional.  This is nothing new to us though - we've been referencing it for quite a long time now as .NET developers.  Many people, predating .NET, have even watched MSDN documentation improve dramatically over the years, and most developers that need to write their own documentation seem to want to reproduce the same look and feel.  Many tools have even helped us to generate reasonable facsimiles in the past (such as NDoc).

But have we finally come to the point where we can write our own help topics without having to remember abstract HTML tags and CSS class names?  Can it be transformed automatically into documentation that looks the same as MSDN, or any other style for that matter?  Is there a way to simply specify a unique identifier for another topic and have hyperlinks generated automatically?  What about linking to reference topics?  Is there a way to ensure that topics of a similar nature will all share the same exact structure?

The answer to all of these questions, of course, is yes.  (But wouldn't it be funny if it was no?  I'd probably take a nap.)

Microsoft Assistance Markup Language (MAML)

Microsoft uses Sandcastle internally to generate help topics for the .NET Framework, so it's no wonder that Sandcastle also provides a way to apply structured authoring techniques to conceptual documentation, in much the same way that XML documentation comments are used by developers to write reference documentation.  In Sandcastle, conceptual topics are written in MAML.

MAML is an XML schema that defines various high-level document types, such as How To, Walkthrough, Sample, Glossary, Whitepaper, Troubleshooting and many others.  These document types provide the structure of a help topic, which doesn't change.  What can change though, is how Sandcastle presents this structure when it generates HTML topics.  This means that, for example, the markup in all of your How To topics will look similar, regardless of the presentation style that you choose.  As a matter of fact, the markup in your How To topics will be similar to mine, even if we choose to produce HTML help output in very different styles.

The schema also defines various XML elements that mark up text using a semantic approach.  For example, the ui tag is applied to text that corresponds to a user interface element, such as the text on a button.  Another example is alert, which also requires an attribute named, class that indicates the type of alert, such as note, caution, tip, warning, and others.  Another is country, which you may have already guessed, describes a country!  You would surround text with an application element when it represents the name of an application.  I think you get the idea...  By my count there's well over 40 elements that you can choose from.   And with Visual Studio's XML editor you can actually have IntelliSense tell you what they all are and where it's appropriate, within the topic's structure, to use them.

The beauty of all this is that the Sandcastle presentation style that you choose controls the HTML layout of the MAML document type used by your topic.  It also defines how all of the MAML elements will appear in the HTML.  For example, alert is transformed into an HTML table layout, while ui and application are simply bolded.  Special formatting is not actually applied to text that is specified as being the name of a country, but you could update the transformation to change the HTML markup or possibly just add a CSS rule to apply the formatting that you want, without having to update the actual topic itself.

A MAML Example

Here's a small portion of the Glossary help topic that I've written for my Auto-Input Protection (AIP) project.

<?xml version="1.0" encoding="utf-8"?>
<topic id="14790228-f45b-42d5-9b3e-f6b4ab932b9e" revisionNumber="0">
  <developerGlossaryDocument xmlns="http://ddue.schemas.microsoft.com/authoring/2003/5" 
                             xmlns:xlink="http://www.w3.org/1999/xlink">
    <glossary>
      <title>Glossary</title>
      <glossaryEntry>
        <terms>
          <term>AIP</term>
        </terms>
        <definition>
          <para>
            An acronym that stands for Auto-Input Protection.
          </para>
        </definition>
      </glossaryEntry>
      <glossaryEntry>
        <terms>
          <term>Answer</term>
        </terms>
        <definition>
          <para>
            A user's or bot's response to a challenge.  In AIP, the correct answer is a 
            string of text that matches the text on the CAPTCHA image.  An incorrect 
            answer does not match.
          </para>
        </definition>
      </glossaryEntry>
      <glossaryEntry>
        <terms>
          <term>CAPTCHA</term>
        </terms>
        <definition>
          <para>
            An acronym that stands for Completely Automated Public Turing test to tell 
            Computers and Humans Apart, trademarked by Carnegie Mellon University according 
            to the following article: <externalLink>
              <linkText>CAPTCHA. (2008, March 26).</linkText>
              <linkUri>http://en.wikipedia.org/w/index.php?title=CAPTCHA&amp;oldid=201120981</linkUri>
            </externalLink> In Wikipedia, The Free Encyclopedia. Retrieved 09:01, March 27, 2008.
          </para>
        </definition>
      </glossaryEntry>
      <glossaryEntry>
        <terms>
          <term>Challenge</term>
          <term>Test</term>
        </terms>
        <definition>
          <para>
            A CAPTCHA image, being displayed on a web page, to which a user must respond 
            with an answer by entering the text that they see on the image.  The result 
            is pass or fail.
          </para>
        </definition>
      </glossaryEntry>
    </glossary>
  </developerGlossaryDocument>
</topic>

The following image shows the results of the glossary transformation into HTML, built by DocProject (a tool that I've written to automate Sandcastle inside Visual Studio).  The VS2005 presentation style was used for this example.

And now here's the same exact topic file after being transformed into HTML using the Hana presentation style.

There are a few things to point out about all of this.

First of all, notice that the topic that I've written only uses some very basic XML, yet the output obviously contains additional layout and style, which differs depending upon the presentation style that I've chosen.  In the Hana version I've even left in the default header that warns about pre-release documentation.

You may have also noticed the letter bar and the individual letter sub headers.  Where'd they come from?  These features are not actually part of Sandcastle, but Eric Woodruff and I have added them to the presentation styles by modifying the XSL transformations that convert the MAML Glossary document type into HTML.  The additional behavior automatically detects the glossary terms in the topic and creates the letter bar and headers dynamically.  All of the terms are sorted alphabetically as well (although it's not obvious in my example because they're already in alphabetical order in my topic file).

Pretty cool, right?  You'll be able to get these Glossary updates from the new Sandcastle Styles project on CodePlex, which should go public within a few days after the next Sandcastle release.  This project was started by Paul Selormey, Eric Woodruff and myself.  In the last week we've been diligently working on preparations for our first release, so please check it out when we go live and let us know what you think :)

Linking in MAML

The last thing that I want to point out about the previous example is that it contains a hyperlink to an external web site.  As you can see from my topic, MAML supports an externalLink element that accepts text in a linkText element and a URI in a linkUri element.  It also accepts alternate text in a linkAlternateText element, but that's optional.

Instead of linking to external URIs, you can also link to any of the other topics being documented.  To do that you would use a very simplified version of the XLink specification on a link element, as in the following example:

<link xlink:href="37852294-410f-4bb2-9008-c5fa9dfb4347">Part II</link>

Right, topics are identified by GUIDs.  Currently, Sandcastle also requires that all conceptual topic files are named with a GUID and an .xml extension.  A bit annoying at first, but if you use DocProject it provides a Topic Explorer tool window that makes it easy to find the topic that you're looking for without having to open all of them :)

Notice that in my example the value in the href does not have an .xml file extension specified.  That's because link doesn't reference files, it references topics.  This is important to realize because it's not the same as the way linking works in HTML - this is actually dynamic.  If Sandcastle cannot find a topic that is associated with the specified GUID, then it doesn't generate a hyperlink at all.

This is a bit different from what we're used to in HTML, which allows us to link to anything under the sun using only one tag: a.  So why such weirdness in MAML?  I think the answer to that question is actually quite simple, although for some reason it's easy to miss when first starting out with MAML.  The MAML schema defines elements that apply structure and semantics to text, instead of format and style, like HTML.  For this reason, you wouldn't see a tag named simply, a in MAML because it's not descriptive at all.  Link, on the other hand, is very descriptive.  And since an HTML anchor is meant to provide the source point of a diametric link, its use is actually more limited than XLink.  The XLink specification actually provides a way to establish relationships between one or more resources (at least that's my interpretation of it), which would offer much more flexibility.  So MAML provides a mechanism to link to other topics, not just external URIs, and the XLink implementation provides an explicit way to describe links as being special - they must be processed by Sandcastle.  Currently, Sandcastle doesn't actually seem to use any of XLink's features though aside from what has been deemed as "simple" usage, but maybe that'll change in the future.

But that's not all.  If you want to create a link to an API in your reference documentation, you would use the codeEntityReference element instead.  Yikes!  So now we've got yet another way to link.  But again, keep in mind that MAML is much more expressive than HTML, and that's why we've got different tags for linking to different things.  The benefit being that our intentions are clear when we write our topics so that different styles of linking can be handled differently.

The following XML snippet illustrates all three approaches to linking in MAML topics.  Each example is a child of the relatedTopics element, which, in the Sandcastle world, will eventually become your topic's See Also section.

<relatedTopics>
  <codeEntityReference>T:MyNamespace.MyClass</codeEntityReference>
  <codeEntityReference>P:MyNamespace.MyClass.MyProp</codeEntityReference>
  <codeEntityReference>M:System.IO.File.OpenText(System.String)</codeEntityReference>
  <externalLink>
    <linkText>DocProject</linkText>
    <linkUri>http://www.codeplex.com/DocProject</linkUri>
  </externalLink>
  <link xref="home">My Home Page</link>
  <link xref="Contact Us"/>
  <link vref="/related.aspx">Related web page</link>
  <link xlink:href="14790228-f45b-42d5-9b3e-f6b4ab932b9e">Part II</link>
</relatedTopics>

Notice that there are also two more link types in the example above that I didn't mention previously: link elements with xref and vref attributes.  This type of linking is used instead of externalLink so that only an ID must be specified instead of an entire URL.  The ID is part of an ID-to-URL mapping that is configured elsewhere.  This feature is not actually part of Sandcastle though; it's provided by a custom build component that I've written which, for the next release of DocProject, has been modified to support conceptual builds as well.  The component is called ResolveExternalLinksComponent and it's available as a separate download or as part of DocProject.  Without this build component xref and vref do nothing.

Conclusion

HTML is out.  MAML is in.

Well, it's not actually as substantial of a change as I'm implying - HTML is still being used extensively as the final output for compiling help; however, we no longer have to author help topics in HTML, which is a huge benefit.

So all this stuff might seem really wonderful in print, but I feel that I must warn you: It actually took me a few weeks before I finally started to get rid of that itch to lace my topics with bold and italic phrases where it didn't actually add any value.  When you first start writing MAML it can feel very restrictive, and it is compared to HTML in terms of how quickly you can apply new styles, since to do that you have to leave the actual topic and modify files in the Sandcastle presentation; but it's actually much more expressive in terms of describing information and that's what we should be concentrating on when we write help topics - the information.

What I've learned from writing topics in MAML is that using elements such as ui, userInput, math, date, and many others, as well as externalLink, codeEntityReference and link for linking, ultimately accomplish the same thing as HTML but in a much better way - no more CSS class names to remember or abstract HTML tags like b and i (or strong and em too).  Instead, I can specify exactly what a phrase represents and continue writing.  The format and style is already defined for me by the presentation style that I choose, even if I haven't chosen it yet!  However, if I've already chosen one that mostly fits my needs but I'm not happy with a particular style, I can apply some HTML and CSS to the different MAML elements without having to update anything in the topics themselves.  By reusing the same common tags throughout my documentation, it looks much more professional, it's easier to manage and it's even portable since it's all XML, so if in the future I want to generate Open XML documents instead of HTML, I won't even have to change anything in my topics.

Note that if you want to convert all of your existing HTML topics to MAML in a batch process, I've got a tool called DocToMaml.  It's currently in beta, but it does work.  Any feedback on it will be appreciated :)

For the next version of DocProject 2008 (Beta 3) I'm working on a MAML WYSIWYG editor that is integrated into Visual Studio, so keep your eyes open for that.

If you have any feedback about how MAML and Sandcastle's conceptual build process can be improved please let the Sandcastle team know by submitting a request to the Sandcastle Issue Tracker on CodePlex.

I've just released the first beta version of ContractN, which is a fully managed programming-by-contract framework that uses the CLR's built-in support for aspect-oriented programming (AOP), which provides a way to seamlessly and intuitively describe the required pre- and post-conditions of a type's contract.

I got the idea based on this blog post by Sasha Goldshtein; although, Sasha's implementation and that of ContractN have little in common other than their purpose.  I liked the way Sasha used dynamically-generated code, but other than that I wasn't satisfied having to create the extra stuff that was required to enforce conditions.

Advantages of ContractN

The real benefit of ContractN, IMO, is that there's no overhead in terms of setting up your classes or their consumers, and you don't have to worry about breaking encapsulation or having to create any supporting interfaces or classes.  You just need to derive an object from ContractN.ProgrammingByContract. (It's a shame that this is required though - I hope AOP is eventually introduced as a C# language feature; although, I suspect that the CLR would have to participate as well to alleviate the need for subclassing.)

ProgrammingByContract is an abstract base class that sets up the AOP infrastructure, which is used to intercept method invocations and define conditions as attributes.  After defining a class that inherits from ProgrammingByContract, simply add the appropriate attributes wherever you'd like to enforce certain conditions on the public contract of your type.  Here's a really simple C# 3.0 example:

public class Person : ContractN.ProgrammingByContract
{
  [InRequired, OutRequired]
  public string Name { get; set; }
}

The example illustrates that there are some built-in conditional attributes for basic argument checking, such as InRequiredAttribute, OutRequiredAttribute and ReturnRequiredAttribute, which throw an exception when an argument (or in the case of the latter two, output parameters and a return value) are null.  I plan to add more conditions in the future, and the framework was designed to be flexible enough so that you can define your own conditions as well.

Custom Conditions

To define a new condition, create a class that derives from either PreConditionBaseAttribute, PostConditionBaseAttribute or ConditionBaseAttribute and override the abstract members.  Usage should be intuitive depending upon the implementation that you need, but you can look at the source code for InRequiredAttribute (beta 1) as an example.

Dynamic Conditions

I decided to take a different approach than Sasha in terms of how dynamic conditions are implemented.  I wanted to take advantage of static type checking instead of hard-coding constraints as strings.  PreConditionAttribute and PostConditionAttribute allow you to specify a method name on the decorated type that will be invoked automatically whenever any public method is called.  That also includes constructors and accessors for properties and events.  (Note that these attributes are also valid on a property, constructor, method or event.)  The method signature can vary depending upon your needs; generally it will be a parameterless instance method, though static methods are allowed as well.  The types of arguments that are accepted encapsulate information about a particular invocation.  For example, you could define a method that accepts an argument typed as ContractN.PropertyCall and the method will be invoked automatically whenever a property get or set accessor is invoked, pre- or post-invocation depending upon the attribute used.  Within these methods you can write code that will actually be part of the class that it's meant to constrain, which maintains encapsulation, makes it easy to share code between constraints and allows access to state with compiler support.

Here's a more sophisticated example than above.  It makes use of pre- and post-conditions described above and also implements ad-hoc conditions by providing an implementation of ICondition:

[PreCondition("TestStatic")]
[PreCondition("TestInstance")]
class AdvancedFeatures : ProgrammingByContract, ICondition
{
    #region Public Properties
    public string TestProperty { get; set; }
    #endregion

    #region Methods
    private static void TestStatic(PropertyCall info)
    {
        Console.WriteLine("Pre Static Property: {0}", info);
    }

    private static void TestStatic(MethodCallBase info)
    {
        Console.WriteLine("Pre Static Default: {0}", info);
    }

    private void TestInstance(MethodCallBase info)
    {
        Console.WriteLine("Pre Instance Default: {0}", info);
    }

    public void NoOperation()
    {
    }
    #endregion

    #region IPreCondition Members
    void IPreCondition.Apply(ConstructorCall info)
    {
        Console.WriteLine("Pre: {0}", info);
    }

    void IPreCondition.Apply(MethodCall info)
    {
        Console.WriteLine("Pre: {0}", info);
    }

    void IPreCondition.Apply(PropertyCall info)
    {
        Console.WriteLine("Pre: {0}", info);
    }

    void IPreCondition.Apply(EventRegistrationCall info)
    {
        Console.WriteLine("Pre: {0}", info);
    }
    #endregion

    #region IPostCondition Members
    void IPostCondition.Apply(ConstructorCall info, MethodCallReturn returnInfo)
    {
        Console.WriteLine("Post: {0}", info);
    }

    void IPostCondition.Apply(MethodCall info, MethodCallReturn returnInfo)
    {
        Console.WriteLine("Post: {0}", info);
    }

    void IPostCondition.Apply(PropertyCall info, MethodCallReturn returnInfo)
    {
        Console.WriteLine("Post: {0}", info);
    }
    
    void IPostCondition.Apply(EventRegistrationCall info, MethodCallReturn returnInfo)
    {
        Console.WriteLine("Post: {0}", info);
    }
    #endregion
}

When the above class is used as follows:

AdvancedFeatures features = new AdvancedFeatures();
features.TestProperty = "A test value";
features.NoOperation();

The console output is:

Pre Static Default: ContractN.ConstructorCall
Post: ContractN.ConstructorCall
Pre: ContractN.PropertyCall
Pre Instance Default: ContractN.PropertyCall
Pre Static Property: ContractN.PropertyCall
Post: ContractN.PropertyCall
Pre: ContractN.MethodCall
Pre Instance Default: ContractN.MethodCall
Pre Static Default: ContractN.MethodCall
Post: ContractN.MethodCall

One of the really neat uses for the PreConditionAttribute that I've found is to have ObjectDisposedException automatically thrown instead of having to check for it everywhere.  Here's an example that builds off of the Person class in my first example:

[PreCondition("NotDisposed")]
public class Person : ContractN.ProgrammingByContract, IDisposable
{
    [InRequired, OutRequired]
    public string Name { get; set; }

    private bool disposed;

    private void NotDisposed()
    {
        if (disposed)
            throw new ObjectDisposedException(this.GetType().FullName);
    }

    public void Dispose()
    {
        disposed = true;
    }
}

After the Dispose method is called, public members are no longer accessible to external types.  All calls will result in ObjectDisposedException automatically!

Conclusion

ContractN is more of a hobby project for me now than anything else, but it has potential and I'd really like to see it blossom into something useful.  If you have any suggestions for improvements please let me know and I'll consider them for the next beta release.  (I'm going to establish a timeline on-the-fly as I get ideas for features.)  I have some interesting ideas for new conditions and I hope to blog about them in the future, so don't wander too far off if you're interested...

I've just published the Help! project on CodePlex, which is intended to provide a fully managed compiled help system to replace Help 1.x (.chm).  Please contribute to this project by using the discussions area to post comments and ideas or join the development team.  If you want to join the team then please see the Welcome! thread for information.

For now I'll be using my blog to post news and information about this project.

In case anyone's interested, this post contains my notes for converting the code base of DocProject 1.7.0 to support Visual Studio 2008 Beta 2. After that I've included DocProject's backlog of tasks, which may contain some ideas that you'd be interested in seeing me implement in a subsequent release.

Notes on the Conversion from 1.7.0 to 2008

It was no easy task, though I don't think my experience is any indication of what must be done for the conversion of general 2.0 Framework applications since DocProject relies heavily on Visual Studio itself at runtime, which is uncommon (at least until future apps start using the new Visual Studio Shell).

While logging this information I was also running into various issues building the source code and running the program. In order to ease maintenance I made whatever changes I could to both the 1.7.0 and 2008 code bases to fix the bugs while keeping them in sync. As a result the code base of DocProject 2008 is 99.99% the same as DocProject 1.7.0, with a few special adjustments such as a different hard-coded installation path under the root installation directory, for example. Many of the updates that I logged were simply changes to references of version numbers but there were also many modifications to the InstallPrep and Installer projects. I don't think any bug fixes are logged here, however.

This is not an exhaustive list of what was required for the conversion. If the time comes again that I must convert DocProject's source I'll certainly follow these notes as a guideline, but I'm sure the process will require my attention.

1. After installing DocProject on the build machine, copy the Source folder to "My Documents\Visual Studio 2008\Projects\DaveSexton.DocProject08\" and uninstall DocProject.
2. Update the Installer project to search for "VSORCAS OR VCSORCASEXPRESS OR VBORCASEXPRESS", but rename the search and conditions to use 2008 instead of ORCAS; e.g., VS2008 OR ...
   NOTE: In 1.7.0 RC for VS 2005 the VSORCAS searches were removed to support side-by-side installation. The VS2005 searches should actually be replaced with VS2008 searches instead.
3. Update the DaveSexton.DocProject.InstallPrep.CustomActions.Contents property to return ONLY the Orcas content.
4. Update all of the projects, including the templates, to use the 3.5 framework. Note that the DocSite templates require a temporary modification to the web.config file before hand unless the modification is done manually. To automate it, use the properties dialog - Application tab for C# projects and Compile tab > Advanced Compile Options button for VB projects.
5. Replace the 2.0 framework prereq with the 3.5 framework in the Installer project.
6. Update the DaveSexton.DocProject VCProjectEngine reference to 9.0.0.0
7. Update DaveSexton.DocProject.InstallPrep\PostBuildEvents\DaveSexton.DocProject.bat, and the Sandcastle, Sandcastle/Deployment and HtmlEditor projects' post build events:
       "%programfiles%\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" ...
8. Update DaveSexton.DocProject.InstallPrep:
   1. BootStrapper\vs_piaredist should be changed to vs90_piaredist.
   2. Copy in the vs90_piaredist files and update the Readme.txt file.
   3. Update the Readme.txt file in the project's root to reference vs90_piaredist instead of vs_piaredist.
   4. Contents\Installer\DocProject.AddIn file should be updated to version 9.0 in the HostApplication element.
   5. Replace all occurances of <ContentVersion>1.0</ContentVersion> with <ContentVersion>2.0</ContentVersion> in Contents\Installer\DocProject.vscontent
9. Delete the DaveSexton.DocProject.DocProjectAddIn.GetMSBuildBinPath methods since they are no longer used (see next step)
10. Delete all the code in the DaveSexton.DocProject.MSBuild.MSBuildAnyProject(string) class constructor above and including the local variable named, "vsVersion" and use the parameterless Engine() constructor instead. (There is a warning about this in the Error List when building the DocProject Add-In assembly).
11. Delete Microsoft.Build... references from DaveSexton.DocProject and DaveSexton.DocProject.ExternalUI and replace them with references to the new 3.5 versions.
12. Remove the DaveSexton.DocProject.MSBuild.BuildDocProject.MSBuildBinPath property and all remaining related code, including code that constructs and MSBuild Engine class using a custom bin path. (There will be warnings in the Error List after building the DocProject Add-In assembly).
13. Update the DaveSexton.DocProject.InstallPrep\DaveSexton.DocProject.targets file by removing the MSBuildBinPath attribute from the BuildDocProject and CleanDocProject tasks.
14. Update DocProject and DocSite project templates (found in the InstallPrep's Contents\[template name] folders):
    1. Change version to <ProductVersion>9.0.20706</ProductVersion>
    2. Append <TargetFrameworkVersion>v3.5</TargetFrameworkVersion> to the first PropertyGroup
    3. Change targets reference to <Import Project="$(DocProjectPath)\VS2008\bin\DaveSexton.DocProject.targets" />
       (NOTE: DocProjectPath environment variable must be defined - this is indicated below)
15. Update DocSite templates:
    1. Change type guids to (actually, I don't think this step is necessary):

       C#: <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{fae04ec0-301f-11d3-bf4b-00c04f79efbc}</ProjectTypeGuids>
       VB: <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

    2. Change web app target to <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\WebApplications\Microsoft.WebApplication.targets" />
16. Global search for "8.0" just in case any more updates are required for "9.0" (NOTE: the binding redirect in the External UI project's app.config file should NOT be updated to 9.0.0.0)
17. In the InstallPrep project delete the WiRunSQL.vbs file and the UpdateMSI.bat file, and then remove the Installer project's post build event.
18. In the Installer project:
    1. In the File System Editor, change the Application Folder's DefaultLocation to "[ProgramFilesFolder][Manufacturer]\DocProject"
    2. Create a folder under the Application Folder and name it "VS2008"
    3. Drag each root folder into the VS2008 folder and then drag the files located in the root into VS2008 as well.
    4. Changeg the Version to 1.0.0
    5. Change the ProductCode and the UpgradeCode.
    6. Change the Subject to "DocProject 2008 (Full)"
    7. Change the ProductName to "DocProject 2008"
    8. Open the Registry Editor. Hard-code "DocProject" instead of using [ProductName] used in the key for the InstallPath value (for both the HKCU and HKLM).
    9. In the Registry Editor, change the value of each SafeImports key (DocProject value) to [TARGETDIR]VS2008\bin\DaveSexton.DocProject.targets
    10. In the Registry Editor, change the "8.0" key to "9.0" in each of the paths to the SafeImports values.
    11. For each SafeImports key (DocProject value) also udpate the condition to use 2008 instead of ORCAS; e.g., VS2008, VCS2008EXPRESS and VB2008EXPRESS
    12. Change the VS2008\bin\DaveSexton.DocProject.dll.config file's TargetName property to DaveSexton.DocProject08.dll.config
19. Change the assembly and file version for each project to "0.0.1.0"
20. Append "08" to each project's assembly name (Applications tab in the project properties dialog).
21. Do a global search in the entire solution for "DaveSexton.DocProject (with a single quote prefix). Where appropriate, append "08" to the assembly name of references and upate the version to "0.0.1.0" as well, for all projects - not just DaveSexton.DocProject.dll.
22. Rename the solution to "DaveSexton.DocProject08". Then locate the .sln file in the Installer project's File System Editor under the Source folder. Delete the existing file and add the new 08 .sln file.
23. Rename the DocProject.AddIn and DocProject.vscontent files in the InstallPrep project to DocProject08.AddIn and DocProject08.vscontent. They're located in the Contents\Installer folder. Also update DocProject08.vscontent so that it references the renamed .AddIn file.
24. Append "08" to the name of each template folder and update the Installer\DocProject08.vscontent file to point to the renamed .zip files.
25. Update the InstallPrep project's Install.CustomActions class:
    1. Repeat the previous step on the static fields that reference the template zip files and the .AddIn file.
    2. Update the InstallEnvironmentVariables method to use "targetEnvironmentVariable, Target"
    3. Update the UninstallEnvironmentVariables to use "targetEnvironmentVariable, null"
    4. Delete the TargetBin property
    5. Change the buildPathEnvironmentVariable field to: private const string targetEnvironmentVariable = "DocProjectPath";
    6. Add a new property (after the Target property) named TargetFlavor that returns "Path.Combine(Target, @"VS2008\");"
    7. Update the CreateInstaller method to use TargetFlavor instead of Target.
26. Update the DocProjectAddIn class's InstallPath property to return Path.Combine(installPath, @"VS2008\");
27. Update the BuildOutput.bat file in the root of InstallPrep:
    1. Update the template names (only the second argument must be updated). For example: CALL :ZIP_TEMPLATE "%TemplateDirRoot%\VBDocSiteTemplate" VBDocSiteTemplate08
    2. SET InstallerName=DocProjectInstaller08
28. Build the InstallPrep project once. Then open the Installer project's File System Editor and navigate to Application Folder\VS2008 in the tree:
    1. Delete the DocProjectInstaller.vsi file and replace it with the new DocProjectInstaller08.vsi file found in the InstallPrep project directory.
    2. Open the Setup\AddIn folder and delete DocProject.AddIn. Add DocProject08.AddIn from the Contents\Installer folder.
    3. Open the Setup\Templates folder and delete all of them. Add them back from the Contents\Templates folder (hidden in Solution Explorer), but only choose the 08 versions (previous versions can be deleted since this is a temporary folder and they are no longer in use).
29. Run Code Analysis on each project, including InstallPrep but not the templates or the Installer. In some projects there may be certain rules that should be disabled. For example:
    1. Naming: CA1703 (resource spell-checker)
    2. Naming: CA1704 (identifier spell-checker)
    3. Naming: CA1716 (indentifiers that are keywords, such as "step")
30. Make sure to exclude all detected dependencies in the Installer project before building it.

Make sure that the solution builds without any errors. Then build the InstallPrep project and make sure that it completes without any errors.
(Check the build output window for DSZip errors). Build the Installer project and install DocProject on the build machine.

• Install DSZip on the build machine.
• Make sure to copy the "vs90_piaredist" bootstrapper folder into the "C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bootstrapper\Packages\" folder on the build machine.

Backlog

Here's some stuff that's been on the backburner for quite some time now.

• Auto-merge changes to partial-build items. Concrete engines are responsible, but the BuildEngine class could copy all partial items to a temporary folder before a build. Then after the build completes, call the derived-implementation of protected abstract void MergePartialBuildItem(DocProjectItem oldItem, DocProjectItem newItem) for each file in the temp folder. Finally, each temp file would be deleted. This method should be added to the IBuildEngine interface as well. If an implementation can't or shouldn't merge the changes then it doesn't have to do anything with the temp file, or even open it.
• Create an item template for Sandcastle's build component (the New Project Wizard may have to be responsible for referencing the BuildAssemblerLibrary assembly unless the item template supports some sort of wizard that can do the job – but this needs to be researched)
• Components dialog that served as an interface for the .config files. It would be a list of components found in the Sandcastle assemblies, the project's assembly, and any registered assemblies. The order and parameters could be specified, somehow.
  • Create a wizard step that asks the user to choose from various build components to be included in the project's {root}\Components\ folder. For example, XAMLSyntaxGenerator.cs, which would automatically be registered in the .config file.
  • Allow users to add cusotm build components to a DocProject or DocSite via an item template.
  • List of "registered" assemblies on left (registered assemblies cannot be removed from the list if there are stack components that reference them)
  • Build component stack displayed on right ("New" drop-down button, with "External Component", like a browse button, and "Internal Component", that searches the project's assembly (also with "Add New" as an option, somewhere)
  • Up down arrows that control the order that components are executed in the stack
  • Reset button that will import the configuration file for the project's presentation, replacing all changes (Are you sure? dialog)
• Have a start debug button ">" in the Sandcastle toolbar that will put Visual Studio into debug mode and attach to an external "msbuild.exe /project" process for debugging custom build componenents and the build process component. For DocProjects, this may be possible without a custom button by setting the startup program on the Debug tag, if it's possible to do that from the templates.
• ApiFilter Issues:
  • reference method signatures?
  • discover generic type name? (currently, the id is parsed by searching for .name` where name is the value of the name attribute of the corresponding apidata element)
• API Topic Management Dialog:
  • Add/Save/Load feature: Allow users to add a filter (regex or category) to a list that can be ordered using up/down arrows. Saving the list would create an XML doc. Loading the XML doc would recreate the list in the dialog, which could be applied by clicking the list's Apply button (each individual filter would be processed one at a time, top-down).
• Attribute Filter Dialog for Syntax Generator ("Syntax Attribute Filter")
• Automatic token replacement in code comments:
  • Have a dialog that lists token values that can expand to full HTML, which can be edited by clicking an ellipses button that opens an Html Editor dialog, for each individual token.
  • Users can import and export the list to and from an xml file.
  • Make the token replacement start and end strings (both "$" in the examples below) configurable in the tools options page in case the default values conflict with some other device. The values are required and cannot contain ANY white-space. Token replacement will only occur if the following regular expression is matched: \$[^\s\$]+\$
  • When customized, for example, if the start token is % and the end token is ^ then the expression would be: \%[^\s\%]+\^

• Examples:

  Custom Tokens
  
  $isnull$      "is a <strong>null</strong> reference (<strong>Nothing</strong> in Visual Basic)"
                (this could be built-in, but editable)
  $my-token$    "Hello <strong>World</strong>!"
  more custom...
  

  
  Uneditable Tokens (cannot be modified, imported or exported):

  $time$        DateTime.Now.ToLongTimeString(), in the current culture
  $date$        DateTime.Now.ToLongDateString(), in the current culture
  $datetime$    DateTime.Now.ToString(), in the current culture
  $user$        Expands to the current user's name.  e.g., Administrator
  $name$        Expands to the name of the member or type to which the comment applies.  
                e.g., ToString, IsEnabled, BuildEngineProvider
  $fullname$    Expands to the full $name$.  e.g., MyNamesapce.MyObject.ToString, 
                MyNamesapce.MyObject.IsEnabled, MyNamesapce.BuildEngineProvider
  $type$        Expands to the containing type of the member, or the type itself if the 
                comment applies directory to a type.
  $-type$       Exapnds $type$ to a link.  Same as: <see cref="$type$"/>
  $-x|y$        Expands to a hyperlink: <a href="y" mce_href="y">x</a>
                or <a href="x" mce_href="x">x</a> if the vertical bar is not specified.
  $project$     Expands to the name of the project: {project name}
  $-project$    Expands to the project link:  <a href="R_Project.htm" >{project name}</a>
                (or the GUID form)
  $home$        Expands to a hyperlink with its url designated by the user.  By default, this token 
                will be ignored unless the user enters a valid Uri and name.  (custom editor, I guess)
  $art$         Expands to the virtual artwork path; currently, /Art
  

Several of these ideas are already work items that you can vote on. Please let me know if any of them seem interesting to you. I have several other ideas as well that I haven't written down yet but I will when the time comes.

The only reason why I haven't started on the token replacement feature is because I think Microsoft already has an internal implementation that they use, but I'm just itching to create it myself. I'm waiting now to see if they'll make theirs public first before I attempt to reinvent the wheel, if in fact their implementation supports the feature set that I'm after.