Category Archives: Uncategorized

Getting started with xlang and C++/WinRT

With the unveiling of the xlang project, C++/WinRT is now a completely open source project. While C++/WinRT has always been about providing an open source header-only library for Windows development, the cppwinrt compiler that generates the headers continued as a closed source project. The xlang project represents a rewrite of the C++/WinRT compiler. This isn’t as dramatic as it may seem. I’ve rewritten the C++/WinRT compiler numerous times during its history. I don’t subscribe to the “all of my code is precious” philosophy of software development. Notably, this is the first version of the compiler where I have my very own winmd parser written entirely in portable and standard C++. Previous iterations relied on the metadata APIs inherited from the .NET project and now included in Windows as a set of COM APIs. Tools like MIDL continue to use those APIs, but they are unnecessarily complex, somewhat buggy, and don’t lend themselves to efficient or concise C++ consumption.

I’ll write more specifically about xlang’s metadata library soon, but today I want to show you how you can get started with xlang in general and go from nothing to compiling your own C++ app. And I’ll use a console app for everything to once again reinforce the fact that there are no “store” or app container requirements, as a lot of developers still seem to believe. We are slowly dismantling all of the remaining requirements that code run in an app container and C++/WinRT specifically has never imposed any such restriction. Right, let’s get started.

These instructions are for Windows developers. You can also build xlang on Linux, as our daily build can testify, but I’m still a Windows guy so that’s what I’ll use.

Open a Visual Studio developer command. I would suggest using the latest Visual Studio 2019 preview release as it is slightly less buggy when it comes to handling cmake projects. I’m no cmake expert, but we are using cmake to make it easier to support cross-platform builds if necessary.

In the developer command prompt, clone the xlang repo:

C:\sample> git clone https://github.com/Microsoft/xlang

With the xlang repo in hand, one option is to open the project using Visual Studio:

C:\sample> devenv xlang

Visual Studio’s cmake support is still a bit rough, but it mostly works and is what I use for most of my development on this project.

Alternatively, there is a script that will allow you to build using cmake directly from the command prompt:

C:\sample> xlang\src\scripts\windows\build.cmd -b Release cppwinrt

Once this completes, you should find that the cppwinrt compiler has been built:

C:\sample> dir /s /b cppwinrt.exe
C:\sample\xlang\_build\Windows\x64\Release\tool\cpp\cppwinrt\cppwinrt.exe

You can then copy it locally for convenience:

C:\sample> copy C:\sample\xlang\_build\Windows\x64\Release\tool\cpp\cppwinrt\cppwinrt.exe

And you’re now ready to produce your own headers. One simple way to get started is to target the metadata for your current installation of Windows:

C:\sample> cppwinrt.exe -in local -verbose
tool: C:\sample\cppwinrt.exe (C++/WinRT v2.0.000000.0)
in: C:\WINDOWS\System32\WinMetadata\Windows.AI.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.ApplicationModel.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Data.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Devices.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Foundation.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Gaming.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Globalization.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Graphics.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Management.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Media.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Networking.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Perception.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Security.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Services.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Storage.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.System.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.UI.Xaml.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.UI.winmd
in: C:\WINDOWS\System32\WinMetadata\Windows.Web.winmd
out: C:\sample\
time: 801ms

As you can see, the “local” input directs cppwinrt to read the metadata from the system’s WinMetadata folder. The resulting winrt folder containing all of the C++/WinRT headers and is now available for you to include:

C:\sample> dir /b winrt
base.h
impl
Windows.AI.MachineLearning.h
Windows.AI.MachineLearning.Preview.h
Windows.ApplicationModel.Activation.h
Windows.ApplicationModel.AppExtensions.h
Windows.ApplicationModel.Appointments.AppointmentsProvider.h
Windows.ApplicationModel.Appointments.DataProvider.h
Windows.ApplicationModel.Appointments.h
Windows.ApplicationModel.AppService.h
Windows.ApplicationModel.Background.h
Windows.ApplicationModel.Calls.Background.h
Windows.ApplicationModel.Calls.h
Windows.ApplicationModel.Calls.Provider.h
Windows.ApplicationModel.Chat.h
Windows.ApplicationModel.CommunicationBlocking.h
Windows.ApplicationModel.Contacts.DataProvider.h
Windows.ApplicationModel.Contacts.h
.
.
.

Time to write some code! How about a simple console app that prints out my recent blog posts using the Syndication API:

C:\sample> type app.cpp

#pragma comment(lib, "windowsapp")
#include "winrt/Windows.Foundation.Collections.h"
#include "winrt/Windows.Web.Syndication.h"
#include <stdio.h>

using namespace winrt;
using namespace Windows::Foundation;
using namespace Windows::Web::Syndication;

int main()
{
    Uri uri{ L"https://kennykerr.ca/feed" };
    SyndicationClient client;
    SyndicationFeed feed = client.RetrieveFeedAsync(uri).get();

    for (auto&& item : feed.Items())
    {
        hstring title = item.Title().Text();

        printf("%ls\n", title.c_str());
    }
}

You can then compile it using the Visual C++ compiler as follows:

C:\sample> cl /EHsc /std:c++17 /permissive- /W4 /I . app.cpp

Of course, you can use Clang if you prefer. And then you can simply run the app as follows and, as you can see, this is just a good-old console app:

C:\sample> app.exe
The State of C++ on Windows
xlang
Compilers and Conformance
New in C++/WinRT: Mastering Strong and Weak References
New in C++/WinRT: Async Cancellation Callback for Coroutines
C++/WinRT on Wikipedia
New Features and Changes Coming to C++/WinRT: Header Isolation
C++/WinRT and Beyond
C++/WinRT: Creating Collections Simply and Efficiently
C++/WinRT: Coroutines and the Calling Context

I’m still putting the finishing touches on this update. Once concluded, the xlang version of C++/WinRT will become the next major update to the C++/WinRT project distributed through the regular channels.

And that’s all for today. I hope you enjoy exploring xlang and building apps using C++/WinRT. Stay tuned for more!

The State of C++ on Windows

As the 2019 State of the Union Address appears to be postponed, I thought I would offer you the State of C++ on Windows instead!

C++ continues to be an important programming language in general and the only systems programming language of any consequence on the Windows platform. The Visual Studio 2019 release has some incredible improvements that I hope to share more about in the coming weeks, but right now I want to give you an update on the state of C++/WinRT.

C++/WinRT continues to gain adoption as developers eschew WRL and C++/CX in favor of modern and standard C++. You may have heard that the Windows UI Library was recently open sourced. Did you also notice that it is written entirely in C++/WinRT? That’s just one high-profile example, but there are many teams that have made the switch and we are strongly discouraging anyone from using WRL or C++/CX going forward.

Even as C++/WinRT is used to implement large parts of Xaml, adoption of C++/WinRT for building Xaml apps is slow. It helps to recognize that there are three categories or scenarios where C++/WinRT comes into play most frequently.

1. Using or calling Windows APIs (otherwise known as consuming WinRT types)
2. Authoring Windows APIs (otherwise known as producing WinRT types)
3. Authoring Xaml apps and controls

The first one involves making API calls using C++/WinRT to do things like communicate using Bluetooth, stream and present video, integrate with the Windows shell, and so on. The second involves authoring APIs that provide such capabilities, the graphics APIs, the storage and file system APIs, the networking APIs, and so on. The third involves writing apps and controls built on the Xaml framework. While Xaml is just one of those APIs, it is the dominant UI framework on Windows today and it has an outsized influence over WinRT, so it deserves its own category.

C++/WinRT fully and uncompromisingly supports the first category. Authoring brand new WinRT APIs is a little more involved because a developer must use IDL to define the shape of the API before the developer can implement the API. This is however not substantially more complicated than using C++/CX and, given the many benefits of using C++/WinRT, developers are generally very happy with the trade. It is the third category where things fall apart. Xaml was designed for .NET and assumes and effectively requires reflection to function as intended. While you can write Xaml apps with C++/WinRT, it is pretty cumbersome. This is where the C++ reflection proposal would be very valuable.

Still, I am hopeful that we can turn that around. It pains me that Windows does not have a good UI story for C++ developers. C++ reflection may well help to solve that problem in the long run, but we cannot wait for that to arrive, so we are also exploring some other options that may prove useful in the near future. Some of those include alternative approaches to Xaml that does not require reflection, as well as discussions with DevDiv to get Visual Studio to do a better job of supporting a C++/WinRT developer experience.

Stay tuned for more on C++/WinRT and xlang. A lot has been going on – I’ve just been heads down getting it done, but I plan to start writing again and sharing some of the work that we’ve been doing.

Next: build your first console app (nothing to do with UWP) with xlang and C++/WinRT.

xlang

Back in July I hinted at the fact that I have started working on a new project that builds on C++/WinRT. Well that project is finally out in the open and it’s called xlang: a cross-language, cross-compiler, and cross-platform generalization of WinRT. It has no dependency on Windows or even WinRT and thus sports an open source PAL. Think of it as WinRT but with much greater reach, potentially supporting Windows 7, Linux, Apple platforms, and more.

https://github.com/Microsoft/xlang

It’s far from done but we are developing it in the open, so you will be able to follow along, provide feedback, and even contribute your own language projections and platform support. I will share more information once there’s more there that works, but feel free to reach out and let me know what you think. Just to set expectations, none of this is ready to replace C++/WinRT just yet. My main goal right now is to reach feature parity with C++/WinRT. At that point I will share more information and explain how the xlang tools and libraries work to generate projections and much more. If you’re impatient, xlang already includes a metadata reader that provides an incredibly fast and portable abstraction over the ECMA-335 format that sits at the heart of both WinRT and now xlang. I started writing a new metadata reader in portable C++ long before this project got started as it’s so fundamental to this whole space. It is thus quite mature, and you can have a lot of fun with it. Happy coding!

Compilers and Conformance

As the compiler vendors ramp up investment in C++ features and improved conformance, there is a bit of an arms race with library developers trying to keep up, while maintaining some degree of compatibility with current compilers. Two recent issues highlight this quite well. As of Visual C++ 15.8 Preview 3, Microsoft’s C++ compiler provides dramatically improved conformance. This has had the effect of detecting long-standing bugs in various libraries that previously compiled without issue.

Take for example the version of C++/WinRT that shipped with the 17134 (RS4) build of the Windows SDK. C++/WinRT builds fine without /permissive-, but with the added conformance that this flag provides it now produces the following error:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.17134.0\cppwinrt\winrt/base.h(2185): error C3861: ‘from_abi’: identifier not found

That’s because the from_abi function was used before it was declared. This all happens inside various templates and the Visual C++ compiler has traditionally been rather lazy about resolving such names and so it previously worked and went undetected. Now that the Visual C++ compiler provides two-phase name lookup via /permissive- it can and does detect such bugs. I have been testing with daily builds of the Visual C++ compiler and thus fixed this bug months ago. The fix is thus already available in preview builds of the Windows SDK. Of course, this is not unique to C++/WinRT. I have fixed numerous such bugs in a variety of libraries over the last year as we’ve been getting ready to ship Visual C++ 15.8. I also now do a much better job of leveraging Clang for added validation.

If you are however stuck using Visual C++ 15.8 with the 17134 SDK then you can work around this problem by removing the /permissive- flag (aka Comformance mode) from the C++ compiler options.

Another example is one that’s plagued C++/WinRT from the early days but was actually a compiler bug rather than an undetected library bug. Consider this example:

struct Type {};

template <typename T>
void Use()
{
    Type{};
}

namespace Sample
{
    struct Type {};
}

using namespace Sample;

int main()
{
    Use<int>();
}

The global Type is being used in a function that happens to be a template. Later, Type is defined in the Sample namespace. That’s obviously a different Type and there’s no ambiguity. Indeed, even if the using namespace directive is used to introduce the names from the Sample namespace for unqualified name lookup there should still be no ambiguity unless the code actually refers to an unqualified Type following the using namespace directive. And yet, Visual C++ has historically complained about this:

app.cpp(6): error C2872: ‘Type’: ambiguous symbol

Visual C++ was being lazy, not necessarily a bad thing, and didn’t bother to resolve the Type in the context of the function template’s definition. Instead, it chose to resolve it in the context of its instantiation. That’s definitely a bad thing. Thankfully, using /permissive- forces two-phase name lookup and this error goes away and this correct code finally compiles!

Improved conformance helps everyone. Library developers just happen to be on the front line and work very hard to deal with these issues as quickly as possible. Sometimes delays in the various distribution channels or uncoordinated releases of compilers and libraries can also cause headaches for app developers. Hopefully this helps to shed some light on these issues.

New in C++/WinRT: Mastering Strong and Weak References

Today I’d like to share another feature of C++/WinRT available in build 17709 of the Windows SDK that really helps to build more complex systems simply and correctly. Distinguishing between strong and weak references is often a necessity in reference-counted systems like WinRT. Knowing how to manage those references correctly can mean the difference between a reliable system that runs smoothly and one that crashes unpredictably. C++/WinRT makes this very simple with some helper functions with deep support in the projection.

While I have already talked about how to work with implementations and understand weak references, reference cycles can wreak havoc on well-intentioned designs and until recently there hasn’t been a simple way to deal with that when it comes to your own implementations. Of course, using neither strong nor weak references is just as problematic. Consider this standalone example:

struct App : implements<App, IInspectable>
{
    hstring m_value{ L"Hello world" };

    IAsyncOperation<hstring> Async()
    {
        co_await 5s;
        co_return m_value;
    }
};

It seems simple enough. The App class has an Async method that eventually returns the value. Now consider this main function:

int main()
{
    init_apartment();
    auto app = make_self<App>();

    auto async = app->Async();

    auto result = async.get();
    printf("%ls\n", result.c_str());
}

Does this seem reasonable? Here’s what happens:

1. The app is created.
2. The async object is created (pointing to the app).
3. The get function blocks for a few seconds and then returns the result.
4. No problems.

But what if the app object is destroyed before the async operation completes? Consider this one-line change:

int main()
{
    init_apartment();
    auto app = make_self<App>();

    auto async = app->Async();
    app = nullptr; // <-- oops

    auto result = async.get(); // <-- boom!
    printf("%ls\n", result.c_str());
}

That should be harmless right? After all, the app object is not referred to after that point. Oh, but it is. The async operation attempts to copy the value stored inside the app (via its implicit this pointer). After all, the coroutine is a member function and considers it the current object that it naturally has access to. Here’s what happens now:

1. The app is created.
2. The async object is created (pointing to the app).
3. The app is destroyed.
4. The get function blocks for a few seconds and then… BOOM!

As soon as it attempts to access the variable inside the app object it will crash or do something entirely undefined. The solution is to give the async operation – the coroutine – its own strong reference to the app object. As it stands, the coroutine effectively holds a raw this pointer to the app object. That is not enough to keep the app object alive. The App class may be updated as follows:

struct App : implements<App, IInspectable>
{
    hstring m_value{ L"Hello world" };

    IAsyncOperation<hstring> Async()
    {
        auto strong = get_strong(); // <-- keep alive

        co_await 5s;
        co_return m_value;
    }
};

Now everything works as expected. All outstanding references to the app may disappear but the coroutine ensures that its dependencies are stable. Of course, a strong reference may not always be desired. A weak reference is also possible as follows:

struct App : implements<App, IInspectable>
{
    hstring m_value{ L"Hello world" };

    IAsyncOperation<hstring> Async()
    {
        auto weak = get_weak(); // <-- maybe keep alive

        co_await 5s;

        if (auto strong = weak.get())
        {
            co_return m_value;
        }
        else
        {
            co_return L"";
        }
    }
};

In this case, the coroutine holds a weak reference that will not keep the app from being destroyed if no other strong references remain. The coroutine must then check whether a strong reference can be acquired before using the member variable.

Of course, lifetime issues are not limited to coroutines or concurrency. You can land up in the same boat using traditional callbacks. Consider another standalone example, this time a hypothetical Window class that sports a single event:

struct Window
{
    event<EventHandler<int>> m_event;

    void PointerPressed(EventHandler<int> const& handler)
    {
        m_event.add(handler);
    }

    void RaisePointerPressed()
    {
        m_event(nullptr, 123);
    }
};

I’m using the Windows::Foundation::EventHandler delegate but the parameters don’t matter. It could just as well be any other delegate. Handlers may be registered, all of which will be called when the event is raised. Now consider a typical app that wants to respond to such events:

struct App : implements<App, IInspectable>
{
    hstring m_value{ L"Hello world" };

    void Register(Window& window)
    {
        window.PointerPressed([&](auto&&...)
        {
            printf("%ls\n", m_value.c_str());
        });
    }
};

In many cases, this is in fact perfectly reasonable. For graphical applications, the App object typically outlives the framework that may raise such events. Still, that’s not necessarily the case and it’s important to know how to deal with that. Consider this main function:

int main()
{
    init_apartment();
    Window window;
    auto app = make_self<App>();

    app->Register(window);

    window.RaisePointerPressed();
}

It seems reasonable enough and you should expect this to work reliably, but consider this change:

int main()
{
    init_apartment();
    Window window;
    auto app = make_self<App>();

    app->Register(window);
    app = nullptr; // <-- oops

    window.RaisePointerPressed(); // <-- boom!
}

Suddenly the event handler explodes. Take a closer look at the handler’s registration:

window.PointerPressed([&](auto&&...)
{
    printf("%ls\n", m_value.c_str());
});

The lambda automatically captures the current object by reference. It’s as if you’d written this:

window.PointerPressed([this](auto&&...)
{
    printf("%ls\n", m_value.c_str());
});

Of course, that capture is just a raw pointer and knows nothing of reference-counting. We can capture a strong reference as follows:

window.PointerPressed([this, strong = get_strong()](auto&&...)
{
    printf("%ls\n", m_value.c_str());
});

And you might even want to exclude the automatic capture of the current object as follows:

window.PointerPressed([strong = get_strong()](auto&&...)
{
    printf("%ls\n", strong->m_value.c_str());
});

Notice that the variable is now accessed through the strong capture variable. Alternatively, you can also capture a weak reference:

window.PointerPressed([weak = get_weak()](auto&&...)
{
    if (auto strong = weak.get())
    {
        printf("%ls\n", strong->m_value.c_str());
    }
});

Some folks prefer member functions over lambdas. That works just as well, but of course the syntax for member functions is slightly different. Here’s the potentially dangerous member function handler using a raw object pointer:

struct App : implements<App, IInspectable>
{
    hstring m_value{ L"Hello world" };

    void Register(Window& window)
    {
        window.PointerPressed({ this, &App::Handler });
    }

    void Handler(IInspectable const&, int)
    {
        printf("%ls\n", m_value.c_str());
    }
};

This is just the standard or conventional way to refer to an object and a corresponding member function. Of course, there’s no use calling get_strong or get_weak from within the handler as we did previously with the coroutine. It may well be too late, as the app object may already have been destroyed by the time the event is raised and the handler is caller. Instead, the choice of weak or strong reference must be established at the point at which the handler is registered, and the app object is still known to be alive. Fortunately, the get_strong and get_weak functions can also be applied here. Consider again the event registration:

window.PointerPressed({ this, &App::Handler });

We can use the get_strong function in place of the raw this pointer as follows:

window.PointerPressed({ get_strong(), &App::Handler });

C++/WinRT ensures that the resulting delegate will hold a strong reference to the current object so that the handler can access any member variables without any concern. Similarly, the get_weak function may be used as follows:

window.PointerPressed({ get_weak(), &App::Handler });

In this case, C++/WinRT ensures that the resulting delegate holds a weak reference. This delegate will internally attempt to resolve it to a strong reference at the last minute and will only call the member function if a strong reference is acquired.

And that’s all I have for today. I trust you will find that C++/WinRT provides a great deal of flexibility when dealing with more complex lifetime management scenarios.

New in C++/WinRT: Async Cancellation Callback for Coroutines

Today I’d like to share another new feature in build 17709 of the Windows SDK that happens to be the most frequently requested feature for C++/WinRT’s coroutine support and that is the addition of a cancellation callback. WinRT has a very specific async pattern that provides cancellation support, but that support does not automatically flow to other async objects, as it does with some other async frameworks and libraries. The cancellation callback gives you an efficient mechanism to make this work and integrate or emulate these other async models.

Previously, cancellation within a coroutine was supported in two ways. The first is explicit:

IAsyncAction OneAsync()
{
    auto token = co_await get_cancellation_token();

    while (!token())
    {
        printf("Do some work for 1 second\n");
        co_await 1s;
    }
}

Waiting on the get_cancellation_token function returns a cancellation token with knowledge of the IAsyncAction that the coroutine is producing on your behalf. You can use a function call to query the cancellation state, essentially polling for cancellation. This makes sense if you are performing some compute-bound operation.

The second option is implicit:

IAsyncAction TwoAsync()
{
    while (true)
    {
        printf("Do some work for 1 second\n");
        co_await 1s;
    }
}

Given a co_await expression, the coroutine checks whether it has been cancelled prior to suspension and will short-circuit out of there rather than suspending. This works great, but if suspension occurs prior to cancellation then the coroutine will not actually come to an end until the nested co_await expression completes and the outer coroutine subsequently returns or hits a subsequent co_await expression.

These options also made it rather difficult to integrate with existing concurrency libraries as there was no preemptive hook by which cancellation might be propagated. No longer! There is now a third option that allows you to register a cancellation callback. Imagine you have a nested coroutine that does the actual work. Let’s use TwoAsync above for that. You can now write another coroutine that wraps it up and forwards cancellation preemptively as follows:

IAsyncAction ThirdAsync()
{
    auto token = co_await get_cancellation_token();
    auto nested = TwoAsync();

    token.callback([=]
    {
        nested.Cancel();
    });

    co_await nested;
}

Notice that the coroutine registers a lambda as the callback and then simply suspends and waits for the nested action to complete. There’s no need to poll for cancellation and the cancellation isn’t blocked indefinitely. Yay! Naturally, you can use this to interop with other coroutine or concurrency libraries that know nothing of C++/WinRT.

And that’s all for today. Stay tuned for more about the next major update to C++/WinRT and do give it a try today.