Updating a Service Fabric cluster certificate using thumbprints involves multiple time-consuming cluster upgrades.  This can be avoided by upgrading the cluster to use common-name instead of thumbprints to reference the certificates.

Unlike thumbprints, common-name remains the same between certificate versions, therefore updating to a newer version is as simple as registering the certificate with the KeyVault & VMSS.

Because the cluster configuration does not need to change, there are no cluster upgrades required when updating certificates.  When the new, later expiring certificate has been registered on the VMSS, Service Fabric automatically uses the later expiring version.

Upgrading to Common-Name

As described in the documentation, the secondary thumbprint should be removed before configuring the cluster & VMSS for common-name:

If you have two thumbprint's declared in your template, you need to perform two deployments. The first deployment is done before following the steps in this article. The first deployment sets your thumbprint property in the template to the certificate being used and removes the thumbprintSecondary property. For the second deployment, follow the steps in this article.

The following ARM templates highlight the configuration changes that need to be deployed for the common-name upgrade.  Ensure a certificate with the common-name used in your template is either already installed in the VMSS, or is installed as part of this upgrade.

Cluster:

The following config should be removed from cluster > properties.

"certificate": {
  "thumbprint": "[parameters('certificateThumbprint')]",
  "x509StoreName": "[parameters('certificateStoreValue')]"
},

And replaced with:

"certificateCommonNames": {
  "commonNames": [
    {
      "certificateCommonName": "[parameters('certificateCommonName')]",
      "certificateIssuerThumbprint": ""
    }
  ],
  "x509StoreName": "[parameters('certificateStoreValue')]"
},

VMSS:

Replace the the certificate > thumbprint property with commonNames in virtualMachineProfile > extensionProfile > extensions > [type='ServiceFabricNode']. The result should appear as below:

"certificate": {
  "commonNames": [
    "[parameters('certificateCommonName')]"
  ],
  "x509StoreName": "[parameters('certificateStoreValue')]"
}

Once the above template modifications have been deployed, the cluster is configured to find certificates by common name. We can now easily automate the certificate update process.

Certificate update can be performed by adding the new certificate to the VMSS vault using your favourite Azure API.  The following PowerShell can be used to add the new certificate to an existing KeyVault, and add the certificate to the VMSS vault:

$subscriptionId  = "sub-id"
$vmssResourceGroupName     = "vmss-rg-name"
$vmssName                  = "vmss-name"
$vaultName                 = "kv-name"
$primaryCertName           = "kv-cert-name"
$certFilePath              = "...\.pfx"
$certPassword              = ConvertTo-SecureString -String "password" -AsPlainText -Force

# Sign in to your Azure account and select your subscription
Login-AzAccount -SubscriptionId $subscriptionId

# Update primary certificate within the Key Vault
$primary = Import-AzKeyVaultCertificate `
    -VaultName $vaultName `
    -Name $primaryCertName `
    -FilePath $certFilePath `
    -Password $certPassword

$certConfig = New-AzVmssVaultCertificateConfig -CertificateUrl $primary.SecretId -CertificateStore "My"

# Get VM scale set 
$vmss = Get-AzVmss -ResourceGroupName $vmssResourceGroupName -VMScaleSetName $vmssName

# Add new certificate version
$vmss.VirtualMachineProfile.OsProfile.Secrets[0].VaultCertificates.Add($certConfig)

# Update the VM scale set 
Update-AzVmss -ResourceGroupName $vmssResourceGroupName -Verbose `
    -Name $vmssName -VirtualMachineScaleSet $vmss

Service Fabric will now use the newer, later expiring certificate.  That's it!

Update

After using this process for a few months, the cluster & application heath has been fine.  It was observed however that error events were being raised on the nodes by Service Fabric: Failed to get private key file. x509FindValue: {commonName}, x509StoreName: My, findType: FindBySubjectName, Error E_FAIL

The following SO post is from a user with the same issue:

ServiceFabric standalone: Failed to get private key file

I have a standalone ServiceFabric cluster (3 nodes). I created SSL certificate for server and client authorization. Then I assign certificate thumbprint to a cluster config. Everything work okey( c...

Denis AzarovStack Overflow

Keep updated for a less manual mitigation for this error.

If you have a requirement to consume high frequency time-series data in your application, there are some excellent proprietary time-series database offerings that provide great features and performance.  For many reasons however, using a proprietary system isn't always the most appropriate option - in these cases you might consider building your own storage and retrieval solution.

This post details an efficient storage solution for ingesting and querying high-frequency time-series data. The example uses Service Fabric Actors and Azure CosmosDB, however the concept can easily be adapted to your own specific platform and storage system. Service Fabric Actors provide a turn-based access model, this means that actor state can be used as our cache without having to worry about locking to prevent concurrent writes, even across multiple VMs.

In summary, the solution is a write-back caching strategy that pre-aggregates and compresses time-series into fixed intervals. This reduces the volume of historic records, making historic queries much faster and storage cheaper, while allowing a longer recent history of data to remain in cache for fast access. This strategy is inspired by time-series databases and can easily be implemented using existing resources.

The solution is geared towards high-frequency telemetry, where the number of writes is high, and the reads are most commonly performed against recent history.  It's also adapted to suit append-heavy workloads, and while it supports updating older records, it may not be the right solution if updates are frequent.

Aggregation

Querying large volumes of high-frequency time-series data can be optimised by pre-aggregating data into larger intervals.  For example, querying against a few months worth of 1 minute interval data (at 1440 samples per day) will be less efficient than querying a few months worth of daily data.

The aggregation interval can be adjusted to suit your application, and should be small enough to reduce the number of updates on existing data, but large enough to significantly reduce the number of records being queried.

Compression

The compression algorithm used in this example was developed by Facebook for their Gorilla time-series database.  It takes advantage of the inherent repetitive properties of time-series data, storing the delta of deltas for the timestamp and storing only 'meaningful' bits for values, omitting leading & trailing zeros.  For regular-interval sampling, timestamps are reduced to single control bits that specify the interval is the same as the previous.

This algorithm is very fast and has a high compression ratio for numeric time-series data, however the approach outlined in this post can be implemented using any time-series compression algorithm.

There are many open-source implementations of gorilla compression, this .NET Core implementation of Gorilla compression has been tried and tested.

Write-Back Caching

Write-back cache describes a system that immediately caches new data and only moves to long-term storage after a period of time.  This works well for time-series data, as it's common for recent data to be accessed more frequently than historic data. Requests can be served using a combination of the cache and database, concatenating the data before sending a response.

Write-back caching works well with Gorilla compression, as we can quickly append new values to the latest period of compressed data.  The decreased size of the compressed time-series allows more of it to be stored in cache, meaning we can keep a longer period of recent history, leading to more cache hits.

A cache-miss would mean that the remaining data is queried from the database. This will not be as fast as serving data from the cache, however the aggregation of data into longer intervals means that database queries will run much faster for high-frequency time-series.

Service Fabric Actor & CosmosDB Implementation - Part 1

Service Fabric Actors keep state and logic closely coupled, this makes for a great in-memory cache. Compression reduces the memory footprint of cached data, and the time period that is cached can be shortened if required. This solution maps each distinct series to a single actor instance.

CosmosDB is a good database choice for scale-able volumes of data as it automatically partitions collections as the data volume grows, dependent on choosing an appropriate partition key.  For this solution, a hash of series id and year is a good partitioning scheme, as we'll be querying individual series from within series-specific actors, the addition of year to the partition key means that partitions will not indefinitely grow as time rolls on.

Thanks for reading, Part 2 will dive deeper into the implementation detail, including how updates are handled. Keep posted!

Overview

ASP.NET Core SignalR provides an abstraction over websocket connections, making it very easy to get up and running with "real-time" pub/sub functionality. When using SignalR within a Service Fabric application, it's likely there will be multiple stateless service instances that a client could have a websocket connection with.

This means that when any service in our application publishes events intended for clients, we need a mechanism to distribute these events across each SignalR service instance to ensure all subscribed clients receive the event.

Backplanes

The easiest method of scaling SignalR across multiple servers is to use a backplane. Backplanes broadcast messages to each connected SignalR service, making them unsuitable for some situations, with the potential to become a bottleneck in high traffic environments.

In scenarios that generate high-frequency or user-specific events, it would be a waste of resources to use a backplane that broadcasts every event to every instance of our SignalR service.

In these scenarios, publishers should only send messages to the appropriate connected clients that have subscribed to that particular topic.  To allow scalability & availability while avoiding the overhead of a backplane, we need to broker events only between publisher and subscribers.

Reliable Actors And Topics

Service Fabric Actors and Actor Events provide a ready-made pub/sub event system that can be used between Actors & services.  Importantly, Actor Events supports the publishing of events to specific stateless service instances.  This is very useful in scenarios where SignalR is hosted in a stateless ASP.NET Core service.

The Actor model aligns nicely with pub/sub scenarios, as each Actor instance can be used to represent a specific topic.  For example, we might be building a chat system, where each user is represented by an Actor instance.

Services can subscribe to specific topics by subscribing to events on a specific Actor instance.  This gives us the ability to publish targeted events to specific service instances, rather than using a backplane.

This also keeps our network calls within our Service Fabric cluster, negating the requirement for an external backplane resource with associated cost.

Implementation

Topic Client

In order to leverage Actor Events as a pub/sub system for SignalR, we need to orchestrate subscriptions and events between clients and Actors. We can wrap up this functionality into a TopicClient class, that can:

• Subscribe to Topic Actors and persist actor proxies
• Map subscribed client connection ids to the appropriate proxies
• Receive Topic Actor events & forward to relevant clients using IHubContext
• Unsubscribe from and remove unused Actor proxies

When a Topic Actor publishes an event, any connected TopicClient instances will receive the event.  The TopicClient uses the persisted connection ids and HubContext to forward the message on to the appropriate clients.

The TopicClient will not create multiple subscriptions to the same Actor, therefore multiple clients subscribed to the same topic, on the same service instance will be served by a shared actor proxy.

Topic Hub

This solution is designed for pub/sub scenarios, therefore we'll use a TopicHub base class to provide the common Subscribe, Unsubscribe and OnMessage functionality for a particular type of subscription.

All that's then required is to inherit this base class and specify the types used for each subscription, for example:

Full source, documentation and a working example can be found in this GitHub repository. There's also a Nuget package.

Trade Offs

It's worth noting that Actor Events are described in the documentation as only being "Best effort".  I presume this is down to the transient nature of Actors across nodes between upgrades and fail-overs.  I couldn't find any more detail on that distinction, however as this solution was designed to support some snazzy real-time UI features (nothing business-critical), this risk is acceptable.

Any Actor Proxies within a TopicClient will be lost if our SignalR service stops, however if this happens, client hub connections will also get disconnected and can then re-connect to another service instance which will subscribe to the appropriate Actors.

Event publishing services within our application are not aware of connected clients and will publish to the Topic Actor whether there are any subscribers or not.  This service remoting overhead is acceptable, given that after this point the Actor event will only propagate to Actor Event subscribers.

Summary 🏁

SignalR allows us to create websocket connections between client and server.  Within a Service Fabric application, services hosting SignalR can have multiple instances, however an individual client will only be connected to one instance.

Backplanes offer a quick and easy solution to replicate events across all service instances, however they can become a bottleneck when an application generates high-frequency, user-specific events.

Events can be directly brokered to only the appropriate SignalR service instances using Actor Events in combination with a TopicClient broker.

This solution is scalable and flexible enough to support many different event types that an application may generate.  It also leverages the existing Service Fabric cluster, rather than an external service with potential extra cost.

Full source, documentation and demo SF app can be found here.

While working on a feature-toggling system that allows users to switch preview features on and off, I wanted to create a featureAware enhancer-type HOC that I could use to wrap existing enhancers, e.g. withRouter & connect:

I also wanted this enhancer to work on it's own:

In this case the HOC is used to inject a hasFeature function into enhanced components:

This application already used Redux, and the selected feature-toggles were being persisted to the server.  I wanted feature-toggle updates to instantly show or hide features in the application.

Components using the featureAware enhancer HOC can use the hasFeature function to allow conditional feature-toggle based behaviour.

After a bit of work with types, taking inspiration from withRouter, I created the following HOC that can be used either on it's own or by wrapping other enhancers. It also allows pass-through of component-level props (like ownProps in Redux connect):

I briefly tried, but couldn't seem to avoid that ugly unknown props cast or the cast to React.Component. It works nicely however, the casts don't prevent the correct types from being inferred when using the component.

It is common for web applications to have many different endpoints/actions with different authorisation requirements. For example, the API behind this blog only allows users with a sufficient role to create new posts.

In ASP.NET Core applications, the framework provided [Authorize] attribute allows claims-based authorisation, however this doesn't take into account the resource being accessed. An application may have user-specific resources, be multi-tenant or have some other custom security model. In these cases - authorisation often needs to be granted based on resource-level access.

Imperative-style authorisation using IAuthorizationService can be used in these circumstances, however with this method we lose the descriptive, declarative simplicity of attribute-based authorisation.

This post provides an example of how we can create custom Action Filters and Attributes, to provide a flexible, efficient and descriptive method of resource authorisation.

Authorisation Rule Attributes

We can use attributes to decorate each endpoint with a rule that's specific to the request content and resource being accessed.

The authorisation rule can be simplified to the following interface:

We can then create an attribute class that implements the interface we have defined:

The example above uses the idArgumentName value from the constructor to extract the requested id from context.ActionArguments. It uses an  IItemLookup and IUserIdProvider service (both examples, not framework-derived!) from the service provider, to check if the current user has access to the resource with the id extracted from the request.

The ItemOwner attribute can then be used to decorate endpoints that require the user to own the resource they are accessing:

With the simple IAuthorisationRule interface, it's trivial to create multiple implementations of these attributes, each tailored to a specific resource security requirement.

Authorisation Filter

In it's current state, the example above offers no protection as it will not be run as part of the request pipeline. We can use Action Filters to intercept the request, consume our IAuthorisationRule attributes and check user access before allowing the request to proceed.

Create an Action Filter and register it globally to intercept every request to our application:

Register the filter in Startup

Now that this filter has been registered globally, our earlier example will now become operational:

The ResourceAuthorisationFilter will now run the HasAccess method on each of the IAuthorisationRule type attributes that the action is decorated with. With the current setup, any action not decorated with an IAuthorisationRule type attribute will not be impacted.

With this basic interface, we can easily extend our collection of rules to provide authorisation logic to check user access against new controller actions, with different resource access requirements. This solution provides a neat, reusable, declarative approach to resource-based authorisation.

Considerations

It's important to consider some of the drawbacks when using this declarative approach. We're accessing some information about the resource before the action body is hit, then it's likely that this resource will be accessed again within the action.

This disconnect between the authorisation filter and the action could be the source of security issues if the resources being accessed are different between the two. Security checks taking place in the authorisation filter must cover all request parameters that can subsequently be used to access resources. In any case - it's good practice to create endpoint integration tests that ensure the rules are correctly applied.

🏁 Summary

Resource-based authorisation in ASP.NET Core applications can be performed in a declarative manner using custom attributes and action filters.

Decorating endpoints/actions with attributes in this way greatly simplifies the implementation of authorisation, making it very easy to apply authorisation rules to new endpoints within the application.

Using a common IAuthorisationRule attribute interface alongside a global action filter gives developers the flexibility to create authorisation rules that are specific to the security model of the application.

Full source code and an example can be found in this github repo.