Rise of the Unicorn Transformers

Rise of the Unicorn Transformers

Before time began, there was…the Cube. We know not where it comes from, only that it holds the power to create worlds and fill them with life. That is how our race was born. For a time we lived in harmony, but like all great power, some wanted it for good…others for evil.

Wait a minute. That’s not it.

Meet something brand new however, please welcome a new addition to the Unicorn and Rainbow Family: Field Transformers. Well Field Transforms, I get carried away.

Here’s the TL;DR. Sometimes a config snippet speaks louder than a thousand words.

What is it?

It’s like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
<!--
PREDICATE AND INCLUDE FIELD TRANSFORMS
- These transforms ONLY APPLY to what gets deserialized into Sitecore. Field values on disk/serialized datastore remain complete - respecting only the Rainbow field filter settings
IN OTHER WORDS:
Example: "-title, -text, +ApiUrl[{$apiUrlToken$}]" => All fields get deployed as normal, except "title" and "text". And "ApiUrl" gets a forced value.
Example: "!title, -text" => "Title" gets reset (to standard value), "Text" gets ignored.
Example: "?title, ?text" => ALL fields gets deployed, but fields "Title" and "Text" will only get their value deployed if target field value is empty
Example: "?title, ?text, !apiUrl" => As above, but field "apiUrl" is reset
"!field" => Reset this field unconditionally
"-field" => Ignore this field unconditionally
"?field" => Deploy this field if it has no value in target data store (Sitecore)
"+field[value]" => Force a new value into this field on target data store (Sitecore)
";field" => Force a "Lorem ipsum dolor" value into the field
":field" => Force a longer Lorem ipsum based HTML string into the field (around 2075 characters of Lorem Ipsum, broken up with <p> into sentences).
"$field[settingName]" => Grab the value of the Sitecore Setting `settingName` and force it as a value
-->
<predicate fieldTransforms=";Title,:Text,!Include In Sitemap,+Api Endpoint[{$apiEndPoint$}],?Default Product">
<include name="Sample Data" database="master" path="/sitecore/content/global/sample" fieldTransforms="-Title" /> /* Predicate transforms apply, but "Title" gets ignored on this include definition */
</predicate>
<!-- Clear Workflow fields on local development environments, but force a specific state upstream -->
<predicate>
<include role:require="Standalone" name="Sample Data" database="master" path="/sitecore/content/global/sample" fieldTransforms="!Workflow,!Workflow State" />
<include role:require="ContentManagement" name="Sample Data" database="master" path="/sitecore/content/global/sample" fieldTransforms="+Workflow[{2DE02B52-B95F-404A-A955-C36B290F1B57}],+Workflow State[{5ACE9C7F-8A18-4C77-BC30-03BE5A40E6B6}]"/>
</predicate>

But what is it?

It’s a quality of life enhancemment. While I do not encourage it, I’ve often been met with questions like these:

  • “How can I prevent just this field being deployed?”
  • “How can I make sure this configuration value doesn’t get reset to development settings on each sync?”
  • “How can I make sure this field value doesn’t overwrite what the Content Editors entered?”

If you’ve been asked (or asked) any of these yourself, this is for you.

The intended use-case(s)

  • Keeping certain fields under Source Control, but not letting them “bleed” across to other developers. For instance “Workflow” and “Workflow State”.
  • When deploying content (example content or otherwise) to an upstream environment, an easy way to ensure a specific workflow and state is set/reset on the content items.
  • When deploying, make sure certain field values are only updated if they do not hold active content on the target environment.
  • A way to inject configuration values into Sitecore field values, for use in CI and CD pipelines.

And then you tell me. There are likely tonnes of ways people would want to use this feature, more than I’ve considered. I’ve just provided some new tools - I’ll leave it to you, how to best use them.

Cool. So how do I use it?

Your <predicate> and <include> nodes now accept a new optional attribute; fieldTransforms. I wanted to keep the configuration very light weight and reasonably intuitive, so that’s all you get. At least for now. If the feature takes off, I’ll look into making something more plug-able and expandable.

Field Transforms only affect what gets written into Sitecore. All other operations remain the same - they do not affect what gets written to disk, they do not affect FieldFilter definitions. This is simply a value-replacer on the pipeline between Filesystem and Sitecore.

Example 1

1
2
3
<predicate fieldTransforms="-Title,-Text">
<include name="Content" database="master" path="/sitecore/content/home" />
</predicate>

For all <include>, never deploy the fields Title and Text to Sitecore. Well technically to the sourceDataStore, but this will be Sitecore for 99.99% of you.

Example 2

1
2
3
<predicate fieldTransforms="-Title,-Text">
<include name="Content" database="master" path="/sitecore/content/home" fieldTransforms="+Title[My Forced Title Value]" />
</predicate>

As Example 1, but the "Content" <include> overrides the <predicate> and forces a value "My Forced Field Value" into the Title field.

Example 3

1
2
3
<predicate>
<include name="Content" database="master" path="/sitecore/content/home" fieldTransforms="+Title[My Forced Title Value]" />
</predicate>

As Example 2, but this time there is no <predicate> field transform. All fields will operate as normal, except Title which will get a forced value of "My Forced Title Value".

Nice. What field transforms are available?

Bit of a mixed pot, really.

  • -: Never deploy this field
  • ?: Only deploy this field if there is no value (can be standard value however) in the target field
  • !: Always reset this field (to standard value)

These are the 3 I probably see being most useful. But it doesn’t stop here.

  • ;: Forces the value ""Lorem ipsum dolor"" into the field
  • :: Forces a much longer (2075-ish characters) Lorem Ipsum string into the field (with <p> tags)
  • +[val]: Forces "val" into the field. E.g. +Title[Optimus Prime] will force the value "Optimus Prime" into the Title field.
  • $[setting]: Forces the value of Sitecore.Configuration.Settings.GetSetting("setting") into the field.

Intended use-case for +[val] could be for your build pipelines; e.g. +My Super Service EndPoint[${superServiceEndpoint}] wil force the value of ${superServiceEndpoint} into the field. Assumption being that you’ve run something like a token replacer in your build pipeline and replaced the actual value in the Unicorn configuration file.

Caveats, and gotchas

So here’s the tool. How you use it, is up to you. But be aware of some (perhaps obivous) caveats.

  • It won’t work with Transparent Sync. Transparent Sync never actually writes anything to Sitecore and therefore won’t ever pass by these Field Transforms.
  • If you force a field value (e.g. with + or $) on a versioned field - the field value will get forced onto all versions of the field. All versions, all languages.
  • It does work with Dilithium
  • Remember; it Field Transforms ONLY apply to field values being written to Sitecore. Everything serializes as before - yaml files are not affected by these transforms.

Can I get it now?

Yes. Yes you can. At the time of this writing, Rainbow 2.1.0-pre1 and Unicorn 4.1.0-pre1 have been pushed to NuGet. I still consider it a pre-release until a few more people have had a chance to take it out for a spin.

Please join our Slack Community at https://sitecore.chat/ and join the #unicorn channel if you’re taking this out for a spin. Would love to hear your thoughts and get your feedback.

Share

It's time to put fast:/ query to rest

It’s time to put fast:/ query to rest

It’s been 8 good years. Well it’s been 8 years. It’s time - it’s way overdue in fact - that fast:/ is retired. I’m getting tired of debating and explaining the same thing over and over again on our Community Slack channels, so I figured I would write this post so as to - once and for all - try and rid the Sitecore community of this fast:/ pestilence.

It’s going to be a semi-long post. I apologise. Read it anyway.

TL;DWR: Stop using fast:/ query.

The history

Sitecore Information Architecture Anno 2005-2012ish

So I’ve been around long enough to remember, what fast:/ was introduced to solve. Join me on a trip down memory lane to the days of Sitecore 5 and 6.

Back then, the way we built Sitecore solutions was very different than what we see today. We were building page templates. Datasources was not commonly used and understood so we all, more or less, built our Sitecore sites in a way that every single field on a page would be represented on the page item. I know right? Unthinkable today. But we did. So we did use inheritance, we micro-managed template inheritance and often ended up with field names such as Section Subheading Bottom Right and Product Spot 3 Inner Subheading. I know, right?

Ignore for now the nightmare it was to refactor an Information Architecture such as this (which led to the also-obsolete practice of always addressing fields by their ID instead of their name - back then we had to change names quite often for the whole thing to make any sense to our content editor users). Anyway - ignore all this. We’ve all moved on, but this is how most of us did Sitecore Information Architecture from 2005 to somewhere around 2012/2013 or so. I know, because I vented about it in a blog series in 2013 called The Page Template Mistake.

The Content Editor Performance Problem

Other problems aside, there was a massive performance drawback from approaching Sitecore IA like this. And it came from the only editor available to us at the time - the good old trusted Sitecore Content Editor which stands, to this day, pretty much as it did back then. And then we had these Page Templates. 50 or so fields was not uncommon at all. And not just simple fields mind you, but Droplists, Droplinks, TreeLists and so on. All of them being populated by the source field of their respective field definition items on their respective templates.

And now we’re getting to it; the Content Editor simply could not cope. Switching between an item meant, Sitecore had to go execute all of these queries to fill the Droplists, TreeLists and so on, and both CPU and memory resources were a lot more limited at the time. Sitecore’s caches hadn’t quite evolved to what they are today, CPUs were single or dual-core (Core2 Duo was the latest and greatest). It just wasn’t really handling the job very well.

We needed a quicker and faster (pun sort of implied) way of being able to query these source definitions to fill our Droplists and TreeLists, and that should be easy enough right?

Since we only need the ID and the Item.Name in order for an item to be populated into one of these lists - surely a shortcut could be found. And a shortcut was found. It took a lot of shortcuts (since it needed only the above; the ID and the Name), scaled horribly (but that was alright, this was just to improve the Content Editing experience of a single CM user), ignored Versioning, Language, maybe even Workflows (I forget) - again, all of this was an acceptable trade-off for a faster Content Editing experience.

And thus, Sitecore fast:/ query was born. To my knowledge with Sitecore 6.2, but could have been slightly sooner than that. I can’t find the original blog posts that discussed all of this, nor the Sitecore Forum posts.

Here is the original documentation cookbook: Using Sitecore Fast Query

The Limitations of Sitecore fast:/ query

Bypassing Sitecore’s Data Provider Architecture

So keeping in mind the above; fast:/ was designed to solve a single-user performance problem when the Content Editor had lots and lots of Droplists and TreeLists and so on to populate. I’ll extract some basic facts about fast:/ - all of which can be found in the cookbook documentation referenced above.

Under the heading: “Sitecore Fast Query has the following benefits compared to the standard Sitecore Query:”

Improved performance — queries are executed by the SQL engine and as a result the scalability and performance of the SQL engine is not limited by.NET or by Sitecore.

Let me translate that for you. “We’re pushing the load away from the web server and down onto the SQL server, so only the SQL server performance affects how fast your fast:/ query performs. Not the lack of caching in the Sitecore Content Editor, not the memory restraints of the web server”.

In other words; SQL now becomes your only bottleneck. But that’s not a problem right? Is just a single user editing some content, they don’t switch between items THAT often.

Consumes less memory — Sitecore Query loads every item that it touches into memory (cache) and this can fill the cache with unnecessary information. Sitecore Fast Query only loads the items from the result set and this minimizes the pressure on the cache.

Indeed. Using fast:/ bypasses such resource hogs as caches. Data caches, item caches. Because who needs them? It’s just a Content Editor user switching between an item once in a while, and since we’re constantly editing the content - caches don’t make any sense anyway.

Fundamentally, fast:/ works like this:

Sitecore Fast Query is translated to SQL queries that be executed by the database engine.

So it takes the query format and converts it into an SQL SELECT statement. Slightly simplified, but close enough. Cool right? Yea except this means bypassing the entire DataProvider architecture model of Sitecore, Item Cache, Data Cache and whatever else is required in a scalable solution then and today.

Sitecore Cache Architecture Overview

But that’s ok because it’s just a single Content Editor user switching between items once in a while. Right?

Because of this SQL conversion, only certain attributes are supported. And any complex operations that involve field values of any kind - very quickly degenerate into SQL queries so terrible your SQL Performance Profiler will have nightmares about them for years to come.

There are further limitations not called out in the original document.

  • No support for Language Fallback
  • In fact no Language support at all
  • Sort Order not respected
  • Versions not respected
  • Add more here if you like, the blog accepts PRs

The Scalability Problems of Sitecore fast:/ Query

So fast:/ was never designed to scale. It remains as I have stated numerous times already, a technology that was meant to be utilised by a single Content Editor user switching between items in the Content Editor to (d’uh) edit content.

So what do you think happens, when you put fast:/ queries in your reqular runtime code that executes your website? Remember what I quoted above?

queries are executed by the SQL engine and as a result the scalability and performance of the SQL engine is not limited by.NET or by Sitecore

And here’s the kicker. Almost every blog I’ve ever read, that in any way deals with query performance (including some of these from my References section below) - only measure performance of various query types in a single-user environment. And if there is one thing you cannot do with fast:/ query, is get a handle on it’s performance in that kind of setup. Your live website will be running 50? 100? 200? 400? 800? concurrent sessions. And not only that; this will not be users “switching from item to item once in a while, while editing content”. No they will click and click and mercilessly request new pages, new content, all the time. And expect to get it instantly, too.

So I ask of you this. A technology introduced to improve the Content Editor performance of a single user switching items from time to time while editing content; now being used say… twice per component on your page, 15 or so average components per page, 800 concurrent sessions each requesting a new page every 10 seconds on average. 2400 fast:/ queries per second, bypassing any and all caching and going straight to your SQL server as inline SQL (no stored procedures, no prepared SQL Execution Plan) - 2400 of those per second - how well do you think that is going to work out?

This is NOT what fast:/ was designed to do. Not ever. Neither was your SQL server, for that matter.

And if you don’t want to believe me, ask your SQL server. Also don’t forget to include a mention for your hatred for Item Cache and Data Cache when you submit your next request to your CTO or whoever, for an upgrade in your Azure storage tier. MOAR SSDs right? Solves any problem.

Sitecore fast:/ does not scale. Be VERY aware of this. Even when you’re “performance testing” on your local machine, it might actually come out looking “allright” performance wise. But it just isn’t “allright”. Never. You have to believe me; 13 years I’ve been doing this (Sitecore stuff) - never once have I seen a benefit from a fast:/ query. Nor have I ever ever used one myself; which at least goes to argue that they by no means are essential.

But Then What?

You know what. Since Sitecore 7.0, Sitecore ContentSearch has been built-in to the product. There is nothing you can fast:/ query that you cannot also query using Sitecore ContentSearch - only you’ll get your results about 10 times quicker (notice how I didn’t say faster…) that way. And it scales.

Ignore made-up problems

Ignore made-up problems such as:

  • “I need a real-time view of my data for this operation”. You don’t - this is an architecture problem/fail.
  • “I really need a fast way to find all items of template XXX under this subtree”. Yes. ContentSearch them. And make a better Sitecore Information Architecture.
  • “Calling up an index to get just a few items is overkill”. It isn’t. It just isn’t. It is by many orders of magnitude quicker than selecting just 1 field from 1 row in your SQL server, bypassing all caches.
  • And if you insist, use the LinkDatabase. Most fast:/ I see anyway, is “Get me all items of template XXX at this location”, and LinkDatabase outperforms fast:/ by 10x or more for this operation.
  • “I don’t want to update my local indexes all the time”. Then why do it? I don’t. My indexes update normally, so do yours. It’s only when you’re changing index configuration such as adding new computed fields and so on that a full rebuild of your index would be required.

I Don’t Believe You

Well then don’t. I have no problem being called on-site as a “Super Hotshot Sitecore Performance Troubleshooter”, fixing a few of your broken fast:/ queries and billing your boss a months salary for it. We all got to make a living somehow.

That said; I also want to put proof behind all of this. Thing is - to well and truly set up a test that measures the real impact of fast:/ query under load, is NOT as simple as it may seem. I will try and see if I can get one of Sitecore’s hosting provider partners to help set up a test rig, fill it with appropriate content, and then query the night away. Send me your favourite fast:/ queries if you like, I’ll be happy to include them in the test.

Any volunteers, feel free to reach out. Otherwise I’ll come knocking.

And Help Spread The Word

fast:/ is like the pestilence that just won’t go away. It is so deeply ingrained in the consciousness of many Sitecore Developers (and even - sigh - Sitecore Trainers) and it comes with a flashy fast:/ prefix. Must mean it’s… fast, right? Wrong.

Help spread the word by retweeting this post; add a few words of your own. If nothing else, just do something like “I’m name and I approve of this message”. Or whatever. But let’s help each other out, yea? :-)

I will embed your tweets below.

References

Tweets

Share

How I get the most from Sitecore Symposium (and other events)

How I get the most from Sitecore Symposium

(and other Sitecore events)

In case you’re living under a rock and haven’t noticed yet, Sitecore Symposium 2018 is just around the corner. This year it’s all happening in Orlando, FL.

If you take a look at the published Agenda you will find a near endless wall of back to back sessions, starting from 8 in the morning (7 if you include breakfast) and continuing on until 6 in the evening for the two main event days (the entire event runs over 4 days). And then, of course, there’s the opening reception, the dinner event, the pre-conference seminars. For some of us, there’s even the yearly Sitecore MVP Summit, extending the event throughout the entire week.

Sessions - Sessions everywhere

So how to make the most of all that?

Filter.

I know I should be telling you, print out the agenda, mark the must-see sessions, fill out the rest of the slots with sessions that catches your eye. You (or your employer) paid good money to be here, you best make the very most of your time in Orlando.

But I won’t.

Because I don’t believe this is the way to get the most from an event like this.

Don’t get me wrong, the sessions are all great. I am presenting myself, and I know the amount of work and sweat and tears that go into preparing to speak at an event like this. And I would of course love it, if my session room was filled to the brim with enthusiastic and eager minds coming to listen to what I have to say. I am currently scheduled to speak Thursday, 9.30 am, for those so inclined.

But back to back sessions, from 8 in the morning until the early evening - followed by various social events and socialising? I think not.

I say, take it one step back. Go ahead, print the agenda. Mark the must-see sessions that apply to you. There’ll probably even be an App. But leave it at that. Go where the flow takes you for the rest. Maybe you’ve made some new friends over by the water cooler and they’re all going to see Kam Figy @kamsar showcase the latest and greatest stuff you can do with JSS. Maybe that wasn’t what you planned when you looked at the printed agenda the week before. But join them anyway.

What happens at these events then, if not sessions?

I’ve been around a few years. Doing Sitecore. I’ve been to Symposiums, SUGCONs, User Group events, pretty much all over the world. And I’ll tell you, honestly, the very best experiences I’ve had on these events had nothing to do with the sessions presented. I’ll share one.

Last year at SUGCON in Berlin, I was “over stimulated” by the sessions, by probably staying up a little later than I should the night before, by the crowd, by the sessions, by all the inputs. Everything. So I took a breather and skipped a session (more than once), just hanging out in the lounge area. I bump into Adam Najmanowicz @adamnaj who was - probably - feeling something similar, and we hit up a chat. I spent the next hour debating back and forth with him, the merits and problems of the Sitecore Experience Accellerator (SXA) and had a chance to voice my unique perspective on it all. And he listened. Anyone who’s ever met Adam will know this, he listens. And learns. And provides insight.

Look. My point is not “don’t attend the sessions and you might meet Adam” (although I do recommend meeting him). My point is, something like this will never make it to any official agenda. You can’t plan for this. But if the opportunity presents itself, grab it.

Because the best thing is not on the agenda at all

What’s that you say? I’m saying that what makes these events so special, is the people attending. You. Me. Everyone. To use a term that has been over-used, washed, hung up to dry, washed again - “Networking”.

The best thing about these events. The Sitecore Community.

We don’t get many chances to all come together at the same place at the same time. But the Sitecore Symposium is one such event. Don’t let that opportunity go past you without you noticing. Don’t come back home, being asked “How was it?” and go “Yea was alright. Saw a lot of new stuff”. Come back, smile on your face, and either just stay silent or respond “I made lots of valuable new friends”. And if your boss asks, just tell him or her “I learned lots about the implications of JSS and SXA on the modern CMS world and how it influences the Sitecore 9 roadmap”. You can quote me on that ;-)

I hope to see you in Orlando.

Share

Clouds, Unicorns and Rainbows

Clouds, Unicorns, and Rainbows

Before you say it; “Doesn’t this blog look familiar somehow?”. I know. Ok? I know :-)

While my Blogger based blog “Into The Core” and I go way back, like back to 2005 back, it has not always been a relationship of pure love and merry co-existence.

Blogger has, as you would expect from any service that has been around for almost 20 years, undergone a number of changes over the years. And ever so often this has had subtle yet annoying influence on how my blog there presented itself. Paragraph spacing would sometimes change, leaving my blog posts looking very congested, and forcing me to go back to all of them and redo their formatting. Not cool.

Example of a condensed post I did not bother reformatting

The problem at the core of all this is, of course, that the underlying content in Blogger is stored as HTML. It became clear to me that I wanted to shift onto a blogging platform that was Markdown based. I also wanted full source level control of the content, and I wanted an easy way to move the platform around when - 15 years from now - things will have changed up once again.

Long story short; Kamsar’s post on Hexo made a lot of sense and inspired me to take a closer look at Hexo myself. And here we are. That we also ended up using the same theme (Icarus) for the blog is more due to the fact that I feel it’s the best theme available for Hexo right now, and (probably moreso) that I’m lazy.

The new name?

A generation has passed (ok, an Internet generation perhaps) since I started blogging. The whole IT landscape has changed, not once but many times over. And in this current day and age I guess it should come as no surprise, it’s all about The Cloud. And when it comes to Sitecore this primarily revolves around Azure, PaaS, and everything that comes with it.

All of this, much as I hate to admit it, has sort of rebooted a large part of what I used to “know” and take for granted. When it comes to deploying a multi-server Sitecore solution to Azure that scales across the globe - I’m as much on page 1 as everyone else. Maybe page 2, but you get my meaning.

As I form experience and in the event I feel I can add something valuable to the ongoing conversation about Sitecore in the Cloud, I feel this new blog and platform is a more appropriate base for future blog posts.

But why now?

I don’t blog a whole lot any more. I’ve found other ways to put my knowledge and experience to use, and where there used to be only a handful of us back in 2005 there are now a literal forest of active bloggers out there writing and sharing - which is excellent, of course :-) But these days I only blog when I have something on my mind that I feel is not getting covered (enough) elsewhere. Otherwise I dedicate my time on:

  1. The Sitecore Stack Exchange - of which I am a co-founder and co-moderator.
  2. The Sitecore Slack Community - Where thousands of active Sitecore community members hang out daily.
    1. Read more about our community Slack in Jammykam’s excellent introduction Sitecore Slack Community Guidelines & Help
    2. Sign up for Sitecore Slack here https://www.bit.ly/SitecoreSlack
  3. And thirdly, the main reason I can no longer postpone having a blogging platform I can live with:

Unicorn & Rainbow

As you may or may not know, Kam Figy - author and creator of many beloved open source tools for the Sitecore platform, recently started working for the Mothership where he is now focused on building cool things that will eventually make all of us back-end developers unemployed ;-)

But cool things such as JSS, a whole new career focus, and the fact that time may very well be relative but not so much for us mere Earthlings, has left Kam pressed for time. He asked me, and I accepted, to become a co-pilot on the Unicorn and Rainbow projects.

Co-pilot, mind you. Kam is by no means gone from these projects and I bear no illusion that I could just jump onboard and start filling out those shoes entirely on my own. But it means that I am doing most of the day-to-day and, over time, will be adding more of my own footprint to these projects. I have had nothing but love for Unicorn since I first came across it in 2014.

And Unicorn continues to be my number 1 choice on Sitecore solutions to this day. Now - almost 4 years later - Unicorn has grown from being relatively obscure to becoming a household name in Sitecore solutions across the world. Thousands and thousands of downloads.

I still remember exactly why I fell in love with Unicorn, and why that is still the case to this day. While co-piloting this project, I will keep staying true to that legacy and still work to find ways to improve and better the project for all of us to enjoy.

I certainly have my work cut out for me :-)

Share