• Sitecore JavaScript Services

    Using the Sitecore JSS Manifest for Content Import

    Posted 03/15/2018 by techphoria414

    So it's been a year for me at Sitecore and nothing posted here, time for something a bit fun (and maybe useful?). One of the amazing things I've had the opportunity to contribute to at Sitecore is the revolutionary Sitecore JavaScript Services. If you don't know what I'm talking about, stop now, go read the docs, spin up one of the sample sites, run an import, then come back.

    One of the powerful aspects of Sitecore JSS is its support for code-first Sitecore development from JavaScript. You can define Sitecore IA (templates, renderings, etc) and content via the Manifest API then trigger an import via the JSS CLI. The manifest "definition" code generates a JSON-based manifest, which is placed in an Update Package, installed via Sitecore Ship (by default), and then run through the JSS import pipelines.

    For JSS, this approach is useful for Front-end developers who don't need to learn all the ins-and-outs of Sitecore in order to deploy a fully functional Sitecore site. But recently I was looking for a means to import some test content for another project, and realized its potential as a more general import tool as well.

    Normally a content import (especially a one-time import) is something I would use Sitecore PowerShell Extensions to accomplish. I still love SPE but for my purposes, the JSS Import had some advantages:

    • I could define the templates, in addition to the content items. There's currently no SPE commadlet for creating templates. (Though this may change based on some Slack discussions since then!)
    • The amount of code required to generate an item is much less.
    • I was planning to import from a JSON-based REST API from Setlist.fm. This means that in JavaScript, it's very easy to map/translate the data into the object shape expected by JSS.
    • By enabling the JSS full wipe mode, I could easily clean all imported templates and content on every incremental run of the import, as I added more fields, data, types, etc.
    • I've been doing way more JavaScript than PowerShell recently, so it just seemed easier. :)

    This approach does have some drawbacks:

    • The import currently expects to run in the context of a JSS app, so a JSS app needs to be configured and a JSS app item will be generated as the parent of the imported content.
    • The import process may create some unnecessary items, e.g. a root for the JSS app dictionary.
    • The JSS manifest does not have the full capabilities of the Sitecore data model. For example interlinking between items is difficult or impossible (at the time of writing).
    • The packaging and installation process does have some overhead. Installation via Ship became an issue for me due to the size of the Update Package. Once I scaled the amount of data coming in, I started getting HTTP timeouts from Ship, and had to install via the Update Installation Wizard.

    It still worked great for my test data purposes, and I could see other one-time imports benefiting as well. And it's always interesting to find new, novel uses for a tool you helped create.

    How It's Done

    Start with the JSS Quick Start to get one of the basic sample apps up and running. You'll need to follow the steps to get the JSS server and infrastructure components up and running. Then, if you like, strip it down. Get rid of all the react/angular code, existing *.sitecore.js manifest definitions, etc. Then create manifest definition files for your templates and your content. See my examples below. Enable full wipe mode if you want a clean import each time. Then use the JSS CLI to generate the manifest, create the update package, and install it.

    jss deploy package --includeContent --noFiles
    The includeContent flag is important to ensure your content items are imported. The noFiles flag is used to exclude any JSS code files from the package (we aren't deploying a JSS app, so we don't need them). If you just want to generate the update package, so you can install it via another means, you can just execute
    jss package --includeContent --noFiles

    The Code

    So again, this is a experimental/novel use of JSS, not a typical use case we had in mind when creating the manifest and import. But is it helpful/interesting to you? Have feedback? Let us know in #jss on the Sitecore Community Slack! -Nick

  • Rewriting Dates in Sitecore Analytics for Demo Data

    Posted 09/09/2016 by techphoria414

    Hello everyone! It's been awhile since I've blogged so I thought I'd post this useful bit before next week's MVP Summit and Symposium. Two plugs first though!

    Core Sampler

    If you haven't yet, be sure to check out the new Sitecore podcast from Derek Dysart, Core Sampler. Episode 1 is everyone's favorite Sitecore junkie, Mike Reynolds, and Episode 2 will be someone else very familiar...


    The Sitecore MVPs are leading an effort to help the Sitecore Community give back to the people of Louisiana, who are so graciously hosting us next week. Many many people not far from New Orleans have experienced horrific flooding recently, and are facing a long recovery.

    You can contribute to the effort through a financial donation or by volunteering the day before Symposium.

    And with that, let's get on to the post.

    Adjusting Dates in your xDB Demo Data

    In preparation for demoing Active Commerce at Symposium, we are busy creating JMeter scripts which simulate various user behaviors, conversions, traffic sources, etc. We are really hoping to show the power of the Sitecore marketing and analytics tools for e-commerce sites powered by our product. I won't get into the JMeter scripts here -- that's perhaps a topic for another day. But the result of running these scripts is a huge number of visits, but all on the same day. Not very exemplary of real site traffic. So I set about creating a PowerShell script that allowed us to "rewrite history" and shape the traffic over a given time period.

    The script can be found below. It uses a string of integers to represent the traffic shape, and updates the mongodb analytics data in place. You'll obviously need to rebuild your reporting database (and maybe your analytics indexes??) after running this. It is dependent on the Mdbc (MongoDB Cmdlets for PowerShell) Module.

    If you are looking to generate dummy or demo data for use with Sitecore Analytics, this may be useful for you. Enjoy!

  • Black Art Revisited: Sitecore DataProvider/Import Hybrid with MongoDB

    Posted 09/11/2015 by techphoria414

    tl;dnr -- Combine data import and DataProvider approaches for great justice. Find the source on GitHub.

    My Black Art of Sitecore DataProviders article still gets a lot of hits, which to some degree makes sense -- creating a DataProvider hasn't really changed since Sitecore 6.0 (though I very much need to explore pipeline-based item providers in Sitecore 8). But if I've learned anything myself since then, it's that there are actually very limited circumstances in which you want to implement a DataProvider. Why's that?

    • They're difficult to implement
    • It's difficult to invalidate Sitecore caches when data is updated
    • It's difficult to trigger search index updates when data is updated
    • You can't enrich the content or metadata in Sitecore (e.g. Analytics attributes)
    • You are dependent at runtime on the source system
    • They're even more difficult to implement in a way that performs well

    So the fallback is usually an import via the Item API. But I've never really seen anyone happy with this either, especially for large imports. Why's that?

    • Writing data to Sitecore is SLOW. Even with a BulkUpdateContext.
    • Publishing can be very slow as well, though Sitecore 7.2 improved this significantly.
    • Usually you end up using a BulkUpdateContext, which means needing to trigger a search index rebuild, and potentially a links database rebuild, after every import.
    • Scheduled updates mean that it can be hours and hours before a data change in the source system is reflected in Sitecore.

    Is there a way to integrate data with Sitecore that balances the immediacy of a data provider, with the simplicity and enrichment ability of an import? Maybe! I'm presenting here a POC that combines the two approaches to try and achieve just that. We do this by introducing an intermediary data store that you may already have in your Sitecore 7.5 or 8 environment -- MongoDB.

    Data Provider / Import Hybrid

    The basic process:

    1. Data is pushed frequently from an external system into MongoDB. Writing data to MongoDB should be quick and easy, so it can be done very often, in theory. Maybe the data is already in MongoDB, in which case you are set.
    2. New items (products in this case) are imported frequently into Sitecore. This can be done quickly and more often, in theory, because we are implementing minimal data -- just creating the item, and populating a field with an external ID.
    3. We use a DataProvider with a simple implementation of GetItemFields to provide field data for the item directly from the MongoDB.
    4. To ensure caches are cleared and indexes are updated when data changes, we monitor the MongoDB oplog, a collection that MongoDB maintains to help synchronize data between replica sets.
    5. Content editors can enrich data on the item as needed. Externally managed fields can be denied Field Write to prevent futile edits.

    So, does this work? Glad you asked. I put together a POC and recorded a walkthrough, which you can find below. In the video, I go into more detail on import vs data provider, and some of the potential gotchas of the hybrid approach.

    Again, this is all theoretical. Has not been attempted in a production implementation. But I do think there is potential here, especially given that MongoDB is going to be found in more and more Sitecore environments going forward. Feedback is welcome, as are pull requests. :)

    Full source code can be found on GitHub.

  • Remove a Page from the Sitecore xDB (Sitecore 8 Technical Preview)

    Posted 10/01/2014 by techphoria414

    Note: Information in this post is based on the Sitecore 7.5 and 8 Technical Previews and is subject to change.

    Here's a quick one. I've been doing some JMeter traffic generation on my xDB for a forthcoming post/video on the Path Analyzer, and to push data from session to the xDB quickly, I implemented an "end session" page to hit at the end of each test thread.

    (I will hopefully also get a chance to share my work in JMeter, but in the mean time you should start where I started, with Martina Welander's awesome post.)

    Unfortunately the first time around I forgot to exclude my /EndSession.aspx from analytics, so it was muddying my data. However it was pretty easy to directly remove this from the xDB using a mongodb query.

    Basically, I'm telling mongodb to update documents in the the Interactions collection and remove elements from the Pages array where the URL path is /EndSession.aspx. The final "true" argument tells mongodb to update all documents which match the query (the empty first argument in this case), not just the first it finds. For more info on what's going on here, check the mongodb documentation on the update() method.

    After running this, I had to rebuild the sitecore_analytics_index using the Indexing Manager and rebuilt the reporting database using /sitecore/admin/RebuildReportingDB.aspx.

    This query was used on a Sitecore 8 xDB but based on the Sitecore 7.5 xDB Technical Preview, would work with 7.5 as well.

    - Nick / techphoria414 

    Read more... Pre-Disqus Comments (1)
  • Active Commerce

    Sitecore 8 Technical Preview: Active Commerce and the Experience Explorer

    Posted 09/26/2014 by techphoria414

    Today Sitecore released a Technical Preview of Sitecore 8 to the MVP community, and like good MVPs we are all scrambling to install the preview and write our first blog post on the beauty of Sitecore 8.

    Note: This article and video is based on a Technical Preview of Sitecore 8. Features and functionality are subject to change.

    Sitecore showed many amazing features of this forthcoming version during the Symposium events in Las Vegas and Barcelona. One "lesser" feature (really only in relation to the other amazing features) is the Experience Explorer. The basic idea is to simulate various visit and visitor segments in order to test personalization and other behaviors. Since Active Commerce utilizes the Sitecore Rule Engine for cart promotions, I was curious as to whether the Experience Explorer could be used to test cart discounts with Active Commerce. The answer was most definitely yes -- check it out below.

    It's worth noting that getting Active Commerce running on this Sitecore 8 Technical Preview took no code changes from the POC which I did on Sitecore 7.5. Neither did getting the Experience Explorer to work with our promotion engine. This is another great example of why we built Active Commerce natively within Sitecore, and why we say Active Commerce is "Sitecore e-commerce done right."

    It's worth noting that there is an Experience Explorer Module available for earlier versions of Sitecore as well.

    Great work Sitecore. Can't wait for Sitecore 8 to go gold.

    - Nick / techphoria414

View more