• Sitecore JavaScript Services

    Using the Sitecore JSS Manifest for Content Import

    Posted 03/15/2018 by techphoria414

    So it's been a year for me at Sitecore and nothing posted here, time for something a bit fun (and maybe useful?). One of the amazing things I've had the opportunity to contribute to at Sitecore is the revolutionary Sitecore JavaScript Services. If you don't know what I'm talking about, stop now, go read the docs, spin up one of the sample sites, run an import, then come back.

    One of the powerful aspects of Sitecore JSS is its support for code-first Sitecore development from JavaScript. You can define Sitecore IA (templates, renderings, etc) and content via the Manifest API then trigger an import via the JSS CLI. The manifest "definition" code generates a JSON-based manifest, which is placed in an Update Package, installed via Sitecore Ship (by default), and then run through the JSS import pipelines.

    For JSS, this approach is useful for Front-end developers who don't need to learn all the ins-and-outs of Sitecore in order to deploy a fully functional Sitecore site. But recently I was looking for a means to import some test content for another project, and realized its potential as a more general import tool as well.

    Normally a content import (especially a one-time import) is something I would use Sitecore PowerShell Extensions to accomplish. I still love SPE but for my purposes, the JSS Import had some advantages:

    • I could define the templates, in addition to the content items. There's currently no SPE commadlet for creating templates. (Though this may change based on some Slack discussions since then!)
    • The amount of code required to generate an item is much less.
    • I was planning to import from a JSON-based REST API from Setlist.fm. This means that in JavaScript, it's very easy to map/translate the data into the object shape expected by JSS.
    • By enabling the JSS full wipe mode, I could easily clean all imported templates and content on every incremental run of the import, as I added more fields, data, types, etc.
    • I've been doing way more JavaScript than PowerShell recently, so it just seemed easier. :)

    This approach does have some drawbacks:

    • The import currently expects to run in the context of a JSS app, so a JSS app needs to be configured and a JSS app item will be generated as the parent of the imported content.
    • The import process may create some unnecessary items, e.g. a root for the JSS app dictionary.
    • The JSS manifest does not have the full capabilities of the Sitecore data model. For example interlinking between items is difficult or impossible (at the time of writing).
    • The packaging and installation process does have some overhead. Installation via Ship became an issue for me due to the size of the Update Package. Once I scaled the amount of data coming in, I started getting HTTP timeouts from Ship, and had to install via the Update Installation Wizard.

    It still worked great for my test data purposes, and I could see other one-time imports benefiting as well. And it's always interesting to find new, novel uses for a tool you helped create.

    How It's Done

    Start with the JSS Quick Start to get one of the basic sample apps up and running. You'll need to follow the steps to get the JSS server and infrastructure components up and running. Then, if you like, strip it down. Get rid of all the react/angular code, existing *.sitecore.js manifest definitions, etc. Then create manifest definition files for your templates and your content. See my examples below. Enable full wipe mode if you want a clean import each time. Then use the JSS CLI to generate the manifest, create the update package, and install it.

    jss deploy package --includeContent --noFiles
    The includeContent flag is important to ensure your content items are imported. The noFiles flag is used to exclude any JSS code files from the package (we aren't deploying a JSS app, so we don't need them). If you just want to generate the update package, so you can install it via another means, you can just execute
    jss package --includeContent --noFiles

    The Code

    So again, this is a experimental/novel use of JSS, not a typical use case we had in mind when creating the manifest and import. But is it helpful/interesting to you? Have feedback? Let us know in #jss on the Sitecore Community Slack! -Nick

  • Sitecore MVP

    2017 Sitecore MVPs

    Posted 01/31/2017 by techphoria414

    Not much to say other than CONGRATS to all the 2017 MVPs! Happy to see my friend and former colleague Derek Dysart finally join the ranks. If you haven't been listening to Core Sampler, be sure to add it to your podcast app.

    This will by my sixth year as a Sitecore MVP. Thanks to the community for all your support, and thanks to Sitecore for the recognition.



  • Nominate more Women for Sitecore MVP

    Posted 11/01/2016 by techphoria414

    Hey Sitecore community! It’s our favorite time of the year: MVP Nomination! It’s your opportunity to help ensure that all the great contributors to the community are recognized for their efforts. If you know anyone who's not an MVP who you think deserves it, please take the time to fill out a nomination. If you know an existing MVP who has continued to serve the community, please re-nominate them as well to ensure continued MVP status!

    I want to take this chance though to ask you to nominate and re-nominate women of the Sitecore community in particular. The problems of monoculture, the advantages of diversity, and the challenges that technology companies have with gender diversity are well documented. I don’t think it’s a leap to argue that the Sitecore MVP program faces the same challenges. Diversity of people means diversity of ideas, and more female role models will only help to grow and strengthen the Sitecore community as a whole.

    There has certainly been progress since the conception of the MVP program. Cheers to all the talented women of the community who attended this year’s MVP Summit!

    But I believe we can do more. I’d ask everyone in the Sitecore community:

    • Are there women you know in the Sitecore community who should be nominated or re-nominated? Maybe a blog post that helped you out, or someone who helped you on Slack? Take the time and make the effort to bring more diversity into the MVP program.
    • Is everyone in your organization, regardless of gender, being given an opportunity to take the time and do the work needed to become an MVP?
    • Is your organization making efforts to hire more women in technology roles?

    The problem of gender diversity in technology is obviously bigger than our community. But we have a strong, tightknit and inclusive community that I have to believe rivals that of any other similar software platform. Let’s make it even stronger.


  • Rewriting Dates in Sitecore Analytics for Demo Data

    Posted 09/09/2016 by techphoria414

    Hello everyone! It's been awhile since I've blogged so I thought I'd post this useful bit before next week's MVP Summit and Symposium. Two plugs first though!

    Core Sampler

    If you haven't yet, be sure to check out the new Sitecore podcast from Derek Dysart, Core Sampler. Episode 1 is everyone's favorite Sitecore junkie, Mike Reynolds, and Episode 2 will be someone else very familiar...


    The Sitecore MVPs are leading an effort to help the Sitecore Community give back to the people of Louisiana, who are so graciously hosting us next week. Many many people not far from New Orleans have experienced horrific flooding recently, and are facing a long recovery.

    You can contribute to the effort through a financial donation or by volunteering the day before Symposium.

    And with that, let's get on to the post.

    Adjusting Dates in your xDB Demo Data

    In preparation for demoing Active Commerce at Symposium, we are busy creating JMeter scripts which simulate various user behaviors, conversions, traffic sources, etc. We are really hoping to show the power of the Sitecore marketing and analytics tools for e-commerce sites powered by our product. I won't get into the JMeter scripts here -- that's perhaps a topic for another day. But the result of running these scripts is a huge number of visits, but all on the same day. Not very exemplary of real site traffic. So I set about creating a PowerShell script that allowed us to "rewrite history" and shape the traffic over a given time period.

    The script can be found below. It uses a string of integers to represent the traffic shape, and updates the mongodb analytics data in place. You'll obviously need to rebuild your reporting database (and maybe your analytics indexes??) after running this. It is dependent on the Mdbc (MongoDB Cmdlets for PowerShell) Module.

    If you are looking to generate dummy or demo data for use with Sitecore Analytics, this may be useful for you. Enjoy!

  • Black Art Revisited: Sitecore DataProvider/Import Hybrid with MongoDB

    Posted 09/11/2015 by techphoria414

    tl;dnr -- Combine data import and DataProvider approaches for great justice. Find the source on GitHub.

    My Black Art of Sitecore DataProviders article still gets a lot of hits, which to some degree makes sense -- creating a DataProvider hasn't really changed since Sitecore 6.0 (though I very much need to explore pipeline-based item providers in Sitecore 8). But if I've learned anything myself since then, it's that there are actually very limited circumstances in which you want to implement a DataProvider. Why's that?

    • They're difficult to implement
    • It's difficult to invalidate Sitecore caches when data is updated
    • It's difficult to trigger search index updates when data is updated
    • You can't enrich the content or metadata in Sitecore (e.g. Analytics attributes)
    • You are dependent at runtime on the source system
    • They're even more difficult to implement in a way that performs well

    So the fallback is usually an import via the Item API. But I've never really seen anyone happy with this either, especially for large imports. Why's that?

    • Writing data to Sitecore is SLOW. Even with a BulkUpdateContext.
    • Publishing can be very slow as well, though Sitecore 7.2 improved this significantly.
    • Usually you end up using a BulkUpdateContext, which means needing to trigger a search index rebuild, and potentially a links database rebuild, after every import.
    • Scheduled updates mean that it can be hours and hours before a data change in the source system is reflected in Sitecore.

    Is there a way to integrate data with Sitecore that balances the immediacy of a data provider, with the simplicity and enrichment ability of an import? Maybe! I'm presenting here a POC that combines the two approaches to try and achieve just that. We do this by introducing an intermediary data store that you may already have in your Sitecore 7.5 or 8 environment -- MongoDB.

    Data Provider / Import Hybrid

    The basic process:

    1. Data is pushed frequently from an external system into MongoDB. Writing data to MongoDB should be quick and easy, so it can be done very often, in theory. Maybe the data is already in MongoDB, in which case you are set.
    2. New items (products in this case) are imported frequently into Sitecore. This can be done quickly and more often, in theory, because we are implementing minimal data -- just creating the item, and populating a field with an external ID.
    3. We use a DataProvider with a simple implementation of GetItemFields to provide field data for the item directly from the MongoDB.
    4. To ensure caches are cleared and indexes are updated when data changes, we monitor the MongoDB oplog, a collection that MongoDB maintains to help synchronize data between replica sets.
    5. Content editors can enrich data on the item as needed. Externally managed fields can be denied Field Write to prevent futile edits.

    So, does this work? Glad you asked. I put together a POC and recorded a walkthrough, which you can find below. In the video, I go into more detail on import vs data provider, and some of the potential gotchas of the hybrid approach.

    Again, this is all theoretical. Has not been attempted in a production implementation. But I do think there is potential here, especially given that MongoDB is going to be found in more and more Sitecore environments going forward. Feedback is welcome, as are pull requests. :)

    Full source code can be found on GitHub.

View more