• Sitecore JavaScript Services

    Using the Sitecore JSS Manifest for Content Import

    Posted 03/15/2018 by techphoria414

    So it's been a year for me at Sitecore and nothing posted here, time for something a bit fun (and maybe useful?). One of the amazing things I've had the opportunity to contribute to at Sitecore is the revolutionary Sitecore JavaScript Services. If you don't know what I'm talking about, stop now, go read the docs, spin up one of the sample sites, run an import, then come back.

    One of the powerful aspects of Sitecore JSS is its support for code-first Sitecore development from JavaScript. You can define Sitecore IA (templates, renderings, etc) and content via the Manifest API then trigger an import via the JSS CLI. The manifest "definition" code generates a JSON-based manifest, which is placed in an Update Package, installed via Sitecore Ship (by default), and then run through the JSS import pipelines.

    For JSS, this approach is useful for Front-end developers who don't need to learn all the ins-and-outs of Sitecore in order to deploy a fully functional Sitecore site. But recently I was looking for a means to import some test content for another project, and realized its potential as a more general import tool as well.

    Normally a content import (especially a one-time import) is something I would use Sitecore PowerShell Extensions to accomplish. I still love SPE but for my purposes, the JSS Import had some advantages:

    • I could define the templates, in addition to the content items. There's currently no SPE commadlet for creating templates. (Though this may change based on some Slack discussions since then!)
    • The amount of code required to generate an item is much less.
    • I was planning to import from a JSON-based REST API from Setlist.fm. This means that in JavaScript, it's very easy to map/translate the data into the object shape expected by JSS.
    • By enabling the JSS full wipe mode, I could easily clean all imported templates and content on every incremental run of the import, as I added more fields, data, types, etc.
    • I've been doing way more JavaScript than PowerShell recently, so it just seemed easier. :)

    This approach does have some drawbacks:

    • The import currently expects to run in the context of a JSS app, so a JSS app needs to be configured and a JSS app item will be generated as the parent of the imported content.
    • The import process may create some unnecessary items, e.g. a root for the JSS app dictionary.
    • The JSS manifest does not have the full capabilities of the Sitecore data model. For example interlinking between items is difficult or impossible (at the time of writing).
    • The packaging and installation process does have some overhead. Installation via Ship became an issue for me due to the size of the Update Package. Once I scaled the amount of data coming in, I started getting HTTP timeouts from Ship, and had to install via the Update Installation Wizard.

    It still worked great for my test data purposes, and I could see other one-time imports benefiting as well. And it's always interesting to find new, novel uses for a tool you helped create.

    How It's Done

    Start with the JSS Quick Start to get one of the basic sample apps up and running. You'll need to follow the steps to get the JSS server and infrastructure components up and running. Then, if you like, strip it down. Get rid of all the react/angular code, existing *.sitecore.js manifest definitions, etc. Then create manifest definition files for your templates and your content. See my examples below. Enable full wipe mode if you want a clean import each time. Then use the JSS CLI to generate the manifest, create the update package, and install it.

    jss deploy package --includeContent --noFiles
    The includeContent flag is important to ensure your content items are imported. The noFiles flag is used to exclude any JSS code files from the package (we aren't deploying a JSS app, so we don't need them). If you just want to generate the update package, so you can install it via another means, you can just execute
    jss package --includeContent --noFiles

    The Code

    So again, this is a experimental/novel use of JSS, not a typical use case we had in mind when creating the manifest and import. But is it helpful/interesting to you? Have feedback? Let us know in #jss on the Sitecore Community Slack! -Nick

  • Black Art Revisited: Sitecore DataProvider/Import Hybrid with MongoDB

    Posted 09/11/2015 by techphoria414

    tl;dnr -- Combine data import and DataProvider approaches for great justice. Find the source on GitHub.

    My Black Art of Sitecore DataProviders article still gets a lot of hits, which to some degree makes sense -- creating a DataProvider hasn't really changed since Sitecore 6.0 (though I very much need to explore pipeline-based item providers in Sitecore 8). But if I've learned anything myself since then, it's that there are actually very limited circumstances in which you want to implement a DataProvider. Why's that?

    • They're difficult to implement
    • It's difficult to invalidate Sitecore caches when data is updated
    • It's difficult to trigger search index updates when data is updated
    • You can't enrich the content or metadata in Sitecore (e.g. Analytics attributes)
    • You are dependent at runtime on the source system
    • They're even more difficult to implement in a way that performs well

    So the fallback is usually an import via the Item API. But I've never really seen anyone happy with this either, especially for large imports. Why's that?

    • Writing data to Sitecore is SLOW. Even with a BulkUpdateContext.
    • Publishing can be very slow as well, though Sitecore 7.2 improved this significantly.
    • Usually you end up using a BulkUpdateContext, which means needing to trigger a search index rebuild, and potentially a links database rebuild, after every import.
    • Scheduled updates mean that it can be hours and hours before a data change in the source system is reflected in Sitecore.

    Is there a way to integrate data with Sitecore that balances the immediacy of a data provider, with the simplicity and enrichment ability of an import? Maybe! I'm presenting here a POC that combines the two approaches to try and achieve just that. We do this by introducing an intermediary data store that you may already have in your Sitecore 7.5 or 8 environment -- MongoDB.

    Data Provider / Import Hybrid

    The basic process:

    1. Data is pushed frequently from an external system into MongoDB. Writing data to MongoDB should be quick and easy, so it can be done very often, in theory. Maybe the data is already in MongoDB, in which case you are set.
    2. New items (products in this case) are imported frequently into Sitecore. This can be done quickly and more often, in theory, because we are implementing minimal data -- just creating the item, and populating a field with an external ID.
    3. We use a DataProvider with a simple implementation of GetItemFields to provide field data for the item directly from the MongoDB.
    4. To ensure caches are cleared and indexes are updated when data changes, we monitor the MongoDB oplog, a collection that MongoDB maintains to help synchronize data between replica sets.
    5. Content editors can enrich data on the item as needed. Externally managed fields can be denied Field Write to prevent futile edits.

    So, does this work? Glad you asked. I put together a POC and recorded a walkthrough, which you can find below. In the video, I go into more detail on import vs data provider, and some of the potential gotchas of the hybrid approach.

    Again, this is all theoretical. Has not been attempted in a production implementation. But I do think there is potential here, especially given that MongoDB is going to be found in more and more Sitecore environments going forward. Feedback is welcome, as are pull requests. :)

    Full source code can be found on GitHub.