Advanced Spatial Search with Sitecore and SOLR

I recently did an in-depth API utilizing the spatial search that is provided by SOLR, and in a previous post I go into this a little bit when creating a custom index for storing location data. That said, I wanted to go more in depth on the code side of things, to show the power of the spatial search feature! This is such an amazing feature, and having recently done something with it, I wanted to share my knowledge around it and expand on some of the things not mentioned in the Sitecore documentation.

Building a custom SOLR index in Sitecore

The task involves creating a dedicated Sitecore Index for specific location items using SOLR. Key requirements include indexing relevant fields, configuring custom settings, and validating the setup post-implementation.

by Damien Rincon

Overview

This task will cover more of the code side of things as I mentioned, and will be expanding on the Sitecore documentation.

I will assume the following:

You have read over the Sitecore documentation for spatial API
Have your index already configured to accept coordinate fields – with a custom or OOB field
Have some sort of Google Map to output this data into from the API
You are using Sitecore 9.3 and not XMC (but I will try to add in info for it)
Know your namespaces, I will use YourProject.Features.Search if I need to reference it.

Getting Started

To get started, let’s start with a sample object, as examples help me understand personally. We’ll keep it simple with simply the coordinate field, radius, and a location name. No need to overcomplicate for the example.

      public class SampleLocationBaseItem 
      {
            /// <summary>
            /// Location Pin as Coordinates
            /// </summary>
            public Coordinate LocationPin { get; set; }

            /// <summary>
            /// Base Radius for the location
            /// </summary>
            public float Radius { get; set; }

            /// <summary>
            /// Name of the location
            /// </summary>
            public string LocationName { get; set; }
      }

Note: I personally prefer a base class if I know this could be utilized for other things, so the base for me would be things I know are constants. For each item there will always be a set of coords, radius, and name. Say I build an object for map pins with specific needs, I can then inherit this base, and it is clear this is the base when simply reading the file name.

in theory, you could also store both latitude and longitude separately and then cast them to an instance of the Coordinate object if you have pre-existing fields.

Set up some sort of search object as well, again, nothing crazy, but will be used in our LINQ query to get fields from SOLR.

      public class SampleLocationSearchItem : SearchResultItem
      {
            /// <summary>
            /// Location Pin as Coordinates
            /// </summary>
            [IndexField("location_geo_coords")]
            public Coordinate LocationGeoPin { get; set; }

            /// <summary>
            /// Base Radius for the location
            /// </summary>
            [IndexField("location_geo_rad")]
            public float LocationRadius { get; set; }

            /// <summary>
            /// Name of the location
            /// </summary>
            [IndexField("location_display_name")]
            public string LocationName { get; set; }

            /// <summary>
            /// friendly URL for the location item
            /// </summary>
            [IndexField("location_url_friendly")]
            public string LocationUrl { get; set; }
      }

Here I have mapped out the IndexField on all, matching the SOLR field names that you can assume I made when adding computed fields to my index.

Awesome, we now have an object we will map to, and one for use in our search query. Let’s build out a super simple API with a controller and service.

Building the service

While the next 2 steps really are not necessary, they are here to help illustrate practical use of the spatial service. I find this helpful, however if you don’t need it, you can skip ahead and as I will be doing my best to make it readable without the context here.

Let’s start with a service, this will be where our logic will be. We will also be making an interface for this, and registering the service to our service registry.

Because of how I think, I like to have some clear goals to hit the bare minimum of a piece of functionality. It’s always good, at least in my mind, to have even the smallest of checklists to verify you are meeting the requirements. of base functionality.

The tasks for this section are:

Built out a service that takes in a series of params for latitude, longitude, and radius, and give you the locations in this area
Build out an interface for this service for dependency injection
Write some error handling just in case
Map the data from the query to our custom object
Register the service if needed

So now that we know the basic tasks for creating our service, let’s start scaffolding out something and then also explore just a couple options with the spatial API.

  ...
  using Sitecore.ContentSearch.Data; // Needed for initializing Coordinate
  using YourProject.Features.Search.Data.Models;
  using YourProject.Features.Search.Data.Models; 
  using System; 
  ...

	public class SpatialLocationService: ISpatialLocationService
	{
        // Add in logger
		private readonly ILogger _logger;

		public SpatialService(
			ILogger logger)
		{
			_logger = logger;
		}

		/// <summary>
		/// Using the solr spatial search, get list of locations near a given coordinate and radius.
		/// </summary>
		/// <param name="coords_lat"></param>
		/// <param name="coords_lng"></param>
		/// <param name="coords_radius"></param>
		/// <returns>List of locations near a given coordinate</returns>
		public List<SampleLocationBaseItem> GetLocationsByCoords(
			float coords_lat,
			float coords_lng,
			float coords_radius)
		{
			try
			{
                // Set up context for this query, using our custom index name defined in our 'Constants' file
				using (var context = ContentSearchManager.GetIndex(Constants.CustomLocationsIndex).CreateSearchContext())
				{
				    // Validate the inputs, cannot assume the values will always be good.
					// Stay between -90 and 90 lat.
					if (coords_lat < -90.0 || coords_lat > 90.0f){
						_logger.LogError($"API Error: Invalid latitude provided: {coords_lat}."
						throw new ArgumentOutOfRangeException(nameof(coords_lat), "Latitude must be between -90 and 90.");
					}
					
					// Stay between -180 and 180 lng
					if (coords_lng < -180.0f || coords_lng > 180.0f){
						_logger.LogError($"API Error: Invalid Longitude provided: {coords_lng}.");
						throw new ArgumentOutOfRangeException(nameof(coords_lng), "Latitude must be between -90 and 90.");
					}
					
					// Radius must not be negative, just double check and provide a response. 
					if (coord_radius <= 0){
						_logger.LogError($"API Error: Radius cannot be negative or zero.");
						throw new ArgumentOutOfRangeException(nameof(coords_radius), "Radius cannot be negative or zero.");
					}
				
					// Generate dedicated obj for coords. Needed for query.
					Coordinate contextCoords = new Coordinate(coords_lat, coords_lng);

					// Using spatial query, get item(s) within provided scope lat/lon/rad - using our custom search item defined earlier
					var locationsNear = context.GetQueryable<SampleLocationSearchItem>()
						.Where(c => c.TemplateId == Constants.LocationTemplateId)
						.WithinRadius(c => c.LocationGeoPin , contextCoords, coords_radius, true)
						.OrderByDistance(c => c.LocationGeoPin , contextCoords, true);

					// Map the data now
                    // Empty list to add, it is okay in this example to return empty if none exist
					var locations = new List<SampleLocationBaseItem>();
                    
					// foreach that passes the location into a private method that maps the data. 
					foreach (var location in locationsNear)
					{
						var mappedObject = ObjectMapper(location);
						
						if (mappedObject != null){
							locations.Add(mappedObject);
						}
					}

					return locations;
				}
			}
			catch (Exception e)
			{
				_logger.LogError($"API error: There was a problem with the coordinates, radius, or data. Params: Lat:{coords_lat} | Lng:{coords_lng} | rad:{coords_radius}", e);
				// Re-throw original exception to preserve stacktrace. 
				throw;
			}
		}

        /// <summary>
		/// Map the object. 
		/// </summary>
		/// <param name="resultItem"></param>
		/// <returns> a single object as type SampleLocationBaseItem</returns>
		private SampleLocationBaseItem ObjectMapper(SampleLocationSearchItem resultItem)
        {
			// Null check the resultitem just in case
			if (resultItem == null){
				_logger.LogWarn($"ObjectMapper was passed a null searchresult item.");
				return null;
			}
			
               try {
                        var locationObj = new SampleLocationBaseItem
                        {
                                LocationPin = resultItem.LocationGeoPin,
                                Radius = resultItem.LocationRadius,
                                LocationName= resultItem.LocationName,
                        }
                       
                       return locationObj;
             } 
			catch (Exception e)
			{
               // Do some logging here
				_logger.LogError($"location map error, problem mapping location object.", e);
				return null;
			}
        }

 using YourProject.Features.Search.Data.Models;
... 
 Namespace YourProject.Features.Search.Services.Interfaces{
	/// <summary>
	/// Interface for spatial location service(s).
	/// </summary>
	public interface ISpatialLocationService {
		
		/// <summary>
		/// Using the solr spatial search, get list of locations near a given coordinate and radius.
		/// </summary>
		/// <param name="coords_lat"></param>
		/// <param name="coords_lng"></param>
		/// <param name="coords_radius"></param>
		/// <returns>List of locations near a given coordinate</returns>
		public List<SampleLocationBaseItem> GetLocationsByCoords(float coords_lat, float coords_lng, float coords_radius);
	}
 }

So this is a fairly simple service that checks off most of our boxes.

It has error handling, logging, try-catch blocks at points of anticipated failure, exception handling and proper API responses to display to the end user. We log events that are not going to be the end of the world as ‘warn’ and return null. We do take note of why this failure occurred in the logs for resolution later during the review of logs. Sometimes these things can be simply edge cases of bad data inputs.
Allows us to pass in coordinates into the method and return a list of locations mapped to our object model. These objects will be used later in the JSON output.
we created a method to map the data object too. This was a bit of extra credit in how I am doing it, this allows you to turn it public, add it to the interface, and map that same object should you have a reason to, or if you were to overload the current method, you can map the data the exact same and have one dedicate place to do this.
Finally, we create an interface for our service.

Once you build out the basic service, you’ll want to make the interface, which is honestly a simple task, but for the sake of the example, I will add one here:

...
// I used a folder for my interfaces here
namespace solutionName.Services.Search.Interfaces
{
    public interface ISpatialLocationService
    {
		/// <summary>
		/// Get a list of locations based on coordinates and radius.
		/// </summary>
		/// <param name="coords_lat"></param>
		/// <param name="coords_lng"></param>
		/// <param name="coords_radius"></param>
		/// <returns></returns>
        public List<SampleLocationBaseItem> GetLocationsByCoords(float coords_lat, float coords_lng, float coords_radius)
    }
}

Now with a proper service and interface, you’ll need to register this service, in the app. For me, I have a SearchServiceRegistry.cs file that I put mine into. This just helps me keep my services grouped by utility, but you may group yours however.

Example:

...
		/// <summary>Configure the Search project services.</summary>
		/// <param name="serviceCollection">The service collection.</param>
		/// <returns>The configured service collection.</returns>
		[MethodImpl(MethodImplOptions.NoInlining)]
		public static IServiceCollection useSearchServiceRegistery(
			this IServiceCollection serviceCollection) =>
			serviceCollection
                    .AddScoped<ISpatialLocationService, SpatialLocationService>();

note: I registered this as a Scoped service, but you could run this as a Transient or even Singleton. I chose to use Scoped because it creates one instance per request and will reuse this instance throughout the lifetime of request. Transient may even be a better option over Singleton. You are welcome to dispute this, but that is my logic behind my choice here.

With this simple service built, let’s build out a controller to accept API requests to run against this service.

Building the Controller

Here is the assumed controller, again keeping it simple, this is not the entire point of this task, but it will help visualize things later.

    [RoutePrefix("api/locations")]
	public class LocationSearchApiController : ServicesApiController
	{
		public ISpatialLocationService _spatialSearch;
		public ILogger _logger;

		public LocationSearchApiController(
			ISpatialLocationService spatialSearch,
			ILogger logger)
		{
			_spatialSearch = spatialSearch;
			_logger = logger;
		}


		[HttpGet]
		[Route("near")]
		public IHttpActionResult GetLocationsByCoords(
		float lat,
		float lng,
		float rad)
     	{
            try {
				// Validate the inputs, cannot assume the values will always be good.
				// Stay between -90 and 90 lat.
				if (lat < -90.0 || lat > 90.0f){
					_logger.LogError($"API Error: Invalid latitude provided: {lat}."
					throw new ArgumentOutOfRangeException(nameof(lat), "Latitude must be between -90 and 90.");
				}
				
				// Stay between -180 and 180 lng
				if (lng < -180.0f || lng > 180.0f){
					_logger.LogError($"API Error: Invalid Longitude provided: {lng}.");
					throw new ArgumentOutOfRangeException(nameof(lng), "Latitude must be between -90 and 90.");
				}
				
				// Radius must not be negative, just double check and provide a response. 
				if (radius <= 0){
					_logger.LogError($"API Error: Radius cannot be negative or zero.");
					throw new ArgumentOutOfRangeException(nameof(radius), "Radius cannot be negative or zero.");
				}
				
                // Hit the service, assign to var
				var LocationByCoords = _spatialSearch.FindByCoords(lat, lng, rad);

				return Json(LocationByCoords);
            }
			catch (Exception e)
			{
			      // Log some error here, this is mostly for example. 
				_logger.LogError($"Error in LocationSearchApi:  see exception for details.", e);
				return BadRequest("An error occurred. Please try again later.");
			}
        }
}

Here, we have set up an API endpoint and defined how to reach it. Allowing the coordinates to be passed into the API request as query params based on the variable names. They will then be validated to be within the proper ranges, then passed into our service. The service will return the list of locations, or an empty location list.

just a note: it would be best to have some sort of field validation class for value types you would continue to check, such as the lat/long/radius values.

We add a try/catch for error scenarios. this just returns BadRequest, but you could also do the same if any of the values are null passed in, this was set up just to illustrate some form of error handling.

There is one more step before we can actually test the response. Sitecore has some built in security to not allow calls to the EntityService and ItemService from a remote client. Both having their own set of criteria as far as security is concerned. So we must create the following config override in order to essentially bypass this. Keep in mind, this will exclude it from the security check, so you’ll need to take care when doing so.

Some things I recommend keeping in mind while implementing something like this:

Bad actors exist, try to keep them in a box
- Keep responses clean and user friendly
  - Do not expose any real paths or file names
  - Do not expose server names
- Sanitize any input that can be modified or relies on user input
- Attempt to TryParse any number to make sure it is a number
Fail gracefully, log everything
- This goes somewhat back to the first bullet point, assume that an end user is malicious and trying to break your response to show some sort of stack trace to gain really any insight.
  - A good example is with a login page, and showing “invalid credentials” – if one was to know which credential is invalid, it is easier to narrow down.
- Have good log messages, it is no use to simply know a value is invalid. This could also be done through monitoring tools as well. Just try your best to have some insight on what values were put in, and what made them fail.

Create a config override based on the file located at App_Config\Sitecore\Services.Client\Sitecore.Services.Client.config

The override will be fairly simple with the following:

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
	<sitecore>
		<api>
			<services>
				<configuration>
					<allowedControllers>
						<allowedController desc="LocationSearchAPIController">{controller here}, { namespace here}</allowedController>
						</allowedControllers>
				</configuration>
			</services>
		</api>
	</sitecore>
</configuration>

You can then verify in the showconfig.aspx page if your override is there and working.

Provided there is nothing crazy that happened in between these steps, you should be able to hit 'www.{YOURSITE}.com/api/locations/near?lat=99&lng=99&rad=99' – and get your results.

Exploring the spatial search

Whether you followed along the previous things or not we will touch on some things not really mentioned in the docs, to help you better use this.

To clarify any confusion, I am using Sitecore 9.3 on-premise for my example, it may have some slight difference in v10+ and XMC.

so in the above example, I show how to use the Coordinate object, and the docs do as well, but there really isn’t any mention of where this reference comes from. To help you find this, and hopefully save you time, if you wanted to call a new instance of the Coordinate object is to import the following:

  using Sitecore.ContentSearch.Data; // Needed for referencing Coordinate object

It is not mentioned anywhere, but this is where that object reference is located. I did have to do a little digging to get this, and hopefully this can help someone else later.

To go over the query methods, we can take the example from above, and break it down

					// Using spatial query, get item(s) within provided scope lat/lon/rad - using our custom search item defined earlier
					var locationsNear = context.GetQueryable<SampleLocationSearchItem>()
						.Where(c => c.TemplateId == Constants.LocationTemplateId)
						.WithinRadius(c => c.LocationGeoPin , contextCoords, coords_radius, true)
						.OrderByDistance(c => c.LocationGeoPin , contextCoords, true);

So while I assume it can be read fairly easy, I will break it down a little just to explain what is happening and what else can be done.

Line 3: We are simply looking for the template related to our location object. It’s safe to assume your index has multiple templates, so you can search by that, or any other you see fit. You could also add a check to make sure the itemID does not match the template ID to avoid pulling in templates into your list.

Line 4: This line looks at all items that matched our template, then checks if those items have a geo pin located within the given radius. A side note, this radius by default is in Kilometers, you could convert this into miles if that is what you require.

line 5: These final results are ordered by distance from your initial point, which is the contextCoords. But the bool is the most interesting, in my opinion. The bool allows us to use a bounding box around the radius circle we have given previously. The bounding box feature in the spatial search essentially creates a square that is calculated on the SOLR side of things to determine how big the box is, and then give you results in a box shape, versus a circle. This is especially useful in situations using a square map interface to display results and needing results in the corner if there are any.

Here is a visual example, borrowed from the SOLR docs:

There are various other things you can do within SOLR as well when it comes to their spatial search. I highly recommend checking out the documentation for it, as you’ll see we have not even really scratched the surface is potential, and this is just a really basic implementation so far. It is honestly a really robust feature, so I will definitely be exploring it more in later articles.

Final Notes

Some things I wanted to note that were not enough for their own topic, but worth bringing up.

Should I add caching?

This is heavily dependent on context of your site. Generally, caching can always be beneficial. However, I would cache direct locations perhaps, over trying to cache every result set that could exist between coordinates. For example, you could create a page dedicated to stores that are in Texas, so you create an endpoint for direct hits, and then then yourApi/texas could be cached.

If your caching concern lies in SOLR, then for the most part, queries are kept in a section of SOLR’s memory for as long as they are used often enough. Also, caching can be adjusted in SOLR to fit your site needs. So assuming you hit the same kind of queries fairly frequently, it’ll keep the results in cache on the SOLR server, at least until the next reset.

Even from the SOLR docs surrounding it, they mention there is little benefit:

If you know the filter query (be it spatial or not) is fairly unique and not likely to get a cache hit then specify cache="false" as a local-param as seen in the following example. The only spatial types which stand to benefit from this technique are those with docValues like LatLonPointSpatialField or BBoxField.

It’s most common to put a spatial query into an “fq” parameter – a filter query. By default, Solr will cache the query in the filter cache.

So in short, keep an eye on your performance and SOLR tuning. These will give you your true answer.

If you plan to use Sitecore to cache, then creating dedicated endpoints for result sets is ideal.

Can I use polygons with this?

SOLR supports indexing GeoJSON data for polygons, so yes. There are some really good docs on how to do this and use it.

You can also index each individual points as a box, and not just a radial circle.

Conclusion

To conclude with this, we have a very basic API built utilizing the spatial search. I hope you can build upon this and create something of your own. I hope it answered any potential questions you may have had, or even any misunderstandings.