Integration of Apache Solr into Plone is well-established for powering fast, scalable full-text search with highly relevant results. The many amazing features of Solr like faceting, hit highlighting and suggestions add real value for the user. Even when search is not explicitly the given task, Solr can be extremely helpful as a portal_catalog replacement to filter content for speeding up complex views and generating overview pages and navigation. In this talk I will relate my experiences and outline the ways in which we use Solr with Plone at Syslab.com. I will highlight the benefits of Solr as well as report some caveats and lessons learned over the years. Previous experience with Solr will help you get the most out of hearing this talk but is not a requirement.
 I'm Manuel Reinhardt, I work for CISLA.com. I want to talk about solar and how to use it with PLON. I'm assuming that at least some of you have heard of solar. I'll briefly say what it actually is. It's based on the Lucene indexing and search engine, which is written in Java. Solar adds to that an HTTP interface. You can get XML and JSON over HTTP. It also adds some features like caching, replication. It has a nice web admin interface. And it basically makes a high performance search server out of that Lucene engine. It's made by Apache. So it's available as open source and an Apache license. So if you know a bit of Java, you can also hack around in the code. I said I wanted to talk about solar and PLON. Why would I even want to connect those two? One big argument is speed. Solar is just terrifyingly fast when it comes to indexing and search. The portal catalog is also fast. You can also search. But it has its limitations, especially if you have a lot of content in your set ODB. You have a lot of text in your searchable text. Then at some point, it gets a bit slow. But speed is not the only thing. You also have some great features that solar offers. Things like faceted search, hit highlighting, complex query language, cross-site search, and so many more. I'm going to mention a few of those later on. So say you want to try it and do want to connect solar to PLON. How do you go about it? There are several options. I want to mention three of them. One is collective solar. Its approach is to index all indexes of all content, both in the portal catalog and in solar, just by registering an additional indexer through collective indexing. And then when you query the catalog, it checks whether the searchable text is a part of the query. And if so, it diverts the whole thing to solar, and everything runs through solar. And its response is used as the search results. There's also ALM solar index. I can't say too much about this one. I haven't had too much experience with it. The approach is slightly different. It doesn't tamper so much with the catalog itself. But it implements a single index inside the catalog through solar connector. So you don't store the indexing data somewhere in the ZODB or somewhere. But it constructs queries to solar, and it updates the index there, and gets results from solar. There's also searched, which is an even more low-level approach. It's basically a Python solar API, which is more powerful. You have more options there, more things you can do. But it's also less convenient. You have to do more pedestrian work to get results that are usable in PLONE. Built on top of it, that there is plonintranet.search. That's, as the name suggests, is used in PLONE intranet, or Quave. It's based on Scorched. But it's tuned towards PLONE or PLONE intranet. So it's more convenient to get stuff out of there, and you don't have to do as much yourself. But it still retains some of the great powers and features that the solar query language has. All of these approaches give you access to some awesome features that solar has. I only have time to mention a few. I encourage you to look online for what else solar can do. One thing is boosting. A good search, I would say, is something that lets you find what you're looking for. Or in other words, which shows you stuff that is relevant to your search. And if you have multiple results, as you usually have, the most relevant results should, of course, be shown first so that you see them first. And solar does that by default. Results that come from solar are by default sorted by a score that is usually calculated by things like how often does the search term occur in the searchable text of a document. But you can influence that score. You can specify, actually, what you think is relevant and what you think a high score constitutes. And that's just what boosting does. So you can, more or less, put arbitrary functions into the solar that it uses then to calculate the score. A very valuable real-life example is a date boost. For example, in a social internet setting, maybe you've heard Alex's talk before mine, you're probably looking for stuff that has been added or edited recently, something that's active and that's not been lying around for ages. And to do that, you would want to assign a high score to a recent document and a lower score to an older one. I won't go into too much detail on that formula there, but you see that the modified date is taken and what it basically gives you is a nice curve that assigns a high score to a recent document and then it curves off nicely towards the older stuff. The next feature I want to, I almost say, to highlight is highlighting. You can see an example here. It's in German. Please bear with me. I think you should get the blue thing. This, of course, the title of the result then comes the URL. And that thing there is a snippet from the searchable text. And the thing in square brackets is the search term that I searched for. So you don't only see what documents mention your search query, but you see the actual context, the bit of text that mentions the search term. It's basically what Google also does. If you type anything into Google, you get these small bits of text where you usually can already judge, yes, this sounds interesting or no, this only mentions my search term in passing or it's just inside a boring list or something. So that's usually really helpful. And Solar can do that as well. It can deliver these snippets of text with a bit of configuration that contain the search term. And you can configure how that's displayed. I chose to just put square brackets around them, but you can basically do arbitrary things there to highlight your search term. One warning I want to give there is if you're generating these snippets from the searchable text, which is a sensible thing to do, then it must be stored in Solar in full so that it can process it. So in portal catalog terms, it must be metadata. And that means it would be returned with every response that you get from Solar. So for example, for office documents or something that has a very large searchable text, that can get really big. So if you have hundreds or thousands of search results and deliver those searchable texts for all of them, that is just not going to work because all of that is going over the network. And it's going to take much too long. But luckily, Solar allows you to do that rather easily. You can specify what fields to return. You just have to pay a bit of attention what you're storing in Solar and what you want to return with every response and what you don't want. And finally, if you are setting up a plain site for something like a larger company or something like a university, then usually there is more than just this plain site that you're setting up. There may be other, even other plain sites or other web servers, other document stores or what is there where people need to get information from. And the user might not always know where exactly is this information that I'm searching for. Am I in the right place going to this plain site? Well, they don't really need to care because what you can do is index multiple sources into the same Solar. You can index other plain sites with the approaches that I've mentioned, Collective Solar, for example. You can even crawl arbitrary sites with Nudge, which is another tool from Apache, which works really well with Solar with just a bit of configuration. You can point it at some other website and it crawls it and indexes everything into your Solar. And what you do then to make everything available to be searched from your plain search is to tweak. You tweak the query so that you search for all the paths and not just slash plain or whatever your plain site is called, but for everything that Solar basically has to offer. And you tweak the results page that usually just links to your own plain site, of course, because it assumes everything comes from there. And you do a little check where a result comes from and you put the right domain or host name in to get the external link right. And this is the same example from before. It doesn't only do highlighting, but also does the cross-site search. I hope you can see these different color icons here. These represent different sources of information. They have different host names here. So these are results that come from a Solar that has different sources indexed. And this is indicated to the user here with a little tweak of the results page. So do you get all this for free? Well, of course, there are some things you have to pay attention to. For one thing, everything goes over the network. Solar is an external component. You do HTTP queries. And you have to pay attention a little bit. It mostly comes down to what you save in index processing time versus the time the result spends on the network. And as I've said, Solar is really fast. So if you save enough time there in index processing, and you don't have too much time that the result needs to go over the network, then you've basically won. You've gained some time. So with a Solar in a local network, in the same data center or whatever, it should be fine. But as I said, you have to be careful that you don't send masses and masses of data. For example, exclude the searchable text if it needs to be stored for highlighting. Another thing that comes from Solar being an external component is the transaction integrity. It's not always sure that transaction is atomic, because you have to communicate with the outside when you are doing stuff like re-indexing. There is basic transaction support in collective indexing, and also in collective Solar. But in my experience, there are still situations where something goes wrong. Sometimes Solar does get out of sync. You have old stale data in there. Some update doesn't get through that kind of thing. That's probably not an unsolvable problem. I expect to be work to be done there in the future. But it's also not that often. So maybe it's OK in your use case to just re-index one or two items and forget about it. Another thing is that commits can be slow in Solar. The Solar is optimized for search and for lookup of the indexes, and the trade-off that you have is that indexing can take a bit of time. So if you have some view that updates your content and you need to re-index, then you may wait a little while until the Solar index is updated, and you can render the result and show it to your user. But what you can do about that, Solar has a feature for that as well, which is called asynchronous commits. So instead of waiting for the index to update, you basically just send your data to Solar. Solar says, OK, got it. And you can go on with rendering your view and showing it to your user. And in the background, Solar will update the index and do the commits and everything that it needs to do. But that, of course, means there is a small delay between your rendering of your view and the index being updated. So at the time you show something to the user, the Solar index might not yet be up to date. So you just need to be aware of that. For example, we had a calendar view that was generated from Solar, and that let you create an event. And after you created it, you got back to the calendar view, say the month view. And if you had asynchronous commits activated there, you just not see your event. Because, yes, the data was sent to Solar, but what you were using to generate the calendar view was out of date because it was still processing the update, and the event was just not there. That's not an answerable problem either. Maybe you just need to not use asynchronous commits in situations like this. Or you can do some trick. I mean, after you know what event was just created, so maybe you just add that to the Solar data you're getting. So there are ways to solve this. So in summary, I've mentioned several methods of connecting Solar to Plone. I've only scratched the surface on the awesome features that Solar offers. And I've mentioned some manageable costs and risks, some stuff that you need to be aware of. So in conclusion, if you want to know should I use Solar for my project, I'm tempted to just say yes. But I'll qualify that a little bit. If you subscribe to that philosophy, that query should be really blazingly fast. And indexing is OK if it's a little bit slow. Then yes, you should definitely consider trying Solar, especially if you have a large site with loads and loads of data to index and to search. Or if one of the features that I mentioned, or that you can find online, appeals to you and gives value to the setup and the customer that you're working with, then yes, definitely do look into it. So that's all for now. As I said, there's a lot more to discover. Do check out the Solar user guide and wiki on the Apache websites. There's a good Solar training on the Plain Training website. Or just find me, or one of my colleagues from syslab.com, or check out our websites as well. We do web solutions and intranets and risk assessment. And we've been using Solar for a while. So we're glad to help you get started or to hear stories how you've been using Solar and discuss all that. Or if you have any questions right now, I'm available. Thank you very much. Thank you. Any questions? Thank you for the talk. I've been playing around with Solar myself for the last one and a half years. And what I've noticed is it really takes time, a lot of time, to really know what you're doing with getting to know this whole search engine and all the indexing. There are a lot of moving parts. And a lot of small things can go wrong. I follow the training. Can you give some advice on how to start with knowing this search server part? How did you get into this? That's rather tricky because it really depends on what you need for your use case. I thought that it was rather easy to get the very basic stuff running. And then it really depends. You can branch out and look at the query language, for example, and all the ands and ors and stuff that you can do there. Or you're more interested in highlighting or in boosting. So it's really tricky to give general advice there, I think. But I found that the Solar wiki has a lot of information on almost all the topics that you can imagine. So I find myself there a lot of the time. And yeah, I think that's the most general thing that I can say right now. Yeah, maybe wait for the mic so that we have it. Yeah, I just recently used Solar, and I didn't know it before. But I used it in context with Django. But for there, I generated some integration tests where I actually had the same data every time and started the Solar instance and made the same search query and did some counting on the results to make sure that it was the same thing every time. So it didn't break the indexes and stuff like that. That's also one way to make sure that it works as intended. And a way to learn Solar while developing. Sounds good. Nice talk, very clear. I have a question about this asynchronous indexing. Can you turn it off and on by query? So for example, when you create your event in the calendar, you have the synchronous indexing. And when you upload many, many files, you have a synchronous indexing by specifying some request parameter to Solar. Not that easily as far as I know, because you have to have some commit strategy in place in the Solar instance itself if you don't do automatic commits, but do it asynchronously. So but if you have them configured and everything the asynchronous commits, then I think you can also send an explicit commit now commands to Solar. That would be great. Yeah, because. Yeah, can be handy. My questions? Maybe it's very basic. But when you make a search in Joe, you get the results for which you have permissions. You have right. How do you manage that with Solar? That depends a bit on the approach, whether you're using collective Solar or Scorched or something. But in the end, it's more or less the same. You have an allowed roles and users index in the portal catalog. And you do something like that in Solar as well. So it can just query, give me these results that these users are allowed to see. Is this native or you have to implement it? You basically have to implement like all the other indexes that you have. Yeah. How about unindexing stuff? Do you have any risk in that? Or does that just work? Unindexing, you mean removing from Solar? Yes. So any issues where you can have results available at Solar that are not any more existing in Plow? I remember having some trouble with that in the early stages. But I think by now that's at least in collective Solar. It's implemented pretty well. So if you're deleting objects, then they're getting removed from Solar. But as I mentioned, on rare occasions, you have some junk laying around in Solar where there's no object anymore. But there's, for example, a sync view that synchronizes the portal catalog with Solar and that gets rid of these unwanted entries. Two additional answers to the two questions. The first thing, if you install Solar with the collective Solar and Solar recipe, you get all these indexes in Plow. So when Manuel says you have to implement it, it is implemented in Solar, but it is implemented for you. So everything is already in place and pre-configured. You don't have to do any coding anymore. And on Askors question, normally if you delete something in Plow, it should be properly unindexed. But of course, it is an HTTP request. And if somebody, for example, restarts Solar, in that moment when you delete an object in Plow, you run, of course, into, it's not in sync anymore. And that's where we have this Solar maintenance browser view, which is great. You can re-index stuff repeatedly. You can run the sync, which really makes sure that everything is in place and so on. And it's quite quick, even. Yeah, thanks, Alex. I have two questions. The first thing is you mentioned collective free-cyped Solar, which was, as far as I know, bound to an old version of Solar. So I think the 5.1 something or so was, I think. And Solar actually is now at the 7.0 something version. So has there been an upgrade path? I know with the change in Solar 5.3, they've completely get rid of the old install methods and everything like that. And the other question is, with the newer versions, like 5.3 and so on, they introduce replication, a Solar cloud, and everything so that you can replicate over several servers your Solar instances. So is there anything that we can use now for that with Plone so that you have a high availability set up that even makes it quicker for you? I have not followed the recipe very closely, but I had the impression that a lot has happened there. Some upgrades are there. I'm not sure about the current state, what versions are supported. But upgrading Solar itself, I found it pretty easy. We upgraded, I think, from 3 to 5 recently, and that was pretty painless. I don't think we've used anything from the recipe itself, but native Solar upgrade methods or something. And about replication and Solar clouds, that's definitely on my list. I haven't looked into that yet. I'm afraid. So I can't say anything about it right now. But that does sound exciting. Yeah. Can I chip into that question from my friendly neighbor here over here? Because I've been looking into that. I've been following the recipe and collective Solar a bit more closely over the last half year. And we have a bit of a chicken and egg problem here. Many people start using Solar a bit, the recipe. And the recipe is, at the moment, supporting an old version of Solar, version 4, and version 7 has just been released. But it's a bit of a community chicken and egg problem. As soon as you start using Solar and you get more experience, then you want to do more. You want to upgrade it. And then also larger companies will start to use configuration management and install their own Solar instance using another system. And they won't use the recipe instance anymore. So we're a bit stuck at the moment in the Plone community that we have a nice starter recipe to install Solar server, but it's very old. And many of the advanced users go away from the recipe instance to set up Solar because they have their own advanced setup method. So this is a bit of an issue we have. We might have to talk about it in an open space or maybe at Sprint. OK, so maybe someone feels inspired right now. If you want to contribute to the recipe, please check it out. Please do. Have you used it in the same context with the rel storage? I haven't done that. But I wouldn't know why not. That should work. I was just curious if there are some strange issues if you use it together. No, I won't make any promises, but that sounds totally feasible. Yeah. I don't have any experience. I'm doing multi-site search with Solar and Plone. Like, first in multiple Plone sites to one Solar instance. It sounded like collective Solaris made mainly for one Plone instance at a time. Yeah, you have to do. We do have multiple Plone instances in one Solar core, even distributed over different Solr servers. So that's without any problem. Yeah, my experience, you have to do some tweaks. So maybe those can even be backported at some point into collective Solar to support that a little better. OK, if we had a multiple site to one Solar, would those maintenance be used in the work, or would they kind of break the catalog? Yeah, if you have moved this multiple site in one Solar, you're like, Alex and I will say it, would we still be able to use those maintenance views? Yeah. I think the only thing you need to do is you need to make sure that what you index can still be recognized to belong to a server site. Also, you can see the right, a single code in the website. So the one who will sing or index that, and you see it. I think what you are thinking about is you sync, and it will see what is not in your local portal catalog and then throw everything away in Solar, right? All the other sites. I think there's a danger that that might happen, yes. But I don't know. I think we're definitely taking a look. That's a good point. Yeah, you have to be careful there. OK, so we've had many interesting questions. Let's thank the speaker, please. Thank you. Thank you.
 I'm Manuel Reinhardt, I work for CISLA.com.  I want to talk about solar and how to use it with PLON.  I'm assuming that at least some of you have heard of solar.  I'll briefly say what it actually is.  It's based on the Lucene indexing and search engine, which  is written in Java.  Solar adds to that an HTTP interface.  You can get XML and JSON over HTTP.  It also adds some features like caching, replication.  It has a nice web admin interface.  And it basically makes a high performance search server  out of that Lucene engine.  It's made by Apache.  So it's available as open source and an Apache license.  So if you know a bit of Java, you can also  hack around in the code.  I said I wanted to talk about solar and PLON.  Why would I even want to connect those two?  One big argument is speed.  Solar is just terrifyingly fast when it comes to indexing  and search.  The portal catalog is also fast.  You can also search.  But it has its limitations, especially if you have a lot  of content in your set ODB.  You have a lot of text in your searchable text.  Then at some point, it gets a bit slow.  But speed is not the only thing.  You also have some great features that solar offers.  Things like faceted search, hit highlighting, complex query  language, cross-site search, and so many more.  I'm going to mention a few of those later on.  So say you want to try it and do want to connect solar to  PLON.  How do you go about it?  There are several options.  I want to mention three of them.  One is collective solar.  Its approach is to index all indexes of all content, both  in the portal catalog and in solar, just by registering an  additional indexer through collective indexing.  And then when you query the catalog, it checks whether the  searchable text is a part of the query.  And if so, it diverts the whole thing to solar, and  everything runs through solar.  And its response is used as the search results.  There's also ALM solar index.  I can't say too much about this one.  I haven't had too much experience with it.  The approach is slightly different.  It doesn't tamper so much with the catalog itself.  But it implements a single index inside the catalog through  solar connector.  So you don't store the indexing data somewhere in the  ZODB or somewhere.  But it constructs queries to solar, and it updates the  index there, and gets results from solar.  There's also searched, which is an even more low-level  approach.  It's basically a Python solar API, which is more powerful.  You have more options there, more things you can do.  But it's also less convenient.  You have to do more pedestrian work to get results that  are usable in PLONE.  Built on top of it, that there is plonintranet.search.  That's, as the name suggests, is used in PLONE intranet, or  Quave.  It's based on Scorched.  But it's tuned towards PLONE or PLONE intranet.  So it's more convenient to get stuff out of there, and you  don't have to do as much yourself.  But it still retains some of the great powers and features  that the solar query language has.  All of these approaches give you access to some awesome  features that solar has.  I only have time to mention a few.  I encourage you to look online for what else solar can do.  One thing is boosting.  A good search, I would say, is something that lets you find  what you're looking for.  Or in other words, which shows you stuff that is relevant  to your search.  And if you have multiple results, as you usually have,  the most relevant results should, of course, be shown  first so that you see them first.  And solar does that by default.  Results that come from solar are by default sorted by a  score that is usually calculated by things like how  often does the search term occur in the searchable text of  a document.  But you can influence that score.  You can specify, actually, what you think is relevant and  what you think a high score constitutes.  And that's just what boosting does.  So you can, more or less, put arbitrary functions into the  solar that it uses then to calculate the score.  A very valuable real-life example is a date boost.  For example, in a social internet setting, maybe  you've heard Alex's talk before mine, you're probably  looking for stuff that has been added or edited recently,  something that's active and that's not been lying around  for ages.  And to do that, you would want to assign a high score to a  recent document and a lower score to an older one.  I won't go into too much detail on that formula there, but  you see that the modified date is taken and what it basically  gives you is a nice curve that assigns a high score to a  recent document and then it curves off nicely towards the  older stuff.  The next feature I want to, I almost say, to highlight is  highlighting.  You can see an example here.  It's in German.  Please bear with me.  I think you should get the blue thing.  This, of course, the title of the result then comes the  URL.  And that thing there is a snippet from the searchable  text.  And the thing in square brackets is the search term  that I searched for.  So you don't only see what documents mention your search  query, but you see the actual context, the bit of text that  mentions the search term.  It's basically what Google also does.  If you type anything into Google, you get these small  bits of text where you usually can already judge, yes, this  sounds interesting or no, this only mentions my search term  in passing or it's just inside a boring list or something.  So that's usually really helpful.  And Solar can do that as well.  It can deliver these snippets of text with a bit of  configuration that contain the search term.  And you can configure how that's displayed.  I chose to just put square brackets around them, but you  can basically do arbitrary things there to  highlight your search term.  One warning I want to give there is if you're generating  these snippets from the searchable text, which is a  sensible thing to do, then it must be stored in Solar in  full so that it can process it.  So in portal catalog terms, it must be metadata.  And that means it would be returned with every response  that you get from Solar.  So for example, for office documents or something that  has a very large searchable text, that can get really big.  So if you have hundreds or thousands of search results  and deliver those searchable texts for all of them, that  is just not going to work because all of that is going  over the network.  And it's going to take much too long.  But luckily, Solar allows you to do that rather easily.  You can specify what fields to return.  You just have to pay a bit of attention what you're storing  in Solar and what you want to return with every response  and what you don't want.  And finally, if you are setting up a plain site for  something like a larger company or something like a  university, then usually there is more than just this plain  site that you're setting up.  There may be other, even other plain sites or other web  servers, other document stores or what is there where  people need to get information from.  And the user might not always know where exactly is this  information that I'm searching for.  Am I in the right place going to this plain site?  Well, they don't really need to care because what you can do  is index multiple sources into the same Solar.  You can index other plain sites with the approaches that  I've mentioned, Collective Solar, for example.  You can even crawl arbitrary sites with Nudge, which is  another tool from Apache, which works really well with  Solar with just a bit of configuration.  You can point it at some other website and it crawls it  and indexes everything into your Solar.  And what you do then to make everything available to be  searched from your plain search is to tweak.  You tweak the query so that you search for all the paths and  not just slash plain or whatever your plain site is  called, but for everything that Solar basically has to offer.  And you tweak the results page that usually just links to  your own plain site, of course, because it assumes everything  comes from there.  And you do a little check where a result comes from and you  put the right domain or host name in to get the external link  right.  And this is the same example from before.  It doesn't only do highlighting, but also does the  cross-site search.  I hope you can see these different color icons here.  These represent different sources of information.  They have different host names here.  So these are results that come from a Solar that has  different sources indexed.  And this is indicated to the user here with a little tweak  of the results page.  So do you get all this for free?  Well, of course, there are some things you have to pay  attention to.  For one thing, everything goes over the network.  Solar is an external component.  You do HTTP queries.  And you have to pay attention a little bit.  It mostly comes down to what you save in index processing  time versus the time the result spends on the network.  And as I've said, Solar is really fast.  So if you save enough time there in index processing, and  you don't have too much time that the result needs to go  over the network, then you've basically won.  You've gained some time.  So with a Solar in a local network, in the same data  center or whatever, it should be fine.  But as I said, you have to be careful that you don't send  masses and masses of data.  For example, exclude the searchable text if it needs  to be stored for highlighting.  Another thing that comes from Solar being an external  component is the transaction integrity.  It's not always sure that transaction is atomic, because  you have to communicate with the outside when you are  doing stuff like re-indexing.  There is basic transaction support in collective  indexing, and also in collective Solar.  But in my experience, there are still situations where  something goes wrong.  Sometimes Solar does get out of sync.  You have old stale data in there.  Some update doesn't get through that kind of thing.  That's probably not an unsolvable problem.  I expect to be work to be done there in the future.  But it's also not that often.  So maybe it's OK in your use case to just re-index one or  two items and forget about it.  Another thing is that commits can be slow in Solar.  The Solar is optimized for search and for lookup of the  indexes, and the trade-off that you have is that indexing  can take a bit of time.  So if you have some view that updates your content and you  need to re-index, then you may wait a little while until the  Solar index is updated, and you can render the result and  show it to your user.  But what you can do about that, Solar has a feature for  that as well, which is called asynchronous commits.  So instead of waiting for the index to update, you basically  just send your data to Solar.  Solar says, OK, got it.  And you can go on with rendering your view and showing it  to your user.  And in the background, Solar will update the index and do  the commits and everything that it needs to do.  But that, of course, means there is a small delay between  your rendering of your view and the index being updated.  So at the time you show something to the user, the Solar  index might not yet be up to date.  So you just need to be aware of that.  For example, we had a calendar view that was generated  from Solar, and that let you create an event.  And after you created it, you got back to the calendar view,  say the month view.  And if you had asynchronous commits activated there, you  just not see your event.  Because, yes, the data was sent to Solar, but what you were  using to generate the calendar view was out of date because  it was still processing the update, and the event was  just not there.  That's not an answerable problem either.  Maybe you just need to not use asynchronous commits in  situations like this.  Or you can do some trick.  I mean, after you know what event was just created, so  maybe you just add that to the Solar data you're getting.  So there are ways to solve this.  So in summary, I've mentioned several methods of  connecting Solar to Plone.  I've only scratched the surface on the awesome features  that Solar offers.  And I've mentioned some manageable costs and risks,  some stuff that you need to be aware of.  So in conclusion, if you want to know should I use Solar for  my project, I'm tempted to just say yes.  But I'll qualify that a little bit.  If you subscribe to that philosophy, that query should  be really blazingly fast.  And indexing is OK if it's a little bit slow.  Then yes, you should definitely consider trying Solar,  especially if you have a large site with loads and loads of  data to index and to search.  Or if one of the features that I mentioned, or that you can  find online, appeals to you and gives value to the setup  and the customer that you're working with, then yes,  definitely do look into it.  So that's all for now.  As I said, there's a lot more to discover.  Do check out the Solar user guide and wiki on the Apache  websites.  There's a good Solar training on the Plain Training website.  Or just find me, or one of my colleagues from syslab.com,  or check out our websites as well.  We do web solutions and intranets and risk assessment.  And we've been using Solar for a while.  So we're glad to help you get started or to hear stories how  you've been using Solar and discuss all that.  Or if you have any questions right now, I'm available.  Thank you very much.  Thank you.  Any questions?  Thank you for the talk.  I've been playing around with Solar myself  for the last one and a half years.  And what I've noticed is it really takes time, a lot of time,  to really know what you're doing with getting  to know this whole search engine and all the indexing.  There are a lot of moving parts.  And a lot of small things can go wrong.  I follow the training.  Can you give some advice on how to start with knowing  this search server part?  How did you get into this?  That's rather tricky because it really  depends on what you need for your use case.  I thought that it was rather easy to get the very basic stuff  running.  And then it really depends.  You can branch out and look at the query language, for example,  and all the ands and ors and stuff that you can do there.  Or you're more interested in highlighting or in boosting.  So it's really tricky to give general advice there, I think.  But I found that the Solar wiki has a lot of information  on almost all the topics that you can imagine.  So I find myself there a lot of the time.  And yeah, I think that's the most general thing  that I can say right now.  Yeah, maybe wait for the mic so that we have it.  Yeah, I just recently used Solar, and I didn't know it before.  But I used it in context with Django.  But for there, I generated some integration tests  where I actually had the same data every time  and started the Solar instance and made the same search query  and did some counting on the results  to make sure that it was the same thing every time.  So it didn't break the indexes and stuff like that.  That's also one way to make sure that it works as intended.  And a way to learn Solar while developing.  Sounds good.  Nice talk, very clear.  I have a question about this asynchronous indexing.  Can you turn it off and on by query?  So for example, when you create your event in the calendar,  you have the synchronous indexing.  And when you upload many, many files,  you have a synchronous indexing by specifying some request  parameter to Solar.  Not that easily as far as I know,  because you have to have some commit strategy in place  in the Solar instance itself if you don't do automatic commits,  but do it asynchronously.  So but if you have them configured and everything  the asynchronous commits, then I think  you can also send an explicit commit now commands to Solar.  That would be great.  Yeah, because.  Yeah, can be handy.  My questions?  Maybe it's very basic.  But when you make a search in Joe,  you get the results for which you have permissions.  You have right.  How do you manage that with Solar?  That depends a bit on the approach,  whether you're using collective Solar or Scorched or something.  But in the end, it's more or less the same.  You have an allowed roles and users index in the portal  catalog.  And you do something like that in Solar as well.  So it can just query, give me these results  that these users are allowed to see.  Is this native or you have to implement it?  You basically have to implement like all the other indexes  that you have.  Yeah.  How about unindexing stuff?  Do you have any risk in that?  Or does that just work?  Unindexing, you mean removing from Solar?  Yes.  So any issues where you can have results available at Solar  that are not any more existing in Plow?  I remember having some trouble with that in the early stages.  But I think by now that's at least in collective Solar.  It's implemented pretty well.  So if you're deleting objects, then they're  getting removed from Solar.  But as I mentioned, on rare occasions,  you have some junk laying around in Solar  where there's no object anymore.  But there's, for example, a sync view  that synchronizes the portal catalog with Solar  and that gets rid of these unwanted entries.  Two additional answers to the two questions.  The first thing, if you install Solar  with the collective Solar and Solar recipe,  you get all these indexes in Plow.  So when Manuel says you have to implement it,  it is implemented in Solar, but it is implemented for you.  So everything is already in place and pre-configured.  You don't have to do any coding anymore.  And on Askors question, normally if you delete something  in Plow, it should be properly unindexed.  But of course, it is an HTTP request.  And if somebody, for example, restarts Solar,  in that moment when you delete an object in Plow,  you run, of course, into, it's not in sync anymore.  And that's where we have this Solar maintenance  browser view, which is great.  You can re-index stuff repeatedly.  You can run the sync, which really makes sure  that everything is in place and so on.  And it's quite quick, even.  Yeah, thanks, Alex.  I have two questions.  The first thing is you mentioned collective free-cyped Solar,  which was, as far as I know, bound to an old version of Solar.  So I think the 5.1 something or so was, I think.  And Solar actually is now at the 7.0 something version.  So has there been an upgrade path?  I know with the change in Solar 5.3,  they've completely get rid of the old install methods  and everything like that.  And the other question is, with the newer versions, like 5.3  and so on, they introduce replication, a Solar cloud,  and everything so that you can replicate over several servers  your Solar instances.  So is there anything that we can use now  for that with Plone so that you have a high availability set up  that even makes it quicker for you?  I have not followed the recipe very closely,  but I had the impression that a lot has happened there.  Some upgrades are there.  I'm not sure about the current state, what versions are supported.  But upgrading Solar itself, I found it pretty easy.  We upgraded, I think, from 3 to 5 recently,  and that was pretty painless.  I don't think we've used anything from the recipe itself,  but native Solar upgrade methods or something.  And about replication and Solar clouds,  that's definitely on my list.  I haven't looked into that yet.  I'm afraid.  So I can't say anything about it right now.  But that does sound exciting.  Yeah.  Can I chip into that question from my friendly neighbor  here over here?  Because I've been looking into that.  I've been following the recipe and collective Solar  a bit more closely over the last half year.  And we have a bit of a chicken and egg problem here.  Many people start using Solar a bit, the recipe.  And the recipe is, at the moment,  supporting an old version of Solar, version 4,  and version 7 has just been released.  But it's a bit of a community chicken and egg problem.  As soon as you start using Solar and you get more experience,  then you want to do more.  You want to upgrade it.  And then also larger companies will  start to use configuration management  and install their own Solar instance using another system.  And they won't use the recipe instance anymore.  So we're a bit stuck at the moment in the Plone community  that we have a nice starter recipe to install Solar server,  but it's very old.  And many of the advanced users go away from the recipe  instance to set up Solar because they have their own advanced  setup method.  So this is a bit of an issue we have.  We might have to talk about it in an open space  or maybe at Sprint.  OK, so maybe someone feels inspired right now.  If you want to contribute to the recipe,  please check it out.  Please do.  Have you used it in the same context with the rel storage?  I haven't done that.  But I wouldn't know why not.  That should work.  I was just curious if there are some strange issues  if you use it together.  No, I won't make any promises, but that sounds totally feasible.  Yeah.  I don't have any experience.  I'm doing multi-site search with Solar and Plone.  Like, first in multiple Plone sites to one Solar instance.  It sounded like collective Solaris made mainly  for one Plone instance at a time.  Yeah, you have to do.  We do have multiple Plone instances in one Solar core,  even distributed over different Solr servers.  So that's without any problem.  Yeah, my experience, you have to do some tweaks.  So maybe those can even be backported at some point  into collective Solar to support that a little better.  OK, if we had a multiple site to one Solar,  would those maintenance be used in the work,  or would they kind of break the catalog?  Yeah, if you have moved this multiple site in one Solar,  you're like, Alex and I will say it,  would we still be able to use those maintenance views?  Yeah.  I think the only thing you need to do  is you need to make sure that what you index can still  be recognized to belong to a server site.  Also, you can see the right, a single code in the website.  So the one who will sing or index that, and you see it.  I think what you are thinking about is you sync,  and it will see what is not in your local portal catalog  and then throw everything away in Solar, right?  All the other sites.  I think there's a danger that that might happen, yes.  But I don't know.  I think we're definitely taking a look.  That's a good point.  Yeah, you have to be careful there.  OK, so we've had many interesting questions.  Let's thank the speaker, please.  Thank you.  Thank you.
Day_2__Manuel_Reinhardt_-_Solr_and_Plone_Search_an_hPu60gR5myI_0_119.png Day_2__Manuel_Reinhardt_-_Solr_and_Plone_Search_an_hPu60gR5myI_120_251.png Day_2__Manuel_Reinhardt_-_Solr_and_Plone_Search_an_hPu60gR5myI_252_25631.png Day_2__Manuel_Reinhardt_-_Solr_and_Plone_Search_an_hPu60gR5myI_25632_28547.png Day_2__Manuel_Reinhardt_-_Solr_and_Plone_Search_an_hPu60gR5myI_46248_46365.png