Solr : Same configuration files for master/slave for every environment

A solr core needs so many configuration files. They look different for master and slave. Also some files like data-config.xml are different for environments when you have data sources to index from. With these permutations, you might end up with large sets of config files. Here I have explained how I managed maintain only single set for all servers and all environments.

We always like to take well tested artifact only to upper environment. If we test something and then copy parts of it to suite higher environment can cause following issues

  • First of its tiring to maintain lots of sets. For example master/slave setup in DEV/QA/PROD environments would 6 different sets with each config set containing 4 different XML files
  • Manually syncing all of them is definitely error prone which might lead to lots of downtime
  • Don’t want to maintain passwords for datasources in these XML files. Its a security issue.
  • Many more …

Basic idea is to store all properties in core.properties and use them in configuration files using ${property_name}

Storing master/slave configuration

This part has been explained in Solr documentation as well. We will control which particular configuration to use via master/slave flag.

solr.master_url=http://dev-solrv5-lx01.amdc.mckinsey.com:8080/solr/persons/replication
enable.master=true
enable.slave=false

In solrconfig.xml

       <requestHandler name="/replication" class="solr.ReplicationHandler" >
	  <lst name="master">
	    <str name="enable">${enable.master:false}</str>
	    <str name="replicateAfter">commit</str>
	    <str name="backupAfter">commit</str>
	    <str name="confFiles">solrconfig.xml,schema.xml,stopwords.txt</str>
	 </lst>
	 <lst name="slave">
	    <str name="enable">${enable.slave:false}</str>
	   <str name="masterUrl">${solr.master_url:master}</str>
	 </lst>
	</requestHandler>

Storing datasource information for data-config.xml for solr DIH

This is bit tricky as core.properties are not available in data-config.xml directly like they do in solrconfig.xml. So what we will do is get these properties from core.properties into solrconfig.xml and then pass them to data-config as request parameters in request handler.

core.properties

db.username=user
db.password=secret
db.url=jdbc:oracle:thin:@//oracleDB:1521/data.world

solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
	<lst name="defaults">
		<str name="config">data-config.xml</str>
		<str name="db_username">${db.username:user}</str>
		<str name="db_password">${db.password:secret}</str>
		<str name="db_url">${db.url:profiles}</str>
	</lst>
</requestHandler>

data-config.xml

 <dataSource 
 	driver="oracle.jdbc.driver.OracleDriver" 
 	url="${dataimporter.request.ksr_db_url}" 
 	user="${dataimporter.request.ksr_db_username}" 
 	password="${dataimporter.request.ksr_db_password}" 
 	name="dbSource" batchSize="100"/>
Advertisements

2 thoughts on “Solr : Same configuration files for master/slave for every environment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s