Enhancing Data Discovery with Lucene in DataMapper

We’re thrilled to announce a significant upgrade to our data discovery solution, DataMapper: the implementation of a new indexer, powered by Lucene. This enhancement is set to revolutionise how users search for and find files, combining the power of names, keywords, and eventually, free-text search, with improved precision and speed. 

Facing the growing needs of our users and increasing data complexity, we recognised that our traditional SQL-based solutions were reaching their limits. To address this, we turned to Lucene, known for its powerful and efficient search capabilities, as the ideal choice for our evolving requirements. 

• Enhanced Search Capabilities: Lucene facilitates sophisticated search queries, enabling users to find files by combining names, keywords, and engaging in free-text search. 
• Dynamic Risk Calculation: The risk associated with particular files will be recalculated with any changes to names or keywords, ensuring ongoing accuracy. 
• Efficient Indexing: Prioritising new files, the system also gradually incorporates already scanned files not previously indexed, enhancing the comprehensiveness of our data discovery. 

We began with an analysis of various options, including minor tweaks to our existing SQL solution and more extensive changes like restructuring components and batch file processing. The strategy moving forward involves reindexing all customers with a custom indexer, considering database improvements, and assessing the potential integration of Elastic Search for larger customers. In our research, we explored alternatives such as Elastic Search, Azure Search, and Microsoft Graph Outlook API, each with its unique benefits and limitations. 

The integration of Lucene into DataMapper represents a significant leap forward in data discovery efficiency, accuracy, and scalability. This advanced technology ensures that our users have access to the best tools for effective data management and utilisation. We look forward to keeping you updated as we continue to refine DataMapper, solidifying our commitment to excellence in data discovery technology.