Beyond Basic Filters: RegEx filters

In one of our applications, where vast amounts of data are displayed in extensive grids, we sought ways to enhance the existing grouping and filtering features to help users locate the data they need more efficiently. One idea was to introduce a wildcard mechanism to our filters.

One of our first decisions was, whether to just support regular expressions or instead to write an own custom wildcard mechanism. The former would be easier to implement, but possibly too technical for some users. The latter would be easy to use, but possibly hard to implement. We decided on starting with regular expressions to see where this would lead us, as the initial effort would be minimal.

Understanding Regular Expressions

Regular expressions (or in short RegEx) are powerful constructs used for pattern matching within text. They provide a standardized syntax to describe search patterns, enabling everything from simple string matching to complex pattern recognition.

The primary strength of regular expressions lies in their flexibility. Users can create sophisticated search patterns that would be difficult or impossible to achieve with traditional filter systems. This flexibility makes regex particularly valuable for power users who need to perform complex data analysis.

Challenges and Considerations

Regular expressions come with their own set of challenges. The syntax can be intimidating for business users, and certain characters need to be escaped properly to work as literals. This technical complexity creates a high barrier to entry for non-technical users.

It's crucial to also consider performance implications when implementing RegEx-based filters. Complex regular expressions can drastically slow down or even temporarily freeze your application, especially when working in a browser-context or dealing with large datasets.

Implementation Strategies

Free Text Filters vs. Multi-Select Filters

We discovered a crucial distinction between implementing regex in free text filters versus multi-select filters. In free text filters, the entered value is stored and automatically persisted. This means that without additional safeguards, a user could potentially soft-lock themselves out of the application by entering a complex regular expression that freezes the client.

In contrast, multi-select filters only store the resulting set of selected items. These filters also include a simple UI text filter to help users quickly locate items to select or deselect. This secondary filter does not have a persisted state, so while it is still possible to freeze the application with a complex regex, reloading the application will restore a working state. This approach is not ideal but significantly reduces the risk compared to free text filters.

Handling Invalid Expressions

When allowing users to enter raw regular expressions, it's essential to handle parsing errors gracefully. This adds complexity to managing the filter state, as filters must accommodate failure states and include UI elements to inform users of errors. Default RegEx parsing errors are often technical and not (business-)user-friendly. To balance cost and benefit, we decided to fall back to treating unparsable regular expressions as plain text filters for our proof of concept. This approach ensures a smooth user experience while still leveraging the advantages of regex when used correctly.

Limitations and Edge Cases

One notable limitation of our implementation is that the user's intent, as expressed through the RegEx, is not preserved in the resulting multi-select filter state. Only the selected items are stored, not the original RegEx pattern. Consequently, when new data is added to the system, the filter cannot reapply the original RegEx to the updated dataset automatically.

Our Final Approach

After careful consideration, we determined that regular expressions would be too technical for most of our regular application users. Our data shows that the majority of users only utilize the basic functions of our application. Therefore, we decided to make RegEx filters an opt-in feature, available only to a select group of users based on preexisting roles within our application. For those with access, we provided a concise RegEx cheat sheet for quick reference and included a clear warning about the potential impact on application performance when using this feature.

Integrating regular expressions into our filtering system was a journey filled with both technical challenges and valuable insights. While RegEx offers powerful and flexible filtering capabilities, it also introduces complexity and potential performance issues that must be carefully managed. By making RegEx filters an opt-in feature for power users and providing clear guidance on their use, we struck a balance between advanced functionality and user accessibility. This approach allows us to cater to the needs of our most demanding users while maintaining a smooth experience for the broader user base. Ultimately, the decision to incorporate RegEx filters has enhanced our application's versatility, enabling more sophisticated data analysis and exploration.