Moving from BRD document to technical document requires SME knowledge. SMEs are not easily available and this causes considerable delay in getting the design document done, which in turn impacts the timeline and productivity.
We use NLP extraction techniques to extract from BRD the relevant items required for reports building. We then filtered the text to extract the relevant terms. From these terms, we developed a recommender algorithm that, using historical BRD to technical document mapping, proposes the appropriate rules and data pieces from the data lake. A support-confidence-based set of rules was produced for the purpose of identifying the rules.
Sample Support-Confidence framework based rule-extraction is given below. SPMF, which is an open source tool is used for rule extraction.
Once the rules are extracted and tested for correctness, the system continuously learns from how the underlying transactions change (has the suggested rules been implemented or is there a change in the actual implementation). This again changes the confidence score of the rule and the recommendation for the next cycle changes.
For example, in the Business Requirements Document (BRD) for developing reports, the text given was “Spend in Euro for the selected data range”. There was only one spend item for the card in the underlying data lake. So this was easy to map and suggest as per the historical rules. But another item – “Select Date” – was impossible to map. There are literally hundreds of date elements within the data lake and all of them are heavily used in reports. So this date part has to be done manually – which is the suggestion given by the tool.
These associative rules engines work well only when there is a reasonable “support” which is the first part of the Support-confidence framework. Newer data elements created in the system are unlikely to have support and so we had to combine associative rule mining with rare event mining and apply filters around creation date of the elements identified.
In other words, these solutions don’t fully depend on machine learning or deep learning. There are elements of plain business logic which are combined to give good results to users.