Python connect composed keywords in texts – Artificial Intelligence

The Phrases/Phraser classes of gensim are designed to use their internal, statistically-derived records of what word pairs should be promoted to phrases – not user-supplied pairings. (You could probably poke & prod a Phraser to do what you want, by synthesizing scores/thresholds, but that would be somewhat awkward & kludgey.)

You could, mimic their general approach: (1) operate on lists-of-tokens rather than raw strings; (2) learn & remember token-pairs that should be combined; & (3) perform combination in a single pass. That should work far more efficiently than anything based on doing repeated search-and-replace on a string – which it sounds like you’ve already tried and found wanting.

For example, let’s first create a dictionary, where the keys are tuples of word-pairs that should be combined, and the values are tuples that include both their designated combination-token, and a 2nd item that’s just an empty-tuple. (The reason for this will become clear later.)

keywords = ['machine learning', 'data science', 'artificial intelligence']
texts = [
    'the new machine learning model built by google is revolutionary for the current state of artificial intelligence. it may change the way we are thinking', 
    'data science and artificial intelligence are two different fields, although they are interconnected. scientists from harvard are explaining it in a detailed presentation that could be found on our page.'
]

combinations_dict = {tuple(kwsplit):('_'.join(kwsplit), ()) 
                     for kwsplit in [kwstr.split() for kwstr in keywords]}
combinations_dict

After this step, combinations_dict is:

{('machine', 'learning'): ('machine_learning', ()),
 ('data', 'science'): ('data_science', ()),
 ('artificial', 'intelligence'): ('artificial_intelligence', ())}

Now, we can use a Python generator function to create an iterable transformation of any other sequence-of-tokens, that takes original tokens one-by-one – but before emitting any, adds the next to a buffered candidate pair-of-tokens. If that pair is one that should be combined, a single combined token is yielded – but if not, just the 1st token is emitted, leaving the 2nd to be combined with the next token in a new candidate pair.

For example:

def combining_generator(tokens, comb_dict):
    buff = ()  # start with empty buffer
    for in_tok in tokens:
        buff += (in_tok,)  # add latest to buffer
        if len(buff) < 2:  # grow buffer to 2 tokens if possible
            continue
        # lookup what to do for current pair... 
        # ...defaulting to emit-[0]-item, keep-[1]-item in new buff
        out_tok, buff = comb_dict.get(buff, (buff[0], (buff[1],)))
        yield out_tok 
    if buff:
        yield buff[0]  # last solo token if any

Here we see the reason for the earlier () empty-tuples: that’s the preferred state of the buff after a successful replacement. And driving the result & next-state this way helps us use the form of dict.get(key, default) that supplies a specific value to be used if the key isn’t found.

Now designated combinations can be applied via:

tokenized_texts = [text.split() for text in texts]
retokenized_texts = [list(combining_generator(tokens, combinations_dict)) for tokens in tokenized_texts]
retokenized_texts

…which reports tokenized_texts as:

[
  ['the', 'new', 'machine_learning', 'model', 'built', 'by', 'google', 'is', 'revolutionary', 'for', 'the', 'current', 'state', 'of', 'artificial', 'intelligence.', 'it', 'may', 'change', 'the', 'way', 'we', 'are', 'thinking'], 
  ['data_science', 'and', 'artificial_intelligence', 'are', 'two', 'different', 'fields,', 'although', 'they', 'are', 'interconnected.', 'scientists', 'from', 'harvard', 'are', 'explaining', 'it', 'in', 'a', 'detailed', 'presentation', 'that', 'could', 'be', 'found', 'on', 'our', 'page.']
]

Note that the tokens ('artificial', 'intelligence.') aren’t combined here, as the dirt-simple .split() tokenization used has left the punctuation attached, preventing an exact match to the rule.

Real projects will want to use a more-sophisticated tokenization, that might either strip the punctuation, or retain punctuation as tokens, or do other preprocessing – and as a result would properly pass 'artificial' as a token without the attached '.'. For example a simple tokenization that just retains runs-of-word-characters discarding punctuation would be:

import re
tokenized_texts = [re.findall('w+', text) for text in texts]
tokenized_texts

Another that also keeps any stray non-word/non-space characters (punctuation) as standalone tokens would be:

tokenized_texts = [re.findall(r'w+|(?:[^ws])', text) for text in texts]
tokenized_texts

Either of these alternatives to a simple .split() would ensure your 1st text presents the necessary ('artificial', 'intelligence') pair for combination.