PyMedExt is a library designed to process clinical text. PyMedExt includes basic data wrangling functions to transform text input formated as txt, pymedext,biocxml,biocjson,fhir, or brat into pymedext, biocxml, biocjson, omop or brat.
PyMedExt also includes an easy way to define Annotator.
pip3 install git+https://github.com/equipe22/pymedext_core.git
git clone https://github.com/equipe22/pymedext_core.git
cd pymedext_core/examples
#This script contains the Tutorial
#python3 demo.py
# go in python interactive mode
python3#import dependencies
from pymedextcore import pymedext # contains all pymedextcore objects
import os
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
dataPath=os.getcwd().replace("examples","data/frenchReport/")
resourcePath=os.getcwd().replace("examples","ressources/")
letter=open(dataPath+"letter.txt","r").read()
print(letter)
LetterPyMedExt=pymedext.Document(raw_text= letter, ID="ID_letter01")
LetterPyMedExt.to_dict()if you want to expand PyMedExt and add a new Annotator. Firstly, create a class which extend the annotators.Annotator class. Secondly, you will need to extend two functions.
- init
- annotate_function
the simplest annotator possibles
import re
thisValue="liposarcome"
#find the position of each thisValue in the letter text
for i in re.finditer(thisValue, letter.lower()):
matchPos=i.start()
if matchPos is not []:
span=(int(matchPos), int(matchPos)+len(thisValue))
print(span)Now we will adapt this function to the Annotator class
The init function must contains
- key_input --> the type of Annotation's input used by the Annotator, here "raw_text"
- key_output --> the type of the Annotation's output by the Annotator, here "regex_fast"
- ID --> the tool ID, eventually the tool git repository address and version for Annotation Traceability
- other arguments are specific to the type of the defined Annotator for example, findValues: "list of value to identify in the text"
from pymedextcore import annotators
class findMatches(annotators.Annotator):
"""
Annotator based on linux grep to search regext from a source file
"""
def __init__(self, key_input, key_output, ID, findValues ):
"""FIXME! initialize the annotator
:param key_input: input ['raw_text']
:param key_output: Annotation type here "Liposarcom.V0"
:param ID: regex_fast.version
:param findValues: "list of value to identify in the text"
:returns:
:rtype:
"""
super().__init__(key_input, key_output, ID)
self.findValues=findValues
```
##### annotate_function()
The annotate_function must contains
- _input --> Annotations associated with the Document to annotate
- returns --> Annotations ( a list of annotations object )
```python
def annotate_function(self, _input):
""" main annotation function
:param _input: in this case raw_text
:returns: a list of annotations
:rtype:
"""
logger.debug(_input)
inp = self.get_key_input(_input,0)[0]
annotationsList=[]
for thisValue in self.findValues:
#result = [i.start() for i in re.finditer(thisValue, inp.value.lower())]
for i in re.finditer(thisValue, inp.value.lower()):
matchPos=i.start()
if matchPos is not []:
logger.debug("ok go in loop")
logger.debug(matchPos)
ID = str(uuid.uuid1())
annotationsList.append(annotators.Annotation(type= self.key_output,
value=thisValue, #thisMatch,
span=(int(matchPos), int(matchPos)+len(thisValue)),
source=self.ID,
isEntity=True,
ID=ID,
source_ID = inp.ID))
logger.debug(annotationsList)
return(annotationsList)demoAnnotator = findMatches(key_input = ['raw_text'],
key_output = 'Liposarcom.V0',
ID = "demoreiter", findValues = ["liposarcome"])
# add all your annotators in a list
annotatorsList =[demoAnnotator]
# annotate your document
LetterPyMedExt.annotate(annotatorsList)grep is a linux command-line which allow you to search into plain-text data sets for lines that match a regular expression. The script grepWrapperAnnotator.py located on the examples directory, is a wrapper around grep.
this wrapper takes as resources two files :
- regexResource.txt --> a one column list of words to search in a text
- pivotResource.csv --> a two columns list of words: pattern, normalizewords
The init function must contains
- key_input --> the type of Annotation's input used by the Annotator, here "raw_text"
- key_output --> the type of the Annotation's output by the Annotator, here "regex_fast"
- ID --> the tool ID, eventually the tool git repository address and version
- other arguments are specific to the type of the defined Annotator
from pymedextcore import annotators
class regexFast(annotators.Annotator):
"""
Annotator based on linux grep to search regex from a source file
"""
def __init__(self, key_input, key_output, ID, regexResource, pathToPivot, ignore_syntax=False):
"""FIXME! initialize the annotator
:param key_input: input [raw_text']
:param key_output: either regex_fast or the normalized regex value need to discuss
:param ID: regex_fast.version
:param regexResource: path to regex value file
:param pathToPivot: pivot table between regex and the normalized value
:param ignore_syntax: not used yet
:returns:
:rtype:
"""
super().__init__(key_input, key_output, ID)
self.ignore_syntax=ignore_syntax
self.fileAnnotation=None
self.countValue=None
self.pathToPivot=pathToPivot
self.pivot=dict()
self.cmds=["fgrep -iow -n -b -F -f "+regexResource]
self.loadPivot()The annotate_function must contains
- _input --> Annotations associated with the Document to annotate
- returns --> Annotations ( a list of annotations object )
def annotate_function(self, _input):
""" main annotation function
:param _input: in this case raw_text
:returns: a list of annotations
:rtype:
"""
logger.debug(_input)
#get_key_input: return the annotations oF Documents.annotations which have
# the same type of the i th key_input element
inp = self.get_key_input(_input,0)[0]
fileAnnotation,countValue=self.makeMatch(inp)
countValue=self.setPivot(countValue)
logger.debug(countValue)
annotations=[]
for matchPos in list(fileAnnotation.keys()):
for thisMatch in fileAnnotation[matchPos]:
ID = str(uuid.uuid1())
attributes={"ngram":thisMatch}
annotations.append(annotators.Annotation(type= self.key_output,
value=countValue[thisMatch]["normalized"], #thisMatch,
span=(int(matchPos), int(matchPos)+len(thisMatch)),
source=self.ID,
isEntity=True,
ID=ID,
attributes=attributes,
source_ID = inp.ID))
return(annotations)First, clone the pymedext_core git repository and go to the examples directory
#import dependencies
from grepWrapperAnnotator import regexFast # contains your local annotator
from pymedextcore import pymedext # contains all pymedextcore objects
import os
import logging
logging.basicConfig(level=logging.DEBUG)
resourcePath=os.getcwd().replace("examples","ressources/")
thisDoc=pymedext.Document(raw_text= " a document demo you want to work with and contains evidence of. covid 19, sras, sars ", ID="ID01")
getRegex = regexFast(key_input = ['raw_text'],
key_output = 'regex_fast',
ID = "regex_fast.v1",
regexResource=resourcePath+"regexResource.txt ",
pathToPivot=resourcePath+"pivotResource.csv"
)
# add all your annotators in a list
annotators =[getRegex]
# annotate your document
thisDoc.annotate(annotators)
thisDoc.to_dict()
#write your annotation in PymedExt json
thisDoc.writeJson("outputfile.json")
LetterPyMedExt.annotate(annotators)
LetterPyMedExt.to_dict()
#write your annotation in PymedExt jsonpath="outputfolder"
try:
os.mkdir(path)
except OSError:
print ("Creation of the directory %s failed" % path)
else:
print ("Successfully created the directory %s " % path)
pymedext.brat.savetobrat(LetterPyMedExt,path)this will output three files located on outputfolder:
- xxx.txt --> the raw TextÒÒ
- xxx.ann --> the annotations
- annotation.conf
cat xxx.ann
T0 Liposarcom.V0 246 258 liposarcome
T1 Liposarcom.V0 518 530 liposarcome
T2 regex_fast 445 450 sars-cov-2pymedext -h
usage: pymedext [-h] [-i INPUTFILE] [-o OUTPUT]
[--itype {txt,pymedext,biocxml,biocjson,fhir,brat}]
[--otype {omop,pymedext,bioc,brat}] [-f] [-be BRATEXCLUDE]
[-v]
optional arguments:
-h, --help show this help message and exit
-i INPUTFILE, --inputFile INPUTFILE
path to input folder
-o OUTPUT, --output OUTPUT
enter the output file name
--itype {txt,pymedext,biocxml,biocjson,fhir,brat}
input type
--otype {omop,pymedext,bioc,brat}
output type
-f, --folder if set, the input is consider to be a folder of json
pymedext
-be BRATEXCLUDE, --bratexclude BRATEXCLUDE
list of annotations to exclude from brat
-v, --version show program's version number and exit pymedext -i demo.txt --itype txt -otype pymedext pymedext -i patient-2169591.fhir-bundle.xml --itype fhir -otype pymedext
pymedext -i patient-99912345.fhir-bundle.xml --itype fhir -otype pymedext pymedext -i demo.txt --itype txt -otype pymedext pymedext -i patient-2169591.fhir-bundle.xml --itype fhir -otype pymedext
pymedext -i patient-99912345.fhir-bundle.xml --itype fhir -otype pymedext
cd data
wget https://quaerofrenchmed.limsi.fr/QUAERO_FrenchMed_BioC.zip
unzip QUAERO_FrenchMed_BioC.zip
pymedext -i 7382743.xml --itype biocxml -otype pymedext
pymedext -i biocformat.json --itype biocjson -otype pymedext
pymedext -i QUAERO_BioC/corpus/train/MEDLINE_train_bioc --itype biocjson -otype pymedext
pymedext -i QUAERO_BioC/corpus/train/EMEA_train_bioc --itype biocjson -otype pymedext
#pymedext to bioc, need to be able to construct collection
no example
brat to bioc
It will be done on pymedext_public
- pymedext to omop
- fhir to omop
- fhir to bioc
- brat to omop
- pymedext to doccano
#local install of pymedext packages
make install
check on 21 January 2021
https://docs.docker.com/engine/install/#server
https://docs.docker.com/docker-for-mac/install/#system-requirements
https://docs.docker.com/docker-for-mac/install/#system-requirements trouble to make it work
first create a file config/.git-credentials based on the config/.git-credentials_template http:user:[email protected]
docker build -t pymedext-core:v0.0.2 .
#build docker instance
make build