SM_parser

system req

4x-8x cpu
12gb+ ram
100gm rom (ssd)
os ubuntu server 20.04

how to use

First of all- copy example config to config.py and configure that

bot

List of all current commands:

/listtg
/unstg
/subtg
/help
::bot buttons callback::

listtg

It returns answer like this. this is a list of all subscribtion of backend tg user. left of the list- channel name in the human readeble format. Right- id list. its needed to unsubscribe from source

Список подписок:
Mash | 1117628569
РТУ МИРЭА | 1248077803
КОСМОС РЯДОМ | 1357504861

unstg

it used to unsubscribe from channel. usage:

/unstg link|id

subtg

it used to subscribe from channel. usage:

/subtg link|id

run

bot

cd ./tgbot
python3 botv3.py

parser

win instructions to run flasked parser:

set FLASK_APP=parse.py
python -m flask run

neural network

note: flask parser needed to be run first

cd ./neural_network
python3 seq_to_idx.py

pre install

python -m venv ./venv/

source ./venv/bin/activate

Dependencies

python3 -m pip install absl-py anyio APScheduler argon2-cffi astunparse async-generator attrs Babel backcall bleach cachetools certifi cffi chardet click colorama DAWG-Python decorator defusedxml docopt entrypoints filelock Flask flatbuffers gast google-auth google-auth-oauthlib google-pasta grpcio h5py idna ipykernel ipython ipython-genutils itsdangerous jedi Jinja2 joblib json5 jsonschema jupyter-client jupyter-core jupyter-packaging jupyter-server jupyterlab jupyterlab-pygments jupyterlab-server Keras-Preprocessing Markdown MarkupSafe mistune nbclassic nbclient nbconvert nbformat nest-asyncio nltk notebook numpy oauthlib opt-einsum packaging pandas pandocfilters parso pickleshare pip prometheus-client prompt-toolkit protobuf pyaes pyasn1 pyasn1-modules pycparser Pygments pymongo pymorphy2 pymorphy2-dicts-ru pyparsing pyrsistent python-dateutil python-telegram-bot pytz pywinpty pyzmq regex requests requests-oauthlib rsa sacremoses scikit-learn scipy Send2Trash setuptools six sklearn sniffio telegram Telethon tensorboard tensorboard-plugin-wit tensorflow tensorflow-estimator termcolor terminado testpath threadpoolctl tokenizers torch tornado tqdm traitlets transformers typing-extensions tzlocal urllib3 wcwidth webencodings Werkzeug wheel wrapt 

Versions

Package                Version
---------------------- ------------------
absl-py                0.12.0
anyio                  2.2.0
APScheduler            3.6.3
argon2-cffi            20.1.0
astunparse             1.6.3
async-generator        1.10
attrs                  20.3.0
Babel                  2.9.0
backcall               0.2.0
bleach                 3.3.0
cachetools             4.2.1
certifi                2020.12.5
cffi                   1.14.5
chardet                4.0.0
click                  7.1.2
colorama               0.4.4
DAWG-Python            0.7.2
decorator              4.4.2
defusedxml             0.7.1
docopt                 0.6.2
entrypoints            0.3
filelock               3.0.12
Flask                  1.1.2
flatbuffers            1.12
gast                   0.3.3
google-auth            1.28.0
google-auth-oauthlib   0.4.3
google-pasta           0.2.0
grpcio                 1.32.0
h5py                   2.10.0
idna                   2.10
ipykernel              5.5.0
ipython                7.21.0
ipython-genutils       0.2.0
itsdangerous           1.1.0
jedi                   0.18.0
Jinja2                 2.11.3
joblib                 1.0.1
json5                  0.9.5
jsonschema             3.2.0
jupyter-client         6.1.12
jupyter-core           4.7.1
jupyter-packaging      0.7.12
jupyter-server         1.5.0
jupyterlab             3.0.12
jupyterlab-pygments    0.1.2
jupyterlab-server      2.3.0
Keras-Preprocessing    1.1.2
Markdown               3.3.4
MarkupSafe             1.1.1
mistune                0.8.4
nbclassic              0.2.6
nbclient               0.5.3
nbconvert              6.0.7
nbformat               5.1.2
nest-asyncio           1.5.1
nltk                   3.5
notebook               6.3.0
numpy                  1.19.5
oauthlib               3.1.0
opt-einsum             3.3.0
packaging              20.9
pandas                 1.2.3
pandocfilters          1.4.3
parso                  0.8.1
pickleshare            0.7.5
pip                    21.0.1
prometheus-client      0.9.0
prompt-toolkit         3.0.18
protobuf               3.15.6
pyaes                  1.6.1
pyasn1                 0.4.8
pyasn1-modules         0.2.8
pycparser              2.20
Pygments               2.8.1
pymongo                3.11.3
pymorphy2              0.9.1
pymorphy2-dicts-ru     2.4.417127.4579844
pyparsing              2.4.7
pyrsistent             0.17.3
python-dateutil        2.8.1
python-telegram-bot    13.4.1
pytz                   2021.1
pywin32                300
pywinpty               0.5.7
pyzmq                  22.0.3
regex                  2021.3.17
requests               2.25.1
requests-oauthlib      1.3.0
rsa                    4.7.2
sacremoses             0.0.43
scikit-learn           0.24.1
scipy                  1.6.1
Send2Trash             1.5.0
setuptools             49.2.1
six                    1.15.0
sklearn                0.0
sniffio                1.2.0
telegram               0.0.1
Telethon               1.21.1
tensorboard            2.4.1
tensorboard-plugin-wit 1.8.0
tensorflow             2.4.1
tensorflow-estimator   2.4.0
termcolor              1.1.0
terminado              0.9.3
testpath               0.4.4
threadpoolctl          2.1.0
tokenizers             0.10.1
torch                  1.8.0
tornado                6.1
tqdm                   4.59.0
traitlets              5.0.5
transformers           4.4.2
typing-extensions      3.7.4.3
tzlocal                2.1
urllib3                1.26.4
wcwidth                0.2.5
webencodings           0.5.1
Werkzeug               1.0.1
wheel                  0.36.2
wrapt                  1.12.1