leftyer/A-Tune
1
0
Fork 0
mirror of https://gitee.com/openeuler/A-Tune.git synced 2025-12-06 08:08:59 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
master
A-Tune/Dockerfile-atune-engine
hubin 701f6e45c9 Dockerfile-atune-engine: fix error that model path not exists 
Signed-off-by: hubin <hubin73@huawei.com>
2022-05-31 19:17:24 +08:00

210 lines
12 KiB
Plaintext
Raw Permalink Blame History

# Copyright (c) 2021 Huawei Technologies Co., Ltd.
# A-Tune is licensed under the Mulan PSL v2.
# You can use this software according to the terms and conditions of the Mulan PSL v2.
# You may obtain a copy of Mulan PSL v2 at:
# http://license.coscl.org.cn/MulanPSL2
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
# PURPOSE.
# See the Mulan PSL v2 for more details.
# Create: 2021-01-14
#
# Dockerfile for building atune-engine docker image.
#
# Usage:
# docker build -f Dockerfile-atune-engine -t atune-engine .
# docker run -p [<HOST_IP>:]<HOST_PORT>:3838 -e ENGINE_TLS=false atune-engine
#
# Supported running arguments:
# -e PORT=<CONTAINER_PORT> The atune-engine running port inside container, should be consistent with the second port of -p, default 3838
#
# -e ENGINE_TLS=<true/false> Whether using TLS to secure atuned client connection, default true
# -e ENGINE_TLS_CA_CERT_FILE=<path> TLS ca.crt file path, default /etc/atuned/engine_certs/ca.crt
# -e ENGINE_TLS_SERVER_CERT_FILE=<path> TLS server.crt file path, default /etc/atuned/engine_certs/server.crt
# -e ENGINE_TLS_SERVER_KEY_FILE=<path> TLS server.key file path, default /etc/atuned/engine_certs/server.key
# -e ENGINE_IP_ADDR=<ip addr> Binded IP/domain of newly generated TLS certificates, default localhost
#
# -e DB_ENABLE=<true/false> Whether using database to store atune-engine data, default false
# -e DB_HOST=<ip address> Database host ip address, default localhost
# -e DB_PORT=<port number> Database running port, default 5432
# -e DB_NAME=<database name> Database name used to store atune-engine tables, default atune_db
# -e DB_USER_NAME=<username> Database login username, default admin
# -e DB_USER_PASSWD=<password> Database login password, default no password, add if necessary
#
# Running example:
# 1. In atune-engine server host, build atune-engine docker image:
# docker build -f Dockerfile-atune-engine -t atune-engine:latest .
#
# 2. Run atune-engine service container at container port 3838, which is mapped to host port 3737:
# docker run -p 3737:3838 -e ENGINE_TLS=false atune-engine
#
# The output will be like this if the atune-engine container is successfully started:
# Starting enhanced syslogd: rsyslogd.
# * Serving Flask app "analysis.app" (lazy loading)
# * Environment: production
# WARNING: This is a development server. Do not use it in a production deployment.
# Use a production WSGI server instead.
# * Debug mode: off
#
# 3. In atuned client host, install A-tune service, and update atuned configuration(/etc/atuned/atuned.cnf):
# engine_host = <atune-engine host ip>
# engine_port = 3737
#
# 4. In atuned client host, start atuned service and run default atune analysis:
# systemctl daemon-reload
# systemctl start atuned
# atune-adm analysis
#
# 5. In atune-engine host, check analysis log in atune-engine container:
# docker ps
# docker exec -it <atune-engine container ID> /bin/bash
# cat /var/log/messages
#
# You will get log record like following if atune-engine successfully finishes the analysis:
# ...
# Jan 15 07:04:33 f1e85f6c7a4d kernel: [347948.341795] docker0: port 2(veth8472648) entered disabled state
# Jan 15 07:04:33 f1e85f6c7a4d kernel: [347948.351181] device veth8472648 left promiscuous mode
# Jan 15 07:04:33 f1e85f6c7a4d kernel: [347948.354295] docker0: port 2(veth8472648) entered disabled state
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,965 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:37] : ImmutableMultiDict([('file', <FileStorage: 'test-1610703229683.csv' ('application/octet-stream')>)])
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,967 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:47] : /var/atune_data/analysis//test-1610703229683.csv
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,967 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:48] : 9.85.179.193
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,973 [INFO] classification [/home/A-Tune/analysis/../analysis/engine/classification.py:42] : {'modelpath': '/usr/libexec/atuned/analysis/models', 'data': '/var/atune_data/analysis//test-1610703229683.csv', 'model': None}
# Jan 15 09:33:50 f1e85f6c7a4d atuned: 2021-01-15 09:33:50,779 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:37] : ImmutableMultiDict([('file', <FileStorage: 'test-1610442581644.log' ('application/octet-stream')>)])
# Jan 15 09:33:50 f1e85f6c7a4d atuned: 2021-01-15 09:33:50,780 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:47] : /var/atune_data/analysis//test-1610442581644.log
# Jan 15 09:33:50 f1e85f6c7a4d atuned: 2021-01-15 09:33:50,780 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:48] : 9.85.179.193
#
# Running example for using database:
# docker run -p 3737:3838 -e ENGINE_TLS=false -e DB_ENABLE=true -e DB_HOST=<database ip> -e DB_PORT=5432 -e DB_NAME=atune_db \
# -e DB_USER_NAME=admin -e DB_USER_PASSWD=Ha123456# atune-engine
#
# Running example for using TLS:
# docker run -p <host ip>:3838:3838 -v /path/to/existing/tls/certs:/etc/atuned/engine_certs atune-engine
# here /path/to/existing/tls/certs is the atune-engine host directory where ca.crt, server.crt, server.key files are provided.
# these certificate files should be consisted with those in atuned client machine.
#
# docker run -p <host ip>:3838:3838 -e ENGINE_IP_ADDR=<host ip> -v /path/to/generate/new/tls/certs:/etc/atuned/engine_certs atune-engine
# here /path/to/generate/new/tls/certs is an empty atune-engine host directory where ca.crt, server.crt, server.key files will be generated.
# later we need to copy ca.crt, client.crt, client.key in the directory to atuned client machine and update atuned configuration.
#
# base image
FROM python:3.8-slim
# update Debian source
RUN echo 'deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster main' > /etc/apt/sources.list && \
echo 'deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main' >> /etc/apt/sources.list && \
echo 'deb https://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main' >> /etc/apt/sources.list
# update pip source
RUN pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
# install python3 library dependencies and rsyslog service
RUN apt update && \
apt install -y --no-install-recommends rsyslog gcc g++ && \
pip3 install --no-cache-dir Flask-RESTful scipy==1.5.4 pandas scikit-learn==0.23.2 scikit-optimize xgboost pyyaml && \
apt purge -y gcc g++ && \
apt clean
# in this RUN, we pull latest code from https://gitee.com/openeuler/A-Tune, and generate models
# then keep necessary files, remove useless files and cache to reduce image size
RUN cd ~ && \
apt install -y --no-install-recommends git && \
git -c http.sslVerify=false clone https://gitee.com/openeuler/A-Tune.git --depth=1 && \
cd ~/A-Tune/tools && \
mkdir -p ../analysis/models/ && \
python3 generate_models.py && \
mkdir -p /usr/libexec/atuned/analysis && \
mv ~/A-Tune/analysis/models /usr/libexec/atuned/analysis/ && \
chmod -R 750 /usr/libexec/atuned/analysis/* && \
cd ~/A-Tune && \
rm -rf analysis/__pycache__ analysis/engine/__pycache__ analysis/engine/utils/__pycache__ analysis/optimizer/__pycache__/&& \
rm -rf analysis/app_rest.py analysis/atuned/ analysis/dataset/ && \
mkdir -p /etc/atuned && \
cp ~/A-Tune/misc/engine.cnf /etc/atuned/ && \
mkdir -p ~/atune-engine && \
cp -r ~/A-Tune/analysis ~/atune-engine && \
cp -r ~/A-Tune/License ~/atune-engine && \
mkdir -p ~/atune-engine/tools && \
cp ~/A-Tune/tools/encrypt.py ~/atune-engine/tools/ && \
cd ~ && \
rm -rf ~/A-Tune && \
apt purge -y git && \
apt clean
# default ENV variables in /etc/atuned/engine.cnf
ENV PORT=3838
ENV ENGINE_TLS=true
ENV ENGINE_TLS_CA_CERT_FILE=/etc/atuned/engine_certs/ca.crt
ENV ENGINE_TLS_SERVER_CERT_FILE=/etc/atuned/engine_certs/server.crt
ENV ENGINE_TLS_SERVER_KEY_FILE=/etc/atuned/engine_certs/server.key
ENV ENGINE_IP_ADDR=localhost
ENV DB_ENABLE=false
ENV DB_HOST=localhost
ENV DB_PORT=5432
ENV DB_NAME=atune_db
ENV DB_USER_NAME=admin
# atuned client should communicate with atune-engine through <host ip>:3838
EXPOSE ${PORT}/tcp
# remove imklog module in rsyslogd service to avoid root permission need
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf
# update engine.cnf, add tls certificates and database support if needed, start rsyslog service and atune-engine
CMD sed -i "/^engine_host = /c\engine_host = 0.0.0.0" /etc/atuned/engine.cnf && \
sed -i "/^engine_port = /c\engine_port = $PORT" /etc/atuned/engine.cnf && \
sed -i "/^engine_tls = /c\engine_tls = $ENGINE_TLS" /etc/atuned/engine.cnf && \
sed -i "/^tlsenginecacertfile = /c\tlsenginecacertfile = $ENGINE_TLS_CA_CERT_FILE" /etc/atuned/engine.cnf && \
sed -i "/^tlsengineservercertfile = /c\tlsengineservercertfile = $ENGINE_TLS_SERVER_CERT_FILE" /etc/atuned/engine.cnf && \
sed -i "/^tlsengineserverkeyfile = /c\tlsengineserverkeyfile = $ENGINE_TLS_SERVER_KEY_FILE" /etc/atuned/engine.cnf && \
(if [ $ENGINE_TLS = true ]; then \
ENGINE_CERT_PATH=`dirname $ENGINE_TLS_CA_CERT_FILE`; \
(if [ ! -f $ENGINE_TLS_CA_CERT_FILE ]; then \
mkdir -p $ENGINE_CERT_PATH; \
openssl genrsa -out $ENGINE_CERT_PATH/ca.key 2048; \
openssl req -new -x509 -days 3650 -subj "/CN=ca" -key $ENGINE_CERT_PATH/ca.key -out $ENGINE_CERT_PATH/ca.crt; \
fi); \
(if [ ! -f $ENGINE_TLS_SERVER_CERT_FILE ] || [ ! -f $ENGINE_TLS_SERVER_KEY_FILE ]; then \
(for name in server client; do \
openssl genrsa -out $ENGINE_CERT_PATH/$name.key 2048; \
(if [ $ENGINE_IP_ADDR = localhost ]; then \
openssl req -new -subj "/CN=localhost" -key $ENGINE_CERT_PATH/$name.key -out $ENGINE_CERT_PATH/$name.csr; \
openssl x509 -req -sha256 -CA $ENGINE_CERT_PATH/ca.crt -CAkey $ENGINE_CERT_PATH/ca.key -CAcreateserial -days 3650 \
-in $ENGINE_CERT_PATH/$name.csr -out $ENGINE_CERT_PATH/$name.crt; \
else \
openssl req -new -subj "/CN=${ENGINE_IP_ADDR}" -key $ENGINE_CERT_PATH/$name.key -out $ENGINE_CERT_PATH/$name.csr; \
echo "subjectAltName=IP:${ENGINE_IP_ADDR}" > $ENGINE_CERT_PATH/extfile.cnf; \
openssl x509 -req -sha256 -CA $ENGINE_CERT_PATH/ca.crt -CAkey $ENGINE_CERT_PATH/ca.key -CAcreateserial -days 3650 \
-extfile $ENGINE_CERT_PATH/extfile.cnf -in $ENGINE_CERT_PATH/$name.csr -out $ENGINE_CERT_PATH/$name.crt; \
fi); \
done); \
rm -rf $ENGINE_CERT_PATH/*.srl $ENGINE_CERT_PATH/*.csr $ENGINE_CERT_PATH/extfile.cnf; \
fi); \
fi) && \
sed -i "/^db_enable = /c\db_enable = $DB_ENABLE" /etc/atuned/engine.cnf && \
sed -i "/^db_host = /c\db_host = $DB_HOST" /etc/atuned/engine.cnf && \
sed -i "/^db_port = /c\db_port = $DB_PORT" /etc/atuned/engine.cnf && \
sed -i "/^db_name = /c\db_name = $DB_NAME" /etc/atuned/engine.cnf && \
(if [ $DB_ENABLE = true ]; then \
apt update && \
apt install -y gcc libpq-dev && \
pip3 install sqlalchemy cryptography psycopg2-binary; \
fi) && \
sed -i "/user_name = /c\user_name = $DB_USER_NAME" /etc/atuned/engine.cnf && \
(if [ $DB_USER_PASSWD ]; then \
res=`python3 ~/atune-engine/tools/encrypt.py -e $DB_USER_PASSWD | awk '{print $2}'`; \
pwd=`echo $res | cut -d' ' -f 1`; \
key=`echo $res | cut -d' ' -f 2`; \
iv=`echo $res | cut -d' ' -f 3`; \
sed -i "/user_passwd = /c\user_passwd = $pwd" /etc/atuned/engine.cnf; \
sed -i "/passwd_key = /c\passwd_key = $key" /etc/atuned/engine.cnf; \
sed -i "/passwd_iv = /c\passwd_iv = $iv" /etc/atuned/engine.cnf; \
fi) && \
service rsyslog start && \
python3 ~/atune-engine/analysis/app_engine.py /etc/atuned/engine.cnf
Reference in New Issue View Git Blame Copy Permalink