Docker ImageからContainerを作るコマンドのまとめ - minus9d's diary の続きです。今回はKeras環境を備えるDocker Imageを作成する練習をしてみました。普通であればTensorflow公式のDocker Imageを使うのが正しいと思いますが、練習なので気にしないことにします。
この記事では私の試行錯誤の過程をそのまま記述しています。最終的なDockerfileは末尾に記載しています。
最初のDockerfile
まず、以下のようなDockerfileを作成します。
FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04
RUN apt-get update && apt-get install -y \
python3-pip \
wget
RUN pip3 install tensorflow-gpu && pip3 install keras
RUN wget https://raw.githubusercontent.com/keras-team/keras/master/examples/cifar10_cnn.py
CMD python3 cifar10_cnn.py
以下、Dockerfileの簡単な解説です。
一行目のFROM nvidia/cudaで、Docker Hub にあるNVIDIA公式のCUDAの環境をベースイメージとして指定します。
二行目のapt-getで、あとで必要になるバイナリの習得を行います。つい手癖でapt-get updateの後はapt-get upgradeしたくなりますが、Best practices for writing Dockerfilesによるとapt-get upgradeは避けるべきだそうです。また、apt-get updateと、apt-get install -y XXXは同じRUN命令で同時に実行すべきだそうです。
三行目でtensorflow-gpuとkerasをインストールします。
四行目でkerasのサンプルスクリプトをダウンロードします。
次に、このDockerfileをもとにDocker Imageを作成します。このDockerfileと同じディレクトリにて
$ docker build -t mykerasimage .
とすることで、mykerasimageという名前のDocker Imageを作成します。
次に、このDocker Imageから、コンテナを作成して実行します。
$ docker run -it mykerasimage
とすると、Dockerfileの最終行に書いたpython3 cifar10_cnn.pyがコンテナ内で実行されます。
以下はそのときのログです。
$ docker run -it mykerasimage
Using TensorFlow backend.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
170500096/170498071 [==============================] - 130s 1us/step
x_train shape: (50000, 32, 32, 3)
50000 train samples
10000 test samples
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
Using real-time data augmentation.
2020-04-07 14:16:02.853936: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-07 14:16:02.857300: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-04-07 14:16:02.857318: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
2020-04-07 14:16:02.857334: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
2020-04-07 14:16:02.875675: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2020-04-07 14:16:02.876032: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x453cdb0 executing computations on platform Host. Devices:
2020-04-07 14:16:02.876049: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2020-04-07 14:16:02.915462: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
Epoch 1/100
1563/1563 [==============================] - 63s 40ms/step - loss: 1.8523 - accuracy: 0.3178 - val_loss: 1.5431 - val_accuracy: 0.4363
Epoch 2/100
1563/1563 [==============================] - 63s 40ms/step - loss: 1.5708 - accuracy: 0.4255 - val_loss: 1.3986 - val_accuracy: 0.4933
Epoch 3/100
1563/1563 [==============================] - 62s 40ms/step - loss: 1.4585 - accuracy: 0.4721 - val_loss: 1.3288 - val_accuracy: 0.5212
Epoch 4/100
1563/1563 [==============================] - 62s 40ms/step - loss: 1.3764 - accuracy: 0.5028 - val_loss: 1.2310 - val_accuracy: 0.5604
（以下略）
このログをよく見ると、Could not dlopen library 'libcuda.so.1' failed call to cuInit: UNKNOWN ERROR (303) などの不穏なエラーが並んでいることがわかります。これは、私が NVIDIA Docker の環境を作らなかったことが原因だと思われます。ただ、これらのエラーが出るにもかかわらずサンプルファイルは実行できているようにも見え、謎です…。
NVIDIA Dockerの導入
NVIDIA Dockerの導入は、NVIDIAの中の人が記した NVIDIA Docker って今どうなってるの？ (19.11版) - Qiita が歴史的経緯も含めてまとめられていてわかりやすいです。今回は「Docker 19.03 以降の環境で前だけを見て生きる場合」に従い、NVIDIA Dockerの追加インストールを行いました。
そして今度は、コンテナ内で使用するGPUを指定するオプション--gpuを付与して、改めてコンテナを作ります。
$ docker run --gpus all -it mykerasimage
以下はその実行結果です。さきほどの不穏なエラーは消えましたが、代わりにlibcudart.so.10.0が見つからないというエラーなどが出ました。
（略） Using real-time data augmentation. 2020-04-07 14:30:25.187308: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-04-07 14:30:25.191128: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1 2020-04-07 14:30:25.281694: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-07 14:30:25.282141: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x34f5f90 executing computations on platform CUDA. Devices: 2020-04-07 14:30:25.282156: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1 2020-04-07 14:30:25.303626: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2020-04-07 14:30:25.303940: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3e90720 executing computations on platform Host. Devices: 2020-04-07 14:30:25.303972: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> 2020-04-07 14:30:25.304181: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-07 14:30:25.304464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683 pciBusID: 0000:01:00.0 2020-04-07 14:30:25.304599: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-07 14:30:25.304673: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-07 14:30:25.304742: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-07 14:30:25.304810: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-07 14:30:25.304880: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-07 14:30:25.304950: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-07 14:30:25.395108: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2020-04-07 14:30:25.395193: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices... 2020-04-07 14:30:25.395243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-04-07 14:30:25.395275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2020-04-07 14:30:25.395302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2020-04-07 14:30:25.472855: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. Epoch 1/100 1563/1563 [==============================] - 63s 40ms/step - loss: 1.9029 - accuracy: 0.3008 - val_loss: 1.5870 - val_accuracy: 0.4238 Epoch 2/100 1563/1563 [==============================] - 63s 40ms/step - loss: 1.5875 - accuracy: 0.4180 - val_loss: 1.3819 - val_accuracy: 0.4944
$ docker run --gpus all -it mykerasimage bashでコンテナの中を探ってみると、libcudart.so.10.0は存在せず、かわりに/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2があるのみでした。また、
tensorflow-gpuのバージョンが1.14.0と古いことに気づきました。tensorflow-gpu 1.14.0はCUDA 10.0を要求するにもかかわらず、CUDA 10.2のベースイメージを用意してしまったのが失敗でした。
ベースイメージの見直しとpip3の更新
この反省をもとに、現時点で最新である tensorflow-gpu 2.1 と、これが要求するCUDA 10.1の組み合わせを実現するDockerfileを書き直すことにしました。
書き直したDockerfileを以下に示します。ベースイメージをCUDA 10.1に変更し、かつ、pip3を最新版に更新（現時点で20.0.2）してから、tensorflow-gpuとkerasをバージョン指定で入れるようにしました。
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
RUN apt-get update && apt-get install -y \
python3-pip \
wget
RUN pip3 install --upgrade pip
RUN pip3 install tensorflow-gpu && pip3 install keras
RUN wget https://raw.githubusercontent.com/keras-team/keras/master/examples/cifar10_cnn.py
CMD python3 cifar10_cnn.py
このDockerfileをもとにDocker Imageを再作成し、docker run --gpus all -it mykerasimageしたときのログを以下に示します。まだlibnvinfer.so.6, libnvinfer_plugin.so.6 などがロードできない旨のエラーが出ていますが、それ以外は正しく動いていそうです。
$ docker run --gpus all -it mykerasimage Using TensorFlow backend. 2020-04-08 01:11:33.204351: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-08 01:11:33.204485: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2020-04-08 01:11:33.204496: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz 170500096/170498071 [==============================] - 91s 1us/step x_train shape: (50000, 32, 32, 3) 50000 train samples 10000 test samples 2020-04-08 01:13:07.198910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-04-08 01:13:07.217257: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.217949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1 coreClock: 1.683GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s 2020-04-08 01:13:07.219716: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-04-08 01:13:07.273601: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-04-08 01:13:07.308268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-04-08 01:13:07.317546: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-04-08 01:13:07.384937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-04-08 01:13:07.393726: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-04-08 01:13:07.493477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-04-08 01:13:07.493865: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.495793: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.497123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0 2020-04-08 01:13:07.497893: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-04-08 01:13:07.538725: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2020-04-08 01:13:07.539661: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x40f4d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-04-08 01:13:07.539710: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-04-08 01:13:07.637951: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.638329: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x417adc0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-04-08 01:13:07.638342: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1 2020-04-08 01:13:07.638458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.638711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1 coreClock: 1.683GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s 2020-04-08 01:13:07.638738: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-04-08 01:13:07.638750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-04-08 01:13:07.638759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-04-08 01:13:07.638770: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-04-08 01:13:07.638785: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-04-08 01:13:07.638808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-04-08 01:13:07.638827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-04-08 01:13:07.638878: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.639220: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.639473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0 2020-04-08 01:13:07.639501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-04-08 01:13:07.641096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-04-08 01:13:07.641107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 2020-04-08 01:13:07.641113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N 2020-04-08 01:13:07.641218: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.641555: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-08 01:13:07.641833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7011 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1) Using real-time data augmentation. Epoch 1/100 2020-04-08 01:13:11.232517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-04-08 01:13:11.745019: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 1563/1563 [==============================] - 19s 12ms/step - loss: 1.8952 - accuracy: 0.3029 - val_loss: 1.5868 - val_accuracy: 0.4233 Epoch 2/100 1563/1563 [==============================] - 17s 11ms/step - loss: 1.6196 - accuracy: 0.4084 - val_loss: 1.4305 - val_accuracy: 0.4763 Epoch 3/100 1563/1563 [==============================] - 17s 11ms/step - loss: 1.4915 - accuracy: 0.4593 - val_loss: 1.3402 - val_accuracy: 0.5222
