.ipynb .pdf

repository open issue

Binder

Contents

RE task 기초 베이스라인

• Task : KLUE-RE

• 담당자: 이승미 님

• 최종수정일: 21-09-15

• 본 자료는 가짜연구소 3기 KLUE 로 모델 평가하기 크루 활동으로 작성됨

00 Relation Task

• 목표 : - 모델이 적합하게 두 개체의 관계를 이해했는지 평가하기에 적합

• 데이터 구성 : 위키피디아, 위키트리(뉴스 도메인), 정책 브리핑(뉴스 도메인)

  • Train : 32470

  • Dev : 7765

  • Test : 7766

• 평가지표 :

  • Micro F1 : Major Negative class(no relation) 을 제외하고 score 측정

  • AUPRC(area under the precision-recall curve) : x축을 Recall, y축을 Precision으로 설정하여 그린 곡선 아래의 면적 값인 모델 평가

• Relation

  • person-related relations 18개

  • organization-related relations 11개

  • no_relation

  

01 init

공식 KLUE github 의 KLUE-baseline 을 바탕으로 작성

---

Install packages

공식 KLUE github에서 pipline과 dataset을 clon

!git clone --recursive https://github.com/KLUE-benchmark/KLUE-Baseline.git

Cloning into 'KLUE-Baseline'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (74/74), done.
remote: Compressing objects: 100% (63/63), done.
remote: Total 74 (delta 18), reused 63 (delta 10), pack-reused 0
Unpacking objects: 100% (74/74), done.
Submodule 'data' (https://github.com/KLUE-benchmark/KLUE) registered for path 'data'
Cloning into '/content/KLUE-Baseline/data'...
remote: Enumerating objects: 154, done.        
remote: Counting objects: 100% (154/154), done.        
remote: Compressing objects: 100% (134/134), done.        
remote: Total 154 (delta 42), reused 42 (delta 4), pack-reused 0        
Receiving objects: 100% (154/154), 41.65 MiB | 14.77 MiB/s, done.
Resolving deltas: 100% (42/42), done.
Submodule path 'data': checked out '1cc52e64c0e0b6915577244f7439c55a42199a64'

!pip install -r KLUE-Baseline/requirements.txt

Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
Collecting torch==1.7.0
  Downloading torch-1.7.0-cp37-cp37m-manylinux1_x86_64.whl (776.7 MB)
     |████████████████████████████████| 776.7 MB 4.6 kB/s 
?25hCollecting transformers==3.5.1
  Downloading transformers-3.5.1-py3-none-any.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 38.0 MB/s 
?25hCollecting pytorch-lightning==1.1.0
  Downloading pytorch_lightning-1.1.0-py3-none-any.whl (665 kB)
     |████████████████████████████████| 665 kB 38.7 MB/s 
?25hCollecting fsspec==2021.4.0
  Downloading fsspec-2021.4.0-py3-none-any.whl (108 kB)
     |████████████████████████████████| 108 kB 51.7 MB/s 
?25hCollecting overrides==3.1.0
  Downloading overrides-3.1.0.tar.gz (11 kB)
Collecting scikit-learn==0.24.1
  Downloading scikit_learn-0.24.1-cp37-cp37m-manylinux2010_x86_64.whl (22.3 MB)
     |████████████████████████████████| 22.3 MB 1.6 MB/s 
?25hCollecting seqeval
  Downloading seqeval-1.2.2.tar.gz (43 kB)
     |████████████████████████████████| 43 kB 2.2 MB/s 
?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.0->-r KLUE-Baseline/requirements.txt (line 2)) (3.7.4.3)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from torch==1.7.0->-r KLUE-Baseline/requirements.txt (line 2)) (0.16.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch==1.7.0->-r KLUE-Baseline/requirements.txt (line 2)) (1.19.5)
Collecting dataclasses
  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)
Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (3.17.3)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (4.62.3)
Collecting tokenizers==0.9.3
  Downloading tokenizers-0.9.3-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 40.4 MB/s 
?25hCollecting sentencepiece==0.1.91
  Downloading sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 47.5 MB/s 
?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (3.0.12)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (2019.12.20)
Collecting sacremoses
  Downloading sacremoses-0.0.46-py3-none-any.whl (895 kB)
     |████████████████████████████████| 895 kB 43.2 MB/s 
?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (21.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (2.23.0)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (2.6.0)
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 33.7 MB/s 
?25hCollecting PyYAML>=5.1
  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
     |████████████████████████████████| 636 kB 37.8 MB/s 
?25hRequirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.1->-r KLUE-Baseline/requirements.txt (line 12)) (1.0.1)
Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.1->-r KLUE-Baseline/requirements.txt (line 12)) (1.4.1)
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-2.2.0-py3-none-any.whl (12 kB)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (3.3.4)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (0.6.1)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (0.12.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (0.4.6)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (57.4.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (1.8.0)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (1.40.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (1.0.1)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (0.37.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (1.35.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (1.15.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (4.2.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (4.7.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (1.3.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (4.8.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (0.4.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (2021.5.30)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (1.24.3)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (3.1.1)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r KLUE-Baseline/requirements.txt (line 4)) (3.5.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (2.4.7)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.5.1->-r KLUE-Baseline/requirements.txt (line 3)) (7.1.2)
Building wheels for collected packages: overrides, future, seqeval
  Building wheel for overrides (setup.py) ... ?25l?25hdone
  Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10186 sha256=1856d718b4354bc7d87fb0bb6922c02fa647b49d6d85eb24130fe7eb759daf04
  Stored in directory: /root/.cache/pip/wheels/3a/0d/38/01a9bc6e20dcfaf0a6a7b552d03137558ba1c38aea47644682
  Building wheel for future (setup.py) ... ?25l?25hdone
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=4baa443ac4f5e2ab6838916ae98878d6d9a8a7102e2a0d6bcc643f2b7c3dd535
  Stored in directory: /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
  Building wheel for seqeval (setup.py) ... ?25l?25hdone
  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16181 sha256=2bb9dfeb18c376a6ced4469c857d32915b45bf1384f165603823aba304bcb75e
  Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
Successfully built overrides future seqeval
Installing collected packages: threadpoolctl, future, dataclasses, torch, tokenizers, sentencepiece, scikit-learn, sacremoses, PyYAML, fsspec, transformers, seqeval, pytorch-lightning, overrides
  Attempting uninstall: future
    Found existing installation: future 0.16.0
    Uninstalling future-0.16.0:
      Successfully uninstalled future-0.16.0
  Attempting uninstall: torch
    Found existing installation: torch 1.9.0+cu102
    Uninstalling torch-1.9.0+cu102:
      Successfully uninstalled torch-1.9.0+cu102
  Attempting uninstall: scikit-learn
    Found existing installation: scikit-learn 0.22.2.post1
    Uninstalling scikit-learn-0.22.2.post1:
      Successfully uninstalled scikit-learn-0.22.2.post1
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 3.13
    Uninstalling PyYAML-3.13:
      Successfully uninstalled PyYAML-3.13
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.10.0+cu102 requires torch==1.9.0, but you have torch 1.7.0 which is incompatible.
torchtext 0.10.0 requires torch==1.9.0, but you have torch 1.7.0 which is incompatible.
Successfully installed PyYAML-5.4.1 dataclasses-0.6 fsspec-2021.4.0 future-0.18.2 overrides-3.1.0 pytorch-lightning-1.1.0 sacremoses-0.0.46 scikit-learn-0.24.1 sentencepiece-0.1.91 seqeval-1.2.2 threadpoolctl-2.2.0 tokenizers-0.9.3 torch-1.7.0 transformers-3.5.1

!pip install pytorch-lightning==1.2.2 torchtext==0.8.0 torch==1.7.1

Collecting pytorch-lightning==1.2.2
  Downloading pytorch_lightning-1.2.2-py3-none-any.whl (816 kB)
     |████████████████████████████████| 816 kB 10.9 MB/s 
?25hCollecting torchtext==0.8.0
  Downloading torchtext-0.8.0-cp37-cp37m-manylinux1_x86_64.whl (6.9 MB)
     |████████████████████████████████| 6.9 MB 1.2 MB/s 
?25hCollecting torch==1.7.1
  Downloading torch-1.7.1-cp37-cp37m-manylinux1_x86_64.whl (776.8 MB)
     |████████████████████████████████| 776.8 MB 17 kB/s 
?25hCollecting PyYAML!=5.4.*,>=5.1
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
     |████████████████████████████████| 269 kB 43.1 MB/s 
?25hRequirement already satisfied: fsspec[http]>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (2021.4.0)
Requirement already satisfied: tqdm>=4.41.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (4.62.3)
Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (1.19.5)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (2.6.0)
Requirement already satisfied: future>=0.17.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (0.18.2)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torchtext==0.8.0) (2.23.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1) (3.7.4.3)
Collecting aiohttp
  Downloading aiohttp-3.7.4.post0-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 39.6 MB/s 
?25hRequirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.35.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (57.4.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.12.0)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.17.3)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.40.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.8.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.6.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.0.1)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.37.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.3.4)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.15.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (4.7.2)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (4.2.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.3.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (4.8.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.4.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (2021.5.30)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.1.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning==1.2.2) (21.2.0)
Collecting multidict<7.0,>=4.5
  Downloading multidict-5.1.0-cp37-cp37m-manylinux2014_x86_64.whl (142 kB)
     |████████████████████████████████| 142 kB 43.8 MB/s 
?25hCollecting async-timeout<4.0,>=3.0
  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting yarl<2.0,>=1.0
  Downloading yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294 kB)
     |████████████████████████████████| 294 kB 45.3 MB/s 
?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.5.0)
Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (setup.py) ... ?25l?25hdone
  Created wheel for PyYAML: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44635 sha256=a549b6783a2c91e8df5d04fb4c3a55b7d289885ace3bcb6db176cedde3f7583c
  Stored in directory: /root/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
Successfully built PyYAML
Installing collected packages: multidict, yarl, async-timeout, aiohttp, torch, PyYAML, torchtext, pytorch-lightning
  Attempting uninstall: torch
    Found existing installation: torch 1.7.0
    Uninstalling torch-1.7.0:
      Successfully uninstalled torch-1.7.0
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 5.4.1
    Uninstalling PyYAML-5.4.1:
      Successfully uninstalled PyYAML-5.4.1
  Attempting uninstall: torchtext
    Found existing installation: torchtext 0.10.0
    Uninstalling torchtext-0.10.0:
      Successfully uninstalled torchtext-0.10.0
  Attempting uninstall: pytorch-lightning
    Found existing installation: pytorch-lightning 1.1.0
    Uninstalling pytorch-lightning-1.1.0:
      Successfully uninstalled pytorch-lightning-1.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.10.0+cu102 requires torch==1.9.0, but you have torch 1.7.1 which is incompatible.
Successfully installed PyYAML-5.3.1 aiohttp-3.7.4.post0 async-timeout-3.0.1 multidict-5.1.0 pytorch-lightning-1.2.2 torch-1.7.1 torchtext-0.8.0 yarl-1.6.3

dataset

KLUE RE 데이터셋

• Train : klue-re-v1.1_train.json

• Dev : klue-re-v1.1_dev.json

import os
import json

dataset_path = os.path.join('/content/KLUE-Baseline/data/klue_benchmark/klue-re-v1.1/klue-re-v1.1_train.json')
with open(dataset_path, "r", encoding="utf-8") as f:
    dataset = json.load(f)

• label

• object_entity
  subject entity

  • index

  • type

  • word

• source

• sentence

dataset[0]

{'guid': 'klue-re-v1_train_00000',
 'label': 'no_relation',
 'object_entity': {'end_idx': 18,
  'start_idx': 13,
  'type': 'PER',
  'word': '조지 해리슨'},
 'sentence': '〈Something〉는 조지 해리슨이 쓰고 비틀즈가 1969년 앨범 《Abbey Road》에 담은 노래다.',
 'source': 'wikipedia',
 'subject_entity': {'end_idx': 26,
  'start_idx': 24,
  'type': 'ORG',
  'word': '비틀즈'}}

Input

• 문장

• Entity Span에 대한 정보

  • entity의 위치 (start idx, end idx)

  • entity의 위치를 표현하기 위해서 entity special token을 이용 ex) <obj>마크 주커버그<obj> 는 2004년 <subj> 페이스북 </subj> 을 설립했다.

import sys

sys.path.insert(0, '/content/KLUE-Baseline/klue_baseline/data')

import argparse
import json
import logging
import os
from typing import Any, List, Optional, Tuple

import torch
from overrides import overrides
from torch.utils.data import TensorDataset
from transformers import PreTrainedTokenizer

from base import DataProcessor, InputExample, InputFeatures, KlueDataModule

Checks tokenizer type \

• 해당 baselinde에서는 wordpiece, sentencepiece 만 지원

import logging
from typing import List, Optional, Union

import transformers
from transformers import PreTrainedTokenizer

logger = logging.getLogger(__name__)

def check_tokenizer_type(tokenizer: PreTrainedTokenizer) -> str:

    if isinstance(tokenizer, transformers.XLMRobertaTokenizer):
        logger.info(f"Using {type(tokenizer).__name__} for fixing tokenization result")
        return "xlm-sp"  # Sentencepiece
    elif isinstance(tokenizer, transformers.BertTokenizer):
        logger.info(f"Using {type(tokenizer).__name__} for fixing tokenization result")
        return "bert-wp"  # Wordpiece (including BertTokenizer & ElectraTokenizer)
    else:
        logger.warn(
            "If you are using other tokenizer (e.g. bbpe), you should change code in `fix_tokenization_error()`"
        )
        return "other"

<obj> <subj> 토큰 추가

class KlueREProcessor(DataProcessor):

    datamodule_type = KlueDataModule

    def __init__(self, tokenizer: PreTrainedTokenizer) -> None:
        super().__init__(self, tokenizer)

        # special tokens to mark the subject/object entity boundaries
        self.subject_start_marker = "<subj>"
        self.subject_end_marker = "</subj>"
        self.object_start_marker = "<obj>"
        self.object_end_marker = "</obj>"

        self.tokenizer.add_special_tokens(
            {
                "additional_special_tokens": [
                    self.subject_start_marker,
                    self.subject_end_marker,
                    self.object_start_marker,
                    self.object_end_marker,
                ]
            }
        )

        # Load relation class
        relation_class_file_path = os.path.join('/content/KLUE-Baseline/data/klue_benchmark/klue-re-v1.1/relation_list.json')

        with open(relation_class_file_path, "r", encoding="utf-8") as f:
            self.relation_class = json.load(f)["relations"]

        # Check type of tokenizer
        self.tokenizer_type = check_tokenizer_type(tokenizer)

    overrides
    def get_labels(self) -> Any:
        return self.relation_class
    

    def _create(self, data) -> List[InputExample]:
        examples = []
        guid = data["guid"]
        text = data["sentence"]
        subject_entity = data["subject_entity"]
        object_entity = data["object_entity"]
        label = data["label"]

        text = self._mark_entity_spans(
            text=text,
            subject_range=(int(subject_entity["start_idx"]), int(subject_entity["end_idx"])),
            object_range=(int(object_entity["start_idx"]), int(object_entity["end_idx"])),
        )
        examples.append(InputExample(guid=guid, text_a=text, label=label))

        return examples

    def _mark_entity_spans(
        self,
        text: str,
        subject_range: Tuple[int, int],
        object_range: Tuple[int, int],
    ) -> str:
        """Adds entity markers to the text to identify the subject/object entities.
        Args:
            text: Original sentence
            subject_range: Pair of start and end indices of subject entity
            object_range: Pair of start and end indices of object entity
        Returns:
            A string of text with subject/object entity markers
        """
        if subject_range < object_range:
            segments = [
                text[: subject_range[0]],
                self.subject_start_marker,
                text[subject_range[0] : subject_range[1] + 1],
                self.subject_end_marker,
                text[subject_range[1] + 1 : object_range[0]],
                self.object_start_marker,
                text[object_range[0] : object_range[1] + 1],
                self.object_end_marker,
                text[object_range[1] + 1 :],
            ]
        elif subject_range > object_range:
            segments = [
                text[: object_range[0]],
                self.object_start_marker,
                text[object_range[0] : object_range[1] + 1],
                self.object_end_marker,
                text[object_range[1] + 1 : subject_range[0]],
                self.subject_start_marker,
                text[subject_range[0] : subject_range[1] + 1],
                self.subject_end_marker,
                text[subject_range[1] + 1 :],
            ]
        else:
            raise ValueError("Entity boundaries overlap.")

        marked_text = "".join(segments)

        return marked_text

    def _convert_example_to_features(self, examples: List[InputExample]) -> List[InputFeatures]:
        max_length = 128
        if max_length is None:
            max_length = self.tokenizer.max_len

        label_map = {label: i for i, label in enumerate(self.get_labels())}
        labels = [label_map[example.label] for example in examples]

        def fix_tokenization_error(text: str, tokenizer_type: str) -> Any:
            """Fix the tokenization due to the `obj` and `subj` marker inserted
            in the middle of a word.
            Example:
                >>> text = "<obj>조지 해리슨</obj>이 쓰고 <subj>비틀즈</subj>가"
                >>> tokens = ['<obj>', '조지', '해리', '##슨', '</obj>', '이', '쓰', '##고', '<subj>', '비틀즈', '</subj>', '가']
                >>> fix_tokenization_error(text, tokenizer_type="bert-wp")
                ['<obj>', '조지', '해리', '##슨', '</obj>', '##이', '쓰', '##고', '<subj>', '비틀즈', '</subj>', '##가']
            """
            tokens = self.tokenizer.tokenize(text)
            # subject
            if text[text.find(self.subject_end_marker) + len(self.subject_end_marker)] != " ":
                space_idx = tokens.index(self.subject_end_marker) + 1
                if tokenizer_type == "xlm-sp":
                    if tokens[space_idx] == "▁":
                        tokens.pop(space_idx)
                    elif tokens[space_idx].startswith("▁"):
                        tokens[space_idx] = tokens[space_idx][1:]
                elif tokenizer_type == "bert-wp":
                    if not tokens[space_idx].startswith("##") and "가" <= tokens[space_idx][0] <= "힣":
                        tokens[space_idx] = "##" + tokens[space_idx]

            # object
            if text[text.find(self.object_end_marker) + len(self.object_end_marker)] != " ":
                space_idx = tokens.index(self.object_end_marker) + 1
                if tokenizer_type == "xlm-sp":
                    if tokens[space_idx] == "▁":
                        tokens.pop(space_idx)
                    elif tokens[space_idx].startswith("▁"):
                        tokens[space_idx] = tokens[space_idx][1:]
                elif tokenizer_type == "bert-wp":
                    if not tokens[space_idx].startswith("##") and "가" <= tokens[space_idx][0] <= "힣":
                        tokens[space_idx] = "##" + tokens[space_idx]

            return tokens

        tokenized_examples = [fix_tokenization_error(example.text_a, self.tokenizer_type) for example in examples]
        batch_encoding = self.tokenizer.batch_encode_plus(
            [(self.tokenizer.convert_tokens_to_ids(tokens), None) for tokens in tokenized_examples],
            max_length=max_length,
            padding="max_length",
            truncation=True,
        )

        features = []
        for i in range(len(examples)):
            inputs = {k: batch_encoding[k][i] for k in batch_encoding}

            feature = InputFeatures(**inputs, label=labels[i])
            features.append(feature)

        for i in range(1):
            print("*** Example ***")
            print("guid: %s" % (examples[i].guid))
            print("origin example: %s" % examples[i])
            print("origin tokens: %s" % self.tokenizer.tokenize(examples[i].text_a))
            print("fixed tokens: %s" % tokenized_examples[i])
            print("features: %s" % features[i])

        return features

    def _create_dataset(self, data) -> TensorDataset:
        examples = self._create(data)
        features = self._convert_example_to_features(examples)

        all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
        all_attention_mask = torch.tensor([f.attention_mask for f in features], dtype=torch.long)
        # Some model does not make use of token type ids (e.g. RoBERTa)
        all_token_type_ids = torch.tensor(
            [0 if f.token_type_ids is None else f.token_type_ids for f in features], dtype=torch.long
        )
        all_labels = torch.tensor([f.label for f in features], dtype=torch.long)

        return TensorDataset(all_input_ids, all_attention_mask, all_token_type_ids, all_labels)

데이터 input 형식 확인

data = {
        "guid": "klue-re-v1_dev_00014",
        "sentence": "엠비아이는 ‘서울국제발명전시회’에서 '파워트레인(POWERTRAIN)\"'으로 영예의 대상을 차지했다.",
        "subject_entity": {
            "word": "엠비아이",
            "start_idx": 0,
            "end_idx": 3,
            "type": "ORG"
        },
        "object_entity": {
            "word": "파워트레인",
            "start_idx": 21,
            "end_idx": 25,
            "type": "POH"
        },
        "label": "org:product",
        "source": "wikitree"
    }

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('monologg/koelectra-base-v3-discriminator')

processor = KlueREProcessor(tokenizer)
processor._create_dataset(data)

*** Example ***
guid: klue-re-v1_dev_00014
origin example: InputExample(guid='klue-re-v1_dev_00014', text_a='<subj>엠비아이</subj>는 ‘서울국제발명전시회’에서 \'<obj>파워트레인</obj>(POWERTRAIN)"\'으로 영예의 대상을 차지했다.', text_b=None, label='org:product')
origin tokens: ['<subj>', '엠', '##비아', '##이', '</subj>', '는', '‘', '서울', '##국', '##제', '##발', '##명', '##전', '##시', '##회', '’', '에서', "'", '<obj>', '파워', '##트', '##레인', '</obj>', '(', 'PO', '##W', '##ER', '##TRA', '##IN', ')', '"', "'", '으로', '영예', '##의', '대상', '##을', '차지', '##했', '##다', '.']
fixed tokens: ['<subj>', '엠', '##비아', '##이', '</subj>', '##는', '‘', '서울', '##국', '##제', '##발', '##명', '##전', '##시', '##회', '’', '에서', "'", '<obj>', '파워', '##트', '##레인', '</obj>', '(', 'PO', '##W', '##ER', '##TRA', '##IN', ')', '"', "'", '으로', '영예', '##의', '대상', '##을', '차지', '##했', '##다', '.']
features: InputFeatures(input_ids=[2, 35000, 3134, 8318, 4007, 35001, 4034, 144, 6265, 4113, 4106, 4387, 4282, 4068, 4114, 4213, 145, 6215, 11, 35002, 9198, 4039, 10007, 35003, 12, 21504, 4198, 12204, 25541, 17471, 13, 6, 11, 6214, 20217, 4234, 6391, 4292, 6838, 4398, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=8)

<torch.utils.data.dataset.TensorDataset at 0x7fe5d73b6850>

02 Train

KLUE에서 공개한 사전 학습 시킨 RoBERTa, BERT를 통해서 Train

metric_key

• micro f1

• auprc

!python KLUE-Baseline/run_klue.py \
train \
--task klue-re \
--output_dir ./re \
--data_dir ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1 \
--model_name_or_path klue/roberta-base \
--learning_rate 2e-5 \
--num_train_epochs 4 \
--train_batch_size 32 \
--eval_batch_size 16 \
--warmup_ratio 0.2 \
--patience 10000 \
--max_seq_length 256 \
--metric_key auprc \
--gpus 0 \

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: MetricBase will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
09/29/2021 09:49:28 - INFO - __main__ - Arguments:
09/29/2021 09:49:28 - INFO - __main__ - 	                command : train
09/29/2021 09:49:28 - INFO - __main__ - 	                   task : klue-re
09/29/2021 09:49:28 - INFO - __main__ - 	             output_dir : ./re
09/29/2021 09:49:28 - INFO - __main__ - 	                   gpus : [0]
09/29/2021 09:49:28 - INFO - __main__ - 	                   fp16 : False
09/29/2021 09:49:28 - INFO - __main__ - 	   num_sanity_val_steps : 2
09/29/2021 09:49:28 - INFO - __main__ - 	              tpu_cores : None
09/29/2021 09:49:28 - INFO - __main__ - 	      gradient_clip_val : 1.0
09/29/2021 09:49:28 - INFO - __main__ - 	accumulate_grad_batches : 1
09/29/2021 09:49:28 - INFO - __main__ - 	                   seed : 42
09/29/2021 09:49:28 - INFO - __main__ - 	             metric_key : auprc
09/29/2021 09:49:28 - INFO - __main__ - 	               patience : 10000
09/29/2021 09:49:28 - INFO - __main__ - 	    early_stopping_mode : max
09/29/2021 09:49:28 - INFO - __main__ - 	               data_dir : ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1
09/29/2021 09:49:28 - INFO - __main__ - 	        train_file_name : None
09/29/2021 09:49:28 - INFO - __main__ - 	          dev_file_name : None
09/29/2021 09:49:28 - INFO - __main__ - 	         test_file_name : None
09/29/2021 09:49:28 - INFO - __main__ - 	            num_workers : 4
09/29/2021 09:49:28 - INFO - __main__ - 	       train_batch_size : 32
09/29/2021 09:49:28 - INFO - __main__ - 	        eval_batch_size : 16
09/29/2021 09:49:28 - INFO - __main__ - 	         max_seq_length : 256
09/29/2021 09:49:28 - INFO - __main__ - 	      relation_filename : relation_list.json
09/29/2021 09:49:28 - INFO - __main__ - 	     model_name_or_path : klue/roberta-base
09/29/2021 09:49:28 - INFO - __main__ - 	            config_name : 
09/29/2021 09:49:28 - INFO - __main__ - 	         tokenizer_name : None
09/29/2021 09:49:28 - INFO - __main__ - 	              cache_dir : 
09/29/2021 09:49:28 - INFO - __main__ - 	      encoder_layerdrop : None
09/29/2021 09:49:28 - INFO - __main__ - 	      decoder_layerdrop : None
09/29/2021 09:49:28 - INFO - __main__ - 	                dropout : None
09/29/2021 09:49:28 - INFO - __main__ - 	      attention_dropout : None
09/29/2021 09:49:28 - INFO - __main__ - 	          learning_rate : 2e-05
09/29/2021 09:49:28 - INFO - __main__ - 	           lr_scheduler : linear
09/29/2021 09:49:28 - INFO - __main__ - 	           weight_decay : 0.0
09/29/2021 09:49:28 - INFO - __main__ - 	           adam_epsilon : 1e-08
09/29/2021 09:49:28 - INFO - __main__ - 	           warmup_steps : None
09/29/2021 09:49:28 - INFO - __main__ - 	           warmup_ratio : 0.2
09/29/2021 09:49:28 - INFO - __main__ - 	             max_epochs : 1
09/29/2021 09:49:28 - INFO - __main__ - 	              adafactor : False
09/29/2021 09:49:28 - INFO - __main__ - 	     verbose_step_count : 100
Global seed set to 42
09/29/2021 09:49:28 - INFO - lightning - Global seed set to 42
GPU available: True, used: True
09/29/2021 09:49:28 - INFO - lightning - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
09/29/2021 09:49:28 - INFO - lightning - TPU available: None, using: 0 TPU cores
09/29/2021 09:49:30 - INFO - klue_baseline.data.utils - Using BertTokenizer for fixing tokenization result
09/29/2021 09:49:30 - INFO - klue_baseline.data.base - Creating features from dataset file at ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1
09/29/2021 09:49:30 - INFO - klue_baseline.data.klue_re - Loading from ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1/klue-re-v1.1_train.json
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_train_00000
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_train_00000', text_a='〈Something〉는 <obj>조지 해리슨</obj>이 쓰고 <subj>비틀즈</subj>가 1969년 앨범 《Abbey Road》에 담은 노래다.', text_b=None, label='no_relation')
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin tokens: ['〈', 'So', '##me', '##th', '##ing', '〉', '는', '<obj>', '조지', '해리', '##슨', '</obj>', '이', '쓰', '##고', '<subj>', '비틀즈', '</subj>', '가', '1969', '##년', '앨범', '《', 'Ab', '##be', '##y', 'Ro', '##ad', '》', '에', '담', '##은', '노래', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - fixed tokens: ['〈', 'So', '##me', '##th', '##ing', '〉', '는', '<obj>', '조지', '해리', '##슨', '</obj>', '##이', '쓰', '##고', '<subj>', '비틀즈', '</subj>', '##가', '1969', '##년', '앨범', '《', 'Ab', '##be', '##y', 'Ro', '##ad', '》', '에', '담', '##은', '노래', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 168, 30985, 14451, 7088, 4586, 169, 793, 32002, 8373, 14113, 2234, 32003, 2052, 1363, 2088, 32000, 29830, 32001, 2116, 14879, 2440, 6711, 170, 21406, 26713, 2076, 25145, 5749, 171, 1421, 818, 2073, 4388, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_train_00001
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_train_00001', text_a='호남이 기반인 바른미래당·<obj>대안신당</obj>·<subj>민주평화당</subj>이 우여곡절 끝에 합당해 민생당(가칭)으로 재탄생한다.', text_b=None, label='no_relation')
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin tokens: ['호남', '##이', '기반', '##인', '바른', '##미', '##래', '##당', '·', '<obj>', '대안', '##신', '##당', '</obj>', '·', '<subj>', '민주', '##평', '##화', '##당', '</subj>', '이', '우여곡절', '끝', '##에', '합당', '##해', '민생', '##당', '(', '가칭', ')', '으로', '재', '##탄', '##생', '##한다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - fixed tokens: ['호남', '##이', '기반', '##인', '바른', '##미', '##래', '##당', '·', '<obj>', '대안', '##신', '##당', '</obj>', '·', '<subj>', '민주', '##평', '##화', '##당', '</subj>', '##이', '우여곡절', '끝', '##에', '합당', '##해', '민생', '##당', '(', '가칭', ')', '으로', '재', '##탄', '##생', '##한다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 6409, 2052, 4568, 2179, 6417, 2044, 2315, 2481, 100, 32002, 5605, 2250, 2481, 32003, 100, 32000, 3772, 2139, 2267, 2481, 32001, 2052, 16489, 711, 2170, 12827, 2097, 8646, 2481, 12, 15283, 13, 3603, 1528, 2554, 2065, 4538, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_train_00002
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_train_00002', text_a='K리그2에서 성적 1위를 달리고 있는 <subj>광주FC</subj>는 지난 26일 <obj>한국프로축구연맹</obj>으로부터 관중 유치 성과와 마케팅 성과를 인정받아 ‘풀 스타디움상’과 ‘플러스 스타디움상’을 수상했다.', text_b=None, label='org:member_of')
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin tokens: ['K', '##리그', '##2', '##에서', '성적', '1', '##위', '##를', '달리', '##고', '있', '##는', '<subj>', '광주', '##FC', '</subj>', '는', '지난', '26', '##일', '<obj>', '한국', '##프로', '##축구연맹', '</obj>', '으로부터', '관중', '유치', '성과', '##와', '마케팅', '성과', '##를', '인정받', '##아', '‘', '풀', '스타디움', '##상', '’', '과', '‘', '플러스', '스타디움', '##상', '’', '을', '수상', '##했', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - fixed tokens: ['K', '##리그', '##2', '##에서', '성적', '1', '##위', '##를', '달리', '##고', '있', '##는', '<subj>', '광주', '##FC', '</subj>', '##는', '지난', '26', '##일', '<obj>', '한국', '##프로', '##축구연맹', '</obj>', '##으로부터', '관중', '유치', '성과', '##와', '마케팅', '성과', '##를', '인정받', '##아', '‘', '풀', '스타디움', '##상', '’', '과', '‘', '플러스', '스타디움', '##상', '’', '을', '수상', '##했', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 47, 17665, 2302, 27135, 4610, 21, 2090, 2138, 4214, 2088, 1513, 2259, 32000, 4104, 10904, 32001, 2259, 3625, 4210, 2210, 32002, 3629, 17287, 20212, 32003, 3, 8862, 4415, 4422, 2522, 4852, 4422, 2138, 6157, 2227, 114, 1872, 14198, 2290, 115, 604, 114, 6646, 14198, 2290, 115, 1498, 4812, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=5)
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_train_00003
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_train_00003', text_a='균일가 생활용품점 (주)<subj>아성다이소</subj>(대표 <obj>박정부</obj>)는 코로나19 바이러스로 어려움을 겪고 있는 대구광역시에 행복박스를 전달했다고 10일 밝혔다.', text_b=None, label='org:top_members/employees')
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin tokens: ['균일', '##가', '생활', '##용품', '##점', '(', '주', ')', '<subj>', '아성', '##다이', '##소', '</subj>', '(', '대표', '<obj>', '박정', '##부', '</obj>', ')', '는', '코', '##로나', '##19', '바이러스', '##로', '어려움', '##을', '겪', '##고', '있', '##는', '대구', '##광역시', '##에', '행복', '##박스', '##를', '전달', '##했', '##다고', '10', '##일', '밝혔', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - fixed tokens: ['균일', '##가', '생활', '##용품', '##점', '(', '주', ')', '<subj>', '아성', '##다이', '##소', '</subj>', '(', '대표', '<obj>', '박정', '##부', '</obj>', ')', '는', '코', '##로나', '##19', '바이러스', '##로', '어려움', '##을', '겪', '##고', '있', '##는', '대구', '##광역시', '##에', '행복', '##박스', '##를', '전달', '##했', '##다고', '10', '##일', '밝혔', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 23306, 2116, 3799, 18319, 2532, 12, 1564, 13, 32000, 27930, 24393, 2024, 32001, 12, 3661, 32002, 6580, 2144, 32003, 13, 793, 1726, 11235, 22328, 8151, 2200, 5117, 2069, 585, 2088, 1513, 2259, 3900, 16955, 2170, 4202, 13473, 2138, 4535, 2371, 4683, 3633, 2210, 3705, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=10)
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_train_00004
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_train_00004', text_a='<obj>1967</obj>년 프로 야구 드래프트 1순위로 <subj>요미우리 자이언츠</subj>에게 입단하면서 등번호는 8번으로 배정되었다.', text_b=None, label='no_relation')
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - origin tokens: ['<obj>', '1967', '</obj>', '년', '프로', '야구', '드래프트', '1', '##순위', '##로', '<subj>', '요미우리', '자이언츠', '</subj>', '에게', '입단', '##하면', '##서', '등', '##번', '##호', '##는', '8', '##번', '##으로', '배정', '##되', '##었', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - fixed tokens: ['<obj>', '1967', '</obj>', '##년', '프로', '야구', '드래프트', '1', '##순위', '##로', '<subj>', '요미우리', '자이언츠', '</subj>', '##에게', '입단', '##하면', '##서', '등', '##번', '##호', '##는', '8', '##번', '##으로', '배정', '##되', '##었', '##다', '.']
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 32002, 14925, 32003, 2440, 3701, 4878, 19038, 21, 22247, 2200, 32000, 20289, 20562, 32001, 3, 11989, 5643, 2112, 886, 2517, 2016, 2259, 28, 2517, 6233, 8557, 2496, 2359, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:53 - INFO - klue_baseline.data.base - Prepare train dataset (Count: 32470) 
09/29/2021 09:49:53 - INFO - klue_baseline.data.base - Creating features from dataset file at ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1
09/29/2021 09:49:53 - INFO - klue_baseline.data.klue_re - Loading from ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1/klue-re-v1.1_dev.json
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00000
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00000', text_a="20대 남성 <subj>A</subj>(26)씨가 아버지 치료비를 위해 B(<obj>30</obj>)씨가 모아둔 돈을 훔쳐 인터넷 방송 BJ에게 '별풍선'으로 쏜 사실이 알려졌다.", text_b=None, label='no_relation')
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin tokens: ['20', '##대', '남성', '<subj>', 'A', '</subj>', '(', '26', ')', '씨', '##가', '아버지', '치료비', '##를', '위해', 'B', '(', '<obj>', '30', '</obj>', ')', '씨', '##가', '모아', '##둔', '돈', '##을', '훔쳐', '인터넷', '방송', 'B', '##J', '##에', '##게', "'", '별', '##풍', '##선', "'", '으로', '쏜', '사실', '##이', '알려졌', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - fixed tokens: ['20', '##대', '남성', '<subj>', 'A', '</subj>', '(', '26', ')', '씨', '##가', '아버지', '치료비', '##를', '위해', 'B', '(', '<obj>', '30', '</obj>', ')', '씨', '##가', '모아', '##둔', '돈', '##을', '훔쳐', '인터넷', '방송', 'B', '##J', '##에', '##게', "'", '별', '##풍', '##선', "'", '으로', '쏜', '사실', '##이', '알려졌', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3619, 2104, 4576, 32000, 37, 32001, 12, 4210, 13, 1370, 2116, 4096, 18014, 2138, 3627, 38, 12, 32002, 3740, 32003, 13, 1370, 2116, 5302, 2684, 850, 2069, 12908, 4254, 3861, 38, 2303, 2170, 2318, 11, 1156, 2533, 2020, 11, 3603, 1356, 3669, 2052, 4757, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00001
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00001', text_a='그러나 <subj>심 의원</subj>은 보좌진이 접속 권한을 받아 <obj>정부</obj> 업무추진비 사용 내역 등을 다운받았음에도 정부가 허위 사실을 유포하는 등 국정감사 활동을 방해하고 있다고 반박했고, 김동연 경제부총리 겸 기획재정부 장관과 김재훈 재정정보원장, 기재부 관계자 등을 무고 등 혐의로 전날 맞고발했다.', text_b=None, label='no_relation')
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin tokens: ['그러나', '<subj>', '심', '의원', '</subj>', '은', '보좌', '##진', '##이', '접속', '권한', '##을', '받아', '<obj>', '정부', '</obj>', '업무', '##추', '##진', '##비', '사용', '내역', '등', '##을', '다운', '##받', '##았', '##음', '##에도', '정부', '##가', '허위', '사실', '##을', '유포', '##하', '##는', '등', '국정', '##감사', '활동', '##을', '방해', '##하고', '있', '##다고', '반박', '##했', '##고', ',', '김동연', '경제', '##부', '##총리', '겸', '기획', '##재', '##정부', '장관', '##과', '김재', '##훈', '재정', '##정보', '##원장', ',', '기재', '##부', '관계자', '등', '##을', '무고', '등', '혐의', '##로', '전날', '맞', '##고', '##발', '##했', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - fixed tokens: ['그러나', '<subj>', '심', '의원', '</subj>', '##은', '보좌', '##진', '##이', '접속', '권한', '##을', '받아', '<obj>', '정부', '</obj>', '업무', '##추', '##진', '##비', '사용', '내역', '등', '##을', '다운', '##받', '##았', '##음', '##에도', '정부', '##가', '허위', '사실', '##을', '유포', '##하', '##는', '등', '국정', '##감사', '활동', '##을', '방해', '##하고', '있', '##다고', '반박', '##했', '##고', ',', '김동연', '경제', '##부', '##총리', '겸', '기획', '##재', '##정부', '장관', '##과', '김재', '##훈', '재정', '##정보', '##원장', ',', '기재', '##부', '관계자', '등', '##을', '무고', '등', '혐의', '##로', '전날', '맞', '##고', '##발', '##했', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3678, 32000, 1331, 3712, 32001, 2073, 8541, 2043, 2052, 8407, 5851, 2069, 21372, 32002, 3659, 32003, 4148, 2159, 2043, 2151, 3704, 10140, 886, 2069, 6163, 2757, 2886, 2053, 6509, 3659, 2116, 7441, 3669, 2069, 12502, 2205, 2259, 886, 4567, 27065, 3746, 2069, 6345, 19521, 1513, 4683, 7703, 2371, 2088, 16, 28320, 3674, 2144, 6384, 588, 4417, 2070, 11722, 4166, 2145, 7863, 2034, 4426, 19861, 10293, 16, 7475, 2144, 3810, 886, 2069, 17891, 886, 4456, 2200, 5978, 1047, 2088, 2311, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00002
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00002', text_a="영화 《룸》에서 감금되어 살아가는 여자의 아들 '<subj>잭</subj>' 역으로 주목받으며, 크리틱스 초이스 영화상 최우수 아역연기상, <obj>캐나다 스크린 어워드</obj>(캐나다 영화 & 텔레비전 아카데미상) 영화부문 최우수 남우주연상을 수상하였고, 미국 배우 조합상(SAG) 최우수 남우조연상에 후보 지명되었다.", text_b=None, label='no_relation')
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin tokens: ['영화', '《', '룸', '》', '에서', '감금', '##되', '##어', '살아가', '##는', '여자', '##의', '아들', "'", '<subj>', '잭', '</subj>', "'", '역', '##으로', '주목', '##받', '##으며', ',', '크리', '##틱', '##스', '초이스', '영화', '##상', '최우수', '아역', '##연', '##기', '##상', ',', '<obj>', '캐나다', '스크린', '어워드', '</obj>', '(', '캐나다', '영화', '&', '텔레비전', '아카데미', '##상', ')', '영화', '##부', '##문', '최우수', '남', '##우주', '##연', '##상', '##을', '수상', '##하', '##였', '##고', ',', '미국', '배우', '조합', '##상', '(', 'SA', '##G', ')', '최우수', '남', '##우', '##조', '##연', '##상', '##에', '후보', '지명', '##되', '##었', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - fixed tokens: ['영화', '《', '룸', '》', '에서', '감금', '##되', '##어', '살아가', '##는', '여자', '##의', '아들', "'", '<subj>', '잭', '</subj>', "'", '역', '##으로', '주목', '##받', '##으며', ',', '크리', '##틱', '##스', '초이스', '영화', '##상', '최우수', '아역', '##연', '##기', '##상', ',', '<obj>', '캐나다', '스크린', '어워드', '</obj>', '(', '캐나다', '영화', '&', '텔레비전', '아카데미', '##상', ')', '영화', '##부', '##문', '최우수', '남', '##우주', '##연', '##상', '##을', '수상', '##하', '##였', '##고', ',', '미국', '배우', '조합', '##상', '(', 'SA', '##G', ')', '최우수', '남', '##우', '##조', '##연', '##상', '##에', '후보', '지명', '##되', '##었', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3771, 170, 1006, 171, 3604, 18033, 2496, 2051, 5933, 2259, 3883, 2079, 4200, 11, 32000, 1529, 32001, 11, 1429, 6233, 4439, 2757, 4007, 16, 10869, 2960, 2255, 29366, 3771, 2290, 11896, 25849, 2156, 2015, 2290, 16, 32002, 7066, 8559, 18085, 32003, 12, 7066, 3771, 10, 7361, 9101, 2290, 13, 3771, 2144, 2346, 11896, 723, 21347, 2156, 2290, 2069, 4812, 2205, 2507, 2088, 16, 3666, 4165, 4936, 2290, 12, 21707, 2341, 13, 11896, 723, 2137, 2446, 2156, 2290, 2170, 3733, 7119, 2496, 2359, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00003
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00003', text_a='전라남도(<obj>도지사</obj> 김영록)는 <subj>해양수산부</subj>의 2020년 어촌뉴딜300 공모사업에 15개 연안 시군 70개소 7천61억 원 규모를 신청했다고 14일 밝혔다.', text_b=None, label='no_relation')
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin tokens: ['전라남도', '(', '<obj>', '도지사', '</obj>', '김영', '##록', ')', '는', '<subj>', '해양수산부', '</subj>', '의', '2020', '##년', '어촌', '##뉴', '##딜', '##30', '##0', '공모', '##사업', '##에', '15', '##개', '연안', '시군', '70', '##개', '##소', '7', '##천', '##61', '##억', '원', '규모', '##를', '신청', '##했', '##다고', '14', '##일', '밝혔', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - fixed tokens: ['전라남도', '(', '<obj>', '도지사', '</obj>', '김영', '##록', ')', '는', '<subj>', '해양수산부', '</subj>', '##의', '2020', '##년', '어촌', '##뉴', '##딜', '##30', '##0', '공모', '##사업', '##에', '15', '##개', '연안', '시군', '70', '##개', '##소', '7', '##천', '##61', '##억', '원', '규모', '##를', '신청', '##했', '##다고', '14', '##일', '밝혔', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 22272, 12, 32002, 9035, 32003, 5553, 2351, 13, 793, 32000, 15995, 32001, 2079, 7330, 2440, 17127, 3065, 3169, 7556, 2082, 5637, 9841, 2170, 3749, 2019, 11189, 15773, 4531, 2019, 2024, 27, 2337, 26421, 2028, 1478, 3849, 2138, 4373, 2371, 4683, 3909, 2210, 3705, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00004
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00004', text_a='<subj>부산항만공사</subj>(BPA, 사장 남기찬)는 <obj>27일</obj> 아동의 인권존중과 아동학대 예방에 대한 인식을 제고하기 위해 전 임직원을 대상으로 아동학대 예방교육을 실시했다.', text_b=None, label='no_relation')
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - origin tokens: ['<subj>', '부산항', '##만', '##공사', '</subj>', '(', 'B', '##PA', ',', '사장', '남기', '##찬', ')', '는', '<obj>', '27', '##일', '</obj>', '아동', '##의', '인권', '##존', '##중', '##과', '아동학', '##대', '예방', '##에', '대한', '인식', '##을', '제고', '##하기', '위해', '전', '임직원', '##을', '대상', '##으로', '아동학', '##대', '예방', '##교육', '##을', '실시', '##했', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - fixed tokens: ['<subj>', '부산항', '##만', '##공사', '</subj>', '(', 'B', '##PA', ',', '사장', '남기', '##찬', ')', '는', '<obj>', '27', '##일', '</obj>', '아동', '##의', '인권', '##존', '##중', '##과', '아동학', '##대', '예방', '##에', '대한', '인식', '##을', '제고', '##하기', '위해', '전', '임직원', '##을', '대상', '##으로', '아동학', '##대', '예방', '##교육', '##을', '실시', '##했', '##다', '.']
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 32000, 17003, 2154, 5634, 32001, 12, 38, 19672, 16, 4184, 6804, 2166, 13, 793, 32002, 4207, 2210, 32003, 5332, 2079, 5188, 2869, 2284, 2145, 30333, 2104, 5005, 2170, 3618, 4263, 2069, 7928, 31302, 3627, 1537, 6809, 2069, 3756, 6233, 30333, 2104, 5005, 6177, 2069, 4130, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:49:59 - INFO - klue_baseline.data.base - Prepare dev dataset (Count: 7765) 
09/29/2021 09:49:59 - INFO - klue_baseline.data.base - Creating features from dataset file at ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - Test dataset doesn't exists. So loading dev dataset instead.
09/29/2021 09:49:59 - INFO - klue_baseline.data.klue_re - Loading from ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1/klue-re-v1.1_dev.json
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00000
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00000', text_a="20대 남성 <subj>A</subj>(26)씨가 아버지 치료비를 위해 B(<obj>30</obj>)씨가 모아둔 돈을 훔쳐 인터넷 방송 BJ에게 '별풍선'으로 쏜 사실이 알려졌다.", text_b=None, label='no_relation')
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin tokens: ['20', '##대', '남성', '<subj>', 'A', '</subj>', '(', '26', ')', '씨', '##가', '아버지', '치료비', '##를', '위해', 'B', '(', '<obj>', '30', '</obj>', ')', '씨', '##가', '모아', '##둔', '돈', '##을', '훔쳐', '인터넷', '방송', 'B', '##J', '##에', '##게', "'", '별', '##풍', '##선', "'", '으로', '쏜', '사실', '##이', '알려졌', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - fixed tokens: ['20', '##대', '남성', '<subj>', 'A', '</subj>', '(', '26', ')', '씨', '##가', '아버지', '치료비', '##를', '위해', 'B', '(', '<obj>', '30', '</obj>', ')', '씨', '##가', '모아', '##둔', '돈', '##을', '훔쳐', '인터넷', '방송', 'B', '##J', '##에', '##게', "'", '별', '##풍', '##선', "'", '으로', '쏜', '사실', '##이', '알려졌', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3619, 2104, 4576, 32000, 37, 32001, 12, 4210, 13, 1370, 2116, 4096, 18014, 2138, 3627, 38, 12, 32002, 3740, 32003, 13, 1370, 2116, 5302, 2684, 850, 2069, 12908, 4254, 3861, 38, 2303, 2170, 2318, 11, 1156, 2533, 2020, 11, 3603, 1356, 3669, 2052, 4757, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00001
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00001', text_a='그러나 <subj>심 의원</subj>은 보좌진이 접속 권한을 받아 <obj>정부</obj> 업무추진비 사용 내역 등을 다운받았음에도 정부가 허위 사실을 유포하는 등 국정감사 활동을 방해하고 있다고 반박했고, 김동연 경제부총리 겸 기획재정부 장관과 김재훈 재정정보원장, 기재부 관계자 등을 무고 등 혐의로 전날 맞고발했다.', text_b=None, label='no_relation')
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin tokens: ['그러나', '<subj>', '심', '의원', '</subj>', '은', '보좌', '##진', '##이', '접속', '권한', '##을', '받아', '<obj>', '정부', '</obj>', '업무', '##추', '##진', '##비', '사용', '내역', '등', '##을', '다운', '##받', '##았', '##음', '##에도', '정부', '##가', '허위', '사실', '##을', '유포', '##하', '##는', '등', '국정', '##감사', '활동', '##을', '방해', '##하고', '있', '##다고', '반박', '##했', '##고', ',', '김동연', '경제', '##부', '##총리', '겸', '기획', '##재', '##정부', '장관', '##과', '김재', '##훈', '재정', '##정보', '##원장', ',', '기재', '##부', '관계자', '등', '##을', '무고', '등', '혐의', '##로', '전날', '맞', '##고', '##발', '##했', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - fixed tokens: ['그러나', '<subj>', '심', '의원', '</subj>', '##은', '보좌', '##진', '##이', '접속', '권한', '##을', '받아', '<obj>', '정부', '</obj>', '업무', '##추', '##진', '##비', '사용', '내역', '등', '##을', '다운', '##받', '##았', '##음', '##에도', '정부', '##가', '허위', '사실', '##을', '유포', '##하', '##는', '등', '국정', '##감사', '활동', '##을', '방해', '##하고', '있', '##다고', '반박', '##했', '##고', ',', '김동연', '경제', '##부', '##총리', '겸', '기획', '##재', '##정부', '장관', '##과', '김재', '##훈', '재정', '##정보', '##원장', ',', '기재', '##부', '관계자', '등', '##을', '무고', '등', '혐의', '##로', '전날', '맞', '##고', '##발', '##했', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3678, 32000, 1331, 3712, 32001, 2073, 8541, 2043, 2052, 8407, 5851, 2069, 21372, 32002, 3659, 32003, 4148, 2159, 2043, 2151, 3704, 10140, 886, 2069, 6163, 2757, 2886, 2053, 6509, 3659, 2116, 7441, 3669, 2069, 12502, 2205, 2259, 886, 4567, 27065, 3746, 2069, 6345, 19521, 1513, 4683, 7703, 2371, 2088, 16, 28320, 3674, 2144, 6384, 588, 4417, 2070, 11722, 4166, 2145, 7863, 2034, 4426, 19861, 10293, 16, 7475, 2144, 3810, 886, 2069, 17891, 886, 4456, 2200, 5978, 1047, 2088, 2311, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00002
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00002', text_a="영화 《룸》에서 감금되어 살아가는 여자의 아들 '<subj>잭</subj>' 역으로 주목받으며, 크리틱스 초이스 영화상 최우수 아역연기상, <obj>캐나다 스크린 어워드</obj>(캐나다 영화 & 텔레비전 아카데미상) 영화부문 최우수 남우주연상을 수상하였고, 미국 배우 조합상(SAG) 최우수 남우조연상에 후보 지명되었다.", text_b=None, label='no_relation')
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin tokens: ['영화', '《', '룸', '》', '에서', '감금', '##되', '##어', '살아가', '##는', '여자', '##의', '아들', "'", '<subj>', '잭', '</subj>', "'", '역', '##으로', '주목', '##받', '##으며', ',', '크리', '##틱', '##스', '초이스', '영화', '##상', '최우수', '아역', '##연', '##기', '##상', ',', '<obj>', '캐나다', '스크린', '어워드', '</obj>', '(', '캐나다', '영화', '&', '텔레비전', '아카데미', '##상', ')', '영화', '##부', '##문', '최우수', '남', '##우주', '##연', '##상', '##을', '수상', '##하', '##였', '##고', ',', '미국', '배우', '조합', '##상', '(', 'SA', '##G', ')', '최우수', '남', '##우', '##조', '##연', '##상', '##에', '후보', '지명', '##되', '##었', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - fixed tokens: ['영화', '《', '룸', '》', '에서', '감금', '##되', '##어', '살아가', '##는', '여자', '##의', '아들', "'", '<subj>', '잭', '</subj>', "'", '역', '##으로', '주목', '##받', '##으며', ',', '크리', '##틱', '##스', '초이스', '영화', '##상', '최우수', '아역', '##연', '##기', '##상', ',', '<obj>', '캐나다', '스크린', '어워드', '</obj>', '(', '캐나다', '영화', '&', '텔레비전', '아카데미', '##상', ')', '영화', '##부', '##문', '최우수', '남', '##우주', '##연', '##상', '##을', '수상', '##하', '##였', '##고', ',', '미국', '배우', '조합', '##상', '(', 'SA', '##G', ')', '최우수', '남', '##우', '##조', '##연', '##상', '##에', '후보', '지명', '##되', '##었', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3771, 170, 1006, 171, 3604, 18033, 2496, 2051, 5933, 2259, 3883, 2079, 4200, 11, 32000, 1529, 32001, 11, 1429, 6233, 4439, 2757, 4007, 16, 10869, 2960, 2255, 29366, 3771, 2290, 11896, 25849, 2156, 2015, 2290, 16, 32002, 7066, 8559, 18085, 32003, 12, 7066, 3771, 10, 7361, 9101, 2290, 13, 3771, 2144, 2346, 11896, 723, 21347, 2156, 2290, 2069, 4812, 2205, 2507, 2088, 16, 3666, 4165, 4936, 2290, 12, 21707, 2341, 13, 11896, 723, 2137, 2446, 2156, 2290, 2170, 3733, 7119, 2496, 2359, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00003
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00003', text_a='전라남도(<obj>도지사</obj> 김영록)는 <subj>해양수산부</subj>의 2020년 어촌뉴딜300 공모사업에 15개 연안 시군 70개소 7천61억 원 규모를 신청했다고 14일 밝혔다.', text_b=None, label='no_relation')
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin tokens: ['전라남도', '(', '<obj>', '도지사', '</obj>', '김영', '##록', ')', '는', '<subj>', '해양수산부', '</subj>', '의', '2020', '##년', '어촌', '##뉴', '##딜', '##30', '##0', '공모', '##사업', '##에', '15', '##개', '연안', '시군', '70', '##개', '##소', '7', '##천', '##61', '##억', '원', '규모', '##를', '신청', '##했', '##다고', '14', '##일', '밝혔', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - fixed tokens: ['전라남도', '(', '<obj>', '도지사', '</obj>', '김영', '##록', ')', '는', '<subj>', '해양수산부', '</subj>', '##의', '2020', '##년', '어촌', '##뉴', '##딜', '##30', '##0', '공모', '##사업', '##에', '15', '##개', '연안', '시군', '70', '##개', '##소', '7', '##천', '##61', '##억', '원', '규모', '##를', '신청', '##했', '##다고', '14', '##일', '밝혔', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 22272, 12, 32002, 9035, 32003, 5553, 2351, 13, 793, 32000, 15995, 32001, 2079, 7330, 2440, 17127, 3065, 3169, 7556, 2082, 5637, 9841, 2170, 3749, 2019, 11189, 15773, 4531, 2019, 2024, 27, 2337, 26421, 2028, 1478, 3849, 2138, 4373, 2371, 4683, 3909, 2210, 3705, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00004
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00004', text_a='<subj>부산항만공사</subj>(BPA, 사장 남기찬)는 <obj>27일</obj> 아동의 인권존중과 아동학대 예방에 대한 인식을 제고하기 위해 전 임직원을 대상으로 아동학대 예방교육을 실시했다.', text_b=None, label='no_relation')
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - origin tokens: ['<subj>', '부산항', '##만', '##공사', '</subj>', '(', 'B', '##PA', ',', '사장', '남기', '##찬', ')', '는', '<obj>', '27', '##일', '</obj>', '아동', '##의', '인권', '##존', '##중', '##과', '아동학', '##대', '예방', '##에', '대한', '인식', '##을', '제고', '##하기', '위해', '전', '임직원', '##을', '대상', '##으로', '아동학', '##대', '예방', '##교육', '##을', '실시', '##했', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - fixed tokens: ['<subj>', '부산항', '##만', '##공사', '</subj>', '(', 'B', '##PA', ',', '사장', '남기', '##찬', ')', '는', '<obj>', '27', '##일', '</obj>', '아동', '##의', '인권', '##존', '##중', '##과', '아동학', '##대', '예방', '##에', '대한', '인식', '##을', '제고', '##하기', '위해', '전', '임직원', '##을', '대상', '##으로', '아동학', '##대', '예방', '##교육', '##을', '실시', '##했', '##다', '.']
09/29/2021 09:50:04 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 32000, 17003, 2154, 5634, 32001, 12, 38, 19672, 16, 4184, 6804, 2166, 13, 793, 32002, 4207, 2210, 32003, 5332, 2079, 5188, 2869, 2284, 2145, 30333, 2104, 5005, 2170, 3618, 4263, 2069, 7928, 31302, 3627, 1537, 6809, 2069, 3756, 6233, 30333, 2104, 5005, 6177, 2069, 4130, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 09:50:04 - INFO - klue_baseline.data.base - Prepare test dataset (Count: 7765) 
Some weights of the model checkpoint at klue/roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at klue/roberta-base and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
09/29/2021 09:50:11 - INFO - __main__ - Start to run the full optimization routine.
09/29/2021 09:50:11 - INFO - pytorch_lightning.accelerators.gpu - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Validation sanity check:   0% 0/2 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_ranking.py:817: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
Epoch 0:   3% 100/2959 [04:19<2:03:49,  2.60s/it, loss=2.57, v_num=2]Step: 100 - Loss: 2.3796730041503906
09/29/2021 09:54:37 - INFO - lightning - Step: 100 - Loss: 2.3796730041503906
Epoch 0:   7% 200/2959 [08:39<1:59:22,  2.60s/it, loss=1.91, v_num=2]Step: 200 - Loss: 1.5856021642684937
09/29/2021 09:58:56 - INFO - lightning - Step: 200 - Loss: 1.5856021642684937
Epoch 0:   9% 260/2959 [10:56<1:53:37,  2.53s/it, loss=1.68, v_num=2]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/486 [00:00<?, ?it/s]
Epoch 0:   9% 280/2959 [11:05<1:46:03,  2.38s/it, loss=1.68, v_num=2]
Epoch 0:  10% 300/2959 [11:14<1:39:40,  2.25s/it, loss=1.68, v_num=2]
Epoch 0:  11% 320/2959 [11:24<1:34:03,  2.14s/it, loss=1.68, v_num=2]
Epoch 0:  11% 340/2959 [11:33<1:29:05,  2.04s/it, loss=1.68, v_num=2]
Epoch 0:  12% 360/2959 [11:43<1:24:39,  1.95s/it, loss=1.68, v_num=2]
Epoch 0:  13% 380/2959 [11:53<1:20:40,  1.88s/it, loss=1.68, v_num=2]
Epoch 0:  14% 400/2959 [12:02<1:17:04,  1.81s/it, loss=1.68, v_num=2]
Epoch 0:  14% 420/2959 [12:12<1:13:48,  1.74s/it, loss=1.68, v_num=2]
Epoch 0:  15% 440/2959 [12:22<1:10:48,  1.69s/it, loss=1.68, v_num=2]
Epoch 0:  16% 460/2959 [12:31<1:08:03,  1.63s/it, loss=1.68, v_num=2]
Epoch 0:  16% 480/2959 [12:41<1:05:32,  1.59s/it, loss=1.68, v_num=2]
Epoch 0:  17% 500/2959 [12:50<1:03:11,  1.54s/it, loss=1.68, v_num=2]
Epoch 0:  18% 520/2959 [13:00<1:01:01,  1.50s/it, loss=1.68, v_num=2]
Epoch 0:  18% 540/2959 [13:10<58:59,  1.46s/it, loss=1.68, v_num=2]  
Epoch 0:  19% 560/2959 [13:19<57:06,  1.43s/it, loss=1.68, v_num=2]
Epoch 0:  20% 580/2959 [13:29<55:20,  1.40s/it, loss=1.68, v_num=2]
Epoch 0:  20% 600/2959 [13:39<53:40,  1.37s/it, loss=1.68, v_num=2]
Epoch 0:  21% 620/2959 [13:48<52:06,  1.34s/it, loss=1.68, v_num=2]
Epoch 0:  22% 640/2959 [13:58<50:37,  1.31s/it, loss=1.68, v_num=2]
Epoch 0:  22% 660/2959 [14:08<49:14,  1.28s/it, loss=1.68, v_num=2]
Epoch 0:  23% 680/2959 [14:17<47:54,  1.26s/it, loss=1.68, v_num=2]
Epoch 0:  24% 700/2959 [14:27<46:38,  1.24s/it, loss=1.68, v_num=2]
Epoch 0:  24% 720/2959 [14:36<45:26,  1.22s/it, loss=1.68, v_num=2]
Epoch 0:  25% 740/2959 [14:46<44:18,  1.20s/it, loss=1.68, v_num=2]
Epoch 0:  26% 760/2959 [14:49<42:53,  1.17s/it, loss=1.68, v_num=2]***** Validation results *****
09/29/2021 10:05:05 - INFO - lightning - ***** Validation results *****
train/loss = 1.5094388723373413
09/29/2021 10:05:05 - INFO - lightning - train/loss = 1.5094388723373413
valid/loss = 1.6732418537139893
09/29/2021 10:05:05 - INFO - lightning - valid/loss = 1.6732418537139893
valid/micro_f1 = 36.80577345465956
09/29/2021 10:05:05 - INFO - lightning - valid/micro_f1 = 36.80577345465956
valid/auprc = 24.167251928098345
09/29/2021 10:05:05 - INFO - lightning - valid/auprc = 24.167251928098345
Epoch 0:  26% 760/2959 [14:50<42:57,  1.17s/it, loss=1.7, v_num=2] 
                                                 /usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
tcmalloc: large alloc 1255776256 bytes == 0x5593b8eb8000 @  0x7fd224e78615 0x5593298ce4cc 0x5593299ae47a 0x5593298d4f0c 0x7fd220379a94 0x7fd22037b864 0x7fd22034b590 0x7fd210c91465 0x7fd210c8d9ca 0x7fd210c92609 0x7fd22034ef2b 0x7fd21ffd4200 0x5593298d2098 0x5593299454d9 0x55932993fced 0x5593298d2bda 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329940c0d 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940c0d 0x5593298d2afa
tcmalloc: large alloc 1569726464 bytes == 0x559403c52000 @  0x7fd224e78615 0x5593298ce4cc 0x5593299ae47a 0x5593298d4f0c 0x7fd220379a94 0x7fd22037b864 0x7fd22034b590 0x7fd210c91465 0x7fd210c8d9ca 0x7fd210c92609 0x7fd22034ef2b 0x7fd21ffd4200 0x5593298d2098 0x5593299454d9 0x55932993fced 0x5593298d2bda 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329940c0d 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940c0d 0x5593298d2afa
Epoch 0:  28% 820/2959 [17:03<44:30,  1.25s/it, loss=1.51, v_num=2]Step: 300 - Loss: 1.2908490896224976
09/29/2021 10:07:21 - INFO - lightning - Step: 300 - Loss: 1.2908490896224976
Epoch 0:  31% 920/2959 [21:22<47:23,  1.39s/it, loss=1.28, v_num=2]Step: 400 - Loss: 1.2117730379104614
09/29/2021 10:11:40 - INFO - lightning - Step: 400 - Loss: 1.2117730379104614
Epoch 0:  34% 1020/2959 [25:41<48:50,  1.51s/it, loss=1.12, v_num=2]Step: 500 - Loss: 0.9035272002220154
09/29/2021 10:15:59 - INFO - lightning - Step: 500 - Loss: 0.9035272002220154
Epoch 0:  35% 1040/2959 [25:57<47:53,  1.50s/it, loss=1.12, v_num=2]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/486 [00:00<?, ?it/s]
Epoch 0:  36% 1060/2959 [26:05<46:44,  1.48s/it, loss=1.12, v_num=2]
Epoch 0:  36% 1080/2959 [26:15<45:40,  1.46s/it, loss=1.12, v_num=2]
Epoch 0:  37% 1100/2959 [26:24<44:38,  1.44s/it, loss=1.12, v_num=2]
Epoch 0:  38% 1120/2959 [26:34<43:38,  1.42s/it, loss=1.12, v_num=2]
Epoch 0:  39% 1140/2959 [26:44<42:39,  1.41s/it, loss=1.12, v_num=2]
Epoch 0:  39% 1160/2959 [26:53<41:42,  1.39s/it, loss=1.12, v_num=2]
Epoch 0:  40% 1180/2959 [27:03<40:47,  1.38s/it, loss=1.12, v_num=2]
Epoch 0:  41% 1200/2959 [27:13<39:53,  1.36s/it, loss=1.12, v_num=2]
Epoch 0:  41% 1220/2959 [27:22<39:01,  1.35s/it, loss=1.12, v_num=2]
Epoch 0:  42% 1240/2959 [27:32<38:10,  1.33s/it, loss=1.12, v_num=2]
Epoch 0:  43% 1260/2959 [27:42<37:21,  1.32s/it, loss=1.12, v_num=2]
Epoch 0:  43% 1280/2959 [27:51<36:32,  1.31s/it, loss=1.12, v_num=2]
Epoch 0:  44% 1300/2959 [28:01<35:45,  1.29s/it, loss=1.12, v_num=2]
Epoch 0:  45% 1320/2959 [28:10<34:59,  1.28s/it, loss=1.12, v_num=2]
Epoch 0:  45% 1340/2959 [28:20<34:14,  1.27s/it, loss=1.12, v_num=2]
Epoch 0:  46% 1360/2959 [28:30<33:30,  1.26s/it, loss=1.12, v_num=2]
Epoch 0:  47% 1380/2959 [28:39<32:47,  1.25s/it, loss=1.12, v_num=2]
Epoch 0:  47% 1400/2959 [28:49<32:05,  1.24s/it, loss=1.12, v_num=2]
Epoch 0:  48% 1420/2959 [28:59<31:24,  1.22s/it, loss=1.12, v_num=2]
Epoch 0:  49% 1440/2959 [29:08<30:44,  1.21s/it, loss=1.12, v_num=2]
Epoch 0:  49% 1460/2959 [29:18<30:05,  1.20s/it, loss=1.12, v_num=2]
Epoch 0:  50% 1480/2959 [29:28<29:26,  1.19s/it, loss=1.12, v_num=2]
Epoch 0:  51% 1500/2959 [29:37<28:49,  1.19s/it, loss=1.12, v_num=2]
Epoch 0:  51% 1520/2959 [29:47<28:12,  1.18s/it, loss=1.12, v_num=2]
Epoch 0:  52% 1540/2959 [29:50<27:29,  1.16s/it, loss=1.12, v_num=2]***** Validation results *****
09/29/2021 10:20:06 - INFO - lightning - ***** Validation results *****
train/loss = 1.600659966468811
09/29/2021 10:20:06 - INFO - lightning - train/loss = 1.600659966468811
valid/loss = 1.1474024057388306
09/29/2021 10:20:06 - INFO - lightning - valid/loss = 1.1474024057388306
valid/micro_f1 = 51.87015982863734
09/29/2021 10:20:06 - INFO - lightning - valid/micro_f1 = 51.87015982863734
valid/auprc = 34.998676373770564
09/29/2021 10:20:06 - INFO - lightning - valid/auprc = 34.998676373770564
Epoch 0:  52% 1540/2959 [29:51<27:30,  1.16s/it, loss=1.12, v_num=2]
                                                 /usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
tcmalloc: large alloc 1569726464 bytes == 0x559403c52000 @  0x7fd224e78615 0x5593298ce4cc 0x5593299ae47a 0x5593298d4f0c 0x7fd220379a94 0x7fd22037b864 0x7fd22034b590 0x7fd210c91465 0x7fd210c8d9ca 0x7fd210c92609 0x7fd22034ef2b 0x7fd21ffd4200 0x5593298d2098 0x5593299454d9 0x55932993fced 0x5593298d2bda 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329940c0d 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940c0d 0x5593298d2afa
Epoch 0:  55% 1640/2959 [34:05<27:25,  1.25s/it, loss=1.05, v_num=2] Step: 600 - Loss: 0.7766194939613342
09/29/2021 10:24:23 - INFO - lightning - Step: 600 - Loss: 0.7766194939613342
Epoch 0:  59% 1740/2959 [38:25<26:54,  1.32s/it, loss=0.867, v_num=2]Step: 700 - Loss: 0.9821751713752747
09/29/2021 10:28:42 - INFO - lightning - Step: 700 - Loss: 0.9821751713752747
Epoch 0:  61% 1800/2959 [40:57<26:22,  1.37s/it, loss=0.937, v_num=2]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/486 [00:00<?, ?it/s]
Epoch 0:  62% 1820/2959 [41:06<25:43,  1.36s/it, loss=0.937, v_num=2]
Epoch 0:  62% 1840/2959 [41:15<25:05,  1.35s/it, loss=0.937, v_num=2]
Epoch 0:  63% 1860/2959 [41:25<24:28,  1.34s/it, loss=0.937, v_num=2]
Epoch 0:  64% 1880/2959 [41:35<23:52,  1.33s/it, loss=0.937, v_num=2]
Epoch 0:  64% 1900/2959 [41:44<23:16,  1.32s/it, loss=0.937, v_num=2]
Epoch 0:  65% 1920/2959 [41:54<22:40,  1.31s/it, loss=0.937, v_num=2]
Epoch 0:  66% 1940/2959 [42:04<22:05,  1.30s/it, loss=0.937, v_num=2]
Epoch 0:  66% 1960/2959 [42:13<21:31,  1.29s/it, loss=0.937, v_num=2]
Epoch 0:  67% 1980/2959 [42:23<20:57,  1.28s/it, loss=0.937, v_num=2]
Epoch 0:  68% 2000/2959 [42:33<20:24,  1.28s/it, loss=0.937, v_num=2]
Epoch 0:  68% 2020/2959 [42:42<19:51,  1.27s/it, loss=0.937, v_num=2]
Epoch 0:  69% 2040/2959 [42:52<19:18,  1.26s/it, loss=0.937, v_num=2]
Epoch 0:  70% 2060/2959 [43:01<18:46,  1.25s/it, loss=0.937, v_num=2]
Epoch 0:  70% 2080/2959 [43:11<18:15,  1.25s/it, loss=0.937, v_num=2]
Epoch 0:  71% 2100/2959 [43:21<17:44,  1.24s/it, loss=0.937, v_num=2]
Epoch 0:  72% 2120/2959 [43:30<17:13,  1.23s/it, loss=0.937, v_num=2]
Epoch 0:  72% 2140/2959 [43:40<16:42,  1.22s/it, loss=0.937, v_num=2]
Epoch 0:  73% 2160/2959 [43:50<16:12,  1.22s/it, loss=0.937, v_num=2]
Epoch 0:  74% 2180/2959 [43:59<15:43,  1.21s/it, loss=0.937, v_num=2]
Epoch 0:  74% 2200/2959 [44:09<15:14,  1.20s/it, loss=0.937, v_num=2]
Epoch 0:  75% 2220/2959 [44:18<14:45,  1.20s/it, loss=0.937, v_num=2]
Epoch 0:  76% 2240/2959 [44:28<14:16,  1.19s/it, loss=0.937, v_num=2]
Epoch 0:  76% 2260/2959 [44:38<13:48,  1.19s/it, loss=0.937, v_num=2]
Epoch 0:  77% 2280/2959 [44:47<13:20,  1.18s/it, loss=0.937, v_num=2]
Epoch 0:  78% 2300/2959 [44:50<12:50,  1.17s/it, loss=0.937, v_num=2]***** Validation results *****
09/29/2021 10:35:07 - INFO - lightning - ***** Validation results *****
train/loss = 0.6126298904418945
09/29/2021 10:35:07 - INFO - lightning - train/loss = 0.6126298904418945
valid/loss = 1.0093520879745483
09/29/2021 10:35:07 - INFO - lightning - valid/loss = 1.0093520879745483
valid/micro_f1 = 55.12799339388934
09/29/2021 10:35:07 - INFO - lightning - valid/micro_f1 = 55.12799339388934
valid/auprc = 40.783971212749776
09/29/2021 10:35:07 - INFO - lightning - valid/auprc = 40.783971212749776
Epoch 0:  78% 2300/2959 [44:51<12:51,  1.17s/it, loss=0.953, v_num=2]
                                                 /usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
tcmalloc: large alloc 1569726464 bytes == 0x559403c52000 @  0x7fd224e78615 0x5593298ce4cc 0x5593299ae47a 0x5593298d4f0c 0x7fd220379a94 0x7fd22037b864 0x7fd22034b590 0x7fd210c91465 0x7fd210c8d9ca 0x7fd210c92609 0x7fd22034ef2b 0x7fd21ffd4200 0x5593298d2098 0x5593299454d9 0x55932993fced 0x5593298d2bda 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940915 0x55932993f9ee 0x5593298d2bda 0x559329940c0d 0x55932993f9ee 0x5593298d2bda 0x559329944d00 0x5593298d2afa 0x559329940c0d 0x5593298d2afa
Epoch 0:  80% 2360/2959 [46:49<11:53,  1.19s/it, loss=0.871, v_num=2]Step: 800 - Loss: 1.2578922510147095
09/29/2021 10:37:07 - INFO - lightning - Step: 800 - Loss: 1.2578922510147095
Epoch 0:  83% 2460/2959 [51:09<10:22,  1.25s/it, loss=0.866, v_num=2]Step: 900 - Loss: 0.9722408056259155
09/29/2021 10:41:26 - INFO - lightning - Step: 900 - Loss: 0.9722408056259155
Epoch 0:  87% 2560/2959 [55:28<08:38,  1.30s/it, loss=0.812, v_num=2]Step: 1000 - Loss: 1.1005289554595947
09/29/2021 10:45:46 - INFO - lightning - Step: 1000 - Loss: 1.1005289554595947
Epoch 0:  87% 2580/2959 [55:59<08:13,  1.30s/it, loss=0.812, v_num=2]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/486 [00:00<?, ?it/s]
Epoch 0:  88% 2600/2959 [56:07<07:45,  1.30s/it, loss=0.812, v_num=2]
Epoch 0:  89% 2620/2959 [56:17<07:17,  1.29s/it, loss=0.812, v_num=2]
Epoch 0:  89% 2640/2959 [56:27<06:49,  1.28s/it, loss=0.812, v_num=2]
Epoch 0:  90% 2660/2959 [56:36<06:21,  1.28s/it, loss=0.812, v_num=2]
Epoch 0:  91% 2680/2959 [56:46<05:54,  1.27s/it, loss=0.812, v_num=2]
Epoch 0:  91% 2700/2959 [56:56<05:27,  1.27s/it, loss=0.812, v_num=2]
Epoch 0:  92% 2720/2959 [57:05<05:01,  1.26s/it, loss=0.812, v_num=2]
Epoch 0:  93% 2740/2959 [57:15<04:34,  1.25s/it, loss=0.812, v_num=2]
Epoch 0:  93% 2760/2959 [57:25<04:08,  1.25s/it, loss=0.812, v_num=2]
Epoch 0:  94% 2780/2959 [57:34<03:42,  1.24s/it, loss=0.812, v_num=2]
Epoch 0:  95% 2800/2959 [57:44<03:16,  1.24s/it, loss=0.812, v_num=2]
Epoch 0:  95% 2820/2959 [57:53<02:51,  1.23s/it, loss=0.812, v_num=2]
Epoch 0:  96% 2840/2959 [58:03<02:25,  1.23s/it, loss=0.812, v_num=2]
Epoch 0:  97% 2860/2959 [58:13<02:00,  1.22s/it, loss=0.812, v_num=2]
Epoch 0:  97% 2880/2959 [58:22<01:36,  1.22s/it, loss=0.812, v_num=2]
Epoch 0:  98% 2900/2959 [58:32<01:11,  1.21s/it, loss=0.812, v_num=2]
Epoch 0:  99% 2920/2959 [58:42<00:47,  1.21s/it, loss=0.812, v_num=2]
Epoch 0:  99% 2940/2959 [58:51<00:22,  1.20s/it, loss=0.812, v_num=2]
Epoch 0: 100% 2959/2959 [59:08<00:00,  1.20s/it, loss=0.812, v_num=2]
Validating:  82% 400/486 [03:11<00:41,  2.08it/s]
Validating:  86% 420/486 [03:21<00:31,  2.08it/s]
Validating:  91% 440/486 [03:30<00:22,  2.08it/s]
Validating:  95% 460/486 [03:40<00:12,  2.08it/s]
Validating:  99% 480/486 [03:50<00:02,  2.08it/s]
Validating: 100% 486/486 [03:52<00:00,  2.08it/s]***** Validation results *****
09/29/2021 10:50:08 - INFO - lightning - ***** Validation results *****
train/loss = 0.9427295327186584
09/29/2021 10:50:08 - INFO - lightning - train/loss = 0.9427295327186584
valid/loss = 1.0403474569320679
09/29/2021 10:50:08 - INFO - lightning - valid/loss = 1.0403474569320679
valid/micro_f1 = 54.50158347157292
09/29/2021 10:50:08 - INFO - lightning - valid/micro_f1 = 54.50158347157292
valid/auprc = 43.516728906364634
09/29/2021 10:50:08 - INFO - lightning - valid/auprc = 43.516728906364634
Epoch 0: 100% 2959/2959 [59:53<00:00,  1.21s/it, loss=0.843, v_num=2]
                                                 /usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
Epoch 0: 100% 2959/2959 [1:00:12<00:00,  1.22s/it, loss=0.843, v_num=2]
09/29/2021 10:50:29 - INFO - pytorch_lightning.accelerators.gpu - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Testing: 100% 486/486 [03:53<00:00,  2.08it/s]***** Test results *****
09/29/2021 10:54:23 - INFO - lightning - ***** Test results *****
train/loss = 1.0006123781204224
09/29/2021 10:54:23 - INFO - lightning - train/loss = 1.0006123781204224
valid/loss = 1.0403474569320679
09/29/2021 10:54:23 - INFO - lightning - valid/loss = 1.0403474569320679
valid/micro_f1 = 54.50158347157292
09/29/2021 10:54:23 - INFO - lightning - valid/micro_f1 = 54.50158347157292
valid/auprc = 43.516728906364634
09/29/2021 10:54:23 - INFO - lightning - valid/auprc = 43.516728906364634
Testing: 100% 486/486 [03:54<00:00,  2.08it/s]
--------------------------------------------------------------------------------
 - valid/loss : 1.0403474569320679
 - valid/micro_f1 : 54.50158347157292
 - valid/auprc : 43.516728906364634
--------------------------------------------------------------------------------

03 Test

!python KLUE-Baseline/run_klue.py \
evaluate \
--task klue-re \
--output_dir ./re/klue-re \
--data_dir ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1 \
--model_name_or_path ./re/klue-re/version_2/transformers \
--eval_batch_size 32 \

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: MetricBase will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
09/29/2021 11:08:23 - INFO - __main__ - Arguments:
09/29/2021 11:08:23 - INFO - __main__ - 	                command : evaluate
09/29/2021 11:08:23 - INFO - __main__ - 	                   task : klue-re
09/29/2021 11:08:23 - INFO - __main__ - 	             output_dir : ./re/klue-re
09/29/2021 11:08:23 - INFO - __main__ - 	                   gpus : None
09/29/2021 11:08:23 - INFO - __main__ - 	                   fp16 : False
09/29/2021 11:08:23 - INFO - __main__ - 	   num_sanity_val_steps : 2
09/29/2021 11:08:23 - INFO - __main__ - 	              tpu_cores : None
09/29/2021 11:08:23 - INFO - __main__ - 	      gradient_clip_val : 1.0
09/29/2021 11:08:23 - INFO - __main__ - 	accumulate_grad_batches : 1
09/29/2021 11:08:23 - INFO - __main__ - 	                   seed : 42
09/29/2021 11:08:23 - INFO - __main__ - 	             metric_key : loss
09/29/2021 11:08:23 - INFO - __main__ - 	               patience : 5
09/29/2021 11:08:23 - INFO - __main__ - 	    early_stopping_mode : max
09/29/2021 11:08:23 - INFO - __main__ - 	               data_dir : ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1
09/29/2021 11:08:23 - INFO - __main__ - 	        train_file_name : None
09/29/2021 11:08:23 - INFO - __main__ - 	          dev_file_name : None
09/29/2021 11:08:23 - INFO - __main__ - 	         test_file_name : None
09/29/2021 11:08:23 - INFO - __main__ - 	            num_workers : 4
09/29/2021 11:08:23 - INFO - __main__ - 	       train_batch_size : 32
09/29/2021 11:08:23 - INFO - __main__ - 	        eval_batch_size : 16
09/29/2021 11:08:23 - INFO - __main__ - 	         max_seq_length : 128
09/29/2021 11:08:23 - INFO - __main__ - 	      relation_filename : relation_list.json
09/29/2021 11:08:23 - INFO - __main__ - 	     model_name_or_path : ./re/klue-re/version_2/transformers
09/29/2021 11:08:23 - INFO - __main__ - 	            config_name : 
09/29/2021 11:08:23 - INFO - __main__ - 	         tokenizer_name : None
09/29/2021 11:08:23 - INFO - __main__ - 	              cache_dir : 
09/29/2021 11:08:23 - INFO - __main__ - 	      encoder_layerdrop : None
09/29/2021 11:08:23 - INFO - __main__ - 	      decoder_layerdrop : None
09/29/2021 11:08:23 - INFO - __main__ - 	                dropout : None
09/29/2021 11:08:23 - INFO - __main__ - 	      attention_dropout : None
09/29/2021 11:08:23 - INFO - __main__ - 	          learning_rate : 5e-05
09/29/2021 11:08:23 - INFO - __main__ - 	           lr_scheduler : linear
09/29/2021 11:08:23 - INFO - __main__ - 	           weight_decay : 0.0
09/29/2021 11:08:23 - INFO - __main__ - 	           adam_epsilon : 1e-08
09/29/2021 11:08:23 - INFO - __main__ - 	           warmup_steps : None
09/29/2021 11:08:23 - INFO - __main__ - 	           warmup_ratio : None
09/29/2021 11:08:23 - INFO - __main__ - 	             max_epochs : 4
09/29/2021 11:08:23 - INFO - __main__ - 	              adafactor : False
09/29/2021 11:08:23 - INFO - __main__ - 	     verbose_step_count : 100
Global seed set to 42
09/29/2021 11:08:23 - INFO - lightning - Global seed set to 42
GPU available: True, used: False
09/29/2021 11:08:23 - INFO - lightning - GPU available: True, used: False
TPU available: None, using: 0 TPU cores
09/29/2021 11:08:23 - INFO - lightning - TPU available: None, using: 0 TPU cores
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: GPU available but not used. Set the --gpus flag when calling the script.
  warnings.warn(*args, **kwargs)
09/29/2021 11:08:23 - INFO - klue_baseline.data.utils - Using BertTokenizer for fixing tokenization result
09/29/2021 11:08:23 - INFO - klue_baseline.data.base - Creating features from dataset file at ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1
09/29/2021 11:08:23 - INFO - klue_baseline.data.klue_re - Loading from ./KLUE-Baseline/data/klue_benchmark/klue-re-v1.1/klue-re-v1.1_dev.json
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00000
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00000', text_a="20대 남성 <subj>A</subj>(26)씨가 아버지 치료비를 위해 B(<obj>30</obj>)씨가 모아둔 돈을 훔쳐 인터넷 방송 BJ에게 '별풍선'으로 쏜 사실이 알려졌다.", text_b=None, label='no_relation')
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin tokens: ['20', '##대', '남성', '<subj>', 'A', '</subj>', '(', '26', ')', '씨', '##가', '아버지', '치료비', '##를', '위해', 'B', '(', '<obj>', '30', '</obj>', ')', '씨', '##가', '모아', '##둔', '돈', '##을', '훔쳐', '인터넷', '방송', 'B', '##J', '##에', '##게', "'", '별', '##풍', '##선', "'", '으로', '쏜', '사실', '##이', '알려졌', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - fixed tokens: ['20', '##대', '남성', '<subj>', 'A', '</subj>', '(', '26', ')', '씨', '##가', '아버지', '치료비', '##를', '위해', 'B', '(', '<obj>', '30', '</obj>', ')', '씨', '##가', '모아', '##둔', '돈', '##을', '훔쳐', '인터넷', '방송', 'B', '##J', '##에', '##게', "'", '별', '##풍', '##선', "'", '으로', '쏜', '사실', '##이', '알려졌', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3619, 2104, 4576, 32000, 37, 32001, 12, 4210, 13, 1370, 2116, 4096, 18014, 2138, 3627, 38, 12, 32002, 3740, 32003, 13, 1370, 2116, 5302, 2684, 850, 2069, 12908, 4254, 3861, 38, 2303, 2170, 2318, 11, 1156, 2533, 2020, 11, 3603, 1356, 3669, 2052, 4757, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00001
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00001', text_a='그러나 <subj>심 의원</subj>은 보좌진이 접속 권한을 받아 <obj>정부</obj> 업무추진비 사용 내역 등을 다운받았음에도 정부가 허위 사실을 유포하는 등 국정감사 활동을 방해하고 있다고 반박했고, 김동연 경제부총리 겸 기획재정부 장관과 김재훈 재정정보원장, 기재부 관계자 등을 무고 등 혐의로 전날 맞고발했다.', text_b=None, label='no_relation')
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin tokens: ['그러나', '<subj>', '심', '의원', '</subj>', '은', '보좌', '##진', '##이', '접속', '권한', '##을', '받아', '<obj>', '정부', '</obj>', '업무', '##추', '##진', '##비', '사용', '내역', '등', '##을', '다운', '##받', '##았', '##음', '##에도', '정부', '##가', '허위', '사실', '##을', '유포', '##하', '##는', '등', '국정', '##감사', '활동', '##을', '방해', '##하고', '있', '##다고', '반박', '##했', '##고', ',', '김동연', '경제', '##부', '##총리', '겸', '기획', '##재', '##정부', '장관', '##과', '김재', '##훈', '재정', '##정보', '##원장', ',', '기재', '##부', '관계자', '등', '##을', '무고', '등', '혐의', '##로', '전날', '맞', '##고', '##발', '##했', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - fixed tokens: ['그러나', '<subj>', '심', '의원', '</subj>', '##은', '보좌', '##진', '##이', '접속', '권한', '##을', '받아', '<obj>', '정부', '</obj>', '업무', '##추', '##진', '##비', '사용', '내역', '등', '##을', '다운', '##받', '##았', '##음', '##에도', '정부', '##가', '허위', '사실', '##을', '유포', '##하', '##는', '등', '국정', '##감사', '활동', '##을', '방해', '##하고', '있', '##다고', '반박', '##했', '##고', ',', '김동연', '경제', '##부', '##총리', '겸', '기획', '##재', '##정부', '장관', '##과', '김재', '##훈', '재정', '##정보', '##원장', ',', '기재', '##부', '관계자', '등', '##을', '무고', '등', '혐의', '##로', '전날', '맞', '##고', '##발', '##했', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3678, 32000, 1331, 3712, 32001, 2073, 8541, 2043, 2052, 8407, 5851, 2069, 21372, 32002, 3659, 32003, 4148, 2159, 2043, 2151, 3704, 10140, 886, 2069, 6163, 2757, 2886, 2053, 6509, 3659, 2116, 7441, 3669, 2069, 12502, 2205, 2259, 886, 4567, 27065, 3746, 2069, 6345, 19521, 1513, 4683, 7703, 2371, 2088, 16, 28320, 3674, 2144, 6384, 588, 4417, 2070, 11722, 4166, 2145, 7863, 2034, 4426, 19861, 10293, 16, 7475, 2144, 3810, 886, 2069, 17891, 886, 4456, 2200, 5978, 1047, 2088, 2311, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00002
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00002', text_a="영화 《룸》에서 감금되어 살아가는 여자의 아들 '<subj>잭</subj>' 역으로 주목받으며, 크리틱스 초이스 영화상 최우수 아역연기상, <obj>캐나다 스크린 어워드</obj>(캐나다 영화 & 텔레비전 아카데미상) 영화부문 최우수 남우주연상을 수상하였고, 미국 배우 조합상(SAG) 최우수 남우조연상에 후보 지명되었다.", text_b=None, label='no_relation')
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin tokens: ['영화', '《', '룸', '》', '에서', '감금', '##되', '##어', '살아가', '##는', '여자', '##의', '아들', "'", '<subj>', '잭', '</subj>', "'", '역', '##으로', '주목', '##받', '##으며', ',', '크리', '##틱', '##스', '초이스', '영화', '##상', '최우수', '아역', '##연', '##기', '##상', ',', '<obj>', '캐나다', '스크린', '어워드', '</obj>', '(', '캐나다', '영화', '&', '텔레비전', '아카데미', '##상', ')', '영화', '##부', '##문', '최우수', '남', '##우주', '##연', '##상', '##을', '수상', '##하', '##였', '##고', ',', '미국', '배우', '조합', '##상', '(', 'SA', '##G', ')', '최우수', '남', '##우', '##조', '##연', '##상', '##에', '후보', '지명', '##되', '##었', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - fixed tokens: ['영화', '《', '룸', '》', '에서', '감금', '##되', '##어', '살아가', '##는', '여자', '##의', '아들', "'", '<subj>', '잭', '</subj>', "'", '역', '##으로', '주목', '##받', '##으며', ',', '크리', '##틱', '##스', '초이스', '영화', '##상', '최우수', '아역', '##연', '##기', '##상', ',', '<obj>', '캐나다', '스크린', '어워드', '</obj>', '(', '캐나다', '영화', '&', '텔레비전', '아카데미', '##상', ')', '영화', '##부', '##문', '최우수', '남', '##우주', '##연', '##상', '##을', '수상', '##하', '##였', '##고', ',', '미국', '배우', '조합', '##상', '(', 'SA', '##G', ')', '최우수', '남', '##우', '##조', '##연', '##상', '##에', '후보', '지명', '##되', '##었', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 3771, 170, 1006, 171, 3604, 18033, 2496, 2051, 5933, 2259, 3883, 2079, 4200, 11, 32000, 1529, 32001, 11, 1429, 6233, 4439, 2757, 4007, 16, 10869, 2960, 2255, 29366, 3771, 2290, 11896, 25849, 2156, 2015, 2290, 16, 32002, 7066, 8559, 18085, 32003, 12, 7066, 3771, 10, 7361, 9101, 2290, 13, 3771, 2144, 2346, 11896, 723, 21347, 2156, 2290, 2069, 4812, 2205, 2507, 2088, 16, 3666, 4165, 4936, 2290, 12, 21707, 2341, 13, 11896, 723, 2137, 2446, 2156, 2290, 2170, 3733, 7119, 2496, 2359, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00003
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00003', text_a='전라남도(<obj>도지사</obj> 김영록)는 <subj>해양수산부</subj>의 2020년 어촌뉴딜300 공모사업에 15개 연안 시군 70개소 7천61억 원 규모를 신청했다고 14일 밝혔다.', text_b=None, label='no_relation')
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin tokens: ['전라남도', '(', '<obj>', '도지사', '</obj>', '김영', '##록', ')', '는', '<subj>', '해양수산부', '</subj>', '의', '2020', '##년', '어촌', '##뉴', '##딜', '##30', '##0', '공모', '##사업', '##에', '15', '##개', '연안', '시군', '70', '##개', '##소', '7', '##천', '##61', '##억', '원', '규모', '##를', '신청', '##했', '##다고', '14', '##일', '밝혔', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - fixed tokens: ['전라남도', '(', '<obj>', '도지사', '</obj>', '김영', '##록', ')', '는', '<subj>', '해양수산부', '</subj>', '##의', '2020', '##년', '어촌', '##뉴', '##딜', '##30', '##0', '공모', '##사업', '##에', '15', '##개', '연안', '시군', '70', '##개', '##소', '7', '##천', '##61', '##억', '원', '규모', '##를', '신청', '##했', '##다고', '14', '##일', '밝혔', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 22272, 12, 32002, 9035, 32003, 5553, 2351, 13, 793, 32000, 15995, 32001, 2079, 7330, 2440, 17127, 3065, 3169, 7556, 2082, 5637, 9841, 2170, 3749, 2019, 11189, 15773, 4531, 2019, 2024, 27, 2337, 26421, 2028, 1478, 3849, 2138, 4373, 2371, 4683, 3909, 2210, 3705, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - *** Example ***
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - guid: klue-re-v1_dev_00004
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin example: InputExample(guid='klue-re-v1_dev_00004', text_a='<subj>부산항만공사</subj>(BPA, 사장 남기찬)는 <obj>27일</obj> 아동의 인권존중과 아동학대 예방에 대한 인식을 제고하기 위해 전 임직원을 대상으로 아동학대 예방교육을 실시했다.', text_b=None, label='no_relation')
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - origin tokens: ['<subj>', '부산항', '##만', '##공사', '</subj>', '(', 'B', '##PA', ',', '사장', '남기', '##찬', ')', '는', '<obj>', '27', '##일', '</obj>', '아동', '##의', '인권', '##존', '##중', '##과', '아동학', '##대', '예방', '##에', '대한', '인식', '##을', '제고', '##하기', '위해', '전', '임직원', '##을', '대상', '##으로', '아동학', '##대', '예방', '##교육', '##을', '실시', '##했', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - fixed tokens: ['<subj>', '부산항', '##만', '##공사', '</subj>', '(', 'B', '##PA', ',', '사장', '남기', '##찬', ')', '는', '<obj>', '27', '##일', '</obj>', '아동', '##의', '인권', '##존', '##중', '##과', '아동학', '##대', '예방', '##에', '대한', '인식', '##을', '제고', '##하기', '위해', '전', '임직원', '##을', '대상', '##으로', '아동학', '##대', '예방', '##교육', '##을', '실시', '##했', '##다', '.']
09/29/2021 11:08:28 - INFO - klue_baseline.data.klue_re - features: InputFeatures(input_ids=[0, 32000, 17003, 2154, 5634, 32001, 12, 38, 19672, 16, 4184, 6804, 2166, 13, 793, 32002, 4207, 2210, 32003, 5332, 2079, 5188, 2869, 2284, 2145, 30333, 2104, 5005, 2170, 3618, 4263, 2069, 7928, 31302, 3627, 1537, 6809, 2069, 3756, 6233, 30333, 2104, 5005, 6177, 2069, 4130, 2371, 2062, 18, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
09/29/2021 11:08:28 - INFO - klue_baseline.data.base - Prepare dev dataset (Count: 7765) 
Testing:  74% 360/486 [33:20<11:32,  5.49s/it]

previous

BERT로 RE task 다루기

next

Dependency Parsing