KoELECTRA 모델로 fine-tuning 하기¶
Task : KLUE-NER
담당자: 전명준 님
최종수정일: 21-09-15
본 자료는 가짜연구소 3기 KLUE 로 모델 평가하기 크루 활동으로 작성됨
진행 상황¶
validation set 3 epoch 기준
entity_macro_f1 = 84.86452184810163
character_macro_f1 = 92.31535366001408
validation loss가 지속적으로 감소되어 early stopping 기준이 되지 않아 중간 체크포인트 저장
fine-tuning 과정¶
사용 모델 : KoELECTRA
주요 파라미터
max length : 256
learning_rate 3e-5
num_train_epochs 5
train_batch_size 16
initialize¶
cd ./drive/MyDrive/Colab Notebooks/klue/KLUE-baseline-main
/content/drive/MyDrive/Colab Notebooks/klue/KLUE-baseline-main
ls
data/ model/ README_ko.md run_all.sh
klue_baseline/ mypy.ini README.md run_klue.py
LICENSE.md ner/ requirements-dev.txt setup.cfg
Makefile pyproject.toml requirements.txt wos/
!pip install -r requirements.txt
Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
Collecting transformers==3.5.1
Downloading transformers-3.5.1-py3-none-any.whl (1.3 MB)
|████████████████████████████████| 1.3 MB 4.2 MB/s
?25hCollecting pytorch-lightning==1.1.0
Downloading pytorch_lightning-1.1.0-py3-none-any.whl (665 kB)
|████████████████████████████████| 665 kB 43.4 MB/s
?25hCollecting fsspec==2021.4.0
Downloading fsspec-2021.4.0-py3-none-any.whl (108 kB)
|████████████████████████████████| 108 kB 44.2 MB/s
?25hCollecting overrides==3.1.0
Downloading overrides-3.1.0.tar.gz (11 kB)
Collecting scikit-learn==0.24.1
Downloading scikit_learn-0.24.1-cp37-cp37m-manylinux2010_x86_64.whl (22.3 MB)
|████████████████████████████████| 22.3 MB 19.7 MB/s
?25hCollecting seqeval
Downloading seqeval-1.2.2.tar.gz (43 kB)
|████████████████████████████████| 43 kB 2.0 MB/s
?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (2.23.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (21.0)
Collecting tokenizers==0.9.3
Downloading tokenizers-0.9.3-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
|████████████████████████████████| 2.9 MB 40.8 MB/s
?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (3.0.12)
Collecting sentencepiece==0.1.91
Downloading sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
|████████████████████████████████| 1.1 MB 38.5 MB/s
?25hRequirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (3.17.3)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (4.62.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (2019.12.20)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 2)) (1.19.5)
Collecting sacremoses
Downloading sacremoses-0.0.45-py3-none-any.whl (895 kB)
|████████████████████████████████| 895 kB 58.2 MB/s
?25hCollecting PyYAML>=5.1
Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
|████████████████████████████████| 636 kB 53.7 MB/s
?25hRequirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (2.6.0)
Collecting future>=0.17.1
Downloading future-0.18.2.tar.gz (829 kB)
|████████████████████████████████| 829 kB 54.0 MB/s
?25hRequirement already satisfied: torch>=1.3 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.9.0+cu102)
Collecting threadpoolctl>=2.0.0
Downloading threadpoolctl-2.2.0-py3-none-any.whl (12 kB)
Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.1->-r requirements.txt (line 11)) (1.4.1)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.1->-r requirements.txt (line 11)) (1.0.1)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.39.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (0.4.5)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.8.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.0.1)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (0.6.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.34.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (0.12.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (3.3.4)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (0.37.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (57.4.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.15.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (4.2.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (1.3.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (4.6.4)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (0.4.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 2)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 2)) (2021.5.30)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 2)) (1.24.3)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (3.1.1)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.3->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (3.7.4.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.1.0->-r requirements.txt (line 3)) (3.5.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==3.5.1->-r requirements.txt (line 2)) (2.4.7)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.5.1->-r requirements.txt (line 2)) (7.1.2)
Building wheels for collected packages: overrides, future, seqeval
Building wheel for overrides (setup.py) ... ?25l?25hdone
Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10186 sha256=65c0837cbc1d8726e7cf6708d08c98ce5fc061e1e795f270af90ee9cab72a7a0
Stored in directory: /root/.cache/pip/wheels/3a/0d/38/01a9bc6e20dcfaf0a6a7b552d03137558ba1c38aea47644682
Building wheel for future (setup.py) ... ?25l?25hdone
Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=c2988347f84bcb1db502b80074597dae18e29ede2ab7681004c3f7cf9fc11893
Stored in directory: /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
Building wheel for seqeval (setup.py) ... ?25l?25hdone
Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16181 sha256=20b732873d6ade81e373d4304036a27cc979baccb2f0b113414e7c417ae5beb3
Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
Successfully built overrides future seqeval
Installing collected packages: threadpoolctl, tokenizers, sentencepiece, scikit-learn, sacremoses, PyYAML, future, fsspec, transformers, seqeval, pytorch-lightning, overrides
Attempting uninstall: scikit-learn
Found existing installation: scikit-learn 0.22.2.post1
Uninstalling scikit-learn-0.22.2.post1:
Successfully uninstalled scikit-learn-0.22.2.post1
Attempting uninstall: PyYAML
Found existing installation: PyYAML 3.13
Uninstalling PyYAML-3.13:
Successfully uninstalled PyYAML-3.13
Attempting uninstall: future
Found existing installation: future 0.16.0
Uninstalling future-0.16.0:
Successfully uninstalled future-0.16.0
Successfully installed PyYAML-5.4.1 fsspec-2021.4.0 future-0.18.2 overrides-3.1.0 pytorch-lightning-1.1.0 sacremoses-0.0.45 scikit-learn-0.24.1 sentencepiece-0.1.91 seqeval-1.2.2 threadpoolctl-2.2.0 tokenizers-0.9.3 transformers-3.5.1
!pip install torchtext==0.8.0 torch==1.7.1 pytorch-lightning==1.2.2
Requirement already satisfied: torchtext==0.8.0 in /usr/local/lib/python3.7/dist-packages (0.8.0)
Collecting torch==1.7.1
Downloading torch-1.7.1-cp37-cp37m-manylinux1_x86_64.whl (776.8 MB)
|████████████████████████████████| 776.8 MB 17 kB/s
?25hRequirement already satisfied: pytorch-lightning==1.2.2 in /usr/local/lib/python3.7/dist-packages (1.2.2)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torchtext==0.8.0) (2.23.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from torchtext==0.8.0) (4.62.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchtext==0.8.0) (1.19.5)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1) (3.7.4.3)
Requirement already satisfied: PyYAML!=5.4.*,>=5.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (5.3.1)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (2.6.0)
Requirement already satisfied: fsspec[http]>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (2021.4.0)
Requirement already satisfied: future>=0.17.1 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning==1.2.2) (0.18.2)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from fsspec[http]>=0.8.1->pytorch-lightning==1.2.2) (3.7.4.post0)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.17.3)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.39.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.0.1)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.12.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.34.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (57.4.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.4.5)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.3.4)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.8.0)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.37.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.15.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (4.2.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (4.7.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (1.3.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (4.6.4)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (0.4.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (2021.5.30)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (1.24.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.8.0) (2.10)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.1.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning==1.2.2) (5.1.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning==1.2.2) (1.6.3)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning==1.2.2) (21.2.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning==1.2.2) (3.0.1)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.2.2) (3.5.0)
Installing collected packages: torch
Attempting uninstall: torch
Found existing installation: torch 1.9.0+cu102
Uninstalling torch-1.9.0+cu102:
Successfully uninstalled torch-1.9.0+cu102
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.10.0+cu102 requires torch==1.9.0, but you have torch 1.7.1 which is incompatible.
Successfully installed torch-1.7.1
train¶
!python run_klue.py \
train \
--task klue-ner \
--output_dir ./ner_output \
--data_dir ./ner \
--model_name_or_path monologg/koelectra-base-v3-discriminator \
--learning_rate 3e-5 \
--num_train_epochs 5 \
--train_batch_size 16 \
--eval_batch_size 16 \
--max_seq_length 256 \
--gradient_accumulation_steps 1 \
--warmup_ratio 0.1 \
--weight_decay 0.01 \
--max_grad_norm 1.0 \
--patience 100000 \
--metric_key character_macro_f1 \
--gpus 0 \
# --parallel_decoding \
# --truncate \
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: MetricBase will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
warnings.warn(*args, **kwargs)
08/22/2021 06:56:38 - INFO - __main__ - Arguments:
08/22/2021 06:56:38 - INFO - __main__ - command : train
08/22/2021 06:56:38 - INFO - __main__ - task : klue-ner
08/22/2021 06:56:38 - INFO - __main__ - output_dir : ./ner_output
08/22/2021 06:56:38 - INFO - __main__ - gpus : [0]
08/22/2021 06:56:38 - INFO - __main__ - fp16 : False
08/22/2021 06:56:38 - INFO - __main__ - num_sanity_val_steps : 2
08/22/2021 06:56:38 - INFO - __main__ - tpu_cores : None
08/22/2021 06:56:38 - INFO - __main__ - gradient_clip_val : 1.0
08/22/2021 06:56:38 - INFO - __main__ - accumulate_grad_batches : 1
08/22/2021 06:56:38 - INFO - __main__ - seed : 42
08/22/2021 06:56:38 - INFO - __main__ - metric_key : character_macro_f1
08/22/2021 06:56:38 - INFO - __main__ - patience : 100000
08/22/2021 06:56:38 - INFO - __main__ - early_stopping_mode : max
08/22/2021 06:56:38 - INFO - __main__ - data_dir : ./ner
08/22/2021 06:56:38 - INFO - __main__ - train_file_name : None
08/22/2021 06:56:38 - INFO - __main__ - dev_file_name : None
08/22/2021 06:56:38 - INFO - __main__ - test_file_name : None
08/22/2021 06:56:38 - INFO - __main__ - num_workers : 4
08/22/2021 06:56:38 - INFO - __main__ - train_batch_size : 16
08/22/2021 06:56:38 - INFO - __main__ - eval_batch_size : 16
08/22/2021 06:56:38 - INFO - __main__ - max_seq_length : 256
08/22/2021 06:56:38 - INFO - __main__ - model_name_or_path : monologg/koelectra-base-v3-discriminator
08/22/2021 06:56:38 - INFO - __main__ - config_name :
08/22/2021 06:56:38 - INFO - __main__ - tokenizer_name : None
08/22/2021 06:56:38 - INFO - __main__ - cache_dir :
08/22/2021 06:56:38 - INFO - __main__ - encoder_layerdrop : None
08/22/2021 06:56:38 - INFO - __main__ - decoder_layerdrop : None
08/22/2021 06:56:38 - INFO - __main__ - dropout : None
08/22/2021 06:56:38 - INFO - __main__ - attention_dropout : None
08/22/2021 06:56:38 - INFO - __main__ - learning_rate : 3e-05
08/22/2021 06:56:38 - INFO - __main__ - lr_scheduler : linear
08/22/2021 06:56:38 - INFO - __main__ - weight_decay : 0.01
08/22/2021 06:56:38 - INFO - __main__ - adam_epsilon : 1e-08
08/22/2021 06:56:38 - INFO - __main__ - warmup_steps : None
08/22/2021 06:56:38 - INFO - __main__ - warmup_ratio : 0.1
08/22/2021 06:56:38 - INFO - __main__ - max_epochs : 5
08/22/2021 06:56:38 - INFO - __main__ - adafactor : False
08/22/2021 06:56:38 - INFO - __main__ - verbose_step_count : 100
Global seed set to 42
08/22/2021 06:56:38 - INFO - lightning - Global seed set to 42
GPU available: True, used: True
08/22/2021 06:56:38 - INFO - lightning - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
08/22/2021 06:56:38 - INFO - lightning - TPU available: None, using: 0 TPU cores
08/22/2021 06:56:42 - INFO - klue_baseline.data.utils - Using ElectraTokenizer for fixing tokenization result
08/22/2021 06:56:42 - INFO - klue_baseline.data.base - Creating features from dataset file at ./ner
08/22/2021 06:56:42 - INFO - klue_baseline.data.klue_ner - Loading from ./ner/klue-ner-v1.1_train.tsv
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_train_00000_wikitree
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 6342, 33383, 11028, 7005, 2693, 4314, 4527, 4325, 4104, 4073, 4129, 2630, 4317, 4473, 4031, 4469, 4149, 4200, 25, 4011, 4036, 8198, 4073, 4034, 11329, 7779, 8331, 2019, 4062, 4482, 4239, 4106, 4110, 6411, 4279, 4031, 4239, 3771, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 2, 2, 12, 2, 3, 3, 3, 3, 12, 12, 2, 3, 3, 3, 3, 12, 12, 10, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_train_00001_nsmc
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 3757, 27789, 4129, 13038, 4292, 6395, 11335, 4283, 6588, 10561, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_train_00002_nsmc
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 14984, 4172, 11651, 4162, 4073, 4034, 13810, 2773, 25, 4217, 4110, 11772, 4451, 4176, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_train_00003_wikitree
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 6294, 9515, 4292, 9181, 4239, 13493, 6662, 4258, 4069, 9039, 4031, 4195, 4071, 23, 4031, 4034, 3092, 10749, 25, 4217, 4501, 4629, 8377, 4292, 9289, 4325, 6236, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 12, 4, 5, 5, 5, 5, 5, 5, 5, 10, 11, 12, 12, 12, 6, 7, 7, 7, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_train_00004_nsmc
08/22/2021 06:57:07 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 3675, 18772, 11184, 11852, 4283, 7144, 4192, 3809, 4309, 4275, 4234, 2297, 4525, 4200, 5745, 4112, 2089, 4543, 4234, 20482, 5, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 12, 12, 12, 0, 1, 1, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:07 - INFO - klue_baseline.data.base - Prepare train dataset (Count: 21008)
08/22/2021 06:57:07 - INFO - klue_baseline.data.base - Creating features from dataset file at ./ner
08/22/2021 06:57:07 - INFO - klue_baseline.data.klue_ner - Loading from ./ner/klue-ner-v1.1_dev.tsv
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00000-wikitree
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 6345, 4112, 2489, 12486, 17875, 6832, 4292, 6369, 4283, 2731, 4561, 4234, 2462, 12, 7393, 13, 2087, 12486, 6426, 2141, 4117, 12, 7578, 13, 3071, 2446, 6427, 4282, 4292, 2024, 4112, 6746, 4239, 7288, 4491, 12981, 4398, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 4, 12, 12, 12, 12, 12, 12, 12, 12, 0, 12, 12, 12, 12, 10, 12, 12, 12, 12, 0, 12, 12, 10, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00001-wikitree
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 38, 3071, 4034, 8544, 20315, 9032, 4174, 4007, 13129, 4880, 2048, 13830, 3084, 2967, 3123, 4480, 4219, 16, 6770, 4007, 3311, 4112, 6335, 4073, 10384, 2967, 3249, 4031, 4172, 4292, 6908, 4820, 4138, 4118, 37, 3071, 4110, 2373, 7848, 4279, 4200, 3083, 4494, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 0, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 0, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00002-wikitree
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 8099, 37, 4303, 4234, 21, 4291, 4027, 22637, 6974, 4112, 6834, 10261, 4469, 16, 8099, 38, 4303, 4112, 8853, 4469, 4007, 4480, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 10, 11, 11, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00003-wikitree
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 11, 13444, 11, 3231, 25, 21530, 7111, 6599, 6249, 4073, 4129, 11333, 6584, 14849, 6388, 4139, 11, 27999, 33267, 11, 3232, 14985, 4239, 11504, 6394, 4239, 16, 6232, 6238, 4366, 9470, 4151, 16982, 2630, 4073, 8438, 16503, 4172, 4282, 4292, 13521, 4576, 6216, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 6, 7, 7, 2, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 6, 7, 7, 12, 12, 6, 12, 12, 12, 10, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00004-wikitree
08/22/2021 06:57:13 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 20286, 4189, 15491, 12, 32828, 13, 3170, 6804, 28226, 4049, 8697, 12, 33501, 4290, 13, 3238, 9960, 4007, 8518, 17312, 2967, 3249, 11649, 11, 3755, 4034, 6365, 4073, 7449, 4200, 4110, 3430, 4112, 8275, 4006, 4112, 13500, 13900, 4292, 24734, 4219, 3249, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 0, 1, 12, 12, 0, 12, 12, 12, 0, 1, 12, 12, 0, 1, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:13 - INFO - klue_baseline.data.base - Prepare dev dataset (Count: 5000)
08/22/2021 06:57:13 - INFO - klue_baseline.data.base - Creating features from dataset file at ./ner
08/22/2021 06:57:13 - INFO - klue_baseline.data.klue_ner - Test dataset doesn't exists. So loading dev dataset instead.
08/22/2021 06:57:13 - INFO - klue_baseline.data.klue_ner - Loading from ./ner/klue-ner-v1.1_dev.tsv
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00000-wikitree
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 6345, 4112, 2489, 12486, 17875, 6832, 4292, 6369, 4283, 2731, 4561, 4234, 2462, 12, 7393, 13, 2087, 12486, 6426, 2141, 4117, 12, 7578, 13, 3071, 2446, 6427, 4282, 4292, 2024, 4112, 6746, 4239, 7288, 4491, 12981, 4398, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 4, 12, 12, 12, 12, 12, 12, 12, 12, 0, 12, 12, 12, 12, 10, 12, 12, 12, 12, 0, 12, 12, 10, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00001-wikitree
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 38, 3071, 4034, 8544, 20315, 9032, 4174, 4007, 13129, 4880, 2048, 13830, 3084, 2967, 3123, 4480, 4219, 16, 6770, 4007, 3311, 4112, 6335, 4073, 10384, 2967, 3249, 4031, 4172, 4292, 6908, 4820, 4138, 4118, 37, 3071, 4110, 2373, 7848, 4279, 4200, 3083, 4494, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 0, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 0, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00002-wikitree
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 8099, 37, 4303, 4234, 21, 4291, 4027, 22637, 6974, 4112, 6834, 10261, 4469, 16, 8099, 38, 4303, 4112, 8853, 4469, 4007, 4480, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 10, 11, 11, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00003-wikitree
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 11, 13444, 11, 3231, 25, 21530, 7111, 6599, 6249, 4073, 4129, 11333, 6584, 14849, 6388, 4139, 11, 27999, 33267, 11, 3232, 14985, 4239, 11504, 6394, 4239, 16, 6232, 6238, 4366, 9470, 4151, 16982, 2630, 4073, 8438, 16503, 4172, 4282, 4292, 13521, 4576, 6216, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 6, 7, 7, 2, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 6, 7, 7, 12, 12, 6, 12, 12, 12, 10, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - *** Example ***
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00004-wikitree
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 20286, 4189, 15491, 12, 32828, 13, 3170, 6804, 28226, 4049, 8697, 12, 33501, 4290, 13, 3238, 9960, 4007, 8518, 17312, 2967, 3249, 11649, 11, 3755, 4034, 6365, 4073, 7449, 4200, 4110, 3430, 4112, 8275, 4006, 4112, 13500, 13900, 4292, 24734, 4219, 3249, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 0, 1, 12, 12, 0, 12, 12, 12, 0, 1, 12, 12, 0, 1, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/22/2021 06:57:19 - INFO - klue_baseline.data.base - Prepare test dataset (Count: 5000)
08/22/2021 06:57:19 - INFO - klue_baseline.data.utils - Using ElectraTokenizer for fixing tokenization result
Some weights of the model checkpoint at monologg/koelectra-base-v3-discriminator were not used when initializing ElectraForTokenClassification: ['discriminator_predictions.dense.weight', 'discriminator_predictions.dense.bias', 'discriminator_predictions.dense_prediction.weight', 'discriminator_predictions.dense_prediction.bias']
- This IS expected if you are initializing ElectraForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ElectraForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of ElectraForTokenClassification were not initialized from the model checkpoint at monologg/koelectra-base-v3-discriminator and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
08/22/2021 06:57:30 - INFO - __main__ - Start to run the full optimization routine.
08/22/2021 06:57:30 - INFO - pytorch_lightning.accelerators.gpu - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Epoch 0: 4% 100/2565 [02:30<1:01:59, 1.51s/it, loss=1.76, v_num=1]Step: 100 - Loss: 1.3293771743774414
08/22/2021 07:00:51 - INFO - lightning - Step: 100 - Loss: 1.3293771743774414
Epoch 0: 8% 200/2565 [05:31<1:05:16, 1.66s/it, loss=0.481, v_num=1]Step: 200 - Loss: 0.5640891790390015
08/22/2021 07:03:51 - INFO - lightning - Step: 200 - Loss: 0.5640891790390015
Epoch 0: 12% 300/2565 [08:30<1:04:16, 1.70s/it, loss=0.306, v_num=1]Step: 300 - Loss: 0.3904399275779724
08/22/2021 07:06:51 - INFO - lightning - Step: 300 - Loss: 0.3904399275779724
Epoch 0: 13% 340/2565 [09:38<1:03:04, 1.70s/it, loss=0.273, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 0: 14% 360/2565 [09:48<1:00:03, 1.63s/it, loss=0.273, v_num=1]
Epoch 0: 15% 380/2565 [09:58<57:19, 1.57s/it, loss=0.273, v_num=1]
Epoch 0: 16% 400/2565 [10:08<54:51, 1.52s/it, loss=0.273, v_num=1]
Epoch 0: 16% 420/2565 [10:18<52:36, 1.47s/it, loss=0.273, v_num=1]
Epoch 0: 17% 440/2565 [10:27<50:32, 1.43s/it, loss=0.273, v_num=1]
Epoch 0: 18% 460/2565 [10:37<48:38, 1.39s/it, loss=0.273, v_num=1]
Epoch 0: 19% 480/2565 [10:47<46:53, 1.35s/it, loss=0.273, v_num=1]
Epoch 0: 19% 500/2565 [10:57<45:15, 1.32s/it, loss=0.273, v_num=1]
Epoch 0: 20% 520/2565 [11:07<43:44, 1.28s/it, loss=0.273, v_num=1]
Epoch 0: 21% 540/2565 [11:17<42:19, 1.25s/it, loss=0.273, v_num=1]
Epoch 0: 22% 560/2565 [11:27<41:00, 1.23s/it, loss=0.273, v_num=1]
Epoch 0: 23% 580/2565 [11:37<39:45, 1.20s/it, loss=0.273, v_num=1]
Epoch 0: 23% 600/2565 [11:46<38:35, 1.18s/it, loss=0.273, v_num=1]
Epoch 0: 24% 620/2565 [11:56<37:28, 1.16s/it, loss=0.273, v_num=1]
Epoch 0: 25% 640/2565 [12:06<36:25, 1.14s/it, loss=0.273, v_num=1]
Epoch 0: 26% 660/2565 [12:28<36:01, 1.13s/it, loss=0.273, v_num=1]***** Validation results *****
08/22/2021 07:11:34 - INFO - lightning - ***** Validation results *****
train/loss = 0.16344191133975983
08/22/2021 07:11:34 - INFO - lightning - train/loss = 0.16344191133975983
valid/loss = 0.2464587390422821
08/22/2021 07:11:34 - INFO - lightning - valid/loss = 0.2464587390422821
valid/entity_macro_f1 = 56.34341838116749
08/22/2021 07:11:34 - INFO - lightning - valid/entity_macro_f1 = 56.34341838116749
valid/character_macro_f1 = 68.72206836299816
08/22/2021 07:11:34 - INFO - lightning - valid/character_macro_f1 = 68.72206836299816
Epoch 0: 26% 660/2565 [13:44<39:38, 1.25s/it, loss=0.247, v_num=1]
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
tcmalloc: large alloc 1086128128 bytes == 0x560019262000 @ 0x7f1a9d753615 0x55ff8b0a302c 0x55ff8b18317a 0x55ff8b0a9a7c 0x7f1a98c54a94 0x7f1a98c56864 0x7f1a98c26590 0x7f1a8956c465 0x7f1a895689ca 0x7f1a8956d609 0x7f1a98c29f2b 0x7f1a988af200 0x55ff8b0a6bf8 0x55ff8b11a6f2 0x55ff8b115235 0x55ff8b0a773a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b115d67 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115d67 0x55ff8b0a765a
tcmalloc: large alloc 1357660160 bytes == 0x560059e32000 @ 0x7f1a9d753615 0x55ff8b0a302c 0x55ff8b18317a 0x55ff8b0a9a7c 0x7f1a98c54a94 0x7f1a98c56864 0x7f1a98c26590 0x7f1a8956c465 0x7f1a895689ca 0x7f1a8956d609 0x7f1a98c29f2b 0x7f1a988af200 0x55ff8b0a6bf8 0x55ff8b11a6f2 0x55ff8b115235 0x55ff8b0a773a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b115d67 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115d67 0x55ff8b0a765a
tcmalloc: large alloc 1697079296 bytes == 0x5600e8f0a000 @ 0x7f1a9d753615 0x55ff8b0a302c 0x55ff8b18317a 0x55ff8b0a9a7c 0x7f1a98c54a94 0x7f1a98c56864 0x7f1a98c26590 0x7f1a8956c465 0x7f1a895689ca 0x7f1a8956d609 0x7f1a98c29f2b 0x7f1a988af200 0x55ff8b0a6bf8 0x55ff8b11a6f2 0x55ff8b115235 0x55ff8b0a773a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b115d67 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115d67 0x55ff8b0a765a
Epoch 0: 29% 740/2565 [16:14<40:04, 1.32s/it, loss=0.174, v_num=1]Step: 400 - Loss: 0.1407061517238617
08/22/2021 07:14:35 - INFO - lightning - Step: 400 - Loss: 0.1407061517238617
Epoch 0: 33% 840/2565 [19:14<39:31, 1.37s/it, loss=0.116, v_num=1]Step: 500 - Loss: 0.09795821458101273
08/22/2021 07:17:35 - INFO - lightning - Step: 500 - Loss: 0.09795821458101273
Epoch 0: 37% 940/2565 [22:14<38:26, 1.42s/it, loss=0.123, v_num=1]Step: 600 - Loss: 0.1189262792468071
08/22/2021 07:20:34 - INFO - lightning - Step: 600 - Loss: 0.1189262792468071
Epoch 0: 39% 1000/2565 [24:14<37:55, 1.45s/it, loss=0.0986, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 0: 40% 1020/2565 [24:24<36:57, 1.44s/it, loss=0.0986, v_num=1]
Epoch 0: 41% 1040/2565 [24:34<36:01, 1.42s/it, loss=0.0986, v_num=1]
Epoch 0: 41% 1060/2565 [24:44<35:07, 1.40s/it, loss=0.0986, v_num=1]
Epoch 0: 42% 1080/2565 [24:53<34:14, 1.38s/it, loss=0.0986, v_num=1]
Epoch 0: 43% 1100/2565 [25:03<33:22, 1.37s/it, loss=0.0986, v_num=1]
Epoch 0: 44% 1120/2565 [25:13<32:32, 1.35s/it, loss=0.0986, v_num=1]
Epoch 0: 44% 1140/2565 [25:23<31:44, 1.34s/it, loss=0.0986, v_num=1]
Epoch 0: 45% 1160/2565 [25:33<30:57, 1.32s/it, loss=0.0986, v_num=1]
Epoch 0: 46% 1180/2565 [25:43<30:11, 1.31s/it, loss=0.0986, v_num=1]
Epoch 0: 47% 1200/2565 [25:53<29:26, 1.29s/it, loss=0.0986, v_num=1]
Epoch 0: 48% 1220/2565 [26:03<28:43, 1.28s/it, loss=0.0986, v_num=1]
Epoch 0: 48% 1240/2565 [26:12<28:00, 1.27s/it, loss=0.0986, v_num=1]
Epoch 0: 49% 1260/2565 [26:22<27:19, 1.26s/it, loss=0.0986, v_num=1]
Epoch 0: 50% 1280/2565 [26:32<26:38, 1.24s/it, loss=0.0986, v_num=1]
Epoch 0: 51% 1300/2565 [26:42<25:59, 1.23s/it, loss=0.0986, v_num=1]
Epoch 0: 51% 1320/2565 [26:59<25:27, 1.23s/it, loss=0.0986, v_num=1]***** Validation results *****
08/22/2021 07:26:11 - INFO - lightning - ***** Validation results *****
train/loss = 0.056011758744716644
08/22/2021 07:26:11 - INFO - lightning - train/loss = 0.056011758744716644
valid/loss = 0.11426108330488205
08/22/2021 07:26:11 - INFO - lightning - valid/loss = 0.11426108330488205
valid/entity_macro_f1 = 80.42282265209936
08/22/2021 07:26:11 - INFO - lightning - valid/entity_macro_f1 = 80.42282265209936
valid/character_macro_f1 = 89.62352599459807
08/22/2021 07:26:11 - INFO - lightning - valid/character_macro_f1 = 89.62352599459807
Epoch 0: 51% 1320/2565 [28:21<26:44, 1.29s/it, loss=0.0946, v_num=1]
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
tcmalloc: large alloc 1697079296 bytes == 0x5600e8f0a000 @ 0x7f1a9d753615 0x55ff8b0a302c 0x55ff8b18317a 0x55ff8b0a9a7c 0x7f1a98c54a94 0x7f1a98c56864 0x7f1a98c26590 0x7f1a8956c465 0x7f1a895689ca 0x7f1a8956d609 0x7f1a98c29f2b 0x7f1a988af200 0x55ff8b0a6bf8 0x55ff8b11a6f2 0x55ff8b115235 0x55ff8b0a773a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115b0e 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b115d67 0x55ff8b114c35 0x55ff8b0a773a 0x55ff8b119f40 0x55ff8b0a765a 0x55ff8b115d67 0x55ff8b0a765a
Epoch 0: 54% 1380/2565 [29:53<25:40, 1.30s/it, loss=0.0966, v_num=1]Step: 700 - Loss: 0.10944793373346329
08/22/2021 07:28:14 - INFO - lightning - Step: 700 - Loss: 0.10944793373346329
Epoch 0: 58% 1480/2565 [32:53<24:06, 1.33s/it, loss=0.0849, v_num=1]Step: 800 - Loss: 0.13102607429027557
08/22/2021 07:31:13 - INFO - lightning - Step: 800 - Loss: 0.13102607429027557
Epoch 0: 62% 1580/2565 [35:53<22:22, 1.36s/it, loss=0.0746, v_num=1]Step: 900 - Loss: 0.10563196241855621
08/22/2021 07:34:13 - INFO - lightning - Step: 900 - Loss: 0.10563196241855621
Epoch 0: 65% 1680/2565 [38:30<20:17, 1.38s/it, loss=0.079, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 0: 66% 1700/2565 [38:41<19:41, 1.37s/it, loss=0.079, v_num=1]
Epoch 0: 67% 1720/2565 [38:50<19:05, 1.36s/it, loss=0.079, v_num=1]
Epoch 0: 68% 1740/2565 [39:00<18:29, 1.35s/it, loss=0.079, v_num=1]
Epoch 0: 69% 1760/2565 [39:10<17:55, 1.34s/it, loss=0.079, v_num=1]
Epoch 0: 69% 1780/2565 [39:20<17:21, 1.33s/it, loss=0.079, v_num=1]
Epoch 0: 70% 1800/2565 [39:30<16:47, 1.32s/it, loss=0.079, v_num=1]
Epoch 0: 71% 1820/2565 [39:40<16:14, 1.31s/it, loss=0.079, v_num=1]
Epoch 0: 72% 1840/2565 [39:50<15:41, 1.30s/it, loss=0.079, v_num=1]
Epoch 0: 73% 1860/2565 [40:00<15:09, 1.29s/it, loss=0.079, v_num=1]
Epoch 0: 73% 1880/2565 [40:09<14:38, 1.28s/it, loss=0.079, v_num=1]
Epoch 0: 74% 1900/2565 [40:19<14:06, 1.27s/it, loss=0.079, v_num=1]
Epoch 0: 75% 1920/2565 [40:29<13:36, 1.27s/it, loss=0.079, v_num=1]
Epoch 0: 76% 1940/2565 [40:39<13:05, 1.26s/it, loss=0.079, v_num=1]
Epoch 0: 76% 1960/2565 [40:49<12:36, 1.25s/it, loss=0.079, v_num=1]
Epoch 0: 77% 1980/2565 [40:59<12:06, 1.24s/it, loss=0.079, v_num=1]
Epoch 0: 78% 2000/2565 [41:19<11:40, 1.24s/it, loss=0.079, v_num=1]***** Validation results *****
08/22/2021 07:40:29 - INFO - lightning - ***** Validation results *****
train/loss = 0.11901336163282394
08/22/2021 07:40:29 - INFO - lightning - train/loss = 0.11901336163282394
valid/loss = 0.087803915143013
08/22/2021 07:40:29 - INFO - lightning - valid/loss = 0.087803915143013
valid/entity_macro_f1 = 83.51168832368522
08/22/2021 07:40:29 - INFO - lightning - valid/entity_macro_f1 = 83.51168832368522
valid/character_macro_f1 = 91.65541953879132
08/22/2021 07:40:29 - INFO - lightning - valid/character_macro_f1 = 91.65541953879132
Epoch 0: 78% 2000/2565 [42:39<12:02, 1.28s/it, loss=0.0815, v_num=1]
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
Epoch 0: 79% 2020/2565 [43:33<11:45, 1.29s/it, loss=0.0782, v_num=1]Step: 1000 - Loss: 0.06341170519590378
08/22/2021 07:41:54 - INFO - lightning - Step: 1000 - Loss: 0.06341170519590378
Epoch 0: 83% 2120/2565 [46:32<09:46, 1.32s/it, loss=0.0916, v_num=1]Step: 1100 - Loss: 0.056680820882320404
08/22/2021 07:44:53 - INFO - lightning - Step: 1100 - Loss: 0.056680820882320404
Epoch 0: 87% 2220/2565 [49:31<07:41, 1.34s/it, loss=0.0695, v_num=1]Step: 1200 - Loss: 0.0766267254948616
08/22/2021 07:47:51 - INFO - lightning - Step: 1200 - Loss: 0.0766267254948616
Epoch 0: 90% 2320/2565 [52:30<05:32, 1.36s/it, loss=0.068, v_num=1] Step: 1300 - Loss: 0.13601723313331604
08/22/2021 07:50:51 - INFO - lightning - Step: 1300 - Loss: 0.13601723313331604
Epoch 0: 91% 2340/2565 [53:16<05:07, 1.37s/it, loss=0.068, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 0: 92% 2360/2565 [53:26<04:38, 1.36s/it, loss=0.068, v_num=1]
Epoch 0: 93% 2380/2565 [53:36<04:10, 1.35s/it, loss=0.068, v_num=1]
Epoch 0: 94% 2400/2565 [53:46<03:41, 1.34s/it, loss=0.068, v_num=1]
Epoch 0: 94% 2420/2565 [53:56<03:13, 1.34s/it, loss=0.068, v_num=1]
Epoch 0: 95% 2440/2565 [54:06<02:46, 1.33s/it, loss=0.068, v_num=1]
Epoch 0: 96% 2460/2565 [54:15<02:18, 1.32s/it, loss=0.068, v_num=1]
Epoch 0: 97% 2480/2565 [54:25<01:51, 1.32s/it, loss=0.068, v_num=1]
Epoch 0: 97% 2500/2565 [54:35<01:25, 1.31s/it, loss=0.068, v_num=1]
Epoch 0: 98% 2520/2565 [54:45<00:58, 1.30s/it, loss=0.068, v_num=1]
Epoch 0: 99% 2540/2565 [54:55<00:32, 1.30s/it, loss=0.068, v_num=1]
Epoch 0: 100% 2560/2565 [55:05<00:06, 1.29s/it, loss=0.068, v_num=1]
Epoch 0: 100% 2565/2565 [55:19<00:00, 1.29s/it, loss=0.068, v_num=1]
Validating: 83% 260/313 [02:08<00:26, 2.04it/s]
Validating: 89% 280/313 [02:17<00:16, 2.04it/s]
Validating: 96% 300/313 [02:27<00:06, 2.04it/s]
Validating: 100% 313/313 [02:33<00:00, 2.05it/s]***** Validation results *****
08/22/2021 07:55:11 - INFO - lightning - ***** Validation results *****
train/loss = 0.08980929851531982
08/22/2021 07:55:11 - INFO - lightning - train/loss = 0.08980929851531982
valid/loss = 0.08884353935718536
08/22/2021 07:55:11 - INFO - lightning - valid/loss = 0.08884353935718536
valid/entity_macro_f1 = 83.81046810746847
08/22/2021 07:55:11 - INFO - lightning - valid/entity_macro_f1 = 83.81046810746847
valid/character_macro_f1 = 91.5924944028394
08/22/2021 07:55:11 - INFO - lightning - valid/character_macro_f1 = 91.5924944028394
Epoch 0: 100% 2565/2565 [57:21<00:00, 1.34s/it, loss=0.0703, v_num=1]
Epoch 1: 3% 80/2565 [02:03<1:03:50, 1.54s/it, loss=0.0502, v_num=1]Step: 1400 - Loss: 0.02445843815803528
08/22/2021 07:58:10 - INFO - lightning - Step: 1400 - Loss: 0.02445843815803528
Epoch 1: 7% 180/2565 [05:03<1:06:54, 1.68s/it, loss=0.0571, v_num=1]Step: 1500 - Loss: 0.08885642886161804
08/22/2021 08:01:10 - INFO - lightning - Step: 1500 - Loss: 0.08885642886161804
Epoch 1: 11% 280/2565 [08:03<1:05:47, 1.73s/it, loss=0.0551, v_num=1]Step: 1600 - Loss: 0.06799381226301193
08/22/2021 08:04:11 - INFO - lightning - Step: 1600 - Loss: 0.06799381226301193
Epoch 1: 13% 340/2565 [09:38<1:03:06, 1.70s/it, loss=0.0381, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 1: 14% 360/2565 [09:48<1:00:06, 1.64s/it, loss=0.0381, v_num=1]
Epoch 1: 15% 380/2565 [09:58<57:22, 1.58s/it, loss=0.0381, v_num=1]
Epoch 1: 16% 400/2565 [10:08<54:54, 1.52s/it, loss=0.0381, v_num=1]
Epoch 1: 16% 420/2565 [10:18<52:39, 1.47s/it, loss=0.0381, v_num=1]
Epoch 1: 17% 440/2565 [10:28<50:35, 1.43s/it, loss=0.0381, v_num=1]
Epoch 1: 18% 460/2565 [10:38<48:41, 1.39s/it, loss=0.0381, v_num=1]
Epoch 1: 19% 480/2565 [10:48<46:56, 1.35s/it, loss=0.0381, v_num=1]
Epoch 1: 19% 500/2565 [10:58<45:18, 1.32s/it, loss=0.0381, v_num=1]
Epoch 1: 20% 520/2565 [11:08<43:47, 1.29s/it, loss=0.0381, v_num=1]
Epoch 1: 21% 540/2565 [11:18<42:23, 1.26s/it, loss=0.0381, v_num=1]
Epoch 1: 22% 560/2565 [11:28<41:03, 1.23s/it, loss=0.0381, v_num=1]
Epoch 1: 23% 580/2565 [11:38<39:48, 1.20s/it, loss=0.0381, v_num=1]
Epoch 1: 23% 600/2565 [11:47<38:38, 1.18s/it, loss=0.0381, v_num=1]
Epoch 1: 24% 620/2565 [11:57<37:31, 1.16s/it, loss=0.0381, v_num=1]
Epoch 1: 25% 640/2565 [12:07<36:28, 1.14s/it, loss=0.0381, v_num=1]
Epoch 1: 26% 660/2565 [12:33<36:13, 1.14s/it, loss=0.0381, v_num=1]***** Validation results *****
08/22/2021 08:09:12 - INFO - lightning - ***** Validation results *****
train/loss = 0.02105019986629486
08/22/2021 08:09:12 - INFO - lightning - train/loss = 0.02105019986629486
valid/loss = 0.08123032003641129
08/22/2021 08:09:12 - INFO - lightning - valid/loss = 0.08123032003641129
valid/entity_macro_f1 = 85.48703130310057
08/22/2021 08:09:12 - INFO - lightning - valid/entity_macro_f1 = 85.48703130310057
valid/character_macro_f1 = 92.33728567808713
08/22/2021 08:09:12 - INFO - lightning - valid/character_macro_f1 = 92.33728567808713
Epoch 1: 26% 660/2565 [13:44<39:41, 1.25s/it, loss=0.0373, v_num=1]
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
Epoch 1: 28% 720/2565 [15:44<40:21, 1.31s/it, loss=0.0466, v_num=1]Step: 1700 - Loss: 0.09545575827360153
08/22/2021 08:11:52 - INFO - lightning - Step: 1700 - Loss: 0.09545575827360153
Epoch 1: 32% 820/2565 [18:45<39:55, 1.37s/it, loss=0.0487, v_num=1]Step: 1800 - Loss: 0.0201523769646883
08/22/2021 08:14:52 - INFO - lightning - Step: 1800 - Loss: 0.0201523769646883
Epoch 1: 36% 920/2565 [21:45<38:54, 1.42s/it, loss=0.0466, v_num=1]Step: 1900 - Loss: 0.058898136019706726
08/22/2021 08:17:52 - INFO - lightning - Step: 1900 - Loss: 0.058898136019706726
Epoch 1: 39% 1000/2565 [24:12<37:52, 1.45s/it, loss=0.0506, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 1: 40% 1020/2565 [24:22<36:55, 1.43s/it, loss=0.0506, v_num=1]
Epoch 1: 41% 1040/2565 [24:32<35:59, 1.42s/it, loss=0.0506, v_num=1]
Epoch 1: 41% 1060/2565 [24:42<35:04, 1.40s/it, loss=0.0506, v_num=1]
Epoch 1: 42% 1080/2565 [24:52<34:11, 1.38s/it, loss=0.0506, v_num=1]
Epoch 1: 43% 1100/2565 [25:02<33:20, 1.37s/it, loss=0.0506, v_num=1]
Epoch 1: 44% 1120/2565 [25:12<32:30, 1.35s/it, loss=0.0506, v_num=1]
Epoch 1: 44% 1140/2565 [25:21<31:42, 1.33s/it, loss=0.0506, v_num=1]
Epoch 1: 45% 1160/2565 [25:31<30:55, 1.32s/it, loss=0.0506, v_num=1]
Epoch 1: 46% 1180/2565 [25:41<30:09, 1.31s/it, loss=0.0506, v_num=1]
Epoch 1: 47% 1200/2565 [25:51<29:24, 1.29s/it, loss=0.0506, v_num=1]
Epoch 1: 48% 1220/2565 [26:01<28:41, 1.28s/it, loss=0.0506, v_num=1]
Epoch 1: 48% 1240/2565 [26:11<27:58, 1.27s/it, loss=0.0506, v_num=1]
Epoch 1: 49% 1260/2565 [26:21<27:17, 1.25s/it, loss=0.0506, v_num=1]
Epoch 1: 50% 1280/2565 [26:31<26:37, 1.24s/it, loss=0.0506, v_num=1]
Epoch 1: 51% 1300/2565 [26:40<25:57, 1.23s/it, loss=0.0506, v_num=1]
Epoch 1: 51% 1320/2565 [27:03<25:31, 1.23s/it, loss=0.0506, v_num=1]***** Validation results *****
08/22/2021 08:23:47 - INFO - lightning - ***** Validation results *****
train/loss = 0.05686040222644806
08/22/2021 08:23:47 - INFO - lightning - train/loss = 0.05686040222644806
valid/loss = 0.07933864742517471
08/22/2021 08:23:47 - INFO - lightning - valid/loss = 0.07933864742517471
valid/entity_macro_f1 = 84.34546469771233
08/22/2021 08:23:47 - INFO - lightning - valid/entity_macro_f1 = 84.34546469771233
valid/character_macro_f1 = 92.0663106777237
08/22/2021 08:23:47 - INFO - lightning - valid/character_macro_f1 = 92.0663106777237
Epoch 1: 51% 1320/2565 [28:20<26:43, 1.29s/it, loss=0.044, v_num=1]
Epoch 1: 53% 1360/2565 [29:08<25:48, 1.29s/it, loss=0.0509, v_num=1]Step: 2000 - Loss: 0.025849992409348488
08/22/2021 08:25:16 - INFO - lightning - Step: 2000 - Loss: 0.025849992409348488
Epoch 1: 57% 1460/2565 [32:08<24:19, 1.32s/it, loss=0.0538, v_num=1]Step: 2100 - Loss: 0.06244417652487755
08/22/2021 08:28:16 - INFO - lightning - Step: 2100 - Loss: 0.06244417652487755
Epoch 1: 61% 1560/2565 [35:09<22:38, 1.35s/it, loss=0.0566, v_num=1]Step: 2200 - Loss: 0.04154680669307709
08/22/2021 08:31:17 - INFO - lightning - Step: 2200 - Loss: 0.04154680669307709
Epoch 1: 65% 1680/2565 [38:15<20:09, 1.37s/it, loss=0.0529, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 1: 66% 1700/2565 [38:25<19:32, 1.36s/it, loss=0.0529, v_num=1]
Epoch 1: 67% 1720/2565 [38:35<18:57, 1.35s/it, loss=0.0529, v_num=1]
Epoch 1: 68% 1740/2565 [38:44<18:22, 1.34s/it, loss=0.0529, v_num=1]
Epoch 1: 69% 1760/2565 [38:54<17:47, 1.33s/it, loss=0.0529, v_num=1]
Epoch 1: 69% 1780/2565 [39:04<17:14, 1.32s/it, loss=0.0529, v_num=1]
Epoch 1: 70% 1800/2565 [39:14<16:40, 1.31s/it, loss=0.0529, v_num=1]
Epoch 1: 71% 1820/2565 [39:24<16:07, 1.30s/it, loss=0.0529, v_num=1]
Epoch 1: 72% 1840/2565 [39:34<15:35, 1.29s/it, loss=0.0529, v_num=1]
Epoch 1: 73% 1860/2565 [39:43<15:03, 1.28s/it, loss=0.0529, v_num=1]
Epoch 1: 73% 1880/2565 [39:53<14:32, 1.27s/it, loss=0.0529, v_num=1]
Epoch 1: 74% 1900/2565 [40:03<14:01, 1.27s/it, loss=0.0529, v_num=1]
Epoch 1: 75% 1920/2565 [40:13<13:30, 1.26s/it, loss=0.0529, v_num=1]
Epoch 1: 76% 1940/2565 [40:23<13:00, 1.25s/it, loss=0.0529, v_num=1]
Epoch 1: 76% 1960/2565 [40:32<12:30, 1.24s/it, loss=0.0529, v_num=1]
Epoch 1: 77% 1980/2565 [40:42<12:01, 1.23s/it, loss=0.0529, v_num=1]
Epoch 1: 78% 2000/2565 [41:03<11:35, 1.23s/it, loss=0.0529, v_num=1]***** Validation results *****
08/22/2021 08:37:48 - INFO - lightning - ***** Validation results *****
train/loss = 0.05247874557971954
08/22/2021 08:37:48 - INFO - lightning - train/loss = 0.05247874557971954
valid/loss = 0.0792388767004013
08/22/2021 08:37:48 - INFO - lightning - valid/loss = 0.0792388767004013
valid/entity_macro_f1 = 85.2295801470762
08/22/2021 08:37:48 - INFO - lightning - valid/entity_macro_f1 = 85.2295801470762
valid/character_macro_f1 = 92.2434934933197
08/22/2021 08:37:48 - INFO - lightning - valid/character_macro_f1 = 92.2434934933197
Epoch 1: 78% 2000/2565 [42:21<11:57, 1.27s/it, loss=0.0598, v_num=1]
Step: 2300 - Loss: 0.03342453017830849
08/22/2021 08:38:37 - INFO - lightning - Step: 2300 - Loss: 0.03342453017830849
Epoch 1: 82% 2100/2565 [45:29<10:04, 1.30s/it, loss=0.0626, v_num=1]Step: 2400 - Loss: 0.038280826061964035
08/22/2021 08:41:37 - INFO - lightning - Step: 2400 - Loss: 0.038280826061964035
Epoch 1: 86% 2200/2565 [48:28<08:02, 1.32s/it, loss=0.0488, v_num=1]Step: 2500 - Loss: 0.020692596212029457
08/22/2021 08:44:36 - INFO - lightning - Step: 2500 - Loss: 0.020692596212029457
Epoch 1: 90% 2300/2565 [51:28<05:55, 1.34s/it, loss=0.046, v_num=1]Step: 2600 - Loss: 0.03536217659711838
08/22/2021 08:47:36 - INFO - lightning - Step: 2600 - Loss: 0.03536217659711838
Epoch 1: 91% 2340/2565 [52:41<05:04, 1.35s/it, loss=0.0478, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 1: 92% 2360/2565 [52:51<04:35, 1.34s/it, loss=0.0478, v_num=1]
Epoch 1: 93% 2380/2565 [53:01<04:07, 1.34s/it, loss=0.0478, v_num=1]
Epoch 1: 94% 2400/2565 [53:11<03:39, 1.33s/it, loss=0.0478, v_num=1]
Epoch 1: 94% 2420/2565 [53:21<03:11, 1.32s/it, loss=0.0478, v_num=1]
Epoch 1: 95% 2440/2565 [53:31<02:44, 1.32s/it, loss=0.0478, v_num=1]
Epoch 1: 96% 2460/2565 [53:41<02:17, 1.31s/it, loss=0.0478, v_num=1]
Epoch 1: 97% 2480/2565 [53:50<01:50, 1.30s/it, loss=0.0478, v_num=1]
Epoch 1: 97% 2500/2565 [54:00<01:24, 1.30s/it, loss=0.0478, v_num=1]
Epoch 1: 98% 2520/2565 [54:10<00:58, 1.29s/it, loss=0.0478, v_num=1]
Epoch 1: 99% 2540/2565 [54:20<00:32, 1.28s/it, loss=0.0478, v_num=1]
Epoch 1: 100% 2560/2565 [54:30<00:06, 1.28s/it, loss=0.0478, v_num=1]
Epoch 1: 100% 2565/2565 [54:43<00:00, 1.28s/it, loss=0.0478, v_num=1]
Validating: 83% 260/313 [02:08<00:26, 2.03it/s]
Validating: 89% 280/313 [02:17<00:16, 2.04it/s]
Validating: 96% 300/313 [02:27<00:06, 2.04it/s]
Validating: 100% 313/313 [02:33<00:00, 2.05it/s]***** Validation results *****
08/22/2021 08:52:15 - INFO - lightning - ***** Validation results *****
train/loss = 0.09390611201524734
08/22/2021 08:52:15 - INFO - lightning - train/loss = 0.09390611201524734
valid/loss = 0.07821818441152573
08/22/2021 08:52:15 - INFO - lightning - valid/loss = 0.07821818441152573
valid/entity_macro_f1 = 85.82455802693441
08/22/2021 08:52:15 - INFO - lightning - valid/entity_macro_f1 = 85.82455802693441
valid/character_macro_f1 = 92.36557041019654
08/22/2021 08:52:15 - INFO - lightning - valid/character_macro_f1 = 92.36557041019654
Epoch 1: 100% 2565/2565 [56:48<00:00, 1.33s/it, loss=0.0378, v_num=1]
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:216: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
Epoch 2: 2% 60/2565 [01:37<1:07:45, 1.62s/it, loss=0.0422, v_num=1]Step: 2700 - Loss: 0.01750083453953266
08/22/2021 08:55:23 - INFO - lightning - Step: 2700 - Loss: 0.01750083453953266
Epoch 2: 6% 160/2565 [04:38<1:09:45, 1.74s/it, loss=0.0348, v_num=1]Step: 2800 - Loss: 0.029271313920617104
08/22/2021 08:58:24 - INFO - lightning - Step: 2800 - Loss: 0.029271313920617104
Epoch 2: 10% 260/2565 [07:39<1:07:57, 1.77s/it, loss=0.0372, v_num=1]Step: 2900 - Loss: 0.04126635566353798
08/22/2021 09:01:25 - INFO - lightning - Step: 2900 - Loss: 0.04126635566353798
Epoch 2: 13% 340/2565 [09:57<1:05:07, 1.76s/it, loss=0.0423, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 2: 14% 360/2565 [10:07<1:01:59, 1.69s/it, loss=0.0423, v_num=1]
Epoch 2: 15% 380/2565 [10:17<59:08, 1.62s/it, loss=0.0423, v_num=1]
Epoch 2: 16% 400/2565 [10:27<56:34, 1.57s/it, loss=0.0423, v_num=1]
Epoch 2: 16% 420/2565 [10:36<54:13, 1.52s/it, loss=0.0423, v_num=1]
Epoch 2: 17% 440/2565 [10:46<52:04, 1.47s/it, loss=0.0423, v_num=1]
Epoch 2: 18% 460/2565 [10:56<50:05, 1.43s/it, loss=0.0423, v_num=1]
Epoch 2: 19% 480/2565 [11:06<48:15, 1.39s/it, loss=0.0423, v_num=1]
Epoch 2: 19% 500/2565 [11:16<46:33, 1.35s/it, loss=0.0423, v_num=1]
Epoch 2: 20% 520/2565 [11:26<44:59, 1.32s/it, loss=0.0423, v_num=1]
Epoch 2: 21% 540/2565 [11:36<43:30, 1.29s/it, loss=0.0423, v_num=1]
Epoch 2: 22% 560/2565 [11:46<42:08, 1.26s/it, loss=0.0423, v_num=1]
Epoch 2: 23% 580/2565 [11:55<40:50, 1.23s/it, loss=0.0423, v_num=1]
Epoch 2: 23% 600/2565 [12:05<39:37, 1.21s/it, loss=0.0423, v_num=1]
Epoch 2: 24% 620/2565 [12:15<38:27, 1.19s/it, loss=0.0423, v_num=1]
Epoch 2: 25% 640/2565 [12:25<37:22, 1.16s/it, loss=0.0423, v_num=1]
Epoch 2: 26% 660/2565 [12:46<36:51, 1.16s/it, loss=0.0423, v_num=1]***** Validation results *****
08/22/2021 09:07:00 - INFO - lightning - ***** Validation results *****
train/loss = 0.011222519911825657
08/22/2021 09:07:00 - INFO - lightning - train/loss = 0.011222519911825657
valid/loss = 0.08279494196176529
08/22/2021 09:07:00 - INFO - lightning - valid/loss = 0.08279494196176529
valid/entity_macro_f1 = 84.57399766501517
08/22/2021 09:07:00 - INFO - lightning - valid/entity_macro_f1 = 84.57399766501517
valid/character_macro_f1 = 91.72960090959677
08/22/2021 09:07:00 - INFO - lightning - valid/character_macro_f1 = 91.72960090959677
Epoch 2: 26% 660/2565 [14:05<40:40, 1.28s/it, loss=0.0357, v_num=1]
Epoch 2: 27% 700/2565 [15:04<40:09, 1.29s/it, loss=0.0404, v_num=1]Step: 3000 - Loss: 0.02093154564499855
08/22/2021 09:08:50 - INFO - lightning - Step: 3000 - Loss: 0.02093154564499855
Epoch 2: 31% 800/2565 [18:05<39:55, 1.36s/it, loss=0.0343, v_num=1]Step: 3100 - Loss: 0.05348069965839386
08/22/2021 09:11:52 - INFO - lightning - Step: 3100 - Loss: 0.05348069965839386
Epoch 2: 35% 900/2565 [21:07<39:04, 1.41s/it, loss=0.025, v_num=1] Step: 3200 - Loss: 0.02934170700609684
08/22/2021 09:14:53 - INFO - lightning - Step: 3200 - Loss: 0.02934170700609684
Epoch 2: 39% 1000/2565 [24:03<37:39, 1.44s/it, loss=0.0377, v_num=1]
Validating: 0it [00:00, ?it/s]
Validating: 0% 0/313 [00:00<?, ?it/s]
Epoch 2: 40% 1020/2565 [24:13<36:41, 1.43s/it, loss=0.0377, v_num=1]
Epoch 2: 41% 1040/2565 [24:23<35:46, 1.41s/it, loss=0.0377, v_num=1]
Epoch 2: 41% 1060/2565 [24:33<34:51, 1.39s/it, loss=0.0377, v_num=1]
Epoch 2: 42% 1080/2565 [24:43<33:59, 1.37s/it, loss=0.0377, v_num=1]
Epoch 2: 43% 1100/2565 [24:53<33:08, 1.36s/it, loss=0.0377, v_num=1]
Epoch 2: 44% 1120/2565 [25:03<32:19, 1.34s/it, loss=0.0377, v_num=1]
Epoch 2: 44% 1140/2565 [25:13<31:31, 1.33s/it, loss=0.0377, v_num=1]
Epoch 2: 45% 1160/2565 [25:23<30:44, 1.31s/it, loss=0.0377, v_num=1]
Epoch 2: 46% 1180/2565 [25:32<29:59, 1.30s/it, loss=0.0377, v_num=1]
Epoch 2: 47% 1200/2565 [25:42<29:14, 1.29s/it, loss=0.0377, v_num=1]
Epoch 2: 48% 1220/2565 [25:52<28:31, 1.27s/it, loss=0.0377, v_num=1]
Epoch 2: 48% 1240/2565 [26:02<27:49, 1.26s/it, loss=0.0377, v_num=1]
Epoch 2: 49% 1260/2565 [26:12<27:08, 1.25s/it, loss=0.0377, v_num=1]
Epoch 2: 50% 1280/2565 [26:22<26:28, 1.24s/it, loss=0.0377, v_num=1]
Epoch 2: 51% 1300/2565 [26:32<25:49, 1.22s/it, loss=0.0377, v_num=1]
Epoch 2: 51% 1320/2565 [26:56<25:24, 1.22s/it, loss=0.0377, v_num=1]***** Validation results *****
08/22/2021 09:21:07 - INFO - lightning - ***** Validation results *****
train/loss = 0.01648346148431301
08/22/2021 09:21:07 - INFO - lightning - train/loss = 0.01648346148431301
valid/loss = 0.07890684902667999
08/22/2021 09:21:07 - INFO - lightning - valid/loss = 0.07890684902667999
valid/entity_macro_f1 = 84.86452184810163
08/22/2021 09:21:07 - INFO - lightning - valid/entity_macro_f1 = 84.86452184810163
valid/character_macro_f1 = 92.31535366001408
08/22/2021 09:21:07 - INFO - lightning - valid/character_macro_f1 = 92.31535366001408
Epoch 2: 51% 1320/2565 [28:12<26:36, 1.28s/it, loss=0.031, v_num=1]
Epoch 2: 52% 1340/2565 [28:33<26:06, 1.28s/it, loss=0.0316, v_num=1]Step: 3300 - Loss: 0.010714218951761723
08/22/2021 09:22:19 - INFO - lightning - Step: 3300 - Loss: 0.010714218951761723
Epoch 2: 56% 1440/2565 [31:34<24:40, 1.32s/it, loss=0.0319, v_num=1]Step: 3400 - Loss: 0.010880246758460999
08/22/2021 09:25:20 - INFO - lightning - Step: 3400 - Loss: 0.010880246758460999
Epoch 2: 58% 1500/2565 [33:26<23:44, 1.34s/it, loss=0.0355, v_num=1]
evalutation¶
!python run_klue.py \
Evaluate \
--task klue-ner \
--output_dir ./ner/ner_output \
--data_dir ./ner \
--model_name_or_path ./ner/ner_output/klue-ner/version_1/transformers \
--eval_batch_size 128 \
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: MetricBase will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
warnings.warn(*args, **kwargs)
08/25/2021 13:59:56 - INFO - __main__ - Arguments:
08/25/2021 13:59:56 - INFO - __main__ - command : Evaluate
08/25/2021 13:59:56 - INFO - __main__ - task : klue-ner
08/25/2021 13:59:56 - INFO - __main__ - output_dir : ./ner/ner_output
08/25/2021 13:59:56 - INFO - __main__ - gpus : None
08/25/2021 13:59:56 - INFO - __main__ - fp16 : False
08/25/2021 13:59:56 - INFO - __main__ - num_sanity_val_steps : 2
08/25/2021 13:59:56 - INFO - __main__ - tpu_cores : None
08/25/2021 13:59:56 - INFO - __main__ - gradient_clip_val : 1.0
08/25/2021 13:59:56 - INFO - __main__ - accumulate_grad_batches : 1
08/25/2021 13:59:56 - INFO - __main__ - seed : 42
08/25/2021 13:59:56 - INFO - __main__ - metric_key : loss
08/25/2021 13:59:56 - INFO - __main__ - patience : 5
08/25/2021 13:59:56 - INFO - __main__ - early_stopping_mode : max
08/25/2021 13:59:56 - INFO - __main__ - data_dir : ./ner
08/25/2021 13:59:56 - INFO - __main__ - train_file_name : None
08/25/2021 13:59:56 - INFO - __main__ - dev_file_name : None
08/25/2021 13:59:56 - INFO - __main__ - test_file_name : None
08/25/2021 13:59:56 - INFO - __main__ - num_workers : 4
08/25/2021 13:59:56 - INFO - __main__ - train_batch_size : 32
08/25/2021 13:59:56 - INFO - __main__ - eval_batch_size : 128
08/25/2021 13:59:56 - INFO - __main__ - max_seq_length : 128
08/25/2021 13:59:56 - INFO - __main__ - model_name_or_path : ./ner/ner_output/klue-ner/version_1/transformers
08/25/2021 13:59:56 - INFO - __main__ - config_name :
08/25/2021 13:59:56 - INFO - __main__ - tokenizer_name : None
08/25/2021 13:59:56 - INFO - __main__ - cache_dir :
08/25/2021 13:59:56 - INFO - __main__ - encoder_layerdrop : None
08/25/2021 13:59:56 - INFO - __main__ - decoder_layerdrop : None
08/25/2021 13:59:56 - INFO - __main__ - dropout : None
08/25/2021 13:59:56 - INFO - __main__ - attention_dropout : None
08/25/2021 13:59:56 - INFO - __main__ - learning_rate : 5e-05
08/25/2021 13:59:56 - INFO - __main__ - lr_scheduler : linear
08/25/2021 13:59:56 - INFO - __main__ - weight_decay : 0.0
08/25/2021 13:59:56 - INFO - __main__ - adam_epsilon : 1e-08
08/25/2021 13:59:56 - INFO - __main__ - warmup_steps : None
08/25/2021 13:59:56 - INFO - __main__ - warmup_ratio : None
08/25/2021 13:59:56 - INFO - __main__ - max_epochs : 4
08/25/2021 13:59:56 - INFO - __main__ - adafactor : False
08/25/2021 13:59:56 - INFO - __main__ - verbose_step_count : 100
Global seed set to 42
08/25/2021 13:59:56 - INFO - lightning - Global seed set to 42
GPU available: True, used: False
08/25/2021 13:59:56 - INFO - lightning - GPU available: True, used: False
TPU available: None, using: 0 TPU cores
08/25/2021 13:59:56 - INFO - lightning - TPU available: None, using: 0 TPU cores
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: GPU available but not used. Set the --gpus flag when calling the script.
warnings.warn(*args, **kwargs)
08/25/2021 14:00:00 - INFO - klue_baseline.data.utils - Using ElectraTokenizer for fixing tokenization result
08/25/2021 14:00:00 - INFO - klue_baseline.data.base - Creating features from dataset file at ./ner
08/25/2021 14:00:00 - INFO - klue_baseline.data.klue_ner - Loading from ./ner/klue-ner-v1.1_dev.tsv
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - *** Example ***
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00000-wikitree
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 6345, 4112, 2489, 12486, 17875, 6832, 4292, 6369, 4283, 2731, 4561, 4234, 2462, 12, 7393, 13, 2087, 12486, 6426, 2141, 4117, 12, 7578, 13, 3071, 2446, 6427, 4282, 4292, 2024, 4112, 6746, 4239, 7288, 4491, 12981, 4398, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 4, 12, 12, 12, 12, 12, 12, 12, 12, 0, 12, 12, 12, 12, 10, 12, 12, 12, 12, 0, 12, 12, 10, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - *** Example ***
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00001-wikitree
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 38, 3071, 4034, 8544, 20315, 9032, 4174, 4007, 13129, 4880, 2048, 13830, 3084, 2967, 3123, 4480, 4219, 16, 6770, 4007, 3311, 4112, 6335, 4073, 10384, 2967, 3249, 4031, 4172, 4292, 6908, 4820, 4138, 4118, 37, 3071, 4110, 2373, 7848, 4279, 4200, 3083, 4494, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 0, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 0, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - *** Example ***
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00002-wikitree
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 8099, 37, 4303, 4234, 21, 4291, 4027, 22637, 6974, 4112, 6834, 10261, 4469, 16, 8099, 38, 4303, 4112, 8853, 4469, 4007, 4480, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 10, 11, 11, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - *** Example ***
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00003-wikitree
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 11, 13444, 11, 3231, 25, 21530, 7111, 6599, 6249, 4073, 4129, 11333, 6584, 14849, 6388, 4139, 11, 27999, 33267, 11, 3232, 14985, 4239, 11504, 6394, 4239, 16, 6232, 6238, 4366, 9470, 4151, 16982, 2630, 4073, 8438, 16503, 4172, 4282, 4292, 13521, 4576, 6216, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 12, 12, 12, 12, 6, 7, 7, 2, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 6, 7, 7, 12, 12, 6, 12, 12, 12, 10, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - *** Example ***
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - guid: klue-ner-v1_dev_00004-wikitree
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - features: InputFeatures(input_ids=[2, 20286, 4189, 15491, 12, 32828, 13, 3170, 6804, 28226, 4049, 8697, 12, 33501, 4290, 13, 3238, 9960, 4007, 8518, 17312, 2967, 3249, 11649, 11, 3755, 4034, 6365, 4073, 7449, 4200, 4110, 3430, 4112, 8275, 4006, 4112, 13500, 13900, 4292, 24734, 4219, 3249, 4176, 18, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=[12, 0, 1, 12, 12, 0, 12, 12, 12, 0, 1, 12, 12, 0, 1, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12])
08/25/2021 14:00:08 - INFO - klue_baseline.data.base - Prepare dev dataset (Count: 5000)
08/25/2021 14:00:08 - INFO - klue_baseline.data.utils - Using ElectraTokenizer for fixing tokenization result
Testing: 50% 20/40 [16:08<16:08, 48.41s/it]Traceback (most recent call last):
File "run_klue.py", line 200, in <module>
main()
File "run_klue.py", line 194, in main
trainer.test(task.model, test_dataloaders=task.val_loader)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 922, in test
results = self.__test_given_model(model, test_dataloaders)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 980, in __test_given_model
results = self.fit(model)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 513, in fit
self.dispatch()
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 547, in dispatch
self.accelerator.start_testing(self)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 77, in start_testing
self.training_type_plugin.start_testing(trainer)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 115, in start_testing
self._results = trainer.run_test()
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 793, in run_test
eval_loop_results, _ = self.run_evaluation()
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 732, in run_evaluation
output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 160, in evaluation_step
output = self.trainer.accelerator.test_step(args)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 196, in test_step
return self.training_type_plugin.test_step(*args)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 131, in test_step
return self.lightning_module.test_step(*args, **kwargs)
File "/content/drive/My Drive/Colab Notebooks/klue/KLUE-baseline-main/klue_baseline/models/lightning_base.py", line 171, in test_step
return self.validation_step(batch, batch_idx, data_type=self.eval_dataset_type)
File "/content/drive/My Drive/Colab Notebooks/klue/KLUE-baseline-main/klue_baseline/models/named_entity_recognition.py", line 62, in validation_step
outputs = self(**inputs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/Colab Notebooks/klue/KLUE-baseline-main/klue_baseline/models/named_entity_recognition.py", line 41, in forward
return self.model(**inputs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_electra.py", line 1109, in forward
return_dict,
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_electra.py", line 763, in forward
return_dict=return_dict,
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_electra.py", line 479, in forward
output_attentions,
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_electra.py", line 418, in forward
self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1700, in apply_chunking_to_forward
return forward_fn(*input_tensors)
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_electra.py", line 425, in feed_forward_chunk
layer_output = self.output(intermediate_output, attention_output)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_electra.py", line 363, in forward
hidden_states = self.dense(hidden_states)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 1692, in linear
output = input.matmul(weight.t())
KeyboardInterrupt
Testing: 50%|█████ | 20/40 [17:23<17:23, 52.20s/it]
inference¶
!python run_klue.py \
--model_dir ./ner/ner_output/klue-ner/version_1/transformers \
--eval_batch_size 128 \
data/ Makefile pyproject.toml requirements.txt wos/
inference.py model/ README_ko.md run_all.sh
klue_baseline/ mypy.ini README.md run_klue.py
LICENSE.md ner/ requirements-dev.txt setup.cfg