The dataset is Stanford Natural Language Inference (SNLI), which we regard as a three-way classification tasks. We use an encoder-attention-decoder architecture, and stack two additional birnn upon the final sequence representation. Both GloVe word embedding and character embedding is used for word-level representation. Main experimental results are summarized below.

LN: layer normalizaton; Time: time in seconds per training batch measured from 1k training steps.

tensorflow >= 1.8.1

• download and preprocess dataset

  • The dataset link: https://nlp.stanford.edu/projects/snli/

  • Prepare separate data files:

    We provide a simple processing script convert_to_plain.py in scripts folder. By calling:

    python convert_to_plain.py snli_1.0/[ds].txt
    

    you can get the *.p, *.q, *.l files as in the config.py. [ds] indicates snli_1.0_train.txt, snli_1.0_dev.txt and snli_1.0_test.txt. We only preserve 'entailment', 'neutral', 'contradiction' instances, and others are dropped.

  • Prepare embedding and vocabulary

    Download the pre-trained GloVe embedding. And prepare the char as well as word vocabulary using vocab.py as follows:

    # word embedding & vocabulary
    python vocab.py --embeddings [path-to-glove-embedding] train.p,train.q,dev.p,dev.q,test.p,test.q word_vocab
    # char embedding
    python vocab.py --char train.p,train.q,dev.p,dev.q,test.p,test.q char_vocab
    

  • Download BERT pre-trained embedding (if you plan to work with BERT)

• training and evaluation

  • Train the model as follows:

  # configure your cuda libaray if necessary
  export CUDA_ROOT=XXX
  export PATH=$CUDA_ROOT/bin:$PATH
  export LD_LIBRARY_PATH=$CUDA_ROOT/lib64:$LD_LIBRARY_PATH
  
  # LRN
  python code/run.py --mode train --config config.py --parameters=gpus=[0],cell="lrn",layer_norm=False,output_dir="train_no_ln" >& log.noln
  # LRN + LN
  python code/run.py --mode train --config config.py --parameters=gpus=[0],cell="lrn",layer_norm=True,output_dir="train_ln" >& log.ln
  # LRN + BERT
  python code/run.py --mode train --config config_bert.py --parameters=gpus=[0],cell="lrn",layer_norm=False,output_dir="train_no_ln_bert" >& log.noln.bert
  # LRN + LN + BERT
  python code/run.py --mode train --config config_bert.py --parameters=gpus=[0],cell="lrn",layer_norm=True,output_dir="train_ln_bert" >& log.ln.bert
  

  Other hyperparameter settings are available in the given config.py.

  • Test the model as follows:

  # LRN
  python code/run.py --mode test --config config.py --parameters=gpus=[0],cell="lrn",layer_norm=False,output_dir="train_no_ln/best",test_output="out.noln" >& log.noln.test
  # LRN + LN
  python code/run.py --mode test --config config.py --parameters=gpus=[0],cell="lrn",layer_norm=True,output_dir="train_ln/best",test_output="out.ln" >& log.ln.test
  # LRN + BERT
  python code/run.py --mode test --config config_bert.py --parameters=gpus=[0],cell="lrn",layer_norm=False,output_dir="train_no_ln_bert/best",test_output="out.noln.bert" >& log.noln.bert.test
  # LRN + LN + BERT
  python code/run.py --mode test --config config_bert.py --parameters=gpus=[0],cell="lrn",layer_norm=True,output_dir="train_ln_bert/best",test_output="out.ln.bert" >& log.ln.bert.test
  

Source code structure is adapted from zero.