- SQLova is a neural semantic parser translating natural language utterance to SQL query. The name is originated from the name of our department: Search & QLova (Search & Clova).
- We present the new state-of-the-art semantic parsing model that translates a natural language (NL) utterance into a SQL query.
- The model is evaluated on WikiSQL, a semantic parsing dataset consisting of 80,654 (NL, SQL) pairs over 24,241 tables from Wikipedia.
- We achieve 83.6% logical form accuracy and 89.6% execution accuracy on WikiSQL test set.
The model in a nutshell
Results (Updated at Jan 12, 2019)
|SQLova||81.6 (+5.5)^||87.2 (+3.2)^||80.7 (+5.3)^||86.2 (+2.5)^|
|SQLova-EG||84.2 (+8.2)*||90.2 (+3.0)*||83.6(+8.2)*||89.6 (+2.5)*|
- ^: Compared to current SOTA models that do not use execution guided decoding.
- *: Compared to current SOTA.
- The order of where conditions is ignored in measuring logical form accuracy in our model.
PyTorch 0.4.0or higher.
- Python libraries:
babel, matplotlib, defusedxml, tqdm
- Install minicoda
conda install pytorch torchvision -c pytorch
conda install -c conda-forge records
conda install babel
conda install matplotlib
conda install defusedxml
conda install tqdm
- The code has been tested on Tesla M40 GPU running on Ubuntu 16.04.4 LTS.
python3 train.py --seed 1 --bS 16 --accumulate_gradients 2 --bert_type_abb uS --fine_tune --lr 0.001 --lr_bert 0.00001 --max_seq_leng 222on terminal.
--seed 1: Set the seed of random generator. The accuracies changes by few percent depending on
--bS 16: Set the batch size by 16.
--accumulate_gradients 2: Make the effective batch size be
16 * 2 = 32.
--bert_type_abb uS: Uncased-Base BERT model is used. Use
uLto use Uncased-Large BERT.
--fine_tune: Train BERT. Without this, only the sequence-to-SQL module is trained.
--lr 0.001: Set the learning rate of the sequence-to-SQL module as 0.001.
--lr_bert 0.00001: Set the learning rate of BERT module as 0.00001.
--max_seq_leng 222: Set the maximum number of input token lengths of BERT.
- The model should show ~79% logical accuracy (lx) on dev set after ~12 hrs (~10 epochs). Higher accuracy can be obtained with longer training, by selecting different seed, by using Uncased Large BERT model, or by using execution guided decoding.
--EGargument while running
train.pyto use execution guided decoding.
- Whenever higher logical form accuracy calculated on the dev set, following three files are saved on current folder:
model_best.pt: the checkpoint of the the sequence-to-SQL module.
model_bert_best.pt: the checkpoint of the BERT module.
results_dev.jsonl: json file for official evaluation.
Decoder-Layermodels can be trained similarly (
Evaluation on WikiSQL DEV set
- To calculate logical form and execution accuracies on
devset using official evaluation script,
- Download original WikiSQL dataset.
- tar xvf data.tar.bz2
- Move them under
- Set path on
evaluation_ws.py. This is the file where the path information has added on original
evaluation.pyscript. Or you can use original
evaluation.pyby setting the path to the files by yourself.
python3 evaluation_ws.pyon terminal.
Evaluation on WikiSQL TEST set
- Uncomment line 550-557 of
- Save the output of
- Evaluate with
Load pre-trained SQLova parameters.
- Pretrained SQLova model parameters are uploaded in release. To start from this, uncomment line 562-565 and set paths.
- Pretrained BERT models were downloaded from official repository.
- BERT code is from huggingface-pytorch-pretrained-BERT.
- The sequence-to-SQL model is started from the source code of SQLNet and significantly re-written while maintaining the basic column-attention and sequence-to-set structure of the SQLNet.
- The data is annotated by using
annotate_ws.pywhich is based on
annotate.pyfrom WikiSQL repository. The tokens of natural language guery, and the start and end indices of where-conditions on natural language tokens are annotated.
- Pre-trained BERT parameters can be downloaded from BERT official repository and can be coverted to
ptfile using following script. You need install both pytorch and tensorflow and change
BERT_BASE_DIRto your data directory.
cd sqlova export BERT_BASE_DIR=data/uncased_L-12_H-768_A-12 python bert/convert_tf_checkpoint_to_pytorch.py \ --tf_checkpoint_path $BERT_BASE_DIR/bert_model.ckpt \ --bert_config_file $BERT_BASE_DIR/bert_config.json \ --pytorch_dump_path $BERT_BASE_DIR/pytorch_model.bin
bert/convert_tf_checkpoint_to_pytorch.pyis from the previous version of huggingface-pytorch-pretrained-BERT, and current version of
pytorch-pretrained-BERTis not compatible with the bert model used in this repo due to the difference in variable names (in LayerNorm). See this for the detail.
- For the convenience, the annotated WikiSQL data and the PyTorch-converted pre-trained BERT parameters are available at here.
Copyright 2019-present NAVER Corp. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.