diff --git a/.gitignore b/.gitignore index 9081237..06fed04 100644 --- a/.gitignore +++ b/.gitignore @@ -134,4 +134,9 @@ dmypy.on *.bin *.idx -*.pt \ No newline at end of file +*.pt + +data +data_raw +results +pretrain_data \ No newline at end of file diff --git a/README-ZH.md b/README-ZH.md new file mode 100644 index 0000000..3ce63b7 --- /dev/null +++ b/README-ZH.md @@ -0,0 +1,60 @@ + + +## 动态 +- 2023/05/27 [CPM-Bee](https://github.com/OpenBMB/CPM-Bee) 发布了！ +- 2023/04/12 CPM-Ant 可以在[HuggingFace Transformers](https://huggingface.co/openbmb/cpm-ant-10b)中使用了！ +- 2022/10/12 中英双语模型 [CPM-Ant+](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live) 已经发布！除了能够生成中文/英文文本，现在模型还可以处理问答、摘要和翻译任务！ +- 2022/09/16 [CPM-Ant](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live) 已经发布！ +- 2022/05/29 CPM-Live的训练今天启动! 详情请查看[训练动态](https://live.openbmb.org/home)。 +- 2022/05/25 CPM-Live的[训练计划](./plans/CPM-Live训练计划书.md)现已公布。期待训练开始! + + +## 里程碑 +- **CPM-Bee** (2022/10/13-2023/05/27) [[代码](https://github.com/OpenBMB/CPM-Bee)][[模型](https://github.com/OpenBMB/CPM-Bee#%E6%A8%A1%E5%9E%8B)][[计划书](./plans/CPM-Bee训练计划书.md)] +- **CPM-Ant+** (2022/08/05-2022/10/12) [[代码](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live)][[模型](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live#model-checkpoints)] +- **CPM-Ant** (2022/05/29-2022/08/05) [[代码](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live)][[模型](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live#model-checkpoints)][[网站](https://live.openbmb.org/ant)][[博客](https://www.openbmb.org/en/community/blogs/blogpage?id=98afef2ce45f4fe9a4bc15a66d7ccb92)][[计划书](./plans/CPM-Ant训练计划书.md)] + +## 训练计划 +考虑到数据和计算资源的规模，CPM-Live将从10B模型开始训练并持续学习。 + +### 在训练过程中，我们将进行： + +- **实时**：显示模型训练指标 +- **每天**：发布模型训练日志 +- **每周**：处理社区的讨论和反馈 +- **不定期**：在模型训练期间发布允许公开下载的检查点 + + +### 在训练期间你可以： + +- **提出你的模型倡议**：对模型架构、训练方法或数据源有好的想法？你可以在社区里提出你的模型倡议。如果该倡议得到更多的支持并且实际可行，我们将把它添加到我们正在训练的模型中，这样CPM-Live就可以在大家的帮助下不断学习和进步。 + +- **开发你的应用程序**：基于CPM-Live，你可以向社区提交你初期想法、原型、开发代码或完成的应用程序。我们将在网站上展示最受欢迎的应用程序。 + +- **在论坛上聊天**：你可以在我们的论坛上谈论任何与大模型有关的话题，如学术研究、工程实现、工具使用、应用设计等。无论你是否有经验，我们相信每个人都可以从积极和开放的讨论中受益。 + +- **下载资源**：模型训练完成后，你可以在开放使用许可下自由下载模型参数。CPM-Live使用的是包括商业化许可的开放许可。通过模型压缩和推理加速工具，你可以在自己的电脑上体验大模型的威力! + + + +## 社区 + +我们的[社区](https://github.com/OpenBMB/CPM-Live/discussions) 基于GitHub Discussions。 + +阅读[第一篇帖子](https://github.com/OpenBMB/CPM-Live/discussions/1)，开始你对CPM-Live的探索吧！ + + + + + diff --git a/README.md b/README.md index c809a39..4c0d942 100644 --- a/README.md +++ b/README.md @@ -5,20 +5,29 @@ **Live Training for Open-source Big Models**

- Website • Plan • Discussion + Website • Plan • Discussion • 简体中文 +

- ## What's New +- 2023/05/27 [CPM-Bee](https://github.com/OpenBMB/CPM-Bee) is released! +- 2023/04/12 CPM-Ant has been integrated into [HuggingFace Transformers](https://huggingface.co/openbmb/cpm-ant-10b)! +- 2022/10/12 [CPM-Ant+](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live)a bilingual modelis released! In addition to generating Chinese/English textyou can now use our model for QAsummarization and translation tasks! +- 2022/09/16 [CPM-Ant](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live) is released! - 2022/05/29 The training of CPM-Live has launched today! See [training dynamics](https://live.openbmb.org/home). - 2022/05/25 The [training plan](./plans/CPM-Live训练计划书.md) for CPM-Live is now published. Look forward to the training! +## Milestones + +- **CPM-Bee** (2022/10/13-2023/05/27) [[Code](https://github.com/OpenBMB/CPM-Bee)][[Model](https://github.com/OpenBMB/CPM-Bee#%E6%A8%A1%E5%9E%8B)][[Plan](./plans/CPM-Bee训练计划书.md)] +- **CPM-Ant+** (2022/08/05-2022/10/12) [[Code](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live)][[Model](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live#model-checkpoints)] +- **CPM-Ant** (2022/05/29-2022/08/05) [[Code](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live)][[Model](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live#model-checkpoints)][[Website](https://live.openbmb.org/ant)][[Blog](https://www.openbmb.org/en/community/blogs/blogpage?id=98afef2ce45f4fe9a4bc15a66d7ccb92)][[Plan](./plans/CPM-Ant训练计划书.md)] ## Training Plan -Considering the scale of data and computing resourcesCPM-Live will start with a 10B model trainingwhich we named CPM-Ant. The training of CPM-Ant will start on May 292022and the entire training process is expected to last five months. +Considering the scale of data and computing resourcesCPM-Live will start with a 10B model training. ### During training we will do: @@ -43,8 +52,3 @@ Considering the scale of data and computing resourcesCPM-Live will start with [Our community](https://github.com/OpenBMB/CPM-Live/discussions) is based on GitHub Discussions. Read the [first post](https://github.com/OpenBMB/CPM-Live/discussions/1) and start your exploration on CPM-Live! - - - - - diff --git a/cpm-live/.flake8 b/cpm-live/.flake8 index b357ab3..e4d2c92 100644 --- a/cpm-live/.flake8 +++ b/cpm-live/.flake8 @@ -3,4 +3,5 @@ per-file-ignores = # imported but unused __init__.py: F401 max-line-length = 100 -extend-ignore = E712E203 \ No newline at end of file +extend-ignore = E712E203 +exclude = examples/*.py \ No newline at end of file diff --git a/cpm-live/config/cpm-bee-10b.on b/cpm-live/config/cpm-bee-10b.on new file mode 100644 index 0000000..c34b2e0 --- /dev/null +++ b/cpm-live/config/cpm-bee-10b.on @@ -0,0 +1,14 @@ +{ + "vocab_size": 86583, + "dim_model": 4096, + "dim_ff" : 10240, + "num_layers" : 48, + "num_heads": 32, + "dim_head" : 128, + "dropout_p" : 0.0, + "position_bias_num_buckets" : 256, + "position_bias_num_segment_buckets": 256, + "position_bias_max_distance" : 2048, + "eps" : 1e-6, + "half" : true +} diff --git a/cpm-live/config/cpm-bee-3b.on b/cpm-live/config/cpm-bee-3b.on new file mode 100644 index 0000000..55fd0f2 --- /dev/null +++ b/cpm-live/config/cpm-bee-3b.on @@ -0,0 +1,14 @@ +{ + "vocab_size": 86580, + "dim_model": 2560, + "dim_ff" : 3072, + "num_layers" : 32, + "num_heads": 32, + "dim_head" : 80, + "dropout_p" : 0.0, + "position_bias_num_buckets" : 256, + "position_bias_num_segment_buckets": 256, + "position_bias_max_distance" : 2048, + "eps" : 1e-6, + "half" : true +} diff --git a/cpm-live/cpm_live/__init__.py b/cpm-live/cpm_live/__init__.py index 03e328b..e69de29 100644 --- a/cpm-live/cpm_live/__init__.py +++ b/cpm-live/cpm_live/__init__.py @@ -1,5 +0,0 @@ -from . import models -from . import dataset -from . import utils -from . import tokenizers -from .arguments import get_args diff --git a/cpm-live/cpm_live/arguments.py b/cpm-live/cpm_live/arguments.py index 2f3202c..b1b6d0e 100644 --- a/cpm-live/cpm_live/arguments.py +++ b/cpm-live/cpm_live/arguments.py @@ -29,13 +29,7 @@ def add_training_args(parser: argparse.ArgumentParser): group = parser.add_argument_group("train""training configurations") - group.add_argument( - "--base-path", - type=str, - default=None, - help="Path to the project base directory.", - ) - group.add_argument("--dataset_name"type=strdefault=Nonehelp="Name of the dataset") + group.add_argument("--dataset"type=strdefault="dataset.on"help="Path to dataset") group.add_argument( "--load", type=str, @@ -54,18 +48,14 @@ def add_training_args(parser: argparse.ArgumentParser): default=None, help="Output filename to save checkpoints to.", ) + group.add_argument( - "--save-iters", - type=int, - default=1000, - help="number of iterations between saves", - ) - group.add_argument( - "--log-dir", + "--tensorboard", type=str, - default="logs", - help="tensorboard log directory", + default=None, + help="tensorboard directory", ) + group.add_argument("--inspect-iters"type=intdefault=1000help="number of inspecting") group.add_argument("--batch-size"type=intdefault=32help="Data Loader batch size") group.add_argument("--clip-grad"type=floatdefault=1.0help="gradient clipping") @@ -76,33 +66,12 @@ def add_training_args(parser: argparse.ArgumentParser): help="total number of iterations to train over all training runs", ) group.add_argument("--max-length"type=intdefault=512help="max length of input") - group.add_argument( - "--max-encoder-length", - type=int, - default=512, - help="max length of encoder input", - ) - group.add_argument( - "--max-decoder-length", - type=int, - default=256, - help="max length of decoder input", - ) - group.add_argument( - "--start-step"type=intdefault=0help="step to start or continue training" - ) - group.add_argument("--seed"type=intdefault=1234help="random seed for reproducibility") - group.add_argument( - "--epochs", - type=int, - default=1, - help="total number of epochs to train over all training runs", - ) + group.add_argument("--seed"type=intdefault=1234help="random seed for reproducibility") # Learning rate. group.add_argument("--lr"type=floatdefault=1.0e-4help="initial learning rate") - group.add_argument("--weight-decay"type=floatdefault=1.0e-2help="weight-decay") + group.add_argument("--weight-decay"type=floatdefault=1.0e-2help="weight decay rate") group.add_argument("--loss-scale"type=floatdefault=65536help="loss scale") group.add_argument( @@ -111,13 +80,6 @@ def add_training_args(parser: argparse.ArgumentParser): default=0.01, help="percentage of data to warmup on (.01 = 1% of all " "training iters). Default 0.01", ) - group.add_argument( - "--lr-decay-iters", - type=int, - default=None, - help="number of iterations to decay LR over," - " If None defaults to `--train-iters`*`--epochs`", - ) group.add_argument( "--lr-decay-", type=str, @@ -125,20 +87,62 @@ def add_training_args(parser: argparse.ArgumentParser): choices=["constant""linear""cosine""exponential""noam"], help="learning rate decay function", ) + group.add_argument("--lr-decay-iters"type=intdefault=Nonehelp="lr decay steps") group.add_argument( - "--local_rank", + "--start-step"type=intdefault=0help="step to start or continue training" + ) + + return parser + + +def add_pretrain_args(parser: argparse.ArgumentParser): + group = parser.add_argument_group("pretrain""pretrain configurations") + group.add_argument( + "--save-iters", type=int, + default=1000, + help="number of iterations between saves", + ) + group.add_argument( + "--log-dir", + type=str, default=None, - help="local rank passed from distributed launcher", + help="log directory", ) return parser -def get_args(): +def add_finetune_args(parser: argparse.ArgumentParser): + group = parser.add_argument_group("finetune""fintune configurations") + group.add_argument("--epoch"type=intdefault=1help="number of training epochs") + group.add_argument("--task-name"type=strdefault="task"help="name of training task") + group.add_argument( + "--use-delta", + action="store_true", + default=False, + help="use delta tuning or not" + ) + group.add_argument("--eval_dataset"type=strhelp="path to eval dataset") + group.add_argument( + "--drop-last", + action="store_true", + default=False, + help="drop data from each epoch that cannot be formed into a complete batch at the end", + ) + group.add_argument("--eval-interval"type=intdefault=500help="eval interval") + group.add_argument("--early-stop-patience"type=intdefault=5help="early stop steps") + return parser + + +def get_args(pretrain: bool = Falsefinetune: bool = False): parser = argparse.ArgumentParser() parser = add_model_config_args(parser) parser = add_training_args(parser) + if pretrain: + parser = add_pretrain_args(parser) + if finetune: + parser = add_finetune_args(parser) args = parser.parse_args() return args diff --git a/cpm-live/cpm_live/dataset/distributed_dataset.py b/cpm-live/cpm_live/dataset/distributed_dataset.py index d90c612..a9979da 100644 --- a/cpm-live/cpm_live/dataset/distributed_dataset.py +++ b/cpm-live/cpm_live/dataset/distributed_dataset.py @@ -15,29 +15,35 @@ import io import os -import pickle -from typing import List +import struct +from typing import ListOptionalSet import torch import bisect import bmtrain as bmt - +import on +from .serializer import SerializerPickleSerializer import random import string +import time def _random_string(): return "".join(random.choices(string.ascii_uppercase + string.digitsk=8)) +_DEFAULT_BLOCK_SIZE = 16 << 20 + + class FileInfo: def __init__( self, - file_name: str, - block_begin: int, - block_end: int, - nbytes: int, - nlines: int, + file_name: str = "", + block_begin: int = 0, + block_end: int = 0, + nbytes: int = 0, + nlines: int = 0, mask: bool = False, + block_size: int = _DEFAULT_BLOCK_SIZE, ) -> None: self.file_name = file_name self.block_begin = block_begin @@ -45,34 +51,153 @@ def __init__( self.nbytes = nbytes self.nlines = nlines self.mask = mask + self.block_size = block_size - @classmethod - def _load_from_state(clsversiondata): - if version == 1: - file_nameblock_beginblock_endnbytesnlinesmask = data - return cls(file_nameblock_beginblock_endnbytesnlinesmask) - else: - raise RuntimeError("Unsupported version %d" % version) - - def __reduce__(self): - return ( - FileInfo._load_from_state, - ( - 1, - ( - self.file_name, - self.block_begin, - self.block_end, - self.nbytes, - self.nlines, - self.mask, - ), - ), - ) + def state_dict(self): + return { + "file_name": self.file_name, + "block_begin": self.block_begin, + "block_end": self.block_end, + "nbytes": self.nbytes, + "nlines": self.nlines, + "mask": self.mask, + "block_size": self.block_size, + } + + def load_state_dict(selfd): + self.file_name = d["file_name"] + self.block_begin = d["block_begin"] + self.block_end = d["block_end"] + self.nbytes = d["nbytes"] + self.nlines = d["nlines"] + self.mask = d["mask"] + self.block_size = d["block_size"] + + def dumps(self) -> str: + return on.dumps(self.state_dict()) + + def loads(selfdata: str) -> "FileInfo": + self.load_state_dict(on.loads(data)) + return self + + def dump(selffp: io.TextIOWrapper) -> "FileInfo": + fp.write(self.dumps()) + return self + + def load(selffp: io.TextIOWrapper) -> "FileInfo": + self.loads(fp.read()) + return self + + +def _read_info_list(meta_path: str) -> List[FileInfo]: + info: List[FileInfo] = [] + while True: + try: + with open(meta_path"r"encoding="utf-8") as f: + for line in f.readlines(): + line = line.strip() + if len(line) > 0: + info.append(FileInfo().loads(line)) + return info + except Exception as e: + print("Error: reading info list in _read_info_list!,meta_path={path}err={err}". + format(path=meta_patherr=str(e))) + time.sleep(10) + + +def _write_info_list(meta_path: strinfo: List[FileInfo]): + base_path = os.path.dirname(meta_path) + random_fname = os.path.join(base_path".meta.bin.%s" % _random_string()) + while True: + try: + with open(random_fname"w"encoding="utf-8") as f: + for v in info: + f.write(v.dumps() + "\n") + os.rename(random_fnamemeta_path) + return + except Exception: + print("Error: writing info list!") + time.sleep(10) -_MASK_VALUE = 0x7FFFFFFF -_DEFAULT_BLOCK_SIZE = 16 << 20 +def _filtered_range( + begin: intend: intrank: intworld_size: intfilter_set: Optional[Set[int]] = None +): + begin = begin + (rank + (world_size - (begin % world_size))) % world_size + + if filter_set is not None: + return [i for i in range(beginendworld_size) if i in filter_set] + else: + return [i for i in range(beginendworld_size)] + + +# for some bugs that may exist in hdfs +class SafeFile: + + def __init__(selffnamemode): + self.fname = None + self.mode = None + self._fp = None + self.open_file(fnamemode) + + def read(selfsize=-1): + if self._fp is None: + raise RuntimeError("Dataset is closed") + try: + res = self._fp.read(size) + self.offset = self._fp.tell() + return res + except Exception as e: + print("Error {}: reading blocks in read {}!".format(eself.fname)) + self.open_file(self.fnameself.modeself.offset) + return self.read(size) + + def tell(self): + if self._fp is None: + raise RuntimeError("Dataset is closed") + try: + res = self._fp.tell() + self.offset = res + return res + except Exception as e: + print("Error {}: reading blocks in tell {}!".format(eself.fname)) + self.open_file(self.fnameself.modeself.offset) + return self.tell() + + def seek(selfoffsetwhence=0): + if self._fp is None: + raise RuntimeError("Dataset is closed") + try: + res = self._fp.seek(offsetwhence) + self.offset = self._fp.tell() + return res + except Exception as e: + print("Error {}: reading blocks in seek {}!".format(eself.fname)) + self.open_file(self.fnameself.modeself.offset) + return self.seek(offsetwhence) + + def close(self): + if self._fp is not None: + try: + self._fp.close() + except Exception: + pass + self._fp = None + + def open_file(selffnamemodeoffset=None): + if not os.path.exists(fname): + raise RuntimeError("Dataset does not exist") + try: + self.fname = fname + self.mode = mode + self._fp = open(fnamemode) + if offset is not None: + self._fp.seek(offsetio.SEEK_SET) + self.offset = self._fp.tell() + except Exception as e: + print("Error {}: reading blocks in open_file {}!".format(eself.fname)) + time.sleep(10) + self.open_file(fnamemodeoffset) class DistributedDataset: @@ -100,16 +225,24 @@ def __init__( path: str, rank: int = 0, world_size: int = 1, - block_size=_DEFAULT_BLOCK_SIZE, + serializer: Optional[Serializer] = None, + max_repeat_times: Optional[int] = None, + shuffle: bool = True, ) -> None: # config self._path = path self._rank = rank self._world_size = world_size - self._block_size = block_size + self._max_repeat_times = max_repeat_times + self._repeat_times = 0 + self._shuffle = shuffle + + if serializer is None: + serializer = PickleSerializer() + self.serializer = serializer # dataset meta - self._block_states = torch.tensor([]dtype=torch.int) + self._unused_block: List[int] = [] self._file_info: List[FileInfo] = [] self._file_ends: List[int] = [] self._total_blocks = 0 @@ -124,12 +257,21 @@ def __init__( self._last_mod_time = 0 self._curr_fname = None - self._update_states() + self._update_states(fast_skip=False) + self._repeat_times += 1 def _update_states(selffast_skip: bool = True): meta_path = os.path.join(self._path"meta.bin") - mod_time = os.stat(meta_path).st_mtime + while True: + try: + mod_time = os.stat(meta_path).st_mtime + break + except Exception as e: + print("Error: reading info list in DistributedDataset._update_states" + "meta_path={path}err={err}!".format(path=meta_patherr=str(e))) + time.sleep(10) + if self._last_mod_time < mod_time: # file changed pass @@ -139,12 +281,13 @@ def _update_states(selffast_skip: bool = True): info: List[FileInfo] = [] if os.path.exists(meta_path): - with open(meta_path"rb") as f: - info = pickle.load(f) + info = _read_info_list(meta_path) old_len = len(self._file_info) if old_len > len(info): raise RuntimeError("Dataset meta file: changed unexpectly") + + mask_changed = False for i in range(old_len): if self._file_info[i].file_name != info[i].file_name: raise RuntimeError("Dataset meta file: changed unexpectly") @@ -152,6 +295,8 @@ def _update_states(selffast_skip: bool = True): raise RuntimeError("Dataset meta file: changed unexpectly") if self._file_info[i].block_end != info[i].block_end: raise RuntimeError("Dataset meta file: changed unexpectly") + if self._file_info[i].mask != info[i].mask: + mask_changed = True if info[0].block_begin != 0: raise RuntimeError("Dataset meta file: block error (0)") @@ -159,7 +304,7 @@ def _update_states(selffast_skip: bool = True): if info[i].block_end != info[i + 1].block_begin: raise RuntimeError("Dataset meta file: block error (%d)" % (i + 1)) - if old_len == len(info) and fast_skip: + if (old_len == len(info) and not mask_changed) and fast_skip: # fast skip return @@ -176,32 +321,38 @@ def _update_states(selffast_skip: bool = True): self._nlines = 0 if total_blocks > 0: - masks = torch.full( - (total_blocks,), - _MASK_VALUE, - dtype=torch.int, - device="cpu", - requires_grad=False, - ) - masks[self._rank :: self._world_size] = 0 - for v in info: - if v.mask or (not os.path.exists(self._get_file_path(v.file_name))): - masks[v.block_begin : v.block_end] = _MASK_VALUE - new_block_states = torch.zeros( - total_blocksdtype=torch.intdevice="cpu"requires_grad=False - ) - new_block_states[: self._block_states.size(0)] = self._block_states - new_block_states = torch.maximum(new_block_statesmasks) - - self._block_states = new_block_states + unused_block_set = set(self._unused_block) + nw_unused_block: List[int] = [] + for i in range(len(info)): + v = info[i] + if not v.mask: + if i < old_len: + nw_unused_block.extend( + _filtered_range( + v.block_begin, + v.block_end, + self._rank, + self._world_size, + unused_block_set, + ) + ) + else: + nw_unused_block.extend( + _filtered_range( + v.block_beginv.block_endself._rankself._world_size + ) + ) + + # re-shuffle unused blocks + if self._shuffle: + random.shuffle(nw_unused_block) + self._unused_block = nw_unused_block self._file_ends = [] for v in info: self._file_ends.append(v.block_end) else: - self._block_states = torch.tensor( - []dtype=torch.intdevice="cpu"requires_grad=False - ) + self._unused_block = [] self._file_ends = [] self._total_blocks = total_blocks self._file_info = info @@ -209,27 +360,62 @@ def _update_states(selffast_skip: bool = True): assert len(self._file_ends) == len(self._file_info) def _mask_file(selff: FileInfo): - masks = torch.full( - (self._total_blocks,)0dtype=torch.intdevice="cpu"requires_grad=False - ) - masks[f.block_begin : f.block_end] = _MASK_VALUE - self._block_states = torch.maximum(self._block_statesmasks) + self._unused_block = [ + block_id + for block_id in self._unused_block + if block_id < f.block_begin or block_id >= f.block_end + ] def _get_block_file(selfblock_id: int): # find block in which file file_idx = bisect.bisect_right(self._file_endsblock_id) return self._file_info[file_idx] + def _prepare_new_epoch(self): + if self._max_repeat_times is not None: + if self._repeat_times >= self._max_repeat_times: + raise EOFError("End of dataset") + nw_unused_block: List[int] = [] + for v in self._file_info: + if not v.mask: + nw_unused_block.extend( + _filtered_range(v.block_beginv.block_endself._rankself._world_size) + ) + if self._shuffle: + random.shuffle(nw_unused_block) + self._unused_block = nw_unused_block + self._repeat_times += 1 + def _get_next_block(self): self._update_states() - if self._block_states.size(0) == 0: - raise RuntimeError("Empty dataset") - mn_block: int = self._block_states.argmin().item() # type: ignore - if self._block_states[mn_block].item() == _MASK_VALUE: - raise RuntimeError("Empty dataset") - self._block_states[mn_block] += 1 + if len(self._unused_block) == 0: + self._prepare_new_epoch() + if len(self._unused_block) == 0: + raise RuntimeError("Empty dataset {}".format(self._path)) + + mn_block: int = self._unused_block.pop() return mn_block + def _state_dict(self): + self._update_states() + num_unused_block = len(self._unused_block) + if (self._fp is not None) and (self._curr_block is not None): + curr_block = self._curr_block + curr_f = self._get_block_file(curr_block) + inblock_offset = self._fp.tell() - (curr_block - curr_f.block_begin) * curr_f.block_size + else: + curr_block = -1 + inblock_offset = 0 + + return { + "states": torch.tensor(self._unused_blockdtype=torch.longdevice="cpu"), + "block": torch.tensor( + [curr_blockinblock_offsetnum_unused_blockself._repeat_times], + dtype=torch.long, + device="cpu", + ), + } + def state_dict(self): """Returns a state dict representing the read states of the dataset. @@ -238,32 +424,43 @@ def state_dict(self): >>> dataset.load_state_dict(state) """ self._update_states() - states = torch.where( - self._block_states == _MASK_VALUE, - torch.zeros(self._total_blocksdtype=torch.intdevice="cpu"requires_grad=False), - self._block_states, - ) + num_unused_block = len(self._unused_block) if (self._fp is not None) and (self._curr_block is not None): curr_block = self._curr_block curr_f = self._get_block_file(curr_block) - inblock_offset = self._fp.tell() - (curr_block - curr_f.block_begin) * self._block_size + inblock_offset = self._fp.tell() - (curr_block - curr_f.block_begin) * curr_f.block_size else: curr_block = -1 inblock_offset = 0 with torch.no_grad(): if self._world_size > 1: - gpu_states = states.cuda() - gpu_block = torch.tensor([curr_blockinblock_offset]dtype=torch.long).cuda() - global_states = bmt.distributed.all_reduce(gpu_statesop="sum").cpu() - global_block = bmt.distributed.all_gather(gpu_block).cpu() + gpu_num_unused_block = torch.tensor([num_unused_block]dtype=torch.long).cuda() + max_unused_blocks = ( + bmt.distributed.all_reduce(gpu_num_unused_blockop="max").cpu().item() + ) + gpu_states = torch.full((max_unused_blocks,)-1dtype=torch.long).cuda() + gpu_states[:num_unused_block] = torch.tensor( + self._unused_blockdtype=torch.long + ).cuda() + + gpu_block = torch.tensor( + [curr_blockinblock_offsetnum_unused_blockself._repeat_times], + dtype=torch.long, + ).cuda() + global_states = bmt.distributed.all_gather( + gpu_states + ).cpu() # (world_sizemax_unused_blocks) + global_block = bmt.distributed.all_gather(gpu_block).cpu() # (world_size4) return {"states": global_states"block": global_block} else: return { - "states": states, + "states": torch.tensor([self._unused_block]dtype=torch.longdevice="cpu"), "block": torch.tensor( - [[curr_blockinblock_offset]]dtype=torch.longdevice="cpu" + [[curr_blockinblock_offsetnum_unused_blockself._repeat_times]], + dtype=torch.long, + device="cpu", ), } @@ -278,11 +475,10 @@ def load_state_dict(selfstatestrict: bool = True): >>> state = dataset.state_dict() >>> """ + block_states: torch.LongTensor = state["states"] + block_info: torch.LongTensor = state["block"] - self._block_states = state["states"] - self._update_states(False) - - if state["block"].size(0) != self._world_size: + if block_states.size(0) != self._world_size: if strict: raise ValueError( "world_size changed (%d -> %d)" % (state["block"].size(0)self._world_size) @@ -291,20 +487,48 @@ def load_state_dict(selfstatestrict: bool = True): self._curr_block = None self._fp = None self._curr_fname = None + self._repeat_times = int(block_info[03].item()) + + # re-shuffle unused blocks + nw_unused_block: List[int] = [] + for i in range(block_states.size(0)): + # filter blocks that are not in this rank + num_unused_blocks: int = int(block_info[i2].item()) + nw_unused_block.extend( + [ + block_id + for block_id in block_states[i:num_unused_blocks].tolist() + if block_id % self._world_size == self._rank + ] + ) + if self._shuffle: + random.shuffle(nw_unused_block) + self._unused_block = nw_unused_block else: - curr_block = state["block"][self._rank][0].item() - inblock_offset = state["block"][self._rank][1].item() + curr_blockinblock_offsetnum_unused_blocksself._repeat_times = block_info[ + self._rank + ].tolist() if curr_block == -1: self._curr_block = None else: - self._curr_block = curr_block - f_info = self._get_block_file(self._curr_block) - self._open_file( - f_info.file_name, - (self._curr_block - f_info.block_begin) * self._block_size + inblock_offset, - ) + while True: + try: + self._curr_block = curr_block + f_info = self._get_block_file(self._curr_block) + self._open_file( + f_info.file_name, + (self._curr_block - f_info.block_begin) + * f_info.block_size + + inblock_offset, + ) + self._unused_block = block_states[self._rank:num_unused_blocks].tolist() + break + except Exception: + print("Error: reading block!") + time.sleep(10) # end + self._update_states() def _get_file_path(selffname): return os.path.join(self._pathfname) @@ -314,11 +538,11 @@ def _open_file(selffnameoffset): if self._fp is not None: self._fp.close() self._curr_fname = None - self._fp = open(self._get_file_path(fname)"rb") + # self._fp = open(self._get_file_path(fname)"rb") + self._fp = SafeFile(self._get_file_path(fname)"rb") self._curr_fname = fname else: assert self._fp is not None"Unexpected error" - self._fp.seek(offsetio.SEEK_SET) # move to block def read(self): @@ -332,10 +556,11 @@ def read(self): try: self._open_file( f_info.file_name, - (next_block_id - f_info.block_begin) * self._block_size, + (next_block_id - f_info.block_begin) * f_info.block_size, ) self._curr_block = next_block_id except FileNotFoundError: + print("ERR: reading again!") self._mask_file(f_info) return self.read() # read again @@ -345,7 +570,9 @@ def read(self): MAGIC = self._fp.read(1) if MAGIC == b"\x1F": # correct - return pickle.load(self._fp) + size = struct.unpack("I"self._fp.read(4))[0] + data = self._fp.read(size) + return self.serializer.deserialize(data) elif MAGIC == b"\x00": # end of block self._curr_block = None @@ -359,24 +586,27 @@ def nbytes(self): class SimpleDataset(DistributedDataset): - def __init__(selfpath: strblock_size=_DEFAULT_BLOCK_SIZE) -> None: - super().__init__(path01block_size) - - def _get_next_block(self): - self._update_states() - if self._block_states.size(0) == 0: - raise RuntimeError("Empty dataset") - mn_block: int = self._block_states.argmin().item() # type: ignore - if self._block_states[mn_block].item() >= 1: - raise EOFError("no more data") - self._block_states[mn_block] += 1 - return mn_block + def __init__( + self, + path: str, + serializer: Optional[Serializer] = None, + shuffle: bool = True, + ) -> None: + super().__init__( + path, + 0, + 1, + serializer=serializer, + max_repeat_times=1, + shuffle=shuffle, + ) def __iter__(self): while True: try: data = self.read() except EOFError: + self._repeat_times = 0 break yield data @@ -385,7 +615,7 @@ def __len__(self): class DatasetWriter: - def __init__(selffnameblock_size): + def __init__(selffname: strblock_size: intserializer: Optional[Serializer] = None): self._fname = fname self._block_size = block_size self._fp = open(self._fname"wb") @@ -395,6 +625,10 @@ def __init__(selffnameblock_size): self._nlines = 0 self._nblocks = 1 + if serializer is None: + serializer = PickleSerializer() + self.serializer = serializer + def write(selfdata): """Write a piece of data into dataset. @@ -405,7 +639,8 @@ def write(selfdata): >>> writer.write( "anything you want" ) """ - byte_data = pickle.dumps(data) + byte_data = self.serializer.serialize(data) + byte_data = struct.pack("I"len(byte_data)) + byte_data if self._inblock_offset + 2 + len(byte_data) > self._block_size: self._fp.write( b"\x00" * (self._block_size - self._inblock_offset) @@ -442,10 +677,19 @@ def close(self): class DatasetBuilder: - def __init__(selfpath: strdbname: strblock_size=_DEFAULT_BLOCK_SIZE) -> None: + def __init__( + self, + path: str, + dbname: str, + block_size=_DEFAULT_BLOCK_SIZE, + serializer: Optional[Serializer] = None, + ) -> None: self._block_size = block_size self._path = path self._dbname = dbname + if serializer is None: + serializer = PickleSerializer() + self.serializer = serializer if not os.path.exists(self._path): os.makedirs(self._path) @@ -454,8 +698,7 @@ def __init__(selfpath: strdbname: strblock_size=_DEFAULT_BLOCK_SIZE) -> No info: List[FileInfo] = [] if os.path.exists(meta_path): - with open(meta_path"rb") as f: - info = pickle.load(f) + info = _read_info_list(meta_path) for v in info: if v.file_name == dbname: @@ -466,7 +709,7 @@ def __init__(selfpath: strdbname: strblock_size=_DEFAULT_BLOCK_SIZE) -> No raise ValueError("File exists `%s`" % self._db_path) def __enter__(self): - self._writer = DatasetWriter(self._db_pathself._block_size) + self._writer = DatasetWriter(self._db_pathself._block_sizeself.serializer) return self._writer def __exit__(selfexc_typeexc_valueexc_traceback): @@ -482,8 +725,7 @@ def __exit__(selfexc_typeexc_valueexc_traceback): meta_path = os.path.join(self._path"meta.bin") info: List[FileInfo] = [] if os.path.exists(meta_path): - with open(meta_path"rb") as f: - info = pickle.load(f) + info = _read_info_list(meta_path) last_block = 0 if len(info) > 0: last_block = info[-1].block_end @@ -495,19 +737,22 @@ def __exit__(selfexc_typeexc_valueexc_traceback): self._writer.nbytes, self._writer.nlines, False, + self._block_size, ) ) # atomic write to meta file - random_fname = os.path.join(self._path".meta.bin.%s" % _random_string()) - with open(random_fname"wb") as f: - pickle.dump(infof) - os.rename(random_fnamemeta_path) + _write_info_list(meta_pathinfo) self._writer = None -def build_dataset(path: strdbname: strblock_size: int = _DEFAULT_BLOCK_SIZE): +def build_dataset( + path: str, + dbname: str, + block_size: int = _DEFAULT_BLOCK_SIZE, + serializer: Optional[Serializer] = None, +): """Open the dataset in write mode and returns a writer. Args: @@ -520,4 +765,4 @@ def build_dataset(path: strdbname: strblock_size: int = _DEFAULT_BLOCK_SIZE) >>> for i in range(10): >>> writer.write( { "anything you want" } ) """ # noqa: E501 - return DatasetBuilder(pathdbnameblock_size) + return DatasetBuilder(pathdbnameblock_size=block_sizeserializer=serializer) diff --git a/cpm-live/cpm_live/dataset/serializer.py b/cpm-live/cpm_live/dataset/serializer.py new file mode 100644 index 0000000..ef3865f --- /dev/null +++ b/cpm-live/cpm_live/dataset/serializer.py @@ -0,0 +1,61 @@ +# coding=utf-8 +# Copyright 2020 The OpenBMB team. All rights reserved. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pickle +import on + + +class Serializer: + def __init__(self) -> None: + pass + + def serialize(selfobj) -> bytes: + raise NotImplementedError() + + def deserialize(selfdata: bytes): + raise NotImplementedError() + + +class PickleSerializer(Serializer): + def __init__(self) -> None: + pass + + def serialize(selfobj) -> bytes: + return pickle.dumps(obj) + + def deserialize(selfdata: bytes): + return pickle.loads(data) + + +class JsonSerializer(Serializer): + def __init__(self) -> None: + pass + + def serialize(selfobj) -> bytes: + return on.dumps(objensure_ascii=False).encode("utf-8") + + def deserialize(selfdata: bytes): + return on.loads(data.decode("utf-8")) + + +class RawSerializer(Serializer): + def __init__(self) -> None: + pass + + def serialize(selfobj) -> bytes: + return obj + + def deserialize(selfdata: bytes): + return data diff --git a/cpm-live/cpm_live/dataset/utils.py b/cpm-live/cpm_live/dataset/utils.py index c166a56..2a6ace6 100644 --- a/cpm-live/cpm_live/dataset/utils.py +++ b/cpm-live/cpm_live/dataset/utils.py @@ -14,16 +14,19 @@ # limitations under the License. import os -from typing import List +import struct +from typing import ListOptional from .distributed_dataset import ( SimpleDataset, build_dataset, + _read_info_list, + _write_info_list, _random_string, _DEFAULT_BLOCK_SIZE, FileInfo, ) +from .serializer import RawSerializer import random -import pickle import shutil try: @@ -42,7 +45,8 @@ def shuffle_dataset( path_tgt: str, block_size: int = _DEFAULT_BLOCK_SIZE, bucket_size: int = _DEFAULT_SHUFFLE_BUCKET_SIZE, - progress_bar=False, + progress_bar: bool = False, + output_name: Optional[str] = None, ): """Shuffle one distributed datatasetwrite results to another dataset. @@ -60,7 +64,7 @@ def shuffle_dataset( if progress_bar and not support_tqdm: raise RuntimeError("Requires `tqdm` to enable progress bar.") - ds = SimpleDataset(path_srcblock_size=block_size) + ds = SimpleDataset(path_srcserializer=RawSerializer()) num_buckets = (ds.nbytes + bucket_size - 1) // bucket_size tmp_files = [os.path.join(path_src".tmp.%s" % _random_string()) for _ in range(num_buckets)] @@ -74,7 +78,8 @@ def shuffle_dataset( iterator = tqdm(dsdesc="Shuffle step 1/2") for data in iterator: bucket_id = int(random.random() * num_buckets) - pickle.dump(dataf_tmp[bucket_id]) # write into a random bucket + len_data = len(data) + f_tmp[bucket_id].write(struct.pack("I"len_data) + data) finally: # close all files for fp in f_tmp: @@ -83,7 +88,14 @@ def shuffle_dataset( f_tmp = [] # Step 2: shuffle inside bucket - with build_dataset(path_tgt"%s.shuffle" % _random_string()) as writer: + if output_name is None: + output_name = "%s.shuffle" % _random_string() + with build_dataset( + path_tgt, + output_name, + block_size=block_size, + serializer=RawSerializer(), + ) as writer: iterator = tmp_files if progress_bar: iterator = tqdm(tmp_filesdesc="Shuffle step 2/2") @@ -93,7 +105,12 @@ def shuffle_dataset( data_in_bucket = [] while True: try: - data_in_bucket.append(pickle.load(fp)) + raw_data = fp.read(4) + if len(raw_data) == 0: + # EOF + break + len_data = struct.unpack("I"raw_data)[0] + data_in_bucket.append(fp.read(len_data)) except EOFError: break random.shuffle(data_in_bucket) @@ -125,12 +142,11 @@ def compact_dataset(path: str): info: List[FileInfo] = [] if os.path.exists(meta_path): - with open(meta_path"rb") as f: - info = pickle.load(f) + info = _read_info_list(meta_path) else: raise ValueError("Dataset not exists") - nw_info = [] + nw_info: List[FileInfo] = [] curr_block = 0 for v in info: if not os.path.exists(v.file_name): @@ -146,14 +162,12 @@ def compact_dataset(path: str): v.nbytes, v.nlines, v.mask, + v.block_size, ) ) curr_block += num_file_block - random_fname = os.path.join(path".meta.bin.%s" % _random_string()) - with open(random_fname"wb") as f: - pickle.dump(nw_infof) - os.rename(random_fnamemeta_path) + _write_info_list(meta_pathnw_info) def mask_dataset(path: strdbname: strmask: bool = True): @@ -173,19 +187,14 @@ def mask_dataset(path: strdbname: strmask: bool = True): info: List[FileInfo] = [] if os.path.exists(meta_path): - with open(meta_path"rb") as f: - info = pickle.load(f) + info = _read_info_list(meta_path) else: raise ValueError("Dataset not exists") for v in info: if v.file_name == dbname: v.mask = mask - - random_fname = os.path.join(path".meta.bin.%s" % _random_string()) - with open(random_fname"wb") as f: - pickle.dump(infof) - os.rename(random_fnamemeta_path) + _write_info_list(meta_pathinfo) def merge_dataset(dst: strsrc: str): @@ -195,15 +204,13 @@ def merge_dataset(dst: strsrc: str): info_src: List[FileInfo] = [] if os.path.exists(meta_path_src): - with open(meta_path_src"rb") as f: - info_src = pickle.load(f) + info_src = _read_info_list(meta_path_src) else: raise ValueError("Dataset not exists") info_dst: List[FileInfo] = [] if os.path.exists(meta_path_dst): - with open(meta_path_dst"rb") as f: - info_dst = pickle.load(f) + info_dst = _read_info_list(meta_path_dst) else: raise ValueError("Dataset not exists") @@ -219,6 +226,7 @@ def merge_dataset(dst: strsrc: str): v.nbytes, v.nlines, v.mask, + v.block_size, ) ) curr_block += num_file_block @@ -244,11 +252,9 @@ def merge_dataset(dst: strsrc: str): v.nbytes, v.nlines, v.mask, + v.block_size, ) ) curr_block += num_file_block - random_fname = os.path.join(dst".meta.bin.%s" % _random_string()) - with open(random_fname"wb") as f: - pickle.dump(nw_infof) - os.rename(random_fnamemeta_path_dst) + _write_info_list(meta_path_dstnw_info) diff --git a/cpm-live/cpm_live/generation/__init__.py b/cpm-live/cpm_live/generation/__init__.py new file mode 100644 index 0000000..70de125 --- /dev/null +++ b/cpm-live/cpm_live/generation/__init__.py @@ -0,0 +1 @@ +from .ant import CPMAntBeamSearchCPMAntRandomSamplingCPMAntGeneration diff --git a/cpm-live/cpm_live/generation/ant.py b/cpm-live/cpm_live/generation/ant.py new file mode 100644 index 0000000..a9e5893 --- /dev/null +++ b/cpm-live/cpm_live/generation/ant.py @@ -0,0 +1,385 @@ +import torch +import torch.nn.functional as F +from .generation_utils import BeamHypothesesapply_repetition_penaltytop_k_top_p_filtering +from ..utils import pad + + +class CPMAntGeneration: + def __init__(selfmodeltokenizerprompt_length=32): + model.eval() + self.model = model + self.tokenizer = tokenizer + self.prompt_length = prompt_length + + def _convert_to_tensors(selfinput_texttask_id=2): + model_inputs = {} + input_ids = [self.tokenizer.bos_id] + self.tokenizer.encode(input_text) + input_ids = [j for j in input_ids if j != self.tokenizer.unk_id] + + model_inputs["input"] = [ + x + self.prompt_length * task_id for x in range(self.prompt_length) + ] + input_ids + model_inputs["length"] = len(model_inputs["input"]) + model_inputs["position"] = list(range(len(model_inputs["input"]))) + model_inputs["span"] = [0] * len(model_inputs["input"]) + model_inputs["context"] = [True] * len(model_inputs["input"]) + model_inputs["segment"] = [0] * self.prompt_length + [2] * len(input_ids) + + for key in model_inputs: + model_inputs[key] = torch.tensor(model_inputs[key]).int().unsqueeze(0) + + return model_inputs + + def _process_texts(selftext_list): + input_tensors = list(map(self._convert_to_tensorstext_list)) + keys = set(input_tensors[0].keys()) + padded = {} + for key in keys: + padded[key] = pad(input_tensorskeypadding_side='left').cuda() + return padded + + def generate(selftext_list**kwargs): + model_inputs = self._process_texts(text_list) + with torch.inference_mode(): + result = self._decode(model_inputs**kwargs) + return result + + def _decode(selfmodel_inputs**kwargs): + raise NotImplementedError("_decode is not implemented.") + + +class CPMAntBeamSearch(CPMAntGeneration): + def _decode( + self, + model_inputs, + beam_size=3, + max_length=100, + repetition_penalty=1.0, + repetition_window=None, + **kwargs + ): + """ + Beam search + Args: + model_inputs (dict): input ids. + beam_size (intoptionaldefaults to 3): beam size of beam search. + generate_length (intoptionaldefaults to 100): maximum generation length. + repetition_penalty (floatoptionaldefaults to 1.0): repetition penalty coefficient1.0 means no penalty. + repetition_window (intoptionaldefaults to None): window size of repetition penaltyNone means that all output tokens are penalized. + """ # noqa: E501 + # generate_length + 1 for EOS token + max_length += 1 + + # expand dimmension + batch_size = model_inputs["input"].size(0) + input = ( + model_inputs["input"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + length = ( + model_inputs["length"] + .unsqueeze(1) + .expand(batch_sizebeam_size) + .contiguous() + .view( + batch_size * beam_size, + ) + ) + context = ( + model_inputs["context"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + position = ( + model_inputs["position"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + segment = ( + model_inputs["segment"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + span = ( + model_inputs["span"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + + done = [False for _ in range(batch_size)] + + beam_scores = torch.zeros((batch_sizebeam_size)dtype=torch.floatdevice=input.device) + beam_scores[:1:] = -1e9 + beam_scores = beam_scores.view(-1) + + # generated hypotheses + generated_hyps = [ + BeamHypotheses(beam_sizemax_lengthlength_penalty=1early_stopping=False) + for _ in range(batch_size) + ] + + pred_start_index = input.size(-1) + past_key_values = None + for i in range(max_length + 1): + if i == 0: + logits_past_key_values = self.model.inference( + input=input, + length=length, + context=context, + position=position, + segment=segment, + span=span, + past_key_values=past_key_values, + ) + else: + logits_past_key_values = self.model.inference( + input=input[:-1:], + length=length, + context=context, + position=position, + segment=segment, + span=span, + past_key_values=past_key_values, + ) + + # skip all steps when we are done with each sentence + if all(done): + break + + # (batch * beamseqlenmodel_dim) + logits = logits[:-1:] + + if i == 0: + logits[:self.tokenizer.eos_id] = -float("inf") + logits[:self.tokenizer.newline_id] = -float("inf") + + apply_repetition_penalty( + logits, + batch_size, + beam_size, + input, + repetition_penalty, + pred_start_index, + input.size(-1) - 1, + repetition_window, + ) + scores = F.log_softmax(logitsdim=-1) + + next_scores = scores + beam_scores[:None].expand_as( + scores + ) # (batch_size * beam_sizevocab_size) + + # re-organize to group the beam together (we are keeping top hypothesis accross beams) + next_scores = next_scores.view(batch_size-1) # (batch_sizebeam_size * vocab_size) + next_scoresnext_words = torch.topk( + next_scores2 * beam_sizedim=1largest=Truesorted=True + ) + + assert next_scores.size() == next_words.size() == (batch_size2 * beam_size) + next_batch_beam = [] + + for sent_id in range(batch_size): + # if we are done with this sentence + done[sent_id] = done[sent_id] or generated_hyps[sent_id].is_done( + next_scores[sent_id].max().item()i + ) + if done[sent_id]: + next_batch_beam.extend( + [(0self.tokenizer.pad_id0)] * beam_size + ) # pad the batch + continue + + # next sentence beam content + next_sent_beam = [] + + # next words for this sentence + for idxvalue in zip(next_words[sent_id]next_scores[sent_id]): + + # get beam and word IDs + beam_id = torch.div(idxscores.size(-1)rounding_mode="floor") + word_id = idx % scores.size(-1) + + # end of sentenceor next word + if word_id == self.tokenizer.eos_id or i == max_length: + generated_hyps[sent_id].add( + input[sent_id * beam_size + beam_idpred_start_index:] + .clone() + .cpu() + .tolist(), + value.item(), + ) + else: + next_sent_beam.append((valueword_idsent_id * beam_size + beam_id)) + + # the beam for next step is full + if len(next_sent_beam) == beam_size: + break + + # update next beam content + assert len(next_sent_beam) == 0 if i == max_length else beam_size + if len(next_sent_beam) == 0: + next_sent_beam = [(0self.tokenizer.pad_id0)] * beam_size # pad the batch + next_batch_beam.extend(next_sent_beam) + assert len(next_batch_beam) == beam_size * (sent_id + 1) + + # we have reached the last step + if i == max_length: + break + + # sanity check / prepare next batch + assert len(next_batch_beam) == batch_size * beam_size + beam_scores = beam_scores.new([x[0] for x in next_batch_beam]) + beam_words = input.new([x[1] for x in next_batch_beam]) + beam_idx = length.new([x[2] for x in next_batch_beam]).long() + + # re-order batch and internal states + input = input[beam_idx:] + + past_key_values = [list(each) if each is not None else each for each in past_key_values] # type: ignore # noqa: E501 + for key_value_layer in past_key_values: + if key_value_layer is not None: + key_value_layer[0] = key_value_layer[0][beam_idx] + key_value_layer[1] = key_value_layer[1][beam_idx] + + # update input ids + input = torch.cat([inputbeam_words.unsqueeze(1)]dim=-1) + length += 1 + context = torch.cat( + [contexttorch.ones((context.size(0)1)dtype=torch.intdevice=context.device)], + dim=-1, + ) + position = torch.cat([positionposition[:-1:] + 1]dim=-1) + segment = torch.cat( + [segmentsegment[:-1:]]dim=-1 + ) # segment id always the same as the previous token + span = torch.cat([spanspan[:-1:]]dim=-1) + + # select the best hypotheses + results = [] + for ihypotheses in enumerate(generated_hyps): + best_hyp = max(hypotheses.hypkey=lambda x: x[0])[1] + results.append(best_hyp) + + result_text = list(map(self.tokenizer.decoderesults)) + return result_text + + +class CPMAntRandomSampling(CPMAntGeneration): + def _decode( + self, + model_inputs, + max_length=100, + top_k=0, + top_p=0.9, + temperature=0.9, + repetition_penalty=1.0, + repetition_window=None, + **kwargs + ): + """ + Top-k and top-p sampling. + Args: + model_inputs (dict): input ids + generate_length (intoptionaldefaults to 100): maximum generation length + top_k (intoptionaldefaults to 0): keep only top k tokens with highest probability. 0 means keeping all tokens. + top_p (intoptionaldefaults to 0.9): keep the top tokens with cumulative probability >= top_p. + temperature (intoptionaldefaults to 0.9): the value that can cool down the logits distribution. + repetition_penalty (floatoptionaldefaults to 1.0): repetition penalty coefficient1.0 means no penalty. + repetition_window (intoptionaldefaults to None): window size of repetition penaltyNone means that all output tokens are penalized. + """ # noqa: E501 + # generate_length + 1 for EOS token + max_length += 1 + + input = model_inputs["input"] + length = model_inputs["length"] + context = model_inputs["context"] + position = model_inputs["position"] + segment = model_inputs["segment"] + span = model_inputs["span"] + batch_size = input.size(0) + + pred_start_index = input.size(-1) + past_key_values = None + done = [False for _ in range(batch_size)] + results = [None for _ in range(batch_size)] + for i in range(max_length): + if i == 0: + logits_past_key_values = self.model.inference( + input=input, + length=length, + context=context, + position=position, + segment=segment, + span=span, + past_key_values=past_key_values, + ) + else: + logits_past_key_values = self.model.inference( + input=input[:-1:], + length=length, + context=context, + position=position, + segment=segment, + span=span, + past_key_values=past_key_values, + ) + + logits = logits[:-1:] + + if i == 0: + logits[:self.tokenizer.eos_id] = -float("inf") + logits[:self.tokenizer.newline_id] = -float("inf") + + apply_repetition_penalty( + logits, + batch_size, + 1, + input, + repetition_penalty, + pred_start_index, + input.size(-1) - 1, + repetition_window, + ) + + logits = logits / temperature + logits = top_k_top_p_filtering(logitstop_k=top_ktop_p=top_p) + + probs = F.softmax(logitsdim=-1) + next_token = torch.multinomial(probsnum_samples=1) + + for idx in range(batch_size): + if not done[idx] and ( + next_token[idx].item() == self.tokenizer.eos_id or i == max_length - 1 + ): + done[idx] = True + results[idx] = input[idxpred_start_index:].clone().cpu().tolist() # type: ignore # noqa: E501 + + if sum(done) == batch_size: + break + + # update input ids + input = torch.cat([inputnext_token]dim=-1) + length += 1 + context = torch.cat( + [contexttorch.ones((context.size(0)1)dtype=torch.intdevice=context.device)], + dim=-1, + ) + position = torch.cat([positionposition[:-1:] + 1]dim=-1) + segment = torch.cat( + [segmentsegment[:-1:]]dim=-1 + ) # segment id always the same as the previous token + span = torch.cat([spanspan[:-1:]]dim=-1) + + result_text = list(map(self.tokenizer.decoderesults)) + return result_text diff --git a/cpm-live/cpm_live/generation/bee.py b/cpm-live/cpm_live/generation/bee.py new file mode 100644 index 0000000..5120b1f --- /dev/null +++ b/cpm-live/cpm_live/generation/bee.py @@ -0,0 +1,629 @@ +from typing import AnyDictListTuple +import numpy as np +import torch +import torch.nn.functional as F +from .generation_utils import BeamHypothesesapply_repetition_penalty +from ..tokenizers.bee import CPMBeeTokenizer +from ..models.bee import CPMBee +from ..training_tasks.bee.pretrain import convert_data_to_id +from ..utils import pad + + +class CPMBeeGeneration: + def __init__(selfmodel: CPMBeetokenizer: CPMBeeTokenizer): + model.eval() + self.model = model + self.tokenizer = tokenizer + + def _convert_to_tensors(selfdata: Anyin_context_samples: List[Any] = []): + answer_placeholders = [] + + def _put_placeholder(data: Anypath: List[str] = []): + if isinstance(datadict): + ret = {} + for kv in data.items(): + ret[k] = _put_placeholder(vpath + [k]) + return ret + else: + answer_placeholders.append(path) + return "".format(len(answer_placeholders)) + + data[""] = _put_placeholder(data[""]) + ( + input_ids, + input_id_subs, + context, + segment_ids, + segment_rel, + n_segments, + table_states, + ) = convert_data_to_id(self.tokenizerdatashuffle_answer=Falsemax_depth=8) + + sub_ans_map: Dict[intint] = {} + for fake_idtoken_sub in table_states["token_id_table"][""].items(): + token = table_states["ext_table"][fake_id] + if token.startswith(""): + ans_id = int(token[5:-1]) + sub_ans_map[token_sub] = ans_id + + tmp_input_ids = [] + tmp_input_sub = [] + tmp_input_seg = [] + + predict_segments: List[Tuple[intint]] = [] + for i in range(input_ids.shape[0]): + if context[i] == 0: + if input_ids[i] == self.tokenizer.encoder[""]: + # is ans + # (segment_idans_id) + predict_segments.append((segment_ids[i]sub_ans_map[input_id_subs[i]])) + else: + tmp_input_ids.append(input_ids[i]) + tmp_input_sub.append(input_id_subs[i]) + tmp_input_seg.append(segment_ids[i]) + + if len(predict_segments) == 0: + raise ValueError("No answer to predict") + + input_ids = np.array(tmp_input_idsdtype=np.int32) + input_id_subs = np.array(tmp_input_subdtype=np.int32) + context = np.full_like(tmp_input_ids1dtype=np.int8) + segment_ids = np.array(tmp_input_segdtype=np.int32) + sample_ids = np.zeros(input_ids.shapedtype=np.int32) + segment_rel_offset = np.zeros(input_ids.shapedtype=np.int32) + num_segments = np.full(input_ids.shapen_segmentsdtype=np.int32) + + for isample in enumerate(in_context_samples): + ( + sample_input_ids, + sample_id_subs, + _, + sample_segments, + sample_rel, + n_segments, + table_states, + ) = convert_data_to_id(self.tokenizersampletable_statesmax_depth=8) + input_ids = np.concatenate([input_idssample_input_ids]axis=0) + input_id_subs = np.concatenate([input_id_subssample_id_subs]axis=0) + context = np.concatenate( + [contextnp.ones(sample_input_ids.shapedtype=np.int8)]axis=0 + ) + segment_ids = np.concatenate([segment_idssample_segments]axis=0) + segment_rel_offset = np.concatenate( + [ + segment_rel_offset, + np.full(sample_input_ids.shapesegment_rel.shape[0]dtype=np.int32), + ], + axis=0, + ) + segment_rel = np.concatenate([segment_relsample_rel]axis=0) + sample_ids = np.concatenate( + [sample_idsnp.full(sample_input_ids.shapei + 1dtype=np.int32)]axis=0 + ) + num_segments = np.concatenate( + [num_segmentsnp.full(sample_input_ids.shapen_segmentsdtype=np.int32)]axis=0 + ) + input_pos = np.arange(input_ids.shape[0]dtype=np.int32) + + return ( + input_ids, + input_id_subs, + input_pos, + context, + segment_ids, + segment_rel_offset, + segment_rel, + sample_ids, + num_segments, + predict_segments, + answer_placeholders, + table_states["ext_table"], + table_states["token_id_table"], + ) + + def _process_list(selfdata_list: List[Any]): + pack_tensor = [] + other_info = [] + segment_rel_pack = [] + + batch_ext_table_map: Dict[Tuple[intint]int] = {} + batch_ext_table_ids: List[int] = [] + batch_ext_table_sub: List[int] = [] + + for data in data_list: + ( + input_ids, + input_id_subs, + input_pos, + context, + segment_ids, + segment_rel_offset, + segment_rel, + sample_ids, + num_segments, + predict_segments, + answer_placeholders, + ext_table, + token_id_table, + ) = self._convert_to_tensors(data[]) + rev_ext_table: Dict[intstr] = {} + for tokenmp in token_id_table.items(): + if token == "": + continue + token_id = self.tokenizer.encoder[token] + for fake_idtoken_sub in mp.items(): + if token_sub > 0: + if (token_idtoken_sub) not in batch_ext_table_map: + batch_ext_table_map[(token_idtoken_sub)] = ( + len(batch_ext_table_ids) + self.tokenizer.vocab_size + ) + batch_ext_table_ids.append(token_id) + batch_ext_table_sub.append(token_sub) + rev_ext_table[batch_ext_table_map[(token_idtoken_sub)]] = ext_table[ + fake_id + ] + else: + rev_ext_table[token_id] = ext_table[fake_id] + pack_tensor.append( + { + "input": torch.from_numpy(input_ids).unsqueeze(0), + "input_sub": torch.from_numpy(input_id_subs).unsqueeze(0), + "input_pos": torch.from_numpy(input_pos).unsqueeze(0), + "context": torch.from_numpy(context).unsqueeze(0), + "sample_idx": torch.from_numpy(sample_ids).unsqueeze(0), + "num_segments": torch.from_numpy(num_segments).unsqueeze(0), + "segment": torch.from_numpy(segment_ids).unsqueeze(0), + "segment_rel_offset": torch.from_numpy(segment_rel_offset).unsqueeze(0), + } + ) + segment_rel_pack.append(torch.from_numpy(segment_rel)) + other_info.append( + { + "predict_segments": predict_segments, + "answer_placeholders": answer_placeholders, + "ext_table": rev_ext_table, + } + ) + + keys = set(pack_tensor[0].keys()) + padded = {} + for key in keys: + padded[key] = pad(pack_tensorkey).cuda() + + max_num_rels = 0 + for rel in segment_rel_pack: + max_num_rels = max(max_num_relsrel.size(0)) + padded_rels = torch.zeros(len(segment_rel_pack)max_num_relsdtype=torch.int32) + for irel in enumerate(segment_rel_pack): + padded_rels[i: rel.size(0)] = rel + padded["segment_rel"] = padded_rels.cuda() + padded["batch_ext_table_ids"] = torch.tensor( + batch_ext_table_idsdtype=torch.int32device="cuda" + ) + padded["batch_ext_table_sub"] = torch.tensor( + batch_ext_table_subdtype=torch.int32device="cuda" + ) + return paddedother_info + + def generate(selfdata_list**kwargs): + model_inputsother_info = self._process_list(data_list) + with torch.inference_mode(): + result_ids = self._decode(model_inputsother_info**kwargs) + for sent_idresult in enumerate(result_ids): + ans_result_map: Dict[intList[int]] = {} + for raw_word_idans_id in result: + if ans_id not in ans_result_map: + ans_result_map[ans_id] = [] + ans_result_map[ans_id].append(raw_word_id) + + answer_placeholders = other_info[sent_id]["answer_placeholders"] + ext_table = other_info[sent_id]["ext_table"] + data = data_list[sent_id] + for ans_idtoken_ids in ans_result_map.items(): + if token_ids[-1] == self.tokenizer.eos_id: + token_ids = token_ids[:-1] + text = self.tokenizer.decode(token_idsext_table) + path = answer_placeholders[ans_id - 1] + + if len(path) > 0: + p = data[""] + for part in path[:-1]: + p = p[part] + p[path[-1]] = text + else: + data[""] = text + for ans_id in range(len(answer_placeholders)): + if (ans_id + 1) not in ans_result_map: + path = answer_placeholders[ans_id] + p = data[""] + for part in path[:-1]: + p = p[part] + p[path[-1]] = None + return data_list + + def _decode(selfmodel_inputsother_info**kwargs): + raise NotImplementedError("_decode is not implemented.") + + +class CPMBeeBeamSearch(CPMBeeGeneration): + def _decode( + self, + model_inputs, + other_info, + beam_size=3, + max_length=100, + repetition_penalty=1.0, + repetition_window=None, + ): + """ + Beam search + Args: + model_inputs (dict): input ids. + beam_size (intoptionaldefaults to 3): beam size of beam search. + generate_length (intoptionaldefaults to 100): maximum generation length. + repetition_penalty (floatoptionaldefaults to 1.0): repetition penalty coefficient1.0 means no penalty. + repetition_window (intoptionaldefaults to None): window size of repetition penaltyNone means that all output tokens are penalized. + """ # noqa: E501 + # generate_length + 1 for EOS token + max_length += 1 + + # expand dimmension + batch_size = model_inputs["input"].size(0) + input: torch.Tensor = ( + model_inputs["input"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + input_sub: torch.Tensor = ( + model_inputs["input_sub"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + input_pos: torch.Tensor = ( + model_inputs["input_pos"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + context: torch.Tensor = ( + model_inputs["context"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + sample_ids: torch.Tensor = ( + model_inputs["sample_idx"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + num_segments: torch.Tensor = ( + model_inputs["num_segments"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + segment: torch.Tensor = ( + model_inputs["segment"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + segment_rel_offset: torch.Tensor = ( + model_inputs["segment_rel_offset"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + segment_rel: torch.Tensor = ( + model_inputs["segment_rel"] + .unsqueeze(1) + .expand(batch_sizebeam_size-1) + .contiguous() + .view(batch_size * beam_size-1) + ) + ext_table_ids: torch.Tensor = model_inputs["batch_ext_table_ids"] + ext_table_sub: torch.Tensor = model_inputs["batch_ext_table_sub"] + ext_table_ids_cpu = ext_table_ids.cpu() + ext_table_sub_cpu = ext_table_sub.cpu() + + done = [False for _ in range(batch_size)] + + beam_scores = torch.zeros((batch_sizebeam_size)dtype=torch.floatdevice=input.device) + beam_scores[:1:] = -1e9 + beam_scores = beam_scores.view(-1) + + # generated hypotheses + generated_hyps = [ + BeamHypotheses(beam_sizemax_lengthlength_penalty=1early_stopping=False) + for _ in range(batch_size) + ] + + pred_start_index = input.size(-1) + __past_key_values = self.model.inference( + input=input, + input_sub=input_sub, + position=input_pos, + context=context, + sample_ids=sample_ids, + num_segments=num_segments, + segment=segment, + segment_rel_offset=segment_rel_offset, + segment_rel=segment_rel, + ext_table_ids=ext_table_ids, + ext_table_sub=ext_table_sub, + past_key_values=None, + ) + + beam_states = [] + for sent_id in range(batch_size): + instance_beam_states = [] + + for beam_id in range(beam_size): + instance_beam_states.append( + { + "idx": 0, + "ans": [], + "nx_token_id": self.tokenizer.bos_id, + "nx_token_sub": 0, + "nx_segment_id": other_info[sent_id]["predict_segments"][0][0], + "nx_position": 0, + } + ) + beam_states.append(instance_beam_states) + for i in range(max_length + 1): + tmp_input = [] + tmp_input_sub = [] + tmp_position = [] + tmp_segment = [] + for sent_id in range(batch_size): + for beam_id in range(beam_size): + tmp_input.append(beam_states[sent_id][beam_id]["nx_token_id"]) + tmp_input_sub.append(beam_states[sent_id][beam_id]["nx_token_sub"]) + tmp_position.append(beam_states[sent_id][beam_id]["nx_position"]) + tmp_segment.append(beam_states[sent_id][beam_id]["nx_segment_id"]) + with torch.no_grad(): + input = torch.cat( + [ + input, + torch.tensor(tmp_inputdtype=torch.int32device="cuda").view( + batch_size * beam_size1 + ), + ], + dim=-1, + ) + logits_past_key_values = self.model.inference( + input=input[:-1:], + input_sub=torch.tensor(tmp_input_subdtype=torch.int32device="cuda").view( + batch_size * beam_size1 + ), + position=torch.tensor(tmp_positiondtype=torch.int32device="cuda").view( + batch_size * beam_size1 + ), + context=torch.ones( + batch_size * beam_sizedtype=torch.booldevice="cuda" + ).view(batch_size * beam_size1), + sample_ids=torch.zeros( + batch_size * beam_sizedtype=torch.int32device="cuda" + ).view(batch_size * beam_size1), + num_segments=num_segments[:-1:], + segment=torch.tensor(tmp_segmentdtype=torch.int32device="cuda").view( + batch_size * beam_size1 + ), + segment_rel_offset=segment_rel_offset[:-1:], + segment_rel=segment_rel, + ext_table_ids=ext_table_ids, + ext_table_sub=ext_table_sub, + past_key_values=past_key_values, + ) + logits = logits[:-1:] + + # skip all steps when we are done with each sentence + if all(done): + break + + for sent_id in range(batch_size): + if self.tokenizer.unk_id not in other_info[sent_id]["ext_table"]: + # unk is not allowedmask unk + logits[ + sent_id * beam_size : (sent_id + 1) * beam_sizeself.tokenizer.unk_id + ] = -10000 + ext_ids = set() + for v in other_info[sent_id]["ext_table"].keys(): + ext_ids.add(v) + for ext_id in range( + self.tokenizer.vocab_sizeself.tokenizer.vocab_size + ext_table_ids.size(0) + ): + if ext_id not in ext_ids: + logits[sent_id * beam_size : (sent_id + 1) * beam_sizeext_id] = -10000 + + apply_repetition_penalty( + logits, + batch_size, + beam_size, + input, + repetition_penalty, + pred_start_index, + input.size(-1) - 1, + repetition_window, + ) + scores = F.log_softmax(logitsdim=-1) + next_scores = scores + beam_scores[:None].expand_as( + scores + ) # (batch_size * beam_sizevocab_size) + + # re-organize to group the beam together (we are keeping top hypothesis accross beams) + next_scores = next_scores.view(batch_size-1) # (batch_sizebeam_size * vocab_size) + next_scoresnext_words = torch.topk( + next_scores2 * beam_sizedim=1largest=Truesorted=True + ) + assert next_scores.size() == next_words.size() == (batch_size2 * beam_size) + next_beam_states = [] + + for sent_id in range(batch_size): + # if we are done with this sentence + done[sent_id] = done[sent_id] or generated_hyps[sent_id].is_done( + next_scores[sent_id].max().item()i + ) + if done[sent_id]: + next_beam_states.append( + [ + ( + { + "idx": 0, + "ans": [], + "nx_token_id": 0, + "nx_token_sub": 0, + "nx_segment_id": 0, + "nx_position": 0, + }, + 0, + 0, + ) + ] + * beam_size + ) # pad the batch + continue + + # next sentence beam content + next_instance_beam_states = [] + + # next words for this sentence + for idxvalue in zip(next_words[sent_id]next_scores[sent_id]): + + # get beam and word IDs + beam_id = torch.div(idxscores.size(-1)rounding_mode="floor").item() + word_id = (idx % scores.size(-1)).item() + + curr_info = beam_states[sent_id][beam_id] + # end of sentenceor next word + if ( + word_id == self.tokenizer.eos_id + and (curr_info["idx"] + 1 == len(other_info[sent_id]["predict_segments"])) + ) or i == max_length: + generated_hyps[sent_id].add( + beam_states[sent_id][beam_id]["ans"] + + [ + ( + word_id, + other_info[sent_id]["predict_segments"][curr_info["idx"]][1], + ) + ], + value.item(), + ) + elif word_id == self.tokenizer.eos_id: + next_instance_beam_states.append( + ( + { + "idx": curr_info["idx"] + 1, + "ans": curr_info["ans"] + + [ + ( + word_id, + other_info[sent_id]["predict_segments"][ + curr_info["idx"] + ][1], + ) + ], + "nx_token_id": self.tokenizer.bos_id, + "nx_token_sub": 0, + "nx_segment_id": other_info[sent_id]["predict_segments"][ + curr_info["idx"] + 1 + ][0], + "nx_position": 0, + }, + value.item(), + sent_id * beam_size + beam_id, + ) + ) + + else: + raw_word_id = word_id + word_id_sub = 0 + if word_id >= self.tokenizer.vocab_size: + word_id -= self.tokenizer.vocab_size + word_id_sub = int(ext_table_sub_cpu[word_id].item()) + word_id = int(ext_table_ids_cpu[word_id].item()) + + next_instance_beam_states.append( + ( + { + "idx": curr_info["idx"], + "ans": curr_info["ans"] + + [ + ( + raw_word_id, + other_info[sent_id]["predict_segments"][ + curr_info["idx"] + ][1], + ) + ], + "nx_token_id": word_id, + "nx_token_sub": word_id_sub, + "nx_segment_id": curr_info["nx_segment_id"], + "nx_position": curr_info["nx_position"] + 1, + }, + value.item(), + sent_id * beam_size + beam_id, + ) + ) + + # the beam for next step is full + if len(next_instance_beam_states) == beam_size: + break + + # update next beam content + assert len(next_instance_beam_states) == 0 if i == max_length else beam_size + next_beam_states.append(next_instance_beam_states) + + # we have reached the last step + if i == max_length: + break + + # sanity check / prepare next batch + beam_reorder_idx = [] + beam_new_scores = [] + beam_states = [] + for sent_id in range(batch_size): + instance_beam_states = [] + for beam_id in range(beam_size): + statevaluebeam_idx = next_beam_states[sent_id][beam_id] + beam_reorder_idx.append(beam_idx) + beam_new_scores.append(value) + instance_beam_states.append(state) + beam_states.append(instance_beam_states) + + input = input[beam_reorder_idx:] + beam_scores = torch.tensor(beam_new_scoresdtype=torch.floatdevice=input.device) + for kw in past_key_values.keys(): + if kw == "buffer": + buf_list = past_key_values[kw] + nw_buf_list = [] + for buf in buf_list: + if buf is None: + nw_buf_list.append((NoneNone)) + else: + k_bufv_buf = buf + nw_buf_list.append( + (k_buf[beam_reorder_idx:]v_buf[beam_reorder_idx:]) + ) + past_key_values[kw] = nw_buf_list + else: + past_key_values[kw] = past_key_values[kw][beam_reorder_idx:] + + # select the best hypotheses + results = [] + for sent_idhypotheses in enumerate(generated_hyps): + best_hyp = max(hypotheses.hypkey=lambda x: x[0])[1] + results.append(best_hyp) + return results diff --git a/cpm-live/cpm_live/generation/generation_utils.py b/cpm-live/cpm_live/generation/generation_utils.py new file mode 100644 index 0000000..07867d8 --- /dev/null +++ b/cpm-live/cpm_live/generation/generation_utils.py @@ -0,0 +1,112 @@ +import torch +import torch.nn.functional as F + + +def top_k_top_p_filtering(logitstop_k=0top_p=0.0filter_value=-float("inf")): + # This function has been mostly taken from huggingface conversational ai code at + # https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313 + + if top_k > 0: + # Remove all tokens with a probability less than the last token of the top-k + indices_to_remove = logits < torch.topk(logitstop_k)[0][...-1None] + logits[indices_to_remove] = filter_value + + batch_size = logits.size()[0] + if top_p > 0.0: + logits = logits.view(batch_size-1).contiguous() + for index in range(len(logits)): + + sorted_logitssorted_indices = torch.sort(logits[index].view(-1)descending=True) + cumulative_probs = torch.cumsum(F.softmax(sorted_logitsdim=-1)dim=-1) + # Remove tokens with cumulative probability above the threshold + sorted_indices_to_remove = cumulative_probs > top_p + # Shift the indices to the right to keep also the first token above the threshold + sorted_indices_to_remove[...1:] = sorted_indices_to_remove[...:-1].clone() + sorted_indices_to_remove[...0] = 0 + indices_to_remove = sorted_indices[sorted_indices_to_remove] + logits[index][indices_to_remove] = filter_value + + logits = logits.view(batch_size-1).contiguous() + + return logits + + +def apply_repetition_penalty( + logits, + batch_size, + num_beams, + prev_output_tokens, + repetition_penalty, + start_idx=None, + end_idx=None, + window_size=None, +): + # only conduct repetition penalty for the output + assert repetition_penalty >= 1"repetition penalty coefficient should >= 1" + # repetition penalty (from CTRL paper https://arxiv.org/abs/1909.05858) + for i in range(batch_size * num_beams): + if start_idx is None or end_idx is None: + output_tokens = prev_output_tokens[i].tolist() + else: + if end_idx >= start_idx: + if window_size: + output_tokens = prev_output_tokens[i][ + max(start_idxend_idx + 1 - window_size) : end_idx + 1 + ].tolist() + else: + output_tokens = prev_output_tokens[i][start_idx : end_idx + 1].tolist() + else: + output_tokens = [] + for previous_token in set(output_tokens): + # if score < 0 then repetition penalty has to + # multiplied to reduce the previous token probability + if logits[iprevious_token] < 0: + logits[iprevious_token] *= repetition_penalty + else: + logits[iprevious_token] /= repetition_penalty + + +class BeamHypotheses: + def __init__(selfn_hypmax_lenlength_penaltyearly_stopping): + """ + Initialize n-best list of hypotheses. + """ + self.max_len = max_len + self.length_penalty = length_penalty + self.early_stopping = early_stopping + self.n_hyp = n_hyp + self.hyp = [] + self.worst_score = 1e9 + + def __len__(self): + """ + Number of hypotheses in the list. + """ + return len(self.hyp) + + def add(selfhypsum_logprobs): + """ + Add a new hypothesis to the list. + """ + score = sum_logprobs / len(hyp) ** self.length_penalty + + if len(self) < self.n_hyp or score > self.worst_score: + self.hyp.append((scorehyp)) + if len(self) > self.n_hyp: + sorted_scores = sorted([(sidx) for idx(s_) in enumerate(self.hyp)]) + del self.hyp[sorted_scores[0][1]] + self.worst_score = sorted_scores[1][0] + else: + self.worst_score = min(scoreself.worst_score) + + def is_done(selfbest_sum_logprobscur_len): + """ + If there are enough hypotheses and that none of the hypotheses being generated + can become better than the worst one in the heapthen we are done with this sentence. + """ + if len(self) < self.n_hyp: + return False + elif self.early_stopping: + return True + else: + return self.worst_score >= best_sum_logprobs / cur_len**self.length_penalty diff --git a/cpm-live/cpm_live/layers/__init__.py b/cpm-live/cpm_live/layers/__init__.py index 4ff22c1..dbd8dcd 100644 --- a/cpm-live/cpm_live/layers/__init__.py +++ b/cpm-live/cpm_live/layers/__init__.py @@ -1,5 +1,5 @@ -from .embedding import Embedding -from .position_embedding import SegmentPositionEmbedding +from .embedding import EmbeddingEmbeddingExt +from .position_embedding import SegmentPositionEmbeddingBucketPositionBiasRotaryEmbedding from .linear import Linear from .layernorm import LayerNorm from .attention import Attention diff --git a/cpm-live/cpm_live/layers/attention.py b/cpm-live/cpm_live/layers/attention.py index c227037..241b14e 100644 --- a/cpm-live/cpm_live/layers/attention.py +++ b/cpm-live/cpm_live/layers/attention.py @@ -72,8 +72,8 @@ def forward( len_q = hidden_q.size(1) len_k = hidden_kv.size(1) - h_q = self.project_q(hidden_q) - h_k = self.project_k(hidden_kv) + h_q = self.project_q(hidden_q) / math.sqrt(math.sqrt(self.dim_head)) + h_k = self.project_k(hidden_kv) / math.sqrt(math.sqrt(self.dim_head)) h_v = self.project_v(hidden_kv) h_q = h_q.view(batch_sizelen_qself.num_headsself.dim_head).permute(0213) @@ -86,7 +86,9 @@ def forward( len_k = h_k.size(-2) # (bn_hlen_qd_h) @ (bn_hd_hlen_k) -> (bn_hlen_qlen_k) - score = torch.matmul(h_qh_k.transpose(-1-2)) / math.sqrt(self.dim_head) + score = torch.matmul( + h_qh_k.transpose(-1-2) + ) # / math.sqrt(self.dim_head) moved to line 75~76 score = score + position_bias score = torch.masked_fill( score, diff --git a/cpm-live/cpm_live/layers/blocks.py b/cpm-live/cpm_live/layers/blocks.py index e5db92e..a16abf6 100644 --- a/cpm-live/cpm_live/layers/blocks.py +++ b/cpm-live/cpm_live/layers/blocks.py @@ -91,7 +91,7 @@ def forward( if self.dropout is not None: x = self.dropout(x) - hidden_states = hidden_states + x + hidden_states = (hidden_states + x) / 1.05 if use_cache: return hidden_statescurrent_key_value @@ -154,7 +154,7 @@ def forward( x = self.ffn(x) if self.dropout is not None: x = self.dropout(x) - hidden_states = hidden_states + x + hidden_states = (hidden_states + x) / 1.05 return hidden_states diff --git a/cpm-live/cpm_live/layers/embedding.py b/cpm-live/cpm_live/layers/embedding.py index 5b805b0..ffc305f 100644 --- a/cpm-live/cpm_live/layers/embedding.py +++ b/cpm-live/cpm_live/layers/embedding.py @@ -13,10 +13,12 @@ # See the License for the specific language governing permissions and # limitations under the License. +from typing import Optional import torch import bmtrain as bmt import math import torch.nn.functional as F +from .position_embedding import RotaryEmbedding class Embedding(bmt.DistributedModule): @@ -60,3 +62,56 @@ def projection(selfx: torch.Tensor): """ # noqa: E501 logits = F.linear(x / math.sqrt(self.dim_model)self.weight) return logits + + +class EmbeddingExt(bmt.DistributedModule): + def __init__( + self, + vocab_size: int, + embedding_size: int, + dtype: torch.dtype = torch.half, + init_mean: float = 0.0, + init_std: float = 1, + distance_scale: int = 16, + ): + + super().__init__() + + self.dim_model = embedding_size + self.rotary_emb = RotaryEmbedding( + dim=embedding_sizedistance_scale=distance_scaledtype=dtype + ) + + self.weight = bmt.DistributedParameter( + torch.empty(vocab_sizeembedding_sizedtype=dtype), + init_method=bmt.ParameterInitializer( + torch.nn.init.normal_mean=init_meanstd=init_std + ), + ) + + def forward(selfids: torch.Tensorids_sub: torch.Tensor): + """ + Args: + ids (:obj:`torch.Tensor` of shape ``(batch_sizeseq_len)``): Indices of input sequence tokens. + ids (:obj:`torch.Tensor` of shape ``(batch_size)``): Subscript of input sequence tokens. + Return: + :obj:`torch.Tensor` of shape ``(batch_sizeseq_lenembedding_size)``: The embedding output. + """ # noqa: E501 + + embeds = F.embedding(idsself.weight) / math.sqrt(self.dim_model) + return self.rotary_emb(embedsids_sub) + + def projection(selfx: torch.Tensorext_table: Optional[torch.Tensor] = None): + """ + Projection based on embedding's weight. For exampleembedding map vocab_size to embed_sizethan projection map embed_size back to vocab_size. + Args: + x (:obj:`torch.Tensor` of shape ``(batchseq_lendim_model)``): Input of projection + ext_table (:obj:`torch.Tensor` of shape ``(ext_table_sizedim_model)``): Ext vocab table. + Returns: + :obj:`torch.Tensor` of shape ``(batchseq_lenvocab_size + ext_table_size)``: The projection output. + """ # noqa: E501 + logits = F.linear(x / math.sqrt(self.dim_model)self.weight) + if ext_table is not None: + logits_ext = F.linear(xext_table) + logits = torch.cat([logitslogits_ext]dim=-1) + return logits diff --git a/cpm-live/cpm_live/layers/feedforward.py b/cpm-live/cpm_live/layers/feedforward.py index 8dd3c23..c015cf3 100644 --- a/cpm-live/cpm_live/layers/feedforward.py +++ b/cpm-live/cpm_live/layers/feedforward.py @@ -101,7 +101,7 @@ def __init__( dim_in=dim_ff, dim_out=dim_model, dtype=dtype, - scale_before=True, + scale_before=False, ) def forward(selfx: torch.Tensor): diff --git a/cpm-live/cpm_live/layers/position_embedding.py b/cpm-live/cpm_live/layers/position_embedding.py index 790b527..2cd49f2 100644 --- a/cpm-live/cpm_live/layers/position_embedding.py +++ b/cpm-live/cpm_live/layers/position_embedding.py @@ -13,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. import math +from typing import Union import torch import bmtrain as bmt import torch.nn.functional as F @@ -21,12 +22,12 @@ class SegmentPositionEmbedding(bmt.DistributedModule): def __init__( self, - num_heads, - num_segments=1, - num_buckets=32, - max_distance=128, - bidirectional=False, - dtype=torch.half, + num_heads: int, + num_segments: int = 1, + num_buckets: int = 32, + max_distance: int = 128, + bidirectional: bool = False, + dtype: torch.dtype = torch.half, init_mean: float = 0.0, init_std: float = 1, ): @@ -125,3 +126,129 @@ def _position_bucket( is_smallrelative_position.to(torch.int32)relative_postion_if_large ) return relative_buckets + + +class BucketPositionBias(bmt.DistributedModule): + def __init__( + self, + num_heads: int, + num_buckets: int = 32, + num_segment_bucket: int = 32, + max_distance: int = 128, + dtype: torch.dtype = torch.half, + init_mean: float = 0.0, + init_std: float = 1, + ) -> None: + super().__init__() + + self.num_heads = num_heads + self.num_buckets = num_buckets + self.num_segment_bucket = num_segment_bucket + self.max_distance = max_distance + + self.relative_attention_bias = bmt.DistributedParameter( + torch.empty(num_buckets + num_segment_bucketnum_headsdtype=dtype), + init_method=bmt.ParameterInitializer( + torch.nn.init.normal_mean=init_meanstd=init_std + ), + ) + + def forward( + self, + query_pos: torch.Tensor # (batchlen_q) + key_pos: torch.Tensor # (batchlen_k) + rel_buckets: torch.Tensor # (batchlen_qlen_k) + ): + with torch.no_grad(): + + batch = key_pos.size(0) + keylen = key_pos.size(1) + querylen = query_pos.size(1) + + assert key_pos.size(0) == query_pos.size(0) + assert ( + rel_buckets.size(0) == batch + and rel_buckets.size(1) == querylen + and rel_buckets.size(2) == keylen + ) + + relative_position_bucket = rel_buckets - 1 + self.num_buckets # 与相对位置编码区间不重叠 + + # b*q*k + inner_segment_bucket = self._position_bucket( + key_pos[...None:] - query_pos[...:None], + num_buckets=self.num_buckets, + max_distance=self.max_distance, + ) + relative_position_bucket = torch.where( + rel_buckets == 0, + inner_segment_bucket, + relative_position_bucket, + ) + # (batchlen_qlen_k) + + # (batchlen_qlen_knum_heads) + embeds = F.embedding(relative_position_bucketself.relative_attention_bias) + # (batchnum_headslen_qlen_k) + embeds = embeds.permute(0312).contiguous() + return embeds + + def _position_bucket(selfrelative_positionnum_buckets=32max_distance=128): + relative_buckets = 0 + num_buckets //= 2 + relative_buckets = (relative_position > 0).to(torch.int32) * num_buckets + relative_position = torch.abs(relative_position) + + max_exact = num_buckets // 2 + is_small = relative_position < max_exact + relative_postion_if_large = max_exact + ( + torch.log(relative_position.float() / max_exact) + / math.log(max_distance / max_exact) + * (num_buckets - max_exact) + ).to(torch.int32) + relative_postion_if_large = torch.min( + relative_postion_if_large, + torch.full_like(relative_postion_if_largenum_buckets - 1), + ) + relative_buckets += torch.where( + is_smallrelative_position.to(torch.int32)relative_postion_if_large + ) + return relative_buckets + + +class RotaryEmbedding(bmt.DistributedModule): + def __init__( + self, + dim, + base=10000, + distance_scale: Union[intfloat] = 1, + dtype: torch.dtype = torch.half, + ): + super().__init__() + inv_freq = 1.0 / ( + base ** (torch.arange(0dim2device="cuda"dtype=torch.float32) / dim) + ) + inv_freq = inv_freq.to(dtype) + self.distance_scale = distance_scale + self.dtype = dtype + self.inv_freq = inv_freq + + def forward(selfx: torch.Tensorx_pos: torch.Tensor): + """ + Args: + x (:obj:`torch.Tensor` of shape ``(...dim)``): Inputs. + x_pos (:obj:`torch.Tensor` of shape ``(...)``): Positions of inputs. + """ + x_pos = x_pos * self.distance_scale + freqs = x_pos[...None].to(self.dtype) * self.inv_freq[None:] # (...dim/2) + + # the same implementation as sat + emb = torch.cat((freqsfreqs)dim=-1) # (...dim) + emb_cos = emb.cos() # (...dim) + emb_sin = emb.sin() # (...dim) + + rotate_x = torch.cat( + [-x[...x.size(-1) // 2 :]x[...: x.size(-1) // 2]]dim=-1 + ) # (...dim) + + return x * emb_cos + rotate_x * emb_sin diff --git a/cpm-live/cpm_live/models/__init__.py b/cpm-live/cpm_live/models/__init__.py index d1e4033..ed9ed68 100644 --- a/cpm-live/cpm_live/models/__init__.py +++ b/cpm-live/cpm_live/models/__init__.py @@ -1 +1,4 @@ from .ant import CPMAntConfigCPMAnt +from .bee import CPMBeeConfigCPMBee +from .ant_torch import CPMAntTorch +from .bee_torch import CPMBeeTorch diff --git a/cpm-live/cpm_live/models/ant.py b/cpm-live/cpm_live/models/ant.py index dc246f6..8d2f11c 100644 --- a/cpm-live/cpm_live/models/ant.py +++ b/cpm-live/cpm_live/models/ant.py @@ -12,6 +12,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. # See the License for the specific language governing permissions and # limitations under the License. + from typing import ListOptionalTuple import torch from ..utils import Config @@ -37,6 +38,7 @@ def __init__( prompt_length: int = 32, segment_types: int = 32, mask_modules: Optional[List[Tuple[boolbool]]] = None, + **kwargs, ): super().__init__() @@ -77,12 +79,12 @@ def __init__(selfconfig: CPMAntConfig): mask_modules=config.mask_modules, ) - # self.prompt_embedding = Embedding( - # vocab_size=config.prompt_types * config.prompt_length, - # embedding_size=config.dim_model, - # dtype=config.dtype, - # init_std=0.02, - # ) + self.prompt_embedding = Embedding( + vocab_size=config.prompt_types * config.prompt_length, + embedding_size=config.dim_model, + dtype=config.dtype, + init_std=0.02, + ) self.segment_embedding = Embedding( vocab_size=config.segment_types, @@ -92,7 +94,7 @@ def __init__(selfconfig: CPMAntConfig): ) self.input_embedding = Embedding( - vocab_size=config.vocab_size + config.prompt_length * config.prompt_types, + vocab_size=config.vocab_size, embedding_size=config.dim_model, dtype=config.dtype, init_std=0.02, @@ -117,8 +119,49 @@ def forward( position: torch.Tensor # (batchseqlen) segment: torch.Tensor # (batchseqlen) span: torch.Tensor # (batchseqlen) + ): + + batch = input.size(0) + seqlen = input.size(1) + input_prompt = input[:: self.prompt_length].contiguous() + input_ids = input[:self.prompt_length :].contiguous() + + prompt_states = self.prompt_embedding(input_prompt) + hidden_states = self.input_embedding(input_ids) + segment_states = self.segment_embedding(segment) + hidden_states = torch.cat([prompt_stateshidden_states]1) + segment_states + + with torch.no_grad(): + device = input.device + directional_mask_2d = torch.arange(seqlendevice=device) <= torch.arange( + seqlendevice=device + ).view(-11) + attention_mask = context[:None:] | ( + context[::None].logical_not() & directional_mask_2d.view(1seqlenseqlen) + ) + attention_mask = attention_mask & (span[:None:] == span[::None]) + mask_1d = ( + torch.arange(seqlendevice=device)[None:].repeat(batch1) < length[:None] + ) + attention_mask = ( + mask_1d.view(batchseqlen1) & mask_1d.view(batch1seqlen) & attention_mask + ) + + position_bias = self.position_bias(positionpositionsegmentsegment) + hidden_states = self.encoder(hidden_statesattention_maskposition_bias) + + logits = self.input_embedding.projection(hidden_states) + return logitshidden_states + + def inference( + self, + input: torch.Tensor # (batchseqlen) + length: torch.Tensor # (batch) + context: torch.Tensor # (batchseqlen) + position: torch.Tensor # (batchseqlen) + segment: torch.Tensor # (batchseqlen) + span: torch.Tensor # (batchseqlen) past_key_values=None # num_layers * 2 * (batchnum_headsseqlendim_head) - use_cache=False, ): batch = input.size(0) @@ -126,11 +169,13 @@ def forward( if past_key_values is None: past_length = 0 past_key_values = tuple([None] * self.encoder.num_layers) - input_ids = input.contiguous() + input_prompt = input[:: self.prompt_length].contiguous() + input_ids = input[:self.prompt_length :].contiguous() + prompt_states = self.prompt_embedding(input_prompt) hidden_states = self.input_embedding(input_ids) segment_states = self.segment_embedding(segment) - hidden_states = hidden_states + segment_states + hidden_states = torch.cat([prompt_stateshidden_states]1) + segment_states else: past_length = past_key_values[0][0].size(-2) @@ -148,8 +193,10 @@ def forward( context[::None].logical_not() & directional_mask_2d.view(1seqlenseqlen) ) attention_mask = attention_mask & (span[:None:] == span[::None]) + # mask for left paddding mask_1d = ( - torch.arange(seqlendevice=device)[None:].repeat(batch1) < length[:None] + torch.tensor(list(range(seqlen))[::-1]device=device)[None:].repeat(batch1) + < length[:None] ) attention_mask = ( mask_1d.view(batchseqlen1) & mask_1d.view(batch1seqlen) & attention_mask @@ -157,19 +204,11 @@ def forward( position_bias = self.position_bias(positionpositionsegmentsegment) - if past_length > 0: - attention_mask = attention_mask[:past_length::] - position_bias = position_bias[::past_length::] + attention_mask = attention_mask[:past_length::] + position_bias = position_bias[::past_length::] - if use_cache: - hidden_statespresent_key_values = self.encoder( - hidden_statesattention_maskposition_biasuse_cachepast_key_values - ) - logits = self.input_embedding.projection(hidden_states) - return logitshidden_statespresent_key_values - else: - hidden_states = self.encoder( - hidden_statesattention_maskposition_biasuse_cachepast_key_values - ) - logits = self.input_embedding.projection(hidden_states) - return logitshidden_states + hidden_statespresent_key_values = self.encoder( + hidden_statesattention_maskposition_biasTruepast_key_values + ) + logits = self.input_embedding.projection(hidden_states) + return logitshidden_statespresent_key_values diff --git a/cpm-live/cpm_live/models/ant_torch.py b/cpm-live/cpm_live/models/ant_torch.py new file mode 100644 index 0000000..44a23bf --- /dev/null +++ b/cpm-live/cpm_live/models/ant_torch.py @@ -0,0 +1,169 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import torch +from ..native_layers import EncoderEmbeddingSegmentPositionEmbedding +from .ant import CPMAntConfig + + +class CPMAntTorch(torch.nn.Module): + def __init__(selfconfig: CPMAntConfig): + + super().__init__() + + self.encoder = Encoder( + num_layers=config.num_layers, + dim_model=config.dim_model, + dim_ff=config.dim_ff, + num_heads=config.num_heads, + dim_head=config.dim_head, + dtype=config.dtype, + eps=config.eps, + dropout_p=config.dropout_p, + mask_modules=config.mask_modules, + ) + + self.prompt_embedding = Embedding( + vocab_size=config.prompt_types * config.prompt_length, + embedding_size=config.dim_model, + dtype=config.dtype, + init_std=0.02, + ) + + self.segment_embedding = Embedding( + vocab_size=config.segment_types, + embedding_size=config.dim_model, + dtype=config.dtype, + init_std=0.02, + ) + + self.input_embedding = Embedding( + vocab_size=config.vocab_size, + embedding_size=config.dim_model, + dtype=config.dtype, + init_std=0.02, + ) + + self.position_bias = SegmentPositionEmbedding( + num_heads=config.num_heads, + num_segments=config.segment_types, + num_buckets=config.position_bias_num_buckets, + max_distance=config.position_bias_max_distance, + bidirectional=True, + dtype=config.dtype, + ) + + self.prompt_length = config.prompt_length + + def forward( + self, + input: torch.Tensor # (batchseqlen) + length: torch.Tensor # (batch) + context: torch.Tensor # (batchseqlen) + position: torch.Tensor # (batchseqlen) + segment: torch.Tensor # (batchseqlen) + span: torch.Tensor # (batchseqlen) + ): + + batch = input.size(0) + seqlen = input.size(1) + input_prompt = input[:: self.prompt_length].contiguous() + input_ids = input[:self.prompt_length :].contiguous() + + prompt_states = self.prompt_embedding(input_prompt) + hidden_states = self.input_embedding(input_ids) + segment_states = self.segment_embedding(segment) + hidden_states = torch.cat([prompt_stateshidden_states]1) + segment_states + + with torch.no_grad(): + device = input.device + directional_mask_2d = torch.arange(seqlendevice=device) <= torch.arange( + seqlendevice=device + ).view(-11) + attention_mask = context[:None:] | ( + context[::None].logical_not() & directional_mask_2d.view(1seqlenseqlen) + ) + attention_mask = attention_mask & (span[:None:] == span[::None]) + mask_1d = ( + torch.arange(seqlendevice=device)[None:].repeat(batch1) < length[:None] + ) + attention_mask = ( + mask_1d.view(batchseqlen1) & mask_1d.view(batch1seqlen) & attention_mask + ) + + position_bias = self.position_bias(positionpositionsegmentsegment) + hidden_states = self.encoder(hidden_statesattention_maskposition_bias) + + logits = self.input_embedding.projection(hidden_states) + return logitshidden_states + + def inference( + self, + input: torch.Tensor # (batchseqlen) + length: torch.Tensor # (batch) + context: torch.Tensor # (batchseqlen) + position: torch.Tensor # (batchseqlen) + segment: torch.Tensor # (batchseqlen) + span: torch.Tensor # (batchseqlen) + past_key_values=None # num_layers * 2 * (batchnum_headsseqlendim_head) + ): + + batch = input.size(0) + + if past_key_values is None: + past_length = 0 + past_key_values = tuple([None] * self.encoder.num_layers) + input_prompt = input[:: self.prompt_length].contiguous() + input_ids = input[:self.prompt_length :].contiguous() + + prompt_states = self.prompt_embedding(input_prompt) + hidden_states = self.input_embedding(input_ids) + segment_states = self.segment_embedding(segment) + hidden_states = torch.cat([prompt_stateshidden_states]1) + segment_states + + else: + past_length = past_key_values[0][0].size(-2) + segment_states = self.segment_embedding(segment) + hidden_states = self.input_embedding(input) + segment_states[:-1::] + + seqlen = past_length + input.size(1) + + with torch.no_grad(): + device = input.device + directional_mask_2d = torch.arange(seqlendevice=device) <= torch.arange( + seqlendevice=device + ).view(-11) + attention_mask = context[:None:] | ( + context[::None].logical_not() & directional_mask_2d.view(1seqlenseqlen) + ) + attention_mask = attention_mask & (span[:None:] == span[::None]) + # mask for left paddding + mask_1d = ( + torch.tensor(list(range(seqlen))[::-1]device=device)[None:].repeat(batch1) + < length[:None] + ) + attention_mask = ( + mask_1d.view(batchseqlen1) & mask_1d.view(batch1seqlen) & attention_mask + ) + + position_bias = self.position_bias(positionpositionsegmentsegment) + + attention_mask = attention_mask[:past_length::] + position_bias = position_bias[::past_length::] + + hidden_statespresent_key_values = self.encoder( + hidden_statesattention_maskposition_biasTruepast_key_values + ) + logits = self.input_embedding.projection(hidden_states) + return logitshidden_statespresent_key_values diff --git a/cpm-live/cpm_live/models/bee.py b/cpm-live/cpm_live/models/bee.py new file mode 100644 index 0000000..098ecc6 --- /dev/null +++ b/cpm-live/cpm_live/models/bee.py @@ -0,0 +1,291 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import ListOptionalTuple +from typing_extensions import TypedDict +import torch + +from ..utils import Config +from ..layers import EncoderEmbeddingExtBucketPositionBias +import bmtrain as bmt +from ..utils.gradient_shrink import gradient_shrink + + +class CPMBeeInferenceState(TypedDict): + buffer_position: torch.Tensor + buffer_context: torch.Tensor + buffer_sample_ids: torch.Tensor + buffer_num_segments: torch.Tensor + buffer_segments: torch.Tensor + buffer: List[Tuple[torch.Tensortorch.Tensor]] + + +class CPMBeeConfig(Config): + def __init__( + self, + vocab_size=30720, + dim_model=4096, + num_heads=64, + dim_head=64, + dim_ff=10240, + num_layers=32, + dropout_p=0.0, + position_bias_num_buckets=256, + position_bias_num_segment_buckets=256, + position_bias_max_distance=2048, + eps=1e-6, + half: bool = True, + mask_modules: Optional[List[Tuple[boolbool]]] = None, + ): + + super().__init__() + self.dim_model = dim_model + self.num_heads = num_heads + self.dim_head = dim_head + self.dim_ff = dim_ff + self.num_layers = num_layers + self.position_bias_num_buckets = position_bias_num_buckets + self.position_bias_num_segment_buckets = position_bias_num_segment_buckets + self.position_bias_max_distance = position_bias_max_distance + self.dropout_p = dropout_p + self.eps = eps + if half: + self.dtype = torch.half + else: + self.dtype = torch.float + self.vocab_size = vocab_size + self.mask_modules = mask_modules + + +class CPMBee(bmt.DistributedModule): + def __init__(selfconfig: CPMBeeConfig): + + super().__init__() + + self.encoder = Encoder( + num_layers=config.num_layers, + dim_model=config.dim_model, + dim_ff=config.dim_ff, + num_heads=config.num_heads, + dim_head=config.dim_head, + dtype=config.dtype, + eps=config.eps, + dropout_p=config.dropout_p, + mask_modules=config.mask_modules, + ) + + self.input_embedding = EmbeddingExt( + vocab_size=config.vocab_size, + embedding_size=config.dim_model, + dtype=config.dtype, + init_std=0.02, + ) + + self.position_bias = BucketPositionBias( + num_heads=config.num_heads, + num_buckets=config.position_bias_num_buckets, + num_segment_bucket=config.position_bias_num_segment_buckets, + max_distance=config.position_bias_max_distance, + dtype=config.dtype, + ) + + def forward( + self, + input: torch.Tensor # (batchseqlen) int32 + input_sub: torch.Tensor # (batchseqlen) int32 + length: torch.Tensor # (batch) int32 + context: torch.Tensor # (batchseqlen) bool + sample_ids: torch.Tensor # (batchseq_len) int32 + num_segments: torch.Tensor # (batchseq_len) int32 + segment: torch.Tensor # (batchseqlen) int32 + segment_rel_offset: torch.Tensor # (batchseq_len) int32 + segment_rel: torch.Tensor # (batchnum_segment_bucket) int32 + span: torch.Tensor # (batchseqlen) int32 + ext_table_ids: torch.Tensor # (ext_table_size) int32 + ext_table_sub: torch.Tensor # (ext_table_size) int32 + ): + batch = input.size(0) + seqlen = input.size(1) + # processing masks and position bias bucket + with torch.no_grad(): + device = input.device + + # calc segment bucket + segment_rel_2d = torch.masked_fill( + segment[::None] * num_segments[::None] + + segment[:None:] + + segment_rel_offset[::None], + ~( + (sample_ids[::None] == sample_ids[:None:]) + & (span[:None:] == span[::None]) + ) # not in the same span or sample + 0 # avoid torch.gather overflow + ).view(batchseqlen * seqlen) + + segment_bucket = torch.gather( + input=segment_rel, + dim=1, + index=segment_rel_2d.long(), + ).view(batchseqlenseqlen) + + segment_bucket.masked_fill_( + ~( + (sample_ids[::None] == sample_ids[:None:]) + & (span[:None:] == span[::None]) + ) # not in the same span or sample + 1 # bucket is used for in-context samples + ) + + # directional mask + directional_mask_2d = torch.arange(seqlendevice=device) <= torch.arange( + seqlendevice=device + ).view(-11) + # sample mask + sample_mask_2d = (sample_ids[::None] == 0) | ( + sample_ids[::None] == sample_ids[:None:] + ) + # context mask + attention_mask = context[:None:] | ( + context[::None].logical_not() & directional_mask_2d.view(1seqlenseqlen) + ) + # span mask + attention_mask = ( + attention_mask & sample_mask_2d & (span[:None:] == span[::None]) + ) + # length mask + mask_1d = ( + torch.arange(seqlendevice=device)[None:].repeat(batch1) < length[:None] + ) + attention_mask = ( + mask_1d.view(batchseqlen1) & mask_1d.view(batch1seqlen) & attention_mask + ) + position = torch.arange(seqlendevice=device).expand(batchseqlen) + + hidden_states = self.input_embedding(inputinput_sub) + position_bias = self.position_bias(positionpositionsegment_bucket) + + hidden_states = self.encoder(hidden_statesattention_maskposition_bias) + + ext_table = self.input_embedding(ext_table_idsext_table_sub) + + logits = self.input_embedding.projection(hidden_statesext_table) + return logitshidden_states + + def inference( + self, + input: torch.Tensor # (batchlen_q) int32 + input_sub: torch.Tensor # (batchlen_q) int32 + position: torch.Tensor # (batchlen_q) int32 + context: torch.Tensor # (batchlen_q) bool + sample_ids: torch.Tensor # (batchlen_q) int32 + num_segments: torch.Tensor # (batchlen_q) int32 + segment: torch.Tensor # (batchlen_q) int32 + segment_rel_offset: torch.Tensor # (batchlen_q) int32 + segment_rel: torch.Tensor # (batchnum_segment_bucket) int32 + ext_table_ids: torch.Tensor # (ext_table_size) int32 + ext_table_sub: torch.Tensor # (ext_table_size) int32 + past_key_values: Optional[CPMBeeInferenceState] = None, + ) -> Tuple[torch.Tensortorch.TensorCPMBeeInferenceState]: + with torch.no_grad(): + if past_key_values is None: + present_position = position + present_context = context + present_sample_ids = sample_ids + present_num_segments = num_segments + present_segments = segment + present_buffer = None + else: + present_position = torch.cat([past_key_values["buffer_position"]position]dim=-1) + present_context = torch.cat([past_key_values["buffer_context"]context]dim=-1) + present_sample_ids = torch.cat( + [past_key_values["buffer_sample_ids"]sample_ids]dim=-1 + ) + present_num_segments = torch.cat( + [past_key_values["buffer_num_segments"]num_segments]dim=-1 + ) + present_segments = torch.cat([past_key_values["buffer_segments"]segment]dim=-1) + present_buffer = past_key_values["buffer"] + + batch = input.size(0) + len_q = input.size(1) + len_buffer = present_position.size(1) + + segment_rel_2d = torch.masked_fill( + segment[::None] * num_segments[::None] + + present_segments[:None:] + + segment_rel_offset[::None], + ~( + (sample_ids[::None] == present_sample_ids[:None:]) + ) # not in the same sample + 0 # avoid torch.gather overflow + ).view(batchlen_q * len_buffer) + + segment_bucket = torch.gather( + input=segment_rel, + dim=1, + index=segment_rel_2d.long(), + ).view(batchlen_qlen_buffer) + + segment_bucket.masked_fill_( + ~( + (sample_ids[::None] == present_sample_ids[:None:]) + ) # not in the same span or sample + 1 # bucket is used for in-context samples + ) + + # directional mask + directional_mask_2d = present_position[:None:] <= position[::None] + # sample mask + sample_mask_2d = (sample_ids[::None] == 0) | ( + sample_ids[::None] == present_sample_ids[:None:] + ) + # context mask + attention_mask = present_context[:None:] | ( + context[::None].logical_not() + & directional_mask_2d.view(batchlen_qlen_buffer) + ) + # span mask + attention_mask = attention_mask & sample_mask_2d + # length mask + mask_1d = present_num_segments != 0 + attention_mask = mask_1d.view(batch1len_buffer) & attention_mask + + hidden_states = gradient_shrink(self.input_embedding(inputinput_sub)) + + position_bias = gradient_shrink( + self.position_bias(positionpresent_positionsegment_bucket) + ) + hidden_statespresent_key_values = self.encoder( + hidden_states, + attention_mask, + position_bias, + True, + present_buffer, + ) + ext_table = gradient_shrink(self.input_embedding(ext_table_idsext_table_sub)) + logits = self.input_embedding.projection(hidden_statesext_table) + + return ( + logits, + hidden_states, + { + "buffer_position": present_position, + "buffer_context": present_context, + "buffer_sample_ids": present_sample_ids, + "buffer_num_segments": present_num_segments, + "buffer_segments": present_segments, + "buffer": present_key_values, + }, + ) diff --git a/cpm-live/cpm_live/models/bee_torch.py b/cpm-live/cpm_live/models/bee_torch.py new file mode 100644 index 0000000..a2e1489 --- /dev/null +++ b/cpm-live/cpm_live/models/bee_torch.py @@ -0,0 +1,240 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import OptionalTuple +import torch + +from ..native_layers import EncoderEmbeddingExtBucketPositionBias +from .bee import CPMBeeConfigCPMBeeInferenceState + + +class CPMBeeTorch(torch.nn.Module): + def __init__(selfconfig: CPMBeeConfig): + + super().__init__() + + self.encoder = Encoder( + num_layers=config.num_layers, + dim_model=config.dim_model, + dim_ff=config.dim_ff, + num_heads=config.num_heads, + dim_head=config.dim_head, + dtype=config.dtype, + eps=config.eps, + dropout_p=config.dropout_p, + mask_modules=config.mask_modules, + ) + + self.input_embedding = EmbeddingExt( + vocab_size=config.vocab_size, + embedding_size=config.dim_model, + dtype=config.dtype, + init_std=0.02, + ) + + self.position_bias = BucketPositionBias( + num_heads=config.num_heads, + num_buckets=config.position_bias_num_buckets, + num_segment_bucket=config.position_bias_num_segment_buckets, + max_distance=config.position_bias_max_distance, + dtype=config.dtype, + ) + + def forward( + self, + input: torch.Tensor # (batchseqlen) int32 + input_sub: torch.Tensor # (batchseqlen) int32 + length: torch.Tensor # (batch) int32 + context: torch.Tensor # (batchseqlen) bool + sample_ids: torch.Tensor # (batchseq_len) int32 + num_segments: torch.Tensor # (batchseq_len) int32 + segment: torch.Tensor # (batchseqlen) int32 + segment_rel_offset: torch.Tensor # (batchseq_len) int32 + segment_rel: torch.Tensor # (batchnum_segment_bucket) int32 + span: torch.Tensor # (batchseqlen) int32 + ext_table_ids: torch.Tensor # (ext_table_size) int32 + ext_table_sub: torch.Tensor # (ext_table_size) int32 + ): + batch = input.size(0) + seqlen = input.size(1) + # processing masks and position bias bucket + with torch.no_grad(): + device = input.device + + # calc segment bucket + segment_rel_2d = torch.masked_fill( + segment[::None] * num_segments[::None] + + segment[:None:] + + segment_rel_offset[::None], + ~( + (sample_ids[::None] == sample_ids[:None:]) + & (span[:None:] == span[::None]) + ) # not in the same span or sample + 0 # avoid torch.gather overflow + ).view(batchseqlen * seqlen) + + segment_bucket = torch.gather( + input=segment_rel, + dim=1, + index=segment_rel_2d.long(), + ).view(batchseqlenseqlen) + + segment_bucket.masked_fill_( + ~( + (sample_ids[::None] == sample_ids[:None:]) + & (span[:None:] == span[::None]) + ) # not in the same span or sample + 1 # bucket is used for in-context samples + ) + + # directional mask + directional_mask_2d = torch.arange(seqlendevice=device) <= torch.arange( + seqlendevice=device + ).view(-11) + # sample mask + sample_mask_2d = (sample_ids[::None] == 0) | ( + sample_ids[::None] == sample_ids[:None:] + ) + # context mask + attention_mask = context[:None:] | ( + context[::None].logical_not() & directional_mask_2d.view(1seqlenseqlen) + ) + # span mask + attention_mask = ( + attention_mask & sample_mask_2d & (span[:None:] == span[::None]) + ) + # length mask + mask_1d = ( + torch.arange(seqlendevice=device)[None:].repeat(batch1) < length[:None] + ) + attention_mask = ( + mask_1d.view(batchseqlen1) & mask_1d.view(batch1seqlen) & attention_mask + ) + position = torch.arange(seqlendevice=device).expand(batchseqlen) + + hidden_states = self.input_embedding(inputinput_sub) + position_bias = self.position_bias(positionpositionsegment_bucket) + + hidden_states = self.encoder(hidden_statesattention_maskposition_bias) + + ext_table = self.input_embedding(ext_table_idsext_table_sub) + + logits = self.input_embedding.projection(hidden_statesext_table) + return logitshidden_states + + def inference( + self, + input: torch.Tensor # (batchlen_q) int32 + input_sub: torch.Tensor # (batchlen_q) int32 + position: torch.Tensor # (batchlen_q) int32 + context: torch.Tensor # (batchlen_q) bool + sample_ids: torch.Tensor # (batchlen_q) int32 + num_segments: torch.Tensor # (batchlen_q) int32 + segment: torch.Tensor # (batchlen_q) int32 + segment_rel_offset: torch.Tensor # (batchlen_q) int32 + segment_rel: torch.Tensor # (batchnum_segment_bucket) int32 + ext_table_ids: torch.Tensor # (ext_table_size) int32 + ext_table_sub: torch.Tensor # (ext_table_size) int32 + past_key_values: Optional[CPMBeeInferenceState] = None, + ) -> Tuple[torch.Tensortorch.TensorCPMBeeInferenceState]: + with torch.no_grad(): + if past_key_values is None: + present_position = position + present_context = context + present_sample_ids = sample_ids + present_num_segments = num_segments + present_segments = segment + present_buffer = None + else: + present_position = torch.cat([past_key_values["buffer_position"]position]dim=-1) + present_context = torch.cat([past_key_values["buffer_context"]context]dim=-1) + present_sample_ids = torch.cat( + [past_key_values["buffer_sample_ids"]sample_ids]dim=-1 + ) + present_num_segments = torch.cat( + [past_key_values["buffer_num_segments"]num_segments]dim=-1 + ) + present_segments = torch.cat([past_key_values["buffer_segments"]segment]dim=-1) + present_buffer = past_key_values["buffer"] + + batch = input.size(0) + len_q = input.size(1) + len_buffer = present_position.size(1) + + segment_rel_2d = torch.masked_fill( + segment[::None] * num_segments[::None] + + present_segments[:None:] + + segment_rel_offset[::None], + ~( + (sample_ids[::None] == present_sample_ids[:None:]) + ) # not in the same sample + 0 # avoid torch.gather overflow + ).view(batchlen_q * len_buffer) + + segment_bucket = torch.gather( + input=segment_rel, + dim=1, + index=segment_rel_2d.long(), + ).view(batchlen_qlen_buffer) + + segment_bucket.masked_fill_( + ~( + (sample_ids[::None] == present_sample_ids[:None:]) + ) # not in the same span or sample + 1 # bucket is used for in-context samples + ) + + # directional mask + directional_mask_2d = present_position[:None:] <= position[::None] + # sample mask + sample_mask_2d = (sample_ids[::None] == 0) | ( + sample_ids[::None] == present_sample_ids[:None:] + ) + # context mask + attention_mask = present_context[:None:] | ( + context[::None].logical_not() + & directional_mask_2d.view(batchlen_qlen_buffer) + ) + # span mask + attention_mask = attention_mask & sample_mask_2d + # length mask + mask_1d = present_num_segments != 0 + attention_mask = mask_1d.view(batch1len_buffer) & attention_mask + + hidden_states = self.input_embedding(inputinput_sub) + + position_bias = self.position_bias(positionpresent_positionsegment_bucket) + hidden_statespresent_key_values = self.encoder( + hidden_states, + attention_mask, + position_bias, + True, + present_buffer, + ) + ext_table = self.input_embedding(ext_table_idsext_table_sub) + logits = self.input_embedding.projection(hidden_statesext_table) + + return ( + logits, + hidden_states, + { + "buffer_position": present_position, + "buffer_context": present_context, + "buffer_sample_ids": present_sample_ids, + "buffer_num_segments": present_num_segments, + "buffer_segments": present_segments, + "buffer": present_key_values, + }, + ) diff --git a/cpm-live/cpm_live/native_layers/__init__.py b/cpm-live/cpm_live/native_layers/__init__.py new file mode 100644 index 0000000..dbd8dcd --- /dev/null +++ b/cpm-live/cpm_live/native_layers/__init__.py @@ -0,0 +1,8 @@ +from .embedding import EmbeddingEmbeddingExt +from .position_embedding import SegmentPositionEmbeddingBucketPositionBiasRotaryEmbedding +from .linear import Linear +from .layernorm import LayerNorm +from .attention import Attention +from .feedforward import FeedForward +from .blocks import TransformerBlock +from .transformer import Encoder diff --git a/cpm-live/cpm_live/native_layers/attention.py b/cpm-live/cpm_live/native_layers/attention.py new file mode 100644 index 0000000..5b9f402 --- /dev/null +++ b/cpm-live/cpm_live/native_layers/attention.py @@ -0,0 +1,117 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import OptionalTuple +import torch +import math +from .linear import Linear + + +class Attention(torch.nn.Module): + def __init__( + self, + dim_model: int, + num_heads: int, + dim_head: int, + dtype: torch.dtype = torch.half, + dropout_p: Optional[float] = None, + ) -> None: + + super().__init__() + + self.dim_model = dim_model + self.num_heads = num_heads + self.dim_head = dim_head + + self.project_q = Linear(self.dim_modelself.num_heads * self.dim_headdtype=dtype) + self.project_k = Linear(self.dim_modelself.num_heads * self.dim_headdtype=dtype) + self.project_v = Linear(self.dim_modelself.num_heads * self.dim_headdtype=dtype) + + self.attention_out = Linear(self.num_heads * self.dim_headself.dim_modeldtype=dtype) + + self.softmax = torch.nn.Softmax(dim=-1) + + if dropout_p is not None: + self.dropout = torch.nn.Dropout(p=dropout_p) + else: + self.dropout = None + + def forward( + self, + hidden_q: torch.Tensor, + hidden_kv: torch.Tensor, + attention_mask: torch.BoolTensor, + position_bias: torch.Tensor, + use_cache: bool = False, + past_kv: Optional[Tuple[torch.Tensortorch.Tensor]] = None, + ): + """ + Args: + hidden_q (:obj:`torch.Tensor` of shape ``(batchlen_qdim_model)``): Indices of input sequence tokens. It will be embedded by model's internal embedding lookup matrix. + hidden_kv (:obj:`torch.Tensor` of shape ``(batchlen_kdim_model)``): Length of input sequence before padding. + attention_mask (:obj:`torch.Tensor` of shape ``(batchlen_qlen_k)``): Used to avoid performing attention on padding token indices. + position_bias(:obj:`torch.Tensor` of shape ``(num_headslen_qlen_k)`` or ``(1num_headslen_klen_q)``): Provide positional information about tensor `key_value` and `query`. + Return: + out (:obj:`torch.Tensor` of shape ``(batchlen_qdim_model)``): The attention output. + """ # noqa: E501 + + batch_size = hidden_q.size(0) + len_q = hidden_q.size(1) + len_k = hidden_kv.size(1) + + h_q = self.project_q(hidden_q) + h_k = self.project_k(hidden_kv) + h_v = self.project_v(hidden_kv) + + h_q = h_q.view(batch_sizelen_qself.num_headsself.dim_head).permute(0213) + h_k = h_k.view(batch_sizelen_kself.num_headsself.dim_head).permute(0213) + h_v = h_v.view(batch_sizelen_kself.num_headsself.dim_head).permute(0213) + + if past_kv is not None: + h_k = torch.cat([past_kv[0]h_k]dim=-2) + h_v = torch.cat([past_kv[1]h_v]dim=-2) + len_k = h_k.size(-2) + + # (bn_hlen_qd_h) @ (bn_hd_hlen_k) -> (bn_hlen_qlen_k) + score = torch.matmul(h_qh_k.transpose(-1-2)) / math.sqrt(self.dim_head) + score = score + position_bias + score = torch.masked_fill( + score, + attention_mask.view(batch_size1len_qlen_k) == False, + torch.scalar_tensor(float("-inf")device=score.devicedtype=score.dtype), + ) + + score = self.softmax(score) + + score = torch.masked_fill( + score, + attention_mask.view(batch_size1len_qlen_k) == False, + torch.scalar_tensor(0device=score.devicedtype=score.dtype), + ) + + if self.dropout is not None: + score = self.dropout(score) + + # (bn_hlen_qlen_k) @ (bn_hlen_kd_h) -> (bn_hlen_qd_h) + score = torch.matmul(scoreh_v) + + score = score.view(batch_sizeself.num_headslen_qself.dim_head).permute(0213) + score = score.contiguous().view(batch_sizelen_qself.num_heads * self.dim_head) + + score = self.attention_out(score) + if use_cache: + return score(h_kh_v) + else: + return score diff --git a/cpm-live/cpm_live/native_layers/blocks.py b/cpm-live/cpm_live/native_layers/blocks.py new file mode 100644 index 0000000..9c6efcb --- /dev/null +++ b/cpm-live/cpm_live/native_layers/blocks.py @@ -0,0 +1,248 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import OptionalTuple +import torch +from .layernorm import LayerNorm +from .attention import Attention +from .feedforward import FeedForward + + +class SelfAttentionBlock(torch.nn.Module): + """The whole cross-attention block. A sequence of operation. Consists of layernormself-attention and residual connection. + + Args: + dim_model (int): main dimension of modules in transformer blocks. + num_heads (int): num_heads used in :py:class:`model_center.layer.Attention`. + dim_head (int): dim_head used in :py:class:`model_center.layer.Attention`. + dtype (optional): Defaults to torch.half. + eps (floatoptional): eps used in :py:class:`model_center.layer.LayerNorm`. Defaults to 1e-5. + dropout_p (floatoptional): Defaults to 0. + """ # noqa: E501 + + def __init__( + self, + dim_model: int, + num_heads: int, + dim_head: int, + dtype=torch.half, + eps: float = 1e-6, + dropout_p: Optional[float] = None, + ): + + super().__init__() + + self.layernorm_before_attention = LayerNorm( + dim_model, + dtype=dtype, + eps=eps, + ) + + self.self_attention = Attention( + dim_model=dim_model, + num_heads=num_heads, + dim_head=dim_head, + dtype=dtype, + dropout_p=dropout_p, + ) + + if dropout_p: + self.dropout = torch.nn.Dropout(dropout_p) + else: + self.dropout = None + + def forward( + self, + hidden_states: torch.Tensor, + attention_mask: torch.Tensor, + position_bias: Optional[torch.Tensor] = None, + use_cache: bool = False, + past_key_value: Optional[Tuple[torch.Tensortorch.Tensor]] = None, + ): + """ + Args: + hidden_states (:obj:`torch.Tensor` of shape ``(batchseq_selfdim_model)``): Input of self-attention block. It can be the embedding of a batch of sequences. + attention_mask (:obj:`torch.Tensor` of shape ``(batchseq_selfseq_self)``): Avoid invalid areas to participate in the calculation. + position_bias (:obj:`torch.Tensor` of shape ``(num_headsseq_selfseq_self)``): Provide positional information to self-attention block. + + Return: + :obj:`torch.Tensor` of shape ``(batchseq_selfdim_model)``: The output of attention block. + + """ # noqa: E501 + x = self.layernorm_before_attention(hidden_states) + x = self.self_attention(xxattention_maskposition_biasuse_cachepast_key_value) + if use_cache: + xcurrent_key_value = x + else: + current_key_value = None + + if self.dropout is not None: + x = self.dropout(x) + hidden_states = (hidden_states + x) / 1.05 + + if use_cache: + return hidden_statescurrent_key_value + else: + return hidden_states + + +class FFNBlock(torch.nn.Module): + """The whole feed-forward block. A sequence of operation. Consists of layernormfeed-forward and residual connection. + + Args: + dim_model (int): main dimension of modules in transformer blocks. + dim_ff (int): dim_ff used in :py:class:`model_center.layer.FeedForward`. + dtype (optional): Defaults to torch.half. + eps (floatoptional): eps used in :py:class:`model_center.layer.LayerNorm`. Defaults to 1e-5. + dropout_p (floatoptional): Defaults to 0. + """ # noqa: E501 + + def __init__( + self, + dim_model: int, + dim_ff: int, + dtype=torch.half, + eps: float = 1e-6, + dropout_p: Optional[float] = 0, + ): + super().__init__() + + self.layernorm_before_ffn = LayerNorm( + dim_model, + dtype=dtype, + eps=eps, + ) + + self.ffn = FeedForward( + dim_model, + dim_ff, + dtype=dtype, + dropout_p=dropout_p, + ) + + if dropout_p: + self.dropout = torch.nn.Dropout(dropout_p) + else: + self.dropout = None + + def forward( + self, + hidden_states: torch.Tensor, + ): + """ + Args: + hidden_states (:obj:`torch.Tensor` of shape ``(batchseq_selfdim_model)``): Hidden states before feed forward layer. + + Return: + :obj:`torch.Tensor` of shape ``(batchseq_selfdim_model)``: The output of feed-forward block + + """ # noqa: E501 + x = self.layernorm_before_ffn(hidden_states) + x = self.ffn(x) + if self.dropout is not None: + x = self.dropout(x) + hidden_states = (hidden_states + x) / 1.05 + return hidden_states + + +class TransformerBlock(torch.nn.Module): + """The whole transformer block. A sequence of operation. Consists of self-attention block[cross-attention block] and feed-forward block. + + Args: + dim_model (int): main dimension of modules in transformer blocks. + dim_ff (int): dim_ff used in :py:class:`model_center.layer.FeedForward`. + num_heads (int): num_heads used in :py:class:`model_center.layer.Attention`. + dim_head (int): dim_head used in :py:class:`model_center.layer.Attention`. + dtype (optional): Defaults to torch.half. + eps (floatoptional): eps used in :py:class:`model_center.layer.LayerNorm`. Defaults to 1e-5. + dropout_p (floatoptional): Defaults to 0. + """ # noqa: E501 + + def __init__( + self, + dim_model: int, + dim_ff: int, + num_heads: int, + dim_head: int, + dtype=torch.half, + eps: float = 1e-6, + dropout_p: Optional[float] = None, + mask_att: bool = False, + mask_ffn: bool = False, + ): + super().__init__() + self.mask_att = mask_att + self.mask_ffn = mask_ffn + + if not self.mask_att: + self.self_att = SelfAttentionBlock( + dim_model=dim_model, + num_heads=num_heads, + dim_head=dim_head, + dtype=dtype, + eps=eps, + dropout_p=dropout_p, + ) + + if not self.mask_ffn: + self.ffn = FFNBlock( + dim_model=dim_model, + dim_ff=dim_ff, + dtype=dtype, + eps=eps, + dropout_p=dropout_p, + ) + + def forward( + self, + self_hidden_states: torch.Tensor, + self_attention_mask: torch.Tensor, + self_position_bias: Optional[torch.Tensor] = None, + use_cache: bool = False, + past_key_value: Optional[Tuple[torch.Tensortorch.Tensor]] = None, + ): + """ + Args: + self_hidden_states (:obj:`torch.Tensor` of shape ``(batchseq_selfdim_model)``): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. + self_attention_mask (:obj:`torch.Tensor` of shape ``(batchseq_selfseq_self)``): Avoid invalid areas to participate in the calculation of self-attention. + self_position_bias (:obj:`torch.Tensor` of shape ``(num_headsseq_selfseq_self)``): Provide positional information to self-attention block. + + Return: + :obj:`torch.Tensor` of shape ``(batchseq_selfdim_model)``: The output of transformer block. + + """ # noqa: E501 + # (batchdim_modelseq_self) + current_key_value = None + if not self.mask_att: + hidden_states = self.self_att( + self_hidden_states, + attention_mask=self_attention_mask, + position_bias=self_position_bias, + use_cache=use_cache, + past_key_value=past_key_value, + ) + if use_cache: + hidden_statescurrent_key_value = hidden_states + else: + hidden_states = self_hidden_states + + # (batchdim_modelseq_self) + if not self.mask_ffn: + hidden_states = self.ffn(hidden_states) + + if use_cache: + return hidden_statescurrent_key_value + else: + return hidden_states diff --git a/cpm-live/cpm_live/native_layers/embedding.py b/cpm-live/cpm_live/native_layers/embedding.py new file mode 100644 index 0000000..7f0fb64 --- /dev/null +++ b/cpm-live/cpm_live/native_layers/embedding.py @@ -0,0 +1,110 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +import math +import torch.nn.functional as F +from .position_embedding import RotaryEmbedding +from typing import Optional + + +class Embedding(torch.nn.Module): + def __init__( + self, + vocab_size: int, + embedding_size: int, + dtype: torch.dtype = torch.half, + init_mean: float = 0.0, + init_std: float = 1, + ): + + super().__init__() + + self.dim_model = embedding_size + self.weight = torch.nn.parameter.Parameter( + torch.empty(vocab_sizeembedding_sizedtype=dtype) + ) + + def forward(selfids: torch.Tensor): + """ + Args: + ids (:obj:`torch.Tensor` of shape ``(batch_sizeseq_len)``): Indices of input sequence tokens. + Return: + :obj:`torch.Tensor` of shape ``(batch_sizeseq_lenembedding_size)``: The embedding output. + """ # noqa: E501 + + embeds = F.embedding(idsself.weight) / math.sqrt(self.dim_model) + return embeds + + def projection(selfx: torch.Tensor): + """ + Projection based on embedding's weight. For exampleembedding map vocab_size to embed_sizethan projection map embed_size back to vocab_size. + Args: + x (:obj:`torch.Tensor` of shape ``(batchseq_lendim_model)``): Input of projection + Returns: + :obj:`torch.Tensor` of shape ``(batchseq_lenvocab_output_size)``: The projection output. + """ # noqa: E501 + logits = F.linear(x / math.sqrt(self.dim_model)self.weight) + return logits + + +class EmbeddingExt(torch.nn.Module): + def __init__( + self, + vocab_size: int, + embedding_size: int, + dtype: torch.dtype = torch.half, + init_mean: float = 0.0, + init_std: float = 1, + distance_scale: int = 16, + ): + + super().__init__() + + self.dim_model = embedding_size + self.rotary_emb = RotaryEmbedding( + dim=embedding_sizedistance_scale=distance_scaledtype=dtype + ) + + self.weight = torch.nn.parameter.Parameter( + torch.empty(vocab_sizeembedding_sizedtype=dtype), + ) + + def forward(selfids: torch.Tensorids_sub: torch.Tensor): + """ + Args: + ids (:obj:`torch.Tensor` of shape ``(batch_sizeseq_len)``): Indices of input sequence tokens. + ids (:obj:`torch.Tensor` of shape ``(batch_size)``): Subscript of input sequence tokens. + Return: + :obj:`torch.Tensor` of shape ``(batch_sizeseq_lenembedding_size)``: The embedding output. + """ # noqa: E501 + + embeds = F.embedding(idsself.weight) / math.sqrt(self.dim_model) + return self.rotary_emb(embedsids_sub) + + def projection(selfx: torch.Tensorext_table: Optional[torch.Tensor] = None): + """ + Projection based on embedding's weight. For exampleembedding map vocab_size to embed_sizethan projection map embed_size back to vocab_size. + Args: + x (:obj:`torch.Tensor` of shape ``(batchseq_lendim_model)``): Input of projection + ext_table (:obj:`torch.Tensor` of shape ``(ext_table_sizedim_model)``): Ext vocab table. + Returns: + :obj:`torch.Tensor` of shape ``(batchseq_lenvocab_size + ext_table_size)``: The projection output. + """ # noqa: E501 + logits = F.linear(x / math.sqrt(self.dim_model)self.weight) + if ext_table is not None: + logits_ext = F.linear(xext_table) + logits = torch.cat([logitslogits_ext]dim=-1) + return logits diff --git a/cpm-live/cpm_live/native_layers/feedforward.py b/cpm-live/cpm_live/native_layers/feedforward.py new file mode 100644 index 0000000..b624592 --- /dev/null +++ b/cpm-live/cpm_live/native_layers/feedforward.py @@ -0,0 +1,120 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Optional +import torch +from .linear import Linear + + +class DenseGatedACT(torch.nn.Module): + def __init__( + self, + dim_in: int, + dim_ff: int, + dtype=torch.half, + ): + super().__init__() + + self.w_0 = Linear( + dim_in=dim_in, + dim_out=dim_ff, + dtype=dtype, + scale_before=False, + ) + + self.w_1 = Linear( + dim_in=dim_in, + dim_out=dim_ff, + dtype=dtype, + scale_before=False, + ) + self.act = torch.nn.GELU() + + def forward(selfx: torch.Tensor): + """Transform an input tensor from one feature space to another via a nonlinear operation + + Args: + x (:obj:`torch.Tensor` of shape ``(batchseq_lendim_in)``): Tensor that will be subject to nonlinear operations. + + Return: + out (:obj:`torch.Tensor` of shape ``(batchseq_lendim_ff)``) + + """ # noqa: E501 + gate_score = self.act(self.w_0(x)) + x = self.w_1(x) + + x = gate_score * x + return x + + +class FeedForward(torch.nn.Module): + r"""FeedForward module + + Args: + dim_in (int): input dimension. + dim_ff (int): middle dimension. + dim_out (intoptional): output dimension. Defaults to Nonewhich means dim_in = dim_out. + dtype (optional): Defaults to torch.half. + init_mean (floatoptional): mean of :math:`\mathbf{W}\sim\mathcal{N}(\text{mean}\text{std}^2)` for fully-connected module used in feed-forward layer. Defaults to 0. + init_std (floatoptional): std of :math:`\mathbf{W}\sim\mathcal{N}(\text{mean}\text{std}^2)` for fully-connected module used in feed-forward layer. Defaults to 0.02. + bias (booloptional): whether to use bias term in fully-connected layers used in feed-forward module. Defaults to False. + activate_fn (stroptional): Defaults to `gated_gelu`. + dropout_p (intoptional): Defaults to 0. + """ # noqa: E501 + + def __init__( + self, + dim_model: int, + dim_ff: int, + dtype=torch.half, + dropout_p: Optional[float] = None, + ): + + super().__init__() + + self.w_in = DenseGatedACT( + dim_in=dim_model, + dim_ff=dim_ff, + dtype=dtype, + ) + + if dropout_p is not None: + self.dropout = torch.nn.Dropout(dropout_p) + else: + self.dropout = None + + self.w_out = Linear( + dim_in=dim_ff, + dim_out=dim_model, + dtype=dtype, + scale_before=False, + ) + + def forward(selfx: torch.Tensor): + """ + Args: + x (:obj:`torch.Tensor` of shape ``(batchseq_lendim_in)``): The input of feed-forward module. + + Return: + :obj:`torch.Tensor` of shape ``(batchseq_lendim_out)``: The output of feed-forward module. + """ # noqa: E501 + x = self.w_in(x) + + if self.dropout is not None: + x = self.dropout(x) + + x = self.w_out(x) + + return x diff --git a/cpm-live/cpm_live/native_layers/layernorm.py b/cpm-live/cpm_live/native_layers/layernorm.py new file mode 100644 index 0000000..e8f19e9 --- /dev/null +++ b/cpm-live/cpm_live/native_layers/layernorm.py @@ -0,0 +1,37 @@ +import torch + + [email protected] # type: ignore +def rms_layernorm(hidden: torch.Tensorweight: torch.Tensoreps: float): + old_dtype = hidden.dtype + variance = hidden.to(torch.float32).pow(2).mean(dim=-1keepdim=True) + hidden = (hidden * torch.rsqrt(variance + eps)).to(old_dtype) + return hidden * weight + + +class LayerNorm(torch.nn.Module): + """RMS LayerNorm""" + + def __init__( + self, + dim_norm: int, + dtype: torch.dtype = torch.half, + eps: float = 1e-6, + init_var: float = 1.0, + ): + + super().__init__() + + self.eps = eps + self.dim_norm = dim_norm + self.weight = torch.nn.parameter.Parameter(torch.full((dim_norm,)init_vardtype=dtype)) + + def forward(selfx: torch.Tensor): + """ + Args: + x (:obj:`torch.Tensor` of shape ``(batch_sizeseq_lendim_norm)``): Input tensor that need to be normalized. + Return: + :obj:`torch.Tensor` of shape ``(batch_sizeseq_lendim_norm)``: The layernorm output. + """ # noqa: E501 + assert x.size(-1) == self.dim_norm + return rms_layernorm(xself.weightself.eps) diff --git a/cpm-live/cpm_live/native_layers/linear.py b/cpm-live/cpm_live/native_layers/linear.py new file mode 100644 index 0000000..fe904ae --- /dev/null +++ b/cpm-live/cpm_live/native_layers/linear.py @@ -0,0 +1,51 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +import math +import torch.nn.functional as F + + +class Linear(torch.nn.Module): + def __init__( + self, + dim_in: int, + dim_out: int, + dtype: torch.dtype = torch.half, + init_mean: float = 0.0, + init_std: float = 1, + scale_before: bool = False, + ): + super().__init__() + self.dim_in = self.in_features = dim_in + self.dim_out = self.out_features = dim_out + self.scale_before = scale_before + + self.weight = torch.nn.parameter.Parameter(torch.empty((dim_outdim_in)dtype=dtype)) + + def forward(selfx: torch.Tensor): + """ + Args: + x (:obj:`torch.Tensor` of shape ``(batchseq_lendim_in)``): The input of linear layer + Returns: + :obj:`torch.Tensor` of shape ``(batchseq_lendim_out)``: The output of the linear transform y. + """ # noqa: E501 + if self.scale_before: + x = x / math.sqrt(self.dim_in) + x = F.linear(xself.weight) + else: + x = F.linear(xself.weight) + x = x / math.sqrt(self.dim_in) + return x diff --git a/cpm-live/cpm_live/native_layers/position_embedding.py b/cpm-live/cpm_live/native_layers/position_embedding.py new file mode 100644 index 0000000..3778f3e --- /dev/null +++ b/cpm-live/cpm_live/native_layers/position_embedding.py @@ -0,0 +1,247 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import math +from typing import Union +import torch +import torch.nn.functional as F + + +class SegmentPositionEmbedding(torch.nn.Module): + def __init__( + self, + num_heads, + num_segments=1, + num_buckets=32, + max_distance=128, + bidirectional=False, + dtype=torch.half, + init_mean: float = 0.0, + init_std: float = 1, + ): + + super().__init__() + + self.num_heads = num_heads + self.num_buckets = num_buckets + self.max_distance = max_distance + self.bidirectional = bidirectional + self.num_segments = num_segments + + self.relative_attention_bias = torch.nn.parameter.Parameter( + torch.empty(num_segments * num_segments + num_bucketsnum_headsdtype=dtype) + ) + + def forward( + self, + key_pos: torch.Tensor, + query_pos: torch.Tensor, + key_segment: torch.Tensor, + query_segment: torch.Tensor, + ): + with torch.no_grad(): + + batch = key_pos.size(0) + keylen = key_pos.size(1) + querylen = query_pos.size(1) + + assert key_pos.size(0) == query_pos.size(0) + assert keylen == key_segment.size(1) and querylen == query_segment.size(1) + + key_pos = key_pos.view(batch-1keylen) + query_pos = query_pos.view(batchquerylen-1) + key_segment = key_segment.view(batch-1keylen) + query_segment = query_segment.view(batchquerylen-1) + + relative_position_bucket = self._segment_relative_position_bucket( + query_segmentkey_segment + ) + relative_position_bucket = relative_position_bucket + self.num_buckets # 与相对位置编码区间不重叠 + + # b*q*k + absolute_position_bucket = self._position_bucket( + torch.arange(keylendtype=torch.int32device=relative_position_bucket.device)[ + None: + ] + - torch.arange(querylendtype=torch.int32device=relative_position_bucket.device)[ + :None + ], + bidirectional=self.bidirectional, + num_buckets=self.num_buckets, + max_distance=self.max_distance, + ) + relative_position_bucket = torch.where( + (key_segment == query_segment), + absolute_position_bucket[None::], + relative_position_bucket, + ) + # (batchlen_qlen_k) + + # (batchlen_qlen_knum_heads) + embeds = F.embedding(relative_position_bucketself.relative_attention_bias) + # (batchnum_headslen_qlen_k) + embeds = embeds.permute(0312).contiguous() + return embeds + + def _segment_relative_position_bucket(selfquery_segmentkey_segment): + return query_segment * self.num_segments + key_segment + + def _position_bucket( + selfrelative_positionbidirectional=Truenum_buckets=32max_distance=128 + ): + relative_buckets = 0 + if bidirectional: + num_buckets //= 2 + relative_buckets = (relative_position > 0).to(torch.int32) * num_buckets + relative_position = torch.abs(relative_position) + else: + relative_position = -torch.min(relative_positiontorch.zeros_like(relative_position)) + max_exact = num_buckets // 2 + is_small = relative_position < max_exact + relative_postion_if_large = max_exact + ( + torch.log(relative_position.float() / max_exact) + / math.log(max_distance / max_exact) + * (num_buckets - max_exact) + ).to(torch.int32) + relative_postion_if_large = torch.min( + relative_postion_if_large, + torch.full_like(relative_postion_if_largenum_buckets - 1), + ) + relative_buckets += torch.where( + is_smallrelative_position.to(torch.int32)relative_postion_if_large + ) + return relative_buckets + + +class BucketPositionBias(torch.nn.Module): + def __init__( + self, + num_heads: int, + num_buckets: int = 32, + num_segment_bucket: int = 32, + max_distance: int = 128, + dtype: torch.dtype = torch.half, + init_mean: float = 0.0, + init_std: float = 1, + ) -> None: + super().__init__() + + self.num_heads = num_heads + self.num_buckets = num_buckets + self.num_segment_bucket = num_segment_bucket + self.max_distance = max_distance + + self.relative_attention_bias = torch.nn.parameter.Parameter( + torch.empty(num_buckets + num_segment_bucketnum_headsdtype=dtype) + ) + + def forward( + self, + query_pos: torch.Tensor # (batchlen_q) + key_pos: torch.Tensor # (batchlen_k) + rel_buckets: torch.Tensor # (batchlen_qlen_k) + ): + with torch.no_grad(): + + batch = key_pos.size(0) + keylen = key_pos.size(1) + querylen = query_pos.size(1) + + assert key_pos.size(0) == query_pos.size(0) + assert ( + rel_buckets.size(0) == batch + and rel_buckets.size(1) == querylen + and rel_buckets.size(2) == keylen + ) + + relative_position_bucket = rel_buckets - 1 + self.num_buckets # 与相对位置编码区间不重叠 + + # b*q*k + inner_segment_bucket = self._position_bucket( + key_pos[...None:] - query_pos[...:None], + num_buckets=self.num_buckets, + max_distance=self.max_distance, + ) + relative_position_bucket = torch.where( + rel_buckets == 0, + inner_segment_bucket, + relative_position_bucket, + ) + # (batchlen_qlen_k) + + # (batchlen_qlen_knum_heads) + embeds = F.embedding(relative_position_bucketself.relative_attention_bias) + # (batchnum_headslen_qlen_k) + embeds = embeds.permute(0312).contiguous() + return embeds + + def _position_bucket(selfrelative_positionnum_buckets=32max_distance=128): + relative_buckets = 0 + num_buckets //= 2 + relative_buckets = (relative_position > 0).to(torch.int32) * num_buckets + relative_position = torch.abs(relative_position) + + max_exact = num_buckets // 2 + is_small = relative_position < max_exact + relative_postion_if_large = max_exact + ( + torch.log(relative_position.float() / max_exact) + / math.log(max_distance / max_exact) + * (num_buckets - max_exact) + ).to(torch.int32) + relative_postion_if_large = torch.min( + relative_postion_if_large, + torch.full_like(relative_postion_if_largenum_buckets - 1), + ) + relative_buckets += torch.where( + is_smallrelative_position.to(torch.int32)relative_postion_if_large + ) + return relative_buckets + + +class RotaryEmbedding(torch.nn.Module): + def __init__( + self, + dim, + base=10000, + distance_scale: Union[intfloat] = 1, + dtype: torch.dtype = torch.half, + ): + super().__init__() + inv_freq = 1.0 / ( + base ** (torch.arange(0dim2device="cuda"dtype=torch.float32) / dim) + ) + inv_freq = inv_freq.to(dtype) + self.distance_scale = distance_scale + self.dtype = dtype + self.inv_freq = inv_freq + + def forward(selfx: torch.Tensorx_pos: torch.Tensor): + """ + Args: + x (:obj:`torch.Tensor` of shape ``(...dim)``): Inputs. + x_pos (:obj:`torch.Tensor` of shape ``(...)``): Positions of inputs. + """ + x_pos = x_pos * self.distance_scale + freqs = x_pos[...None].to(self.dtype) * self.inv_freq[None:] # (...dim/2) + + # the same implementation as sat + emb = torch.cat((freqsfreqs)dim=-1) # (...dim) + emb_cos = emb.cos() # (...dim) + emb_sin = emb.sin() # (...dim) + + rotate_x = torch.cat( + [-x[...x.size(-1) // 2 :]x[...: x.size(-1) // 2]]dim=-1 + ) # (...dim) + + return x * emb_cos + rotate_x * emb_sin diff --git a/cpm-live/cpm_live/native_layers/transformer.py b/cpm-live/cpm_live/native_layers/transformer.py new file mode 100644 index 0000000..b21445c --- /dev/null +++ b/cpm-live/cpm_live/native_layers/transformer.py @@ -0,0 +1,125 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from typing import OptionalListTuple + +from .blocks import TransformerBlock +from .layernorm import LayerNorm + + +class Encoder(torch.nn.Module): + """Layers of encoder transformer blocks plus an final layernorm. + + Args: + num_layers (int): number of layers. + dim_model (int): main dimension of modules in transformer blocks. + dim_ff (int): dim_ff used in :py:class:`model_center.layer.FeedForward`. + num_heads (int): num_heads used in :py:class:`model_center.layer.Attention`. + dim_head (int): dim_head used in :py:class:`model_center.layer.Attention`. + dtype (optional): Defaults to torch.half. + eps (floatoptional): eps used in :py:class:`model_center.layer.LayerNorm`. Defaults to 1e-6. + dropout_p (floatoptional): Defaults to 0. + """ # noqa: E501 + + def __init__( + self, + num_layers: int, + dim_model: int, + dim_ff: int, + num_heads: int, + dim_head: int, + dtype: torch.dtype = torch.half, + eps: float = 1e-6, + dropout_p: Optional[float] = None, + mask_modules: Optional[List[Tuple[boolbool]]] = None, + ): + + super().__init__() + + self.num_layers = num_layers + + if mask_modules is not None: + assert ( + len(mask_modules) == num_layers + )"The total number of masks should equal to num_layers" + for mask_module in mask_modules: + assert ( + len(mask_module) == 2 + )"For encodereach mask should be (mask_attmask_ffn)" + else: + mask_modules = [(FalseFalse)] * num_layers + + self.layers = torch.nn.ModuleList( + [ + TransformerBlock( + dim_model=dim_model, + dim_ff=dim_ff, + num_heads=num_heads, + dim_head=dim_head, + dtype=dtype, + eps=eps, + dropout_p=dropout_p, + mask_att=mask_modules[ith][0], + mask_ffn=mask_modules[ith][1], + ) + for ith in range(num_layers) + ] + ) + + self.output_layernorm = LayerNorm(dim_norm=dim_modeldtype=dtypeeps=eps) + + def forward( + self, + hidden_states: torch.Tensor, + attention_mask: torch.Tensor, + position_bias: torch.Tensor, + use_cache: bool = False, + past_key_values: Optional[List[Tuple[torch.Tensortorch.Tensor]]] = None, + ): + """ + Args: + hidden-states (:obj:`torch.Tensor` of shape ``(batchseq_encdim_model)``): Input of encodermight be the embedding of a batch of sequences. + attention_mask (:obj:`torch.Tensor` of shape ``(batchseq_encseq_enc)``): Avoid invalid areas to participate in the calculation + position_bias(:obj:`torch.Tensor` of shape ``(num_headsseq_encseq_enc)``) Provides position information to attention mechanism. + + Return: + :obj:`torch.Tensor` of shape ``(batchseq_encdim_model)``: The encoder output. + + """ # noqa: E501 + if not use_cache: + for layer in self.layers: + hidden_states = layer(hidden_statesattention_maskposition_bias) + hidden_states = self.output_layernorm(hidden_states) + return hidden_states + else: + with torch.no_grad(): + current_key_values = [] + for imodule in enumerate(self.layers): + hidden_states = module( + hidden_states, + attention_mask, + position_bias, + past_key_value=past_key_values[i] if past_key_values else None, + use_cache=use_cache, + ) + if use_cache: + current_key_values.append(hidden_states[1]) + hidden_states = hidden_states[0] + hidden_states = self.output_layernorm(hidden_states) + if use_cache: + return hidden_statescurrent_key_values + else: + return hidden_states diff --git a/cpm-live/cpm_live/tokenizers/__init__.py b/cpm-live/cpm_live/tokenizers/__init__.py index 1c2bb3a..c664def 100644 --- a/cpm-live/cpm_live/tokenizers/__init__.py +++ b/cpm-live/cpm_live/tokenizers/__init__.py @@ -1 +1,2 @@ from .ant import CPMAntTokenizer +from .bee import CPMBeeTokenizer diff --git a/cpm-live/cpm_live/tokenizers/ant.py b/cpm-live/cpm_live/tokenizers/ant.py index bcbf826..d127ec6 100644 --- a/cpm-live/cpm_live/tokenizers/ant.py +++ b/cpm-live/cpm_live/tokenizers/ant.py @@ -1,4 +1,18 @@ # coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + import jieba import pkg_resources import io @@ -111,6 +125,10 @@ def pad_id(self): def unk_id(self): return self.encoder[self.unk_token] + @property + def newline_id(self): + return self.encoder["\n"] + def __len__(self): return len(self.encoder) diff --git a/cpm-live/cpm_live/tokenizers/bee.py b/cpm-live/cpm_live/tokenizers/bee.py new file mode 100644 index 0000000..6078f6b --- /dev/null +++ b/cpm-live/cpm_live/tokenizers/bee.py @@ -0,0 +1,223 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pkg_resources +import io +from typing import IODictListOptionalTuple + + +def load_vocab(fp: IO[bytes]) -> Dict[strint]: + """Loads a vocabulary file into a dictionary.""" + vocab: Dict[strint] = {} + + reader = io.TextIOWrapper(fpencoding="utf-8") + for token in reader.readlines(): + if token[-1] == "\n": + token = token[:-1] + if len(token) == 0: + continue + vocab[token] = len(vocab) + return vocab + + +class Token(object): + def __init__(selftoken: strstart: intis_unk: boolis_special: bool): + self.token = token + self.start = start + self.is_unk = is_unk + self.is_special = is_special + + def __str__(self): + return "Token(token={}start={}is_unk={}is_special={})".format( + self.tokenself.startself.is_unkself.is_special + ) + + def __repr__(self): + return self.__str__() + + +class CPMBeeTokenizer(object): + def __init__( + self, + ): + self.unk_token = "" + self.mask_token = "" + self.bos_token = "" + self.eos_token = "" + self.line_token = "\n" + self.space_token = " " + + self.encoder = load_vocab(pkg_resources.resource_stream("cpm_live""vocabs/bee.txt")) + self.encoder[self.line_token] = self.encoder[""] + self.encoder[self.space_token] = self.encoder[""] + del self.encoder[""] + del self.encoder[""] + + self.decoder = {v: k for kv in self.encoder.items()} + self._special_tokens = { + k: v for kv in self.encoder.items() if k.startswith("<") and k.endswith(">") + } + + self._max_word_len = max([len(x) for x in self.encoder.keys()]) + + def get_piece(selftext: str) -> str: + text = text[: self._max_word_len] + len_text = len(text) + for i in range(len(text)): + sub = text[: len_text - i] + if (sub in self.encoder) and (sub not in self._special_tokens): + return sub + return text[0] + + @property + def vocab_size(self): + return len(self.encoder) + + @property + def eos_id(self): + return self.encoder[self.eos_token] + + @property + def bos_id(self): + return self.encoder[self.bos_token] + + @property + def unk_id(self): + return self.encoder[self.unk_token] + + @property + def mask_id(self): + return self.encoder[self.mask_token] + + def __len__(self): + return len(self.encoder) + + def tokenize(selftext: str) -> List[Token]: + output_tokens: List[Token] = [] + + sentence_split = [""] + is_escape = False + is_special_token = False + for ic in enumerate(text): + if is_special_token: + if c == "<": + raise ValueError("Invalid special token at pos {}".format(i)) + elif c == ">": + # end of special token + sentence_split[-1] += c + is_special_token = False + sentence_split.append("") + else: + sentence_split[-1] += c + else: + if c == "<": + if is_escape: + # case: << + sentence_split[-1] += c + is_escape = False + else: + # case: x< + is_escape = True + else: + if is_escape: + # case str: + return text.replace("<""<<") + + @staticmethod + def unescape(text: str) -> str: + return text.replace("<<""<") + + def encode( + selftext: strpast_table: Dict[intstr] = {} + ) -> Tuple[List[int]Dict[intstr]]: + ext_table_rev: Dict[strint] = {} + ext_table: Dict[intstr] = {} + for idxval in past_table.items(): + ext_table[idx] = val + ext_table_rev[val] = idx + ret = [] + for x in self.tokenize(text): + if x.is_unk or (x.is_special and (x.token not in self.encoder)): + if x.token not in ext_table_rev: + ext_table_rev[x.token] = len(ext_table_rev) + self.vocab_size + ext_table[ext_table_rev[x.token]] = x.token + ret.append(ext_table_rev[x.token]) + elif x.token in self.encoder: + ret.append(self.encoder[x.token]) + else: + raise ValueError("Unknown token `{}` at pos {}".format(x.tokenx.start)) + + return retext_table + + def decode(selftokens: List[int]ext_table: Optional[Dict[intstr]] = None): + """Decode ids into a string.""" + if ext_table is None: + ext_table = {} + ret = [] + for token in tokens: + if token in ext_table: + ret.append(ext_table[token]) + else: + if token >= 0: + w = self.decoder[token] + if w in self._special_tokens: + ret.append(w) + else: + ret.append(self.escape(w)) + return "".join(ret) diff --git a/cpm-live/training_tasks/__init__.py b/cpm-live/cpm_live/training_tasks/__init__.py similarity index 50% rename from cpm-live/training_tasks/__init__.py rename to cpm-live/cpm_live/training_tasks/__init__.py index 0a502f1..3cb9fa0 100644 --- a/cpm-live/training_tasks/__init__.py +++ b/cpm-live/cpm_live/training_tasks/__init__.py @@ -1 +1,2 @@ from . import ant +from . import bee diff --git a/cpm-live/training_tasks/ant/__init__.py b/cpm-live/cpm_live/training_tasks/ant/__init__.py similarity index 100% rename from cpm-live/training_tasks/ant/__init__.py rename to cpm-live/cpm_live/training_tasks/ant/__init__.py diff --git a/cpm-live/training_tasks/ant/pretrain.py b/cpm-live/cpm_live/training_tasks/ant/pretrain.py similarity index 96% rename from cpm-live/training_tasks/ant/pretrain.py rename to cpm-live/cpm_live/training_tasks/ant/pretrain.py index 055b691..5604bfc 100644 --- a/cpm-live/training_tasks/ant/pretrain.py +++ b/cpm-live/cpm_live/training_tasks/ant/pretrain.py @@ -96,8 +96,7 @@ def __get_item_data(selfraw_data): tgt = np.concatenate((np.full(self.prompt_length-100dtype=np.int64)tgt)) inp = np.concatenate( ( - np.arange(self.prompt_lengthdtype=np.int64) + - self.prompt_length * global_task + self.tokenizer.vocab_size, + np.arange(self.prompt_lengthdtype=np.int64) + self.prompt_length * global_task, ctx, ) ) diff --git a/cpm-live/cpm_live/training_tasks/bee/__init__.py b/cpm-live/cpm_live/training_tasks/bee/__init__.py new file mode 100644 index 0000000..684f707 --- /dev/null +++ b/cpm-live/cpm_live/training_tasks/bee/__init__.py @@ -0,0 +1,2 @@ +from .pretrain import MixedDataset +from .finetune import FinetuneDataset diff --git a/cpm-live/cpm_live/training_tasks/bee/finetune.py b/cpm-live/cpm_live/training_tasks/bee/finetune.py new file mode 100644 index 0000000..561e3c2 --- /dev/null +++ b/cpm-live/cpm_live/training_tasks/bee/finetune.py @@ -0,0 +1,71 @@ +from ...tokenizers import CPMBeeTokenizer +from .pretrain import _MixedDatasetBatchPacker_MixedDatasetConfigCPMBeeBatch +from ...dataset import SimpleDataset +import bmtrain as bmt + + +class FinetuneDataset: + def __init__( + self, + dataset_path: str, + batch_size: int, + max_length: int, + tokenizer: CPMBeeTokenizer, + max_depth: int = 16, + task_name: str = "task", + drop_last: bool = False, + ) -> None: + self._world_size = bmt.world_size() + self._rank = bmt.rank() + self._batch_size = batch_size + + self._packer = _MixedDatasetBatchPacker( + batch_size * self._world_sizemax_lengthtokenizermax_depth + ) + self._drop_last = drop_last + + ds = SimpleDataset(dataset_pathshuffle=False) + self._ds_cfg: _MixedDatasetConfig = { + "weight": 1.0, + "path": dataset_path, + "transforms": [], + "task_name": task_name, + "dataset_name": "finetune", + "incontext_weight": [1.0], + "lines": len(ds), + "dataset": ds, + } + + def __batch_iter(self): + while True: + try: + batch = self._packer.add_data(self._ds_cfg) + except EOFError: + break + if batch is None: + continue + yield batch + if len(self._packer) > 0: + batch = self._packer.pack_batch(force=True) + if not self._drop_last: + yield batch + self._ds_cfg["dataset"]._repeat_times = 0 + + def __iter__(self): + batch_st = self._batch_size * self._rank + batch_end = self._batch_size * (self._rank + 1) + for batch in self.__batch_iter(): + batch_size = batch["inputs"].shape[0] + if batch_size <= batch_st: + yield None + else: + ret: CPMBeeBatch = { + kw: val[batch_st:batch_end] # type: ignore + for kwval in batch.items() + if kw not in ["task_names""raw_data""ext_ids""ext_sub"] + } # type: ignore + ret["task_names"] = batch["task_names"] + ret["raw_data"] = batch["raw_data"] + ret["ext_ids"] = batch["ext_ids"] + ret["ext_sub"] = batch["ext_sub"] + yield ret diff --git a/cpm-live/cpm_live/training_tasks/bee/pretrain.py b/cpm-live/cpm_live/training_tasks/bee/pretrain.py new file mode 100644 index 0000000..d3caef2 --- /dev/null +++ b/cpm-live/cpm_live/training_tasks/bee/pretrain.py @@ -0,0 +1,1041 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from collections import OrderedDict +import multiprocessing +import os +from queue import Empty +from typing import AnyCallableDictListOptionalSetTupleUnion +from typing_extensions import TypedDict +from ...dataset import DistributedDataset +from ...tokenizers import CPMBeeTokenizer +from ...utils.config import load_dataset_config +import numpy as np +import time +from numpy.typing import NDArray +import torch +import bmtrain as bmt +import importlib.machinery +import importlib.util +import types +import random + + +class _MixedDatasetConfig(TypedDict): + weight: float + path: str + transforms: Union[List[Dict[strAny]]str] + task_name: str + dataset_name: str + incontext_weight: List[float] + + lines: int + dataset: DistributedDataset + + +CPMBeeInputType = Union[strDict[str"CPMBeeInputType"]] + + +class _DictTree(TypedDict): + value: str + children: List["_DictTree"] + depth: int + segment_id: int + need_predict: bool + + +class _PrevExtTableStates(TypedDict): + ext_table: Dict[intstr] + token_id_table: Dict[strDict[intint]] + + +class _TransformFuncDict(TypedDict): + loader: importlib.machinery.SourceFileLoader + module: types.ModuleType + last_m: float + + +_TransformFunction = Callable[[CPMBeeInputTypeintrandom.Random]CPMBeeInputType] + + +class CPMBeeBatch(TypedDict): + inputs: NDArray[np.int32] + inputs_sub: NDArray[np.int32] + length: NDArray[np.int32] + context: NDArray[np.bool_] + sample_ids: NDArray[np.int32] + num_segments: NDArray[np.int32] + segment_ids: NDArray[np.int32] + segment_rel_offset: NDArray[np.int32] + segment_rel: NDArray[np.int32] + spans: NDArray[np.int32] + target: NDArray[np.int32] + ext_ids: NDArray[np.int32] + ext_sub: NDArray[np.int32] + task_ids: NDArray[np.int32] + task_names: List[str] + raw_data: List[Any] + + +def rel_to_bucket(n_up: intn_down: intmax_depth: int = 8): + ret = n_up * max_depth + n_down + if ret == 0: + return ret + else: + # bucket 1 is reserved for incontext samples + return ret + 1 + + +def convert_data_to_id( + tokenizer: CPMBeeTokenizer, + data: Any, + prev_ext_states: Optional[_PrevExtTableStates] = None, + shuffle_answer: bool = True, + max_depth: int = 8, +): + root: _DictTree = { + "value": "", + "children": [], + "depth": 0, + "segment_id": 0, + "need_predict": False, + } + + segments = [root] + + def _build_dict_tree(data: CPMBeeInputTypedepth: intneed_predict: bool) -> List[_DictTree]: + if isinstance(datadict): + ret_list: List[_DictTree] = [] + curr_items = list(data.items()) + if need_predict and shuffle_answer: + access_idx = np.arange(len(curr_items)) + np.random.shuffle(access_idx) + curr_items = [curr_items[idx] for idx in access_idx] + for kv in curr_items: + child_info: _DictTree = { + "value": k, + "children": [], + "depth": depth, + "segment_id": len(segments), + "need_predict": False # only leaves are contexts + } + segments.append(child_info) + child_info["children"] = _build_dict_tree( + vdepth + 1need_predict or (depth == 1 and k == "") + ) # elements in . + + ret_list.append(child_info) + return ret_list + else: + assert isinstance(datastr)"Invalid data {}".format(data) + ret: _DictTree = { + "value": data, + "children": [], + "depth": depth, + "segment_id": len(segments), + "need_predict": need_predict, + } + segments.append(ret) + return [ret] + + root["children"] = _build_dict_tree(data1False) + + num_segments = len(segments) + segment_rel = np.zeros((num_segments * num_segments,)dtype=np.int32) + + def _build_segment_rel(node: _DictTree) -> List[Tuple[intint]]: + ret: List[Tuple[intint]] = [(node["segment_id"]node["depth"])] + for child in node["children"]: + sub = _build_segment_rel(child) + for seg_id_1depth_1 in sub: + for seg_id_2depth_2 in ret: + n_up = min(depth_1 - node["depth"]max_depth - 1) + n_down = min(depth_2 - node["depth"]max_depth - 1) + segment_rel[seg_id_1 * num_segments + seg_id_2] = rel_to_bucket( + n_upn_downmax_depth=max_depth + ) + segment_rel[seg_id_2 * num_segments + seg_id_1] = rel_to_bucket( + n_downn_upmax_depth=max_depth + ) + ret.extend(sub) + return ret + + _build_segment_rel(root) + + input_ids: List[int] = [] + input_id_subs: List[int] = [] + segment_bound: List[Tuple[intint]] = [] + + ext_table: Dict[intstr] = {} + token_id_table: Dict[strDict[intint]] = {} + + if prev_ext_states is not None: + ext_table = prev_ext_states["ext_table"] + token_id_table = prev_ext_states["token_id_table"] + + for seg in segments: + tokensext_table = tokenizer.encode(seg["value"]ext_table) + + token_id_subs = [] + reid_token_ids = [] + for idx in tokens: + if idx in ext_table: + # unk or special token + token = ext_table[idx] + if token.startswith("<") and token.endswith(">"): + # special token + if "_" in token: + token_name = token[1:-1].split("_"maxsplit=1)[0] + else: + token_name = token[1:-1] + token_name = "<{}>".format(token_name) + else: + token_name = "" + + if token_name not in token_id_table: + token_id_table[token_name] = {} + if idx not in token_id_table[token_name]: + token_id_table[token_name][idx] = len(token_id_table[token_name]) + if token_name not in tokenizer.encoder: + raise ValueError("Invalid token {}".format(token)) + reid_token_ids.append(tokenizer.encoder[token_name]) + token_id_subs.append(token_id_table[token_name][idx]) + else: + reid_token_ids.append(idx) + token_id_subs.append(0) + tokens = [tokenizer.bos_id] + reid_token_ids + token_id_subs = [0] + token_id_subs + if not seg["need_predict"]: + tokens = tokens + [tokenizer.eos_id] + token_id_subs = token_id_subs + [0] + else: + # no eos + pass + begin = len(input_ids) + input_ids.extend(tokens) + input_id_subs.extend(token_id_subs) + end = len(input_ids) + segment_bound.append((beginend)) + + ids = np.array(input_idsdtype=np.int32) + id_subs = np.array(input_id_subsdtype=np.int32) + segs = np.zeros((ids.shape[0],)dtype=np.int32) + context = np.zeros((ids.shape[0],)dtype=np.int8) + for i(beginend) in enumerate(segment_bound): + if not segments[i]["need_predict"]: + context[begin:end] = 1 + segs[begin:end] = i + + curr_ext_table_states: _PrevExtTableStates = { + "ext_table": ext_table, + "token_id_table": token_id_table, + } + return idsid_subscontextsegssegment_relnum_segmentscurr_ext_table_states + + +def _dataset_identity(c: _MixedDatasetConfig): + return "{}.{}".format(c["task_name"]c["dataset_name"]) + + +class _MixedDatasetBatchPacker: + def __init__( + self, + batch_size: int, + max_length: int, + tokenizer: CPMBeeTokenizer, + max_depth: int = 16, + ) -> None: + self._batch_size = batch_size + self._max_length = max_length + self._max_depth = max_depth + self.tokenizer = tokenizer + self._transform_func_table: Dict[str_TransformFuncDict] = {} + + self._inputs: List[NDArray[np.int32]] = [] + self._inputs_sub: List[NDArray[np.int32]] = [] + self._context: List[NDArray[np.int8]] = [] + self._sample_ids: List[NDArray[np.int32]] = [] + self._segments: List[NDArray[np.int32]] = [] + self._num_segments: List[NDArray[np.int32]] = [] + self._segment_rel_offset: List[NDArray[np.int32]] = [] + self._segment_rel: List[NDArray[np.int32]] = [] + self._spans: List[List[int]] = [] + self._task_ids: List[List[str]] = [] + self._raw_data: List[List[Any]] = [] + + def __len__(self): + return len(self._inputs) + + def apply_transform( + self, + data: CPMBeeInputType, + transform: Union[Dict[strAny]Callable[[CPMBeeInputType]CPMBeeInputType]None], + ) -> CPMBeeInputType: + if transform is None: + return data + if not isinstance(transformdict): + # transform function + return transform(data) + + mapping_list: List[Tuple[strstr]] = [] + + def _walk_transform_dict(data: Union[Dict[strAny]str]prefix: str = ""): + if isinstance(datadict): + for kv in data.items(): + if len(prefix) > 0: + _walk_transform_dict(vprefix + "." + k) + else: + _walk_transform_dict(vk) + else: + assert isinstance(datastr)"Invalid transform {}".format(data) + mapping_list.append((prefixdata)) + + _walk_transform_dict(transform) + + expanded_mapping_list: List[Tuple[strAny]] = [] + + def _expand_mapping( + data: CPMBeeInputTypestars: List[str]path: List[str]target: List[str] + ): + if len(path) == 0: + num_stars = 0 + for it in target: + if it == "*": + num_stars += 1 + if num_stars != len(stars): + raise ValueError("Invalid transform {}".format(".".join(target))) + + nw_tgt = [] + num_stars = 0 + for it in target: + if it == "*": + nw_tgt.append(stars[num_stars]) + num_stars += 1 + else: + nw_tgt.append(it) + expanded_mapping_list.append((".".join(nw_tgt)data)) + else: + if not isinstance(datadict): + raise ValueError("Invalid data {}".format(data)) + if path[0] == "*": + for kv in data.items(): + _expand_mapping(vstars + [k]path[1:]target) + else: + _expand_mapping(data[path[0]]starspath[1:]target) + + # expand mapping list + for tgtsrc in mapping_list: + if src.startswith("$"): + # copy from src + _expand_mapping(data[]src[1:].split(".")tgt.split(".")) + else: + if "*" in tgt: + raise ValueError("Constant value is not allowed to have `*` in prefix") + expanded_mapping_list.append((tgtsrc)) + + ret = {} + for tgtval in expanded_mapping_list: + tgt = tgt.split(".") + cur = ret + while len(tgt) > 1: + cur = cur[tgt[0]] + tgt = tgt[1:] + cur[tgt[0]] = val + return ret + + def data_to_id( + self, + data: Any, + prev_ext_states: Optional[_PrevExtTableStates] = None, + shuffle_answer: bool = True, + ): + return convert_data_to_id( + self.tokenizerdataprev_ext_statesshuffle_answerself._max_depth + ) + + def _ensure_transform_function( + selfmodule_name: strtransform_script_path: str + ) -> _TransformFunction: + module_name = "cpm_live.transforms.{}".format(module_name) + if transform_script_path not in self._transform_func_table: + loader = importlib.machinery.SourceFileLoader(module_nametransform_script_path) + spec = importlib.util.spec_from_loader(loader.nameloader) + if spec is None: + raise RuntimeError("spec is none! {}".format(module_name)) + mod = importlib.util.module_from_spec(spec) + self._transform_func_table[transform_script_path] = { + "loader": loader, + "module": mod, + "last_m": 0, + } + + transform_script_info = self._transform_func_table[transform_script_path] + curr_m_time = float( + transform_script_info["loader"].path_stats(transform_script_path)["mtime"] + ) + if curr_m_time > transform_script_info["last_m"]: + transform_script_info["last_m"] = curr_m_time + transform_script_info["loader"].exec_module(transform_script_info["module"]) + transform_func = getattr(transform_script_info["module"]"transform"None) + if transform_func is None: + + def _empty_transform_func(data: CPMBeeInputTypenum_sample: intr: random.Random): + raise NotImplementedError( + "Transform func for dataset {} not implemented".format(module_name) + ) + + return _empty_transform_func + else: + return transform_func + + def build_instance(selfconfig: _MixedDatasetConfig): + _sample_weight = np.array(config["incontext_weight"]dtype=np.float32) + _sample_weight = _sample_weight / _sample_weight.sum() + num_incontext = np.random.choice(_sample_weight.shape[0]p=_sample_weight) + ds = config["dataset"] + transforms = config["transforms"] + if isinstance(transformsstr): + while True: + try: + if not os.path.exists(transforms): + raise RuntimeError( + "transform script file {} not exists".format(transforms) + ) + # load transform script + transform_func = self._ensure_transform_function( + _dataset_identity(config)transforms + ) + seed = random.random() + break + except Exception as e: + print(e) + time.sleep(10) + + def _transform(data: CPMBeeInputType): + r = random.Random(seed) + return transform_func(datanum_incontextr) + transform = _transform + elif len(transforms) == 0: + transform = None + else: + transform = transforms[np.random.choice(len(transforms))] + + raw_data = {} + while True: + inp = ds.read() + inp = self.apply_transform(inptransform) + + ( + input_ids, + input_id_subs, + context, + segment_ids, + segment_rel, + n_segments, + table_states, + ) = self.data_to_id(inp) + if input_ids.shape[0] > self._max_length: + # too long + continue + input_ids = input_ids[: self._max_length] + context = context[: self._max_length] + segment_ids = segment_ids[: self._max_length] + raw_data["input"] = inp + raw_data["samples"] = [] + break + + sample_ids = np.zeros(input_ids.shapedtype=np.int32) + segment_rel_offset = np.zeros(input_ids.shapedtype=np.int32) + num_segments = np.full(input_ids.shapen_segmentsdtype=np.int32) + + for i in range(num_incontext): + if input_ids.shape[0] >= self._max_length: + # early break + break + + sample = ds.read() + sample = self.apply_transform(sampletransform) + ( + sample_input_ids, + sample_id_subs, + _, + sample_segments, + sample_rel, + n_segments, + table_states, + ) = self.data_to_id(sampletable_states) + + if input_ids.shape[0] + sample_input_ids.shape[0] > self._max_length: + # too longbreak + break + raw_data["samples"].append(sample) + input_ids = np.concatenate([input_idssample_input_ids]axis=0) + input_id_subs = np.concatenate([input_id_subssample_id_subs]axis=0) + context = np.concatenate( + [contextnp.ones(sample_input_ids.shapedtype=np.int8)]axis=0 + ) + segment_ids = np.concatenate([segment_idssample_segments]axis=0) + segment_rel_offset = np.concatenate( + [ + segment_rel_offset, + np.full(sample_input_ids.shapesegment_rel.shape[0]dtype=np.int32), + ], + axis=0, + ) + segment_rel = np.concatenate([segment_relsample_rel]axis=0) + sample_ids = np.concatenate( + [sample_idsnp.full(sample_input_ids.shapei + 1dtype=np.int32)]axis=0 + ) + num_segments = np.concatenate( + [num_segmentsnp.full(sample_input_ids.shapen_segmentsdtype=np.int32)]axis=0 + ) + return ( + input_ids, + input_id_subs, + context, + segment_ids, + segment_rel_offset, + segment_rel, + sample_ids, + num_segments, + raw_data, + ) + + def pack_batch(selfforce: bool = False) -> CPMBeeBatch: + # pack batch + if len(self._inputs) < self._batch_size: + if not force: + raise RuntimeError("Batch insufficient") + batch_size = len(self._inputs) + else: + batch_size = self._batch_size + inputs = np.zeros((batch_sizeself._max_length)dtype=np.int32) + inputs_sub = np.zeros((batch_sizeself._max_length)dtype=np.int32) + context = np.zeros((batch_sizeself._max_length)dtype=np.int8) + sample_ids = np.zeros((batch_sizeself._max_length)dtype=np.int32) + segments = np.zeros((batch_sizeself._max_length)dtype=np.int32) + num_segments = np.zeros((batch_sizeself._max_length)dtype=np.int32) + segment_rel_offset = np.zeros((batch_sizeself._max_length)dtype=np.int32) + tgt = np.full((batch_sizeself._max_length)-100dtype=np.int32) + + max_rel = 0 + for i in range(batch_size): + max_rel = max(max_relself._segment_rel[i].shape[0]) + segment_rel = np.zeros((batch_sizemax_rel)dtype=np.int32) + spans = np.zeros((batch_sizeself._max_length)dtype=np.int32) + length = np.zeros((batch_size,)dtype=np.int32) + task_ids = np.zeros((batch_sizeself._max_length)dtype=np.int32) + + all_task_names: Set[str] = set() + for i in range(batch_size): + for task_name in self._task_ids[i]: + all_task_names.add(task_name) + task_names: List[str] = list(all_task_names) + task_name_to_id = {name: i for iname in enumerate(task_names)} + + batch_ext_table_map: Dict[Tuple[intint]int] = {} + batch_ext_table_ids: List[int] = [] + batch_ext_table_sub: List[int] = [] + raw_data_list: List[Any] = [] + for i in range(batch_size): + instance_length = self._inputs[i].shape[0] + rel_size = self._segment_rel[i].shape[0] + inputs[i:instance_length] = self._inputs[i] + inputs_sub[i:instance_length] = self._inputs_sub[i] + context[i:instance_length] = self._context[i] + sample_ids[i:instance_length] = self._sample_ids[i] + segments[i:instance_length] = self._segments[i] + num_segments[i:instance_length] = self._num_segments[i] + segment_rel_offset[i:instance_length] = self._segment_rel_offset[i] + segment_rel[i:rel_size] = self._segment_rel[i] + + span_begin = 0 + for span_id(span_endtask_name) in enumerate(zip(self._spans[i]self._task_ids[i])): + spans[ispan_begin:span_end] = span_id + task_ids[ispan_begin:span_end] = task_name_to_id[task_name] + span_begin = span_end + length[i] = instance_length + raw_data_list.extend(self._raw_data[i]) + + for j in range(instance_length): + idxidx_sub = self._inputs[i][j]self._inputs_sub[i][j] + tgt_idx = idx + if idx_sub > 0: + # need to be in ext table + if (idxidx_sub) not in batch_ext_table_map: + batch_ext_table_map[(idxidx_sub)] = len(batch_ext_table_map) + batch_ext_table_ids.append(idx) + batch_ext_table_sub.append(idx_sub) + tgt_idx = batch_ext_table_map[(idxidx_sub)] + self.tokenizer.vocab_size + if j > 1 and context[ij - 1] == 0: + if idx != self.tokenizer.bos_id: + tgt[ij - 1] = tgt_idx + else: + tgt[ij - 1] = self.tokenizer.eos_id + if context[iinstance_length - 1] == 0: + tgt[iinstance_length - 1] = self.tokenizer.eos_id + + if len(batch_ext_table_map) == 0: + # placeholder + batch_ext_table_ids.append(0) + batch_ext_table_sub.append(1) + + self._inputs = self._inputs[batch_size:] + self._inputs_sub = self._inputs_sub[batch_size:] + self._context = self._context[batch_size:] + self._sample_ids = self._sample_ids[batch_size:] + self._segments = self._segments[batch_size:] + self._num_segments = self._num_segments[batch_size:] + self._segment_rel_offset = self._segment_rel_offset[batch_size:] + self._segment_rel = self._segment_rel[batch_size:] + self._spans = self._spans[batch_size:] + self._task_ids = self._task_ids[batch_size:] + self._raw_data = self._raw_data[batch_size:] + return { + "inputs": inputs, + "inputs_sub": inputs_sub, + "length": length, + "context": context > 0, + "sample_ids": sample_ids, + "num_segments": num_segments, + "segment_ids": segments, + "segment_rel_offset": segment_rel_offset, + "segment_rel": segment_rel, + "spans": spans, + "target": tgt, + "ext_ids": np.array(batch_ext_table_idsdtype=np.int32), + "ext_sub": np.array(batch_ext_table_subdtype=np.int32), + "task_ids": task_ids, + "task_names": task_names, + "raw_data": raw_data_list, + } + + def add_data(selfconfig: _MixedDatasetConfig) -> Optional[CPMBeeBatch]: + ( + input_ids, + input_id_subs, + context, + segment_ids, + segment_rel_offset, + segment_rel, + sample_ids, + num_segments, + raw_data, + ) = self.build_instance(config) + + # add to batch + best_fit: Union[Noneint] = None + best_fit_space: Union[Noneint] = None + for i in range(len(self._inputs)): + space = self._max_length - self._inputs[i].shape[0] + if input_ids.shape[0] <= space: + if best_fit_space is None: + best_fit = i + best_fit_space = space + elif best_fit_space > space: + best_fit = i + best_fit_space = space + if best_fit is None: + # add a new instance + self._inputs.append(input_ids) + self._inputs_sub.append(input_id_subs) + self._context.append(context) + self._sample_ids.append(sample_ids) + self._segments.append(segment_ids) + self._num_segments.append(num_segments) + self._segment_rel_offset.append(segment_rel_offset) + self._segment_rel.append(segment_rel) + self._spans.append([input_ids.shape[0]]) + self._task_ids.append([config["task_name"]]) + self._raw_data.append([raw_data]) + else: + # add to existing instance + self._inputs[best_fit] = np.concatenate([self._inputs[best_fit]input_ids]axis=0) + self._inputs_sub[best_fit] = np.concatenate( + [self._inputs_sub[best_fit]input_id_subs]axis=0 + ) + self._context[best_fit] = np.concatenate([self._context[best_fit]context]axis=0) + self._sample_ids[best_fit] = np.concatenate( + [self._sample_ids[best_fit]sample_ids]axis=0 + ) + self._segments[best_fit] = np.concatenate( + [self._segments[best_fit]segment_ids]axis=0 + ) + self._num_segments[best_fit] = np.concatenate( + [self._num_segments[best_fit]num_segments]axis=0 + ) + self._segment_rel_offset[best_fit] = np.concatenate( + [ + self._segment_rel_offset[best_fit], + segment_rel_offset + self._segment_rel[best_fit].shape[0], + ], + axis=0, + ) + self._segment_rel[best_fit] = np.concatenate( + [self._segment_rel[best_fit]segment_rel]axis=0 + ) + self._spans[best_fit].append(self._inputs[best_fit].shape[0]) + self._task_ids[best_fit].append(config["task_name"]) + self._raw_data[best_fit].append(raw_data) + + if len(self._inputs) > self._batch_size: + return self.pack_batch() + else: + # not ready + return None + + +class _MixedDatasetConfigMananger: + def __init__(selfconfig_path: str) -> None: + self._config_path: str = config_path + self._config: Union[List[_MixedDatasetConfig]None] = None + self._last_m = 0 + + def changed(self): + while True: + try: + m_time = os.stat(self._config_path).st_mtime + if m_time > self._last_m: + # try to load new config + try: + self._config = load_dataset_config(self._config_path) + except Exception as e: + # failed to load config + print( + "Error: load new config in changed" + "self._config_path={path}err={err}" + .format(path=self._config_patherr=str(e)) + ) + + return False + # new config loaded + self._last_m = m_time + return True + return False + except Exception as e: + print("Error: reading info list in _MixedDatasetConfigMananger.changed!" + "self._config_path={path}err={err}" + .format(path=self._config_patherr=str(e))) + time.sleep(30) + + def get_config(self) -> List[_MixedDatasetConfig]: + if self._config is None: + if not self.changed(): + raise RuntimeError("Failed to load config") + if self._config is None: + raise RuntimeError("Failed to load config") + return self._config + + +def _mixed_dataset_process( + config_path: str, + q_cmd: multiprocessing.Queue, + q_cmd_out: multiprocessing.Queue, + q_data: multiprocessing.Queue, + rank: int, + world_size: int, + packer: _MixedDatasetBatchPacker, +): + # ignore SIGINT + import signal + + signal.signal(signal.SIGINTsignal.SIG_IGN) + config_base_path = os.path.dirname(os.path.abspath(config_path)) + + def _convert_to_abs_path(transform_path: str): + if transform_path.startswith("/"): + return transform_path + else: + return os.path.join(config_base_pathtransform_path) + + def _build_sample_weights(config: List[_MixedDatasetConfig]): + if len(config) == 0: + return np.array([]dtype=np.float32) + weights = [c["weight"] * c["lines"] for c in config] + weights = np.array(weightsdtype=np.float32) + sm_weight = weights.sum() + if sm_weight > 0: + weights = weights / sm_weight + return weights + else: + raise RuntimeError("Empty datasets") + + cfg_mgr = _MixedDatasetConfigMananger(config_path) + config = cfg_mgr.get_config() + + for c in config: + ds = DistributedDataset( + _convert_to_abs_path(c["path"]), + rank, + world_size, + ) + + c["lines"] = ds._nlines + c["dataset"] = ds + if "weight" not in c: + c["weight"] = 1.0 + if "transforms" not in c: + c["transforms"] = [] + elif isinstance(c["transforms"]str): + c["transforms"] = _convert_to_abs_path(c["transforms"]) + if "incontext_weight" not in c: + c["incontext_weight"] = [1.0] + + weights = _build_sample_weights(config) + + should_stop = False + should_start = False + + while not should_stop: + # update config first + if cfg_mgr.changed(): + path_ds_map: Dict[str_MixedDatasetConfig] = {} + nw_path_set: Set[str] = set() + + # load new config + nw_config = cfg_mgr.get_config() + + # build path -> dataset map + for c in config: + path_ds_map[_dataset_identity(c)] = c + + # add new datasets + for c in nw_config: + if _dataset_identity(c) in path_ds_map: + # update values only + if "weight" in c: + path_ds_map[_dataset_identity(c)]["weight"] = c["weight"] + if "transform" in c: + if isinstance(c["transforms"]str): + path_ds_map[_dataset_identity(c)]["transforms"] = _convert_to_abs_path( + c["transforms"] + ) + else: + path_ds_map[_dataset_identity(c)]["transforms"] = c["transforms"] + if "incontext_weight" in c: + path_ds_map[_dataset_identity(c)]["incontext_weight"] = c[ + "incontext_weight" + ] + else: + # new dataset + ds = DistributedDataset( + _convert_to_abs_path(c["path"]), + rank, + world_size, + ) + c["lines"] = ds._nlines + c["dataset"] = ds + if "weight" not in c: + c["weight"] = 1.0 + if "transforms" not in c: + c["transforms"] = [] + elif isinstance(c["transforms"]str): + c["transforms"] = _convert_to_abs_path(c["transforms"]) + if "incontext_weight" not in c: + c["incontext_weight"] = [1.0] + path_ds_map[_dataset_identity(c)] = c + nw_path_set.add(_dataset_identity(c)) + + # remove unused datasets + for c in config: + if _dataset_identity(c) not in nw_path_set: + del path_ds_map[_dataset_identity(c)] + + config: List[_MixedDatasetConfig] = [] + for c in nw_config: + config.append(path_ds_map[_dataset_identity(c)]) + del path_ds_map + del nw_path_set + del nw_config + + weights = _build_sample_weights(config) + + # get cmds + while True: + try: + cmd = q_cmd.get_nowait() + except Empty: + break + if cmd == "stop": + should_stop = True + q_cmd_out.put(True) + break + elif cmd == "state_dict": + ret = OrderedDict() + for c in config: + ds_name = _dataset_identity(c) + ret[ds_name] = c["dataset"]._state_dict() + q_cmd_out.put(ret) + elif cmd == "load_state_dict": + state_dict = q_cmd.get() + missing = [] + for c in config: + ds_name = _dataset_identity(c) + if ds_name in state_dict: + c["dataset"].load_state_dict(state_dict[ds_name]strict=False) + else: + # new dataset + missing.append(ds_name) + q_cmd_out.put(missing) + elif cmd == "start": + should_start = True + q_cmd_out.put(True) + else: + raise RuntimeError("Unknown command: {}".format(cmd)) + + if should_stop: + break + + if not should_start: + # wait for start cmd + time.sleep(1) + continue + + if len(config) == 0: + # no dataset available + time.sleep(1) + continue + + if q_data.full(): + # queue full + time.sleep(1) + continue + + # sample a dataset + ds_id: int = 0 + + while True: + ds_id = np.random.choice(weights.shape[0]p=weights) + if config[ds_id]["dataset"]._nlines != config[ds_id]["lines"]: + # dataset size changed + for c in config: + c["lines"] = c["dataset"]._nlines + weights = _build_sample_weights(config) + continue + else: + break + + batch = packer.add_data(config[ds_id]) + if batch is not None: + # new batch comming + q_data.put(batch) + + # clean queue + while True: + try: + q_data.get_nowait() + except Empty: + break + + +class MixedDataset: + def __init__( + self, + config_path: str, + batch_size: int, + max_length: int, + tokenizer: CPMBeeTokenizer, + max_depth: int = 16, + ) -> None: + self._q_cmd = multiprocessing.Queue() + self._q_cmd_out = multiprocessing.Queue() + self._q_data = multiprocessing.Queue(maxsize=1) + self._packer = _MixedDatasetBatchPacker(batch_sizemax_lengthtokenizermax_depth) + self._p = multiprocessing.Process( + target=_mixed_dataset_process, + args=( + config_path, + self._q_cmd, + self._q_cmd_out, + self._q_data, + bmt.rank(), + bmt.world_size(), + self._packer, + ), + ) + self._p.start() + self._closed = False + + def close(self): + if not self._closed: + self._closed = True + self._q_cmd.put("stop") + assert self._q_cmd_out.get()"Failed to stop process" + self._p.join() + + @property + def closed(self): + return self._closed + + def start(self): + self._q_cmd.put("start") + return self._q_cmd_out.get() + + def state_dict(self): + self._q_cmd.put("state_dict") + states = self._q_cmd_out.get() + if not isinstance(statesOrderedDict): + raise RuntimeError("Invalid state dict {}".format(states)) + if bmt.world_size() == 1: + for val in states.values(): + val["states"].unsqueeze_(0) + val["block"].unsqueeze_(0) + return states + + ret = OrderedDict() + for kv in states.items(): + num_unused_block = v["states"].size(0) + gpu_num_unused_block = torch.tensor([num_unused_block]dtype=torch.long).cuda() + max_unused_blocks = ( + bmt.distributed.all_reduce(gpu_num_unused_blockop="max").cpu().item() + ) + if max_unused_blocks == 0: + max_unused_blocks = 1 + gpu_states = torch.full((max_unused_blocks,)-1dtype=torch.long).cuda() + gpu_states[:num_unused_block] = v["states"].cuda() + + gpu_block = v["block"].cuda() + global_states = bmt.distributed.all_gather( + gpu_states + ).cpu() # (world_sizemax_unused_blocks) + global_block = bmt.distributed.all_gather(gpu_block).cpu() # (world_size4) + ret[k] = {"states": global_states"block": global_block} + return ret + + def load_state_dict(selfdata: OrderedDictstrict: bool = False): + self._q_cmd.put("load_state_dict") + self._q_cmd.put(data) + missing = self._q_cmd_out.get() + if strict: + if len(missing) > 0: + raise RuntimeError("Missing dataset state: {}".format(missing)) + return missing + + def get(self) -> CPMBeeBatch: + ret: CPMBeeBatch = self._q_data.get() # type: ignore + if not isinstance(retdict): + raise RuntimeError("Invalid data {}".format(ret)) + return ret + + def __iter__(self): + while True: + yield self.get() + + def __del__(self): + if not self.closed: + try: + self.close() + except Exception: + pass diff --git a/cpm-live/cpm_live/utils/__init__.py b/cpm-live/cpm_live/utils/__init__.py index cca5d9b..3d9e31d 100644 --- a/cpm-live/cpm_live/utils/__init__.py +++ b/cpm-live/cpm_live/utils/__init__.py @@ -1 +1,5 @@ from .config import Config +from .data_utils import pad +from .object import allgather_objects +from .log import LogManagerlogger +from .config import load_dataset_config diff --git a/cpm-live/cpm_live/utils/config.py b/cpm-live/cpm_live/utils/config.py index 7e93ee4..28d2b26 100644 --- a/cpm-live/cpm_live/utils/config.py +++ b/cpm-live/cpm_live/utils/config.py @@ -17,10 +17,35 @@ import os import copy from typing import AnyDictUnion +from .log import logger + + +def load_dataset_config(dataset_path: str): + cfg = on.load(open(dataset_path"r"encoding="utf-8")) + + platform_config_path = os.getenv("PLATFORM_CONFIG_PATH") + if platform_config_path is None: + logger.info( + "no platform_config_path. Directly load dataset_path({dataset_path})" + .format(dataset_path=dataset_path) + ) + return cfg + + path_dict = on.load(open(platform_config_path"r"encoding="utf-8"))["dataset_map"] + logger.info( + "load dataset_path({dataset_path}) with platform_config_path({platform_config_path})" + .format(dataset_path=dataset_pathplatform_config_path=platform_config_path) + ) + for dataset in cfg: + dataset["path"] = os.path.join(path_dict[dataset["dataset_name"]]dataset["path"]) + dataset["transforms"] = os.path.join( + path_dict[dataset["dataset_name"]]dataset["transforms"] + ) + return cfg class Config(object): - """enc_dec model configuration""" + """model configuration""" def __init__(self): super().__init__() diff --git a/cpm-live/cpm_live/utils/data_utils.py b/cpm-live/cpm_live/utils/data_utils.py new file mode 100644 index 0000000..913e062 --- /dev/null +++ b/cpm-live/cpm_live/utils/data_utils.py @@ -0,0 +1,44 @@ +import torch + + +def pad(orig_itemskeypadding_value=0padding_side="left"): + items = [] + if isinstance(orig_items[0][key]list): + assert isinstance(orig_items[0][key][0]torch.Tensor) + for it in orig_items: + for tr in it[key]: + items.append({key: tr}) + else: + assert isinstance(orig_items[0][key]torch.Tensor) + items = orig_items + + batch_size = len(items) + shape = items[0][key].shape + dim = len(shape) + assert dim <= 3 + max_length = max(item[key].shape[-1] for item in items) + min_length = min(item[key].shape[-1] for item in items) + dtype = items[0][key].dtype + + if dim == 1: + return torch.cat([item[key] for item in items]dim=0) + elif dim == 2: + if max_length == min_length: + return torch.cat([item[key] for item in items]dim=0) + tensor = torch.zeros((batch_sizemax_length)dtype=dtype) + padding_value + else: + tensor = torch.zeros((batch_sizemax_lengthshape[-1])dtype=dtype) + padding_value + + for iitem in enumerate(items): + if dim == 2: + if padding_side == "left": + tensor[i-len(item[key][0]) :] = item[key][0].clone() + else: + tensor[i: len(item[key][0])] = item[key][0].clone() + elif dim == 3: + if padding_side == "left": + tensor[i-len(item[key][0]) ::] = item[key][0].clone() + else: + tensor[i: len(item[key][0]):] = item[key][0].clone() + + return tensor diff --git a/cpm-live/cpm_live/utils/export.py b/cpm-live/cpm_live/utils/export.py new file mode 100644 index 0000000..6e4db68 --- /dev/null +++ b/cpm-live/cpm_live/utils/export.py @@ -0,0 +1,56 @@ +import os +import time +import functools +import torch +import bmtrain as bmt +import on +from cpm_live.models import CPMBee +from .log import logger +from typing import ListOptional + + +def rename_if_exists(file_path): + if not os.path.exists(file_path): + return + timestamp = time.strftime('%Y%m%d%H%M%S') + file_dirfile_name = os.path.split(file_path) + file_rootfile_ext = os.path.splitext(file_name) + new_file_name = f"{file_root}_bak_{timestamp}{file_ext}" + new_file_path = os.path.join(file_dirnew_file_name) + try: + os.rename(file_pathnew_file_path) + logger.info(f"File '{file_name}' already exists. Renamed to '{new_file_name}'") + except Exception as e: + logger.warn( + "rename file failed,file_path={file_path}new_file_path={new_file_path},err={err}" + .format(file_path=file_pathnew_file_path=new_file_patherr=str(e))) + + +def rename_if_exists_decorator(func): + @functools.wraps(func) + def wrapper(file_path*args**kwargs): + rename_if_exists(file_path) + return func(file_path*args**kwargs) + return wrapper + + +@rename_if_exists_decorator +def bmt_save(file_path: strmodel: CPMBeeexport_files: Optional[List[str]] = None): + bmt.save(modelfile_path) + if export_files is not None: + export_files.append(file_path) + + +@rename_if_exists_decorator +def torch_save(file_path: strobj: objectexport_files: Optional[List[str]] = None): + torch.save(objfile_path) + if export_files is not None: + export_files.append(file_path) + + +@rename_if_exists_decorator +def on_save(file_path: strobj: objectexport_files: Optional[List[str]] = None): + with open(file_path"w") as data_f: + on.dump(objdata_f) + if export_files is not None: + export_files.append(file_path) diff --git a/cpm-live/cpm_live/utils/gradient_shrink.py b/cpm-live/cpm_live/utils/gradient_shrink.py new file mode 100644 index 0000000..7354d73 --- /dev/null +++ b/cpm-live/cpm_live/utils/gradient_shrink.py @@ -0,0 +1,16 @@ +import torch + + +class OpGradientShrink(torch.autograd.Function): + @staticmethod + def forward(ctxx: torch.Tensoralpha: float): + ctx.alpha = alpha + return x + + @staticmethod + def backward(ctxgrad_output): + return grad_output * ctx.alphaNone + + +def gradient_shrink(x: torch.Tensoralpha: float = 0.1): + return OpGradientShrink.apply(xalpha) diff --git a/cpm-live/cpm_live/utils/log.py b/cpm-live/cpm_live/utils/log.py new file mode 100644 index 0000000..55873ad --- /dev/null +++ b/cpm-live/cpm_live/utils/log.py @@ -0,0 +1,98 @@ +import os +import sys +from typing import AnyDictOptionalTupleUnion +import datetime +import on +import logging +import bmtrain as bmt + + +# Set up the common logger +def _get_logger(): + log = logging.getLogger('__name__') + log.setLevel(logging.INFO) + console_handle = logging.StreamHandler(sys.stdout) + node_name = os.getenv("NODE_NAME"str(bmt.rank())) + console_handle.setFormatter( + logging.Formatter( + '[%(levelname)s][%(asctime)s][{}][%(filename)s:%(lineno)d:%(process)d] - %(message)s' + .format(node_name), + datefmt='%Y-%m-%d %H:%M:%S' + ) + ) + log.addHandler(console_handle) + return log + + +logger = _get_logger() + + +class LogManager: + def __init__(selfpath: str): + if not os.path.exists(path): + os.makedirs(path) + self.path = path + + now = self.get_log_time() + latest_log: Union[Dict[strAny]None] = None + for _ in range(15): + log_name = self.get_log_name(now) + if os.path.exists(log_name): + with open(log_name"r") as flog: + latest_log = on.loads(flog.readlines()[-1]) # get last log + break + now -= datetime.timedelta(days=1) + + if latest_log is None: + self.global_token_pass = 0 + else: + self.global_token_pass = latest_log["token pass"] + + def get_log_time(self) -> datetime.datetime: + return datetime.datetime.utcnow() + datetime.timedelta(hours=16) + + def get_log_name(selfnow: Optional[datetime.datetime] = None): + if now is None: + now = self.get_log_time() + return os.path.join(self.path"log.%s.txt" % now.strftime("%Y%m%d")) + + def write( + self, + time: float, + iteration: int, + loss: float, + lr: float, + lr_scale: float, + time_usage: Dict[strfloat], + mem_usage: Dict[strTuple[floatfloat]], + avg_time: float, + token_max: float, + token_pass: float, + throughout: float, + grad_norm: float, + mask_max: float, + num_gpus: int, + task_loss: Dict[strfloat], + model_inspect: Optional[Any] = None, + ): + with open(self.get_log_name()"a") as fp: + ret = { + "time": time, + "iter": iteration, + "loss": loss, + "lr": lr, + "lr scale": int(lr_scale), + "time usage": time_usage, + "mem usage": mem_usage, + "avg time (s)": avg_time, + "token/max": token_max, + "token pass": token_pass + self.global_token_pass, + "throughout (token/s)": throughout, + "grad_norm": grad_norm, + "mask/max": mask_max, + "num_gpus": num_gpus, + "task_loss": task_loss, + } + if model_inspect is not None: + ret["model_inspect"] = model_inspect + fp.write(on.dumps(retensure_ascii=False) + "\n") diff --git a/cpm-live/cpm_live/utils/object.py b/cpm-live/cpm_live/utils/object.py new file mode 100644 index 0000000..0b32e68 --- /dev/null +++ b/cpm-live/cpm_live/utils/object.py @@ -0,0 +1,28 @@ +import bmtrain as bmt +import pickle +import torch + + +def allgather_objects(obj): + if bmt.world_size() == 1: + return [obj] + + with torch.no_grad(): + data_bytes: bytes = pickle.dumps(obj) + data_length: int = len(data_bytes) + + gpu_data_length = torch.tensor([data_length]device="cuda"dtype=torch.long) + gathered_length = bmt.distributed.all_gather(gpu_data_length).view(-1).cpu() + max_data_length = gathered_length.max().item() + + gpu_data_bytes = torch.zeros(max_data_lengthdtype=torch.uint8device="cuda") + byte_storage = torch.ByteStorage.from_buffer(data_bytes) + gpu_data_bytes[:data_length] = torch.ByteTensor(byte_storage) + + gathered_data = bmt.distributed.all_gather(gpu_data_bytes).cpu() + + ret = [] + for i in range(gathered_data.size(0)): + data_bytes = gathered_data[i: gathered_length[i].item()].numpy().tobytes() + ret.append(pickle.loads(data_bytes)) + return ret diff --git a/cpm-live/cpm_live/vocabs/bee.txt b/cpm-live/cpm_live/vocabs/bee.txt new file mode 100644 index 0000000..b977bda --- /dev/null +++ b/cpm-live/cpm_live/vocabs/bee.txt @@ -0,0 +1,86583 @@ + + + + + + + + + + + +! +" +# +$ +% +& +' +( +) +* ++ +, +- +. +.. +... +/ +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +: +; +< += +> +? +@ +A +B +C +D +E +F +G +H +I +J +K +L +M +N +O +P +Q +R +S +T +U +V +W +X +Y +Z +[ +\ +] +^ +_ +__ +` +a +b +c +d +e +f +g +h +i +j +k +l +m +n +o +p +q +r +s +t +u +v +w +x +y +z +{ +| +} +~ +¢ +£ +¤ +¥ +§ +© +¬ +® +° +± +´ +· +× +Ü +÷ +ü +Α +Β +Γ +Δ +Ε +Ζ +Η +Θ +Ι +Κ +Λ +Μ +Ν +Ξ +Ο +Π +Ρ +Σ +Τ +Υ +Φ +Χ +Ψ +Ω +α +β +γ +δ +ε +ζ +η +θ +ι +κ +λ +μ +ν +ξ +ο +π +ρ +ς +σ +τ +υ +φ +χ +ψ +ω +– +— +‘ +’ +“ +” +† +‡ +• +‣ +‰ +‱ +′ +※ +← +↑ +→ +↓ +↔ +↕ +↖ +↗ +↘ +↙ +∀ +∂ +∃ +∅ +∆ +∇ +∈ +∉ +√ +∝ +∞ +∠ +∧ +∨ +∩ +∪ +∫ +∴ +∵ +≈ +≠ +≤ +≥ +⊂ +⊆ +⊕ +⊙ +■ +▪ +◆ +◇ +○ +◎ +● +◦ +★ +☆ +♀ +♂ +、 +。 +〃 +〈 +〉 +《 +》 +【 +】 +！ +， +： +； +？ +㈰ +〇 +一 +乙 +二 +十 +丁 +厂 +七 +卜 +八 +人 +入 +儿 +匕 +几 +九 +刁 +了 +刀 +力 +乃 +又 +三 +干 +于 +亏 +工 +土 +士 +才 +下 +寸 +大 +丈 +与 +万 +上 +小 +口 +山 +巾 +千 +乞 +川 +亿 +个 +夕 +久 +么 +勺 +凡 +丸 +及 +广 +亡 +门 +丫 +义 +之 +尸 +己 +已 +巳 +弓 +子 +卫 +也 +女 +刃 +飞 +习 +叉 +马 +乡 +丰 +王 +开 +井 +天 +夫 +元 +无 +云 +专 +丐 +扎 +艺 +木 +五 +支 +厅 +不 +犬 +太 +区 +历 +歹 +友 +尤 +匹 +车 +巨 +牙 +屯 +戈 +比 +互 +切 +瓦 +止 +少 +曰 +日 +中 +贝 +冈 +内 +水 +见 +午 +牛 +手 +气 +毛 +壬 +升 +夭 +长 +仁 +什 +片 +仆 +化 +仇 +币 +仍 +仅 +斤 +爪 +反 +介 +父 +从 +仑 +今 +凶 +分 +乏 +公 +仓 +月 +氏 +勿 +欠 +风 +丹 +匀 +乌 +勾 +凤 +六 +文 +亢 +方 +火 +为 +斗 +忆 +计 +订 +户 +认 +冗 +讥 +心 +尺 +引 +丑 +巴 +孔 +队 +办 +以 +允 +予 +邓 +劝 +双 +书 +幻 +玉 +刊 +未 +末 +示 +击 +打 +巧 +正 +扑 +卉 +扒 +功 +扔 +去 +甘 +世 +艾 +古 +节 +本 +术 +可 +丙 +左 +厉 +石 +右 +布 +夯 +戊 +龙 +平 +灭 +轧 +东 +卡 +北 +占 +凸 +卢 +业 +旧 +帅 +归 +旦 +目 +且 +叶 +甲 +申 +叮 +电 +号 +田 +由 +只 +叭 +史 +央 +兄 +叽 +叼 +叫 +叩 +叨 +另 +叹 +冉 +皿 +凹 +囚 +四 +生 +矢 +失 +乍 +禾 +丘 +付 +仗 +代 +仙 +们 +仪 +白 +仔 +他 +斥 +瓜 +乎 +丛 +令 +用 +甩 +印 +尔 +乐 +句 +匆 +册 +卯 +犯 +外 +处 +冬 +鸟 +务 +包 +饥 +主 +市 +立 +冯 +玄 +闪 +兰 +半 +汁 +汇 +头 +汉 +宁 +穴 +它 +讨 +写 +让 +礼 +训 +议 +必 +讯 +记 +永 +司 +尼 +民 +弗 +弘 +出 +辽 +奶 +奴 +召 +加 +皮 +边 +孕 +发 +圣 +对 +台 +矛 +纠 +母 +幼 +丝 +邦 +式 +迂 +刑 +戎 +动 +扛 +寺 +吉 +扣 +考 +托 +老 +巩 +圾 +执 +扩 +扫 +地 +场 +扬 +耳 +芋 +共 +芒 +亚 +芝 +朽 +朴 +机 +权 +过 +臣 +吏 +再 +协 +西 +压 +厌 +戌 +在 +百 +有 +存 +而 +页 +匠 +夸 +夺 +灰 +达 +列 +死 +成 +夹 +夷 +轨 +邪 +尧 +划 +迈 +毕 +至 +此 +贞 +师 +尘 +尖 +劣 +光 +当 +早 +吁 +吐 +吓 +虫 +曲 +团 +吕 +同 +吊 +吃 +因 +吸 +吗 +吆 +屿 +屹 +岁 +帆 +回 +岂 +则 +刚 +网 +肉 +年 +朱 +先 +丢 +廷 +舌 +竹 +迁 +乔 +迄 +伟 +传 +乒 +乓 +休 +伍 +伏 +优 +臼 +伐 +延 +仲 +件 +任 +伤 +价 +伦 +份 +华 +仰 +仿 +伙 +伪 +自 +伊 +血 +向 +似 +后 +行 +舟 +全 +会 +杀 +合 +兆 +企 +众 +爷 +伞 +创 +肌 +肋 +朵 +杂 +危 +旬 +旨 +旭 +负 +匈 +名 +各 +多 +争 +色 +壮 +冲 +妆 +冰 +庄 +庆 +亦 +刘 +齐 +交 +衣 +次 +产 +决 +亥 +充 +妄 +闭 +问 +闯 +羊 +并 +关 +米 +灯 +州 +汗 +污 +江 +汛 +池 +汝 +汤 +忙 +兴 +宇 +守 +宅 +字 +安 +讲 +讳 +军 +讶 +许 +讹 +论 +讼 +农 +讽 +设 +访 +诀 +寻 +那 +迅 +尽 +导 +异 +弛 +孙 +阵 +阳 +收 +阶 +阴 +防 +奸 +如 +妇 +妃 +好 +她 +妈 +戏 +羽 +观 +欢 +买 +红 +驮 +纤 +驯 +约 +级 +纪 +驰 +纫 +巡 +寿 +弄 +麦 +玖 +玛 +形 +进 +戒 +吞 +远 +违 +韧 +运 +扶 +抚 +坛 +技 +坏 +抠 +扰 +扼 +拒 +找 +批 +址 +扯 +走 +抄 +贡 +汞 +坝 +攻 +赤 +折 +抓 +扳 +抡 +扮 +抢 +孝 +坎 +均 +抑 +抛 +投 +坟 +坑 +抗 +坊 +抖 +护 +壳 +志 +块 +扭 +声 +把 +报 +拟 +却 +抒 +劫 +芙 +芜 +苇 +芽 +花 +芹 +芥 +芬 +苍 +芳 +严 +芦 +芯 +劳 +克 +芭 +苏 +杆 +杠 +杜 +材 +村 +杖 +杏 +杉 +巫 +极 +李 +杨 +求 +甫 +匣 +更 +束 +吾 +豆 +两 +酉 +丽 +医 +辰 +励 +否 +还 +尬 +歼 +来 +连 +轩 +步 +卤 +坚 +肖 +旱 +盯 +呈 +时 +吴 +助 +县 +里 +呆 +吱 +吠 +呕 +园 +旷 +围 +呀 +吨 +足 +邮 +男 +困 +吵 +串 +员 +呐 +听 +吟 +吩 +呛 +吻 +吹 +呜 +吭 +吧 +邑 +吼 +囤 +别 +吮 +岖 +岗 +帐 +财 +针 +钉 +牡 +告 +我 +乱 +利 +秃 +秀 +私 +每 +兵 +估 +体 +何 +佐 +佑 +但 +伸 +佃 +作 +伯 +伶 +佣 +低 +你 +住 +位 +伴 +身 +皂 +伺 +佛 +囱 +近 +彻 +役 +返 +余 +希 +坐 +谷 +妥 +含 +邻 +岔 +肝 +肛 +肚 +肘 +肠 +龟 +甸 +免 +狂 +犹 +狈 +角 +删 +条 +彤 +卵 +灸 +岛 +刨 +迎 +饭 +饮 +系 +言 +冻 +状 +亩 +况 +床 +库 +庇 +疗 +吝 +应 +这 +冷 +庐 +序 +辛 +弃 +冶 +忘 +闰 +闲 +间 +闷 +判 +兑 +灶 +灿 +灼 +弟 +汪 +沐 +沛 +汰 +沥 +沙 +汽 +沃 +沦 +汹 +泛 +沧 +没 +沟 +沪 +沈 +沉 +沁 +怀 +忧 +忱 +快 +完 +宋 +宏 +牢 +究 +穷 +灾 +良 +证 +启 +评 +补 +初 +社 +祀 +识 +诈 +诉 +罕 +诊 +词 +译 +君 +灵 +即 +层 +屁 +尿 +尾 +迟 +局 +改 +张 +忌 +际 +陆 +阿 +陈 +阻 +附 +坠 +妓 +妙 +妖 +姊 +妨 +妒 +努 +忍 +劲 +矣 +鸡 +纬 +驱 +纯 +纱 +纲 +纳 +驳 +纵 +纷 +纸 +纹 +纺 +驴 +纽 +奉 +玩 +环 +武 +青 +责 +现 +玫 +表 +规 +抹 +卦 +坷 +坯 +拓 +拢 +拔 +坪 +拣 +坦 +担 +坤 +押 +抽 +拐 +拖 +者 +拍 +顶 +拆 +拎 +拥 +抵 +拘 +势 +抱 +拄 +垃 +拉 +拦 +幸 +拌 +拧 +拂 +拙 +招 +坡 +披 +拨 +择 +抬 +拇 +拗 +其 +取 +茉 +苦 +昔 +苛 +若 +茂 +苹 +苗 +英 +苟 +苑 +苞 +范 +直 +茁 +茄 +茎 +苔 +茅 +枉 +林 +枝 +杯 +枢 +柜 +枚 +析 +板 +松 +枪 +枫 +构 +杭 +杰 +述 +枕 +丧 +或 +画 +卧 +事 +刺 +枣 +雨 +卖 +郁 +矾 +矿 +码 +厕 +奈 +奔 +奇 +奋 +态 +欧 +殴 +垄 +妻 +轰 +顷 +转 +斩 +轮 +软 +到 +非 +叔 +歧 +肯 +齿 +些 +卓 +虎 +虏 +肾 +贤 +尚 +旺 +具 +味 +果 +昆 +国 +哎 +咕 +昌 +呵 +畅 +明 +易 +咙 +昂 +迪 +典 +固 +忠 +呻 +咒 +咋 +咐 +呼 +鸣 +咏 +呢 +咄 +咖 +岸 +岩 +帖 +罗 +帜 +帕 +岭 +凯 +败 +账 +贩 +贬 +购 +贮 +图 +钓 +制 +知 +迭 +氛 +垂 +牧 +物 +乖 +刮 +秆 +和 +季 +委 +秉 +佳 +侍 +岳 +供 +使 +例 +侠 +侥 +版 +侄 +侦 +侣 +侧 +凭 +侨 +佩 +货 +侈 +依 +卑 +的 +迫 +质 +欣 +征 +往 +爬 +彼 +径 +所 +舍 +金 +刹 +命 +肴 +斧 +爸 +采 +觅 +受 +乳 +贪 +念 +贫 +忿 +肤 +肺 +肢 +肿 +胀 +朋 +股 +肮 +肪 +肥 +服 +胁 +周 +昏 +鱼 +兔 +狐 +忽 +狗 +狞 +备 +饰 +饱 +饲 +变 +京 +享 +庞 +店 +夜 +庙 +府 +底 +疟 +疙 +疚 +剂 +卒 +郊 +庚 +废 +净 +盲 +放 +刻 +育 +氓 +闸 +闹 +郑 +券 +卷 +单 +炬 +炒 +炊 +炕 +炎 +炉 +沫 +浅 +法 +泄 +沽 +河 +沾 +泪 +沮 +油 +泊 +沿 +泡 +注 +泣 +泞 +泻 +泌 +泳 +泥 +沸 +沼 +波 +泼 +泽 +治 +怔 +怯 +怖 +性 +怕 +怜 +怪 +怡 +学 +宝 +宗 +定 +宠 +宜 +审 +宙 +官 +空 +帘 +宛 +实 +试 +郎 +诗 +肩 +房 +诚 +衬 +衫 +视 +祈 +话 +诞 +诡 +询 +该 +详 +建 +肃 +录 +隶 +帚 +屉 +居 +届 +刷 +屈 +弧 +弥 +弦 +承 +孟 +陋 +陌 +孤 +陕 +降 +函 +限 +妹 +姑 +姐 +姓 +妮 +始 +姆 +迢 +驾 +叁 +参 +艰 +线 +练 +组 +绅 +细 +驶 +织 +驹 +终 +驻 +绊 +驼 +绍 +绎 +经 +贯 +契 +贰 +奏 +春 +帮 +玷 +珍 +玲 +珊 +玻 +毒 +型 +拭 +挂 +封 +持 +拷 +拱 +项 +垮 +挎 +城 +挟 +挠 +政 +赴 +赵 +挡 +拽 +哉 +挺 +括 +垢 +拴 +拾 +挑 +垛 +指 +垫 +挣 +挤 +拼 +挖 +按 +挥 +挪 +拯 +某 +甚 +荆 +茸 +革 +茬 +荐 +巷 +带 +草 +茧 +茵 +茶 +荒 +茫 +荡 +荣 +荤 +荧 +故 +胡 +荫 +荔 +南 +药 +标 +栈 +柑 +枯 +柄 +栋 +相 +查 +柏 +栅 +柳 +柱 +柿 +栏 +柠 +树 +勃 +要 +柬 +咸 +威 +歪 +研 +砖 +厘 +厚 +砌 +砂 +泵 +砚 +砍 +面 +耐 +耍 +牵 +鸥 +残 +殃 +轴 +轻 +鸦 +皆 +韭 +背 +战 +点 +虐 +临 +览 +竖 +省 +削 +尝 +昧 +盹 +是 +盼 +眨 +哇 +哄 +哑 +显 +冒 +映 +星 +昨 +咧 +昭 +畏 +趴 +胃 +贵 +界 +虹 +虾 +蚁 +思 +蚂 +虽 +品 +咽 +骂 +勋 +哗 +咱 +响 +哈 +哆 +咬 +咳 +咪 +哪 +哟 +炭 +峡 +罚 +贱 +贴 +贻 +骨 +幽 +钙 +钝 +钞 +钟 +钢 +钠 +钥 +钦 +钧 +钩 +钮 +卸 +缸 +拜 +看 +矩 +毡 +氢 +怎 +牲 +选 +适 +秒 +香 +种 +秋 +科 +重 +复 +竿 +段 +便 +俩 +贷 +顺 +修 +俏 +保 +促 +俄 +俐 +侮 +俭 +俗 +俘 +信 +皇 +泉 +鬼 +侵 +禹 +侯 +追 +俊 +盾 +待 +徊 +衍 +律 +很 +须 +叙 +剑 +逃 +食 +盆 +胚 +胧 +胆 +胜 +胞 +胖 +脉 +胎 +勉 +狭 +狮 +独 +狰 +狡 +狱 +狠 +贸 +怨 +急 +饵 +饶 +蚀 +饺 +饼 +峦 +弯 +将 +奖 +哀 +亭 +亮 +度 +迹 +庭 +疮 +疯 +疫 +疤 +咨 +姿 +亲 +音 +帝 +施 +闺 +闻 +闽 +阀 +阁 +差 +养 +美 +姜 +叛 +送 +类 +迷 +籽 +娄 +前 +首 +逆 +兹 +总 +炼 +炸 +烁 +炮 +炫 +烂 +剃 +洼 +洁 +洪 +洒 +柒 +浇 +浊 +洞 +测 +洗 +活 +派 +洽 +染 +洛 +浏 +济 +洋 +洲 +浑 +浓 +津 +恃 +恒 +恢 +恍 +恬 +恤 +恰 +恼 +恨 +举 +觉 +宣 +宦 +室 +宫 +宪 +突 +穿 +窃 +客 +诫 +冠 +诬 +语 +扁 +袄 +祖 +神 +祝 +祠 +误 +诱 +诲 +说 +诵 +垦 +退 +既 +屋 +昼 +屏 +屎 +费 +陡 +逊 +眉 +孩 +陨 +除 +险 +院 +娃 +姥 +姨 +姻 +娇 +姚 +娜 +怒 +架 +贺 +盈 +勇 +怠 +癸 +蚤 +柔 +垒 +绑 +绒 +结 +绕 +骄 +绘 +给 +绚 +骆 +络 +绝 +绞 +骇 +统 +耕 +耘 +耗 +耙 +艳 +泰 +秦 +珠 +班 +素 +匿 +蚕 +顽 +盏 +匪 +捞 +栽 +捕 +埂 +捂 +振 +载 +赶 +起 +盐 +捎 +捍 +捏 +埋 +捉 +捆 +捐 +损 +袁 +捌 +都 +哲 +逝 +捡 +挫 +换 +挽 +挚 +热 +恐 +捣 +壶 +捅 +埃 +挨 +耻 +耿 +耽 +聂 +恭 +莽 +莱 +莲 +莫 +莉 +荷 +获 +晋 +恶 +莹 +莺 +真 +框 +梆 +桂 +桔 +栖 +档 +桐 +株 +桥 +桦 +栓 +桃 +格 +桩 +校 +核 +样 +根 +索 +哥 +速 +逗 +栗 +贾 +酌 +配 +翅 +辱 +唇 +夏 +砸 +砰 +砾 +础 +破 +原 +套 +逐 +烈 +殊 +殉 +顾 +轿 +较 +顿 +毙 +致 +柴 +桌 +虑 +监 +紧 +党 +逞 +晒 +眠 +晓 +哮 +唠 +鸭 +晃 +哺 +晌 +剔 +晕 +蚌 +畔 +蚣 +蚊 +蚪 +蚓 +哨 +哩 +圃 +哭 +哦 +恩 +鸯 +唤 +唁 +哼 +唧 +啊 +唉 +唆 +罢 +峭 +峨 +峰 +圆 +峻 +贼 +贿 +赂 +赃 +钱 +钳 +钻 +钾 +铁 +铃 +铅 +缺 +氧 +氨 +特 +牺 +造 +乘 +敌 +秤 +租 +积 +秧 +秩 +称 +秘 +透 +笔 +笑 +笋 +债 +借 +值 +倚 +俺 +倾 +倒 +倘 +俱 +倡 +候 +赁 +俯 +倍 +倦 +健 +臭 +射 +躬 +息 +倔 +徒 +徐 +殷 +舰 +舱 +般 +航 +途 +拿 +耸 +爹 +舀 +爱 +豺 +豹 +颁 +颂 +翁 +胰 +脆 +脂 +胸 +胳 +脏 +脐 +胶 +脑 +脓 +逛 +狸 +狼 +卿 +逢 +鸵 +留 +鸳 +皱 +饿 +馁 +凌 +凄 +恋 +桨 +浆 +衰 +衷 +高 +郭 +席 +准 +座 +症 +病 +疾 +斋 +疹 +疼 +疲 +脊 +效 +离 +紊 +唐 +瓷 +资 +凉 +站 +剖 +竞 +部 +旁 +旅 +畜 +阅 +羞 +羔 +瓶 +拳 +粉 +料 +益 +兼 +烤 +烘 +烦 +烧 +烛 +烟 +烙 +递 +涛 +浙 +涝 +浦 +酒 +涉 +消 +涡 +浩 +海 +涂 +浴 +浮 +涣 +涤 +流 +润 +涧 +涕 +浪 +浸 +涨 +烫 +涩 +涌 +悖 +悟 +悄 +悍 +悔 +悯 +悦 +害 +宽 +家 +宵 +宴 +宾 +窍 +窄 +容 +宰 +案 +请 +朗 +诸 +诺 +读 +扇 +诽 +袜 +袖 +袍 +被 +祥 +课 +冥 +谁 +调 +冤 +谅 +谆 +谈 +谊 +剥 +恳 +展 +剧 +屑 +弱 +陵 +祟 +陶 +陷 +陪 +娱 +娟 +恕 +娥 +娘 +通 +能 +难 +预 +桑 +绢 +绣 +验 +继 +骏 +球 +琐 +理 +琉 +琅 +捧 +堵 +措 +描 +域 +捺 +掩 +捷 +排 +焉 +掉 +捶 +赦 +堆 +推 +埠 +掀 +授 +捻 +教 +掏 +掐 +掠 +掂 +培 +接 +掷 +控 +探 +据 +掘 +掺 +职 +基 +聆 +勘 +聊 +娶 +著 +菱 +勒 +黄 +菲 +萌 +萝 +菌 +萎 +菜 +萄 +菊 +菩 +萍 +菠 +萤 +营 +乾 +萧 +萨 +菇 +械 +彬 +梦 +婪 +梗 +梧 +梢 +梅 +检 +梳 +梯 +桶 +梭 +救 +曹 +副 +票 +酝 +酗 +厢 +戚 +硅 +硕 +奢 +盔 +爽 +聋 +袭 +盛 +匾 +雪 +辅 +辆 +颅 +虚 +彪 +雀 +堂 +常 +眶 +匙 +晨 +睁 +眯 +眼 +悬 +野 +啪 +啦 +曼 +晦 +晚 +啄 +啡 +距 +趾 +啃 +跃 +略 +蚯 +蛀 +蛇 +唬 +累 +鄂 +唱 +患 +啰 +唾 +唯 +啤 +啥 +啸 +崖 +崎 +崭 +逻 +崔 +帷 +崩 +崇 +崛 +婴 +圈 +铐 +铛 +铝 +铜 +铭 +铲 +银 +矫 +甜 +秸 +梨 +犁 +秽 +移 +笨 +笼 +笛 +笙 +符 +第 +敏 +做 +袋 +悠 +偿 +偶 +偎 +偷 +您 +售 +停 +偏 +躯 +兜 +假 +衅 +徘 +徙 +得 +衔 +盘 +舶 +船 +舵 +斜 +盒 +鸽 +敛 +悉 +欲 +彩 +领 +脚 +脖 +脯 +豚 +脸 +脱 +象 +够 +逸 +猜 +猪 +猎 +猫 +凰 +猖 +猛 +祭 +馅 +馆 +凑 +减 +毫 +烹 +庶 +麻 +庵 +痊 +痒 +痕 +廊 +康 +庸 +鹿 +盗 +章 +竟 +商 +族 +旋 +望 +率 +阎 +阐 +着 +羚 +盖 +眷 +粘 +粗 +粒 +断 +剪 +兽 +焊 +焕 +清 +添 +鸿 +淋 +涯 +淹 +渠 +渐 +淑 +淌 +混 +淮 +淆 +渊 +淫 +渔 +淘 +淳 +液 +淤 +淡 +淀 +深 +涮 +涵 +婆 +梁 +渗 +情 +惜 +惭 +悼 +惧 +惕 +惟 +惊 +惦 +悴 +惋 +惨 +惯 +寇 +寅 +寄 +寂 +宿 +窒 +窑 +密 +谋 +谍 +谎 +谐 +袱 +祷 +祸 +谓 +谚 +谜 +逮 +敢 +尉 +屠 +弹 +隋 +堕 +随 +蛋 +隅 +隆 +隐 +婚 +婶 +婉 +颇 +颈 +绩 +绪 +续 +骑 +绰 +绳 +维 +绵 +绷 +绸 +综 +绽 +绿 +缀 +巢 +琴 +琳 +琢 +琼 +斑 +替 +揍 +款 +堪 +塔 +搭 +堰 +揩 +越 +趁 +趋 +超 +揽 +堤 +提 +博 +揭 +喜 +彭 +揣 +插 +揪 +搜 +煮 +援 +搀 +裁 +搁 +搓 +搂 +搅 +壹 +握 +搔 +揉 +斯 +期 +欺 +联 +葫 +散 +惹 +葬 +募 +葛 +董 +葡 +敬 +葱 +蒋 +蒂 +落 +韩 +朝 +辜 +葵 +棒 +棱 +棋 +椰 +植 +森 +焚 +椅 +椒 +棵 +棍 +椎 +棉 +棚 +棕 +棺 +榔 +椭 +惠 +惑 +逼 +粟 +棘 +酣 +酥 +厨 +厦 +硬 +硝 +确 +硫 +雁 +殖 +裂 +雄 +颊 +雳 +暂 +雅 +翘 +辈 +悲 +紫 +凿 +辉 +敞 +棠 +赏 +掌 +晴 +睐 +暑 +最 +晰 +量 +鼎 +喷 +喳 +晶 +喇 +遇 +喊 +遏 +晾 +景 +畴 +践 +跋 +跌 +跑 +跛 +遗 +蛙 +蛛 +蜓 +蜒 +蛤 +喝 +鹃 +喂 +喘 +喉 +喻 +啼 +喧 +嵌 +幅 +帽 +赋 +赌 +赎 +赐 +赔 +黑 +铸 +铺 +链 +销 +锁 +锄 +锅 +锈 +锋 +锌 +锐 +甥 +掰 +短 +智 +氮 +毯 +氯 +鹅 +剩 +稍 +程 +稀 +税 +筐 +等 +筑 +策 +筛 +筒 +筏 +答 +筋 +筝 +傲 +傅 +牌 +堡 +集 +焦 +傍 +储 +皓 +皖 +粤 +奥 +街 +惩 +御 +循 +艇 +舒 +逾 +番 +释 +禽 +腊 +脾 +腋 +腔 +腕 +鲁 +猩 +猬 +猾 +猴 +惫 +然 +馈 +馋 +装 +蛮 +就 +敦 +斌 +痘 +痢 +痪 +痛 +童 +竣 +阔 +善 +翔 +羡 +普 +粪 +尊 +奠 +道 +遂 +曾 +焰 +港 +滞 +湖 +湘 +渣 +渤 +渺 +湿 +温 +渴 +溃 +溅 +滑 +湃 +渝 +湾 +渡 +游 +滋 +渲 +溉 +愤 +慌 +惰 +愕 +愣 +惶 +愧 +愉 +慨 +割 +寒 +富 +寓 +窜 +窝 +窖 +窗 +窘 +遍 +雇 +裕 +裤 +裙 +禅 +禄 +谢 +谣 +谤 +谦 +犀 +属 +屡 +强 +粥 +疏 +隔 +隙 +隘 +媒 +絮 +嫂 +媚 +婿 +登 +缅 +缆 +缉 +缎 +缓 +缔 +缕 +骗 +编 +骚 +缘 +瑟 +鹉 +瑞 +瑰 +瑙 +魂 +肆 +摄 +摸 +填 +搏 +塌 +鼓 +摆 +携 +搬 +摇 +搞 +塘 +摊 +聘 +斟 +蒜 +勤 +靴 +靶 +鹊 +蓝 +墓 +幕 +蓬 +蓄 +蒲 +蓉 +蒙 +蒸 +献 +椿 +禁 +楚 +楷 +榄 +想 +槐 +榆 +楼 +概 +赖 +酪 +酬 +感 +碍 +碘 +碑 +碎 +碰 +碗 +碌 +尴 +雷 +零 +雾 +雹 +辐 +辑 +输 +督 +频 +龄 +鉴 +睛 +睹 +睦 +瞄 +睫 +睡 +睬 +嗜 +鄙 +嗦 +愚 +暖 +盟 +歇 +暗 +暇 +照 +畸 +跨 +跷 +跳 +跺 +跪 +路 +跤 +跟 +遣 +蜈 +蜗 +蛾 +蜂 +蜕 +嗅 +嗡 +嗓 +署 +置 +罪 +罩 +蜀 +幌 +错 +锚 +锡 +锣 +锤 +锥 +锦 +键 +锯 +锰 +矮 +辞 +稚 +稠 +颓 +愁 +筹 +签 +简 +筷 +毁 +舅 +鼠 +催 +傻 +像 +躲 +魁 +衙 +微 +愈 +遥 +腻 +腰 +腥 +腮 +腹 +腺 +鹏 +腾 +腿 +鲍 +猿 +颖 +触 +解 +煞 +雏 +馍 +馏 +酱 +禀 +痹 +廓 +痴 +痰 +廉 +靖 +新 +韵 +意 +誊 +粮 +数 +煎 +塑 +慈 +煤 +煌 +满 +漠 +滇 +源 +滤 +滥 +滔 +溪 +溜 +漓 +滚 +溢 +溯 +滨 +溶 +溺 +粱 +滩 +慎 +誉 +塞 +寞 +窥 +窟 +寝 +谨 +褂 +裸 +福 +谬 +群 +殿 +辟 +障 +媳 +嫉 +嫌 +嫁 +叠 +缚 +缝 +缠 +缤 +剿 +静 +碧 +璃 +赘 +熬 +墙 +墟 +嘉 +摧 +赫 +截 +誓 +境 +摘 +摔 +撇 +聚 +慕 +暮 +摹 +蔓 +蔑 +蔡 +蔗 +蔽 +蔼 +熙 +蔚 +兢 +模 +槛 +榴 +榜 +榨 +榕 +歌 +遭 +酵 +酷 +酿 +酸 +碟 +碱 +碳 +磁 +愿 +需 +辖 +辗 +雌 +裳 +颗 +瞅 +墅 +嗽 +踊 +蜻 +蜡 +蝇 +蜘 +蝉 +嘛 +嘀 +赚 +锹 +锻 +镀 +舞 +舔 +稳 +熏 +箕 +算 +箩 +管 +箫 +舆 +僚 +僧 +鼻 +魄 +魅 +貌 +膜 +膊 +膀 +鲜 +疑 +孵 +馒 +裹 +敲 +豪 +膏 +遮 +腐 +瘩 +瘟 +瘦 +辣 +彰 +竭 +端 +旗 +精 +粹 +歉 +弊 +熄 +熔 +煽 +潇 +漆 +漱 +漂 +漫 +滴 +漾 +演 +漏 +慢 +慷 +寨 +赛 +寡 +察 +蜜 +寥 +谭 +肇 +褐 +褪 +谱 +隧 +嫩 +翠 +熊 +凳 +骡 +缩 +慧 +撵 +撕 +撒 +撩 +趣 +趟 +撑 +撮 +撬 +播 +擒 +墩 +撞 +撤 +增 +撰 +聪 +鞋 +鞍 +蕉 +蕊 +蔬 +蕴 +横 +槽 +樱 +橡 +樟 +橄 +敷 +豌 +飘 +醋 +醇 +醉 +磕 +磊 +磅 +碾 +震 +霄 +霉 +瞒 +题 +暴 +瞎 +嘻 +嘶 +嘲 +嘹 +影 +踢 +踏 +踩 +踪 +蝶 +蝴 +蝠 +蝎 +蝌 +蝗 +蝙 +嘿 +嘱 +幢 +墨 +镇 +镐 +镑 +靠 +稽 +稻 +黎 +稿 +稼 +箱 +篓 +箭 +篇 +僵 +躺 +僻 +德 +艘 +膝 +膛 +鲤 +鲫 +熟 +摩 +褒 +瘪 +瘤 +瘫 +凛 +颜 +毅 +糊 +遵 +憋 +潜 +澎 +潮 +潭 +鲨 +澳 +潘 +澈 +澜 +澄 +懂 +憔 +懊 +憎 +额 +翩 +褥 +谴 +鹤 +憨 +慰 +劈 +履 +豫 +缭 +撼 +擂 +操 +擅 +燕 +蕾 +薯 +薛 +薇 +擎 +薪 +薄 +颠 +翰 +噩 +橱 +橙 +橘 +整 +融 +瓢 +醒 +霍 +霎 +辙 +冀 +餐 +嘴 +踱 +蹄 +蹂 +蟆 +螃 +器 +噪 +鹦 +赠 +默 +黔 +镜 +赞 +穆 +篮 +篡 +篷 +篱 +儒 +邀 +衡 +膨 +雕 +鲸 +磨 +瘾 +瘸 +凝 +辨 +辩 +糙 +糖 +糕 +燃 +濒 +澡 +激 +懒 +憾 +懈 +窿 +壁 +避 +缰 +缴 +戴 +擦 +藉 +鞠 +藏 +藐 +檬 +檐 +檀 +礁 +磷 +霜 +霞 +瞭 +瞧 +瞬 +瞳 +瞩 +瞪 +曙 +蹋 +蹈 +螺 +蟋 +蟀 +嚎 +赡 +穗 +魏 +簧 +簇 +繁 +徽 +爵 +朦 +臊 +鳄 +癌 +辫 +赢 +糟 +糠 +燥 +懦 +豁 +臀 +臂 +翼 +骤 +藕 +鞭 +藤 +覆 +瞻 +蹦 +嚣 +镰 +翻 +鳍 +鹰 +瀑 +襟 +璧 +戳 +孽 +警 +蘑 +藻 +攀 +曝 +蹲 +蹭 +蹬 +巅 +簸 +簿 +蟹 +颤 +靡 +癣 +瓣 +羹 +鳖 +爆 +疆 +鬓 +壤 +馨 +耀 +躁 +蠕 +嚼 +嚷 +巍 +籍 +鳞 +魔 +糯 +灌 +譬 +蠢 +霸 +露 +霹 +躏 +黯 +髓 +赣 +囊 +镶 +瓤 +罐 +矗 +乂 +乜 +兀 +弋 +孑 +孓 +幺 +亓 +韦 +廿 +丏 +卅 +仄 +厄 +仃 +仉 +仂 +兮 +刈 +爻 +卞 +闩 +讣 +尹 +夬 +爿 +毋 +邗 +邛 +艽 +艿 +札 +叵 +匝 +丕 +匜 +劢 +卟 +叱 +叻 +仨 +仕 +仟 +仡 +仫 +仞 +卮 +氐 +犰 +刍 +邝 +邙 +汀 +讦 +讧 +讪 +讫 +尻 +阡 +尕 +弁 +驭 +匡 +耒 +玎 +玑 +邢 +圩 +圬 +圭 +扦 +圪 +圳 +圹 +扪 +圮 +圯 +芊 +芍 +芄 +芨 +芑 +芎 +芗 +亘 +厍 +夼 +戍 +尥 +乩 +旯 +曳 +岌 +屺 +凼 +囡 +钇 +缶 +氘 +氖 +牝 +伎 +伛 +伢 +佤 +仵 +伥 +伧 +伉 +伫 +囟 +汆 +刖 +夙 +旮 +刎 +犷 +犸 +舛 +凫 +邬 +饧 +汕 +汔 +汐 +汲 +汜 +汊 +忖 +忏 +讴 +讵 +祁 +讷 +聿 +艮 +厾 +阱 +阮 +阪 +丞 +妁 +牟 +纡 +纣 +纥 +纨 +玕 +玙 +抟 +抔 +圻 +坂 +坍 +坞 +抃 +抉 +㧐 +芫 +邯 +芸 +芾 +苈 +苣 +芷 +芮 +苋 +芼 +苌 +苁 +芩 +芪 +芡 +芟 +苄 +苎 +苡 +杌 +杓 +杞 +杈 +忑 +孛 +邴 +邳 +矶 +奁 +豕 +忒 +欤 +轫 +迓 +邶 +忐 +卣 +邺 +旰 +呋 +呒 +呓 +呔 +呖 +呃 +旸 +吡 +町 +虬 +呗 +吽 +吣 +吲 +帏 +岐 +岈 +岘 +岑 +岚 +兕 +囵 +囫 +钊 +钋 +钌 +迕 +氙 +氚 +牤 +佞 +邱 +攸 +佚 +佝 +佟 +佗 +伽 +彷 +佘 +佥 +孚 +豸 +坌 +肟 +邸 +奂 +劬 +狄 +狁 +鸠 +邹 +饨 +饩 +饪 +饫 +饬 +亨 +庑 +庋 +疔 +疖 +肓 +闱 +闳 +闵 +羌 +炀 +沣 +沅 +沔 +沤 +沌 +沏 +沚 +汩 +汨 +沂 +汾 +沨 +汴 +汶 +沆 +沩 +泐 +怃 +怄 +忡 +忤 +忾 +怅 +忻 +忪 +怆 +忭 +忸 +诂 +诃 +诅 +诋 +诌 +诏 +诒 +孜 +陇 +陀 +陂 +陉 +妍 +妩 +妪 +妣 +妊 +妗 +妫 +妞 +姒 +妤 +邵 +劭 +刭 +甬 +邰 +纭 +纰 +纴 +纶 +纾 +玮 +玡 +玭 +玠 +玢 +玥 +玦 +盂 +忝 +匦 +坩 +抨 +拤 +坫 +拈 +垆 +抻 +劼 +拃 +拊 +坼 +坻 +㧟 +坨 +坭 +抿 +坳 +耶 +苷 +苯 +苤 +茏 +苫 +苜 +苴 +苒 +苘 +茌 +苻 +苓 +茚 +茆 +茑 +茓 +茔 +茕 +茀 +苕 +枥 +枇 +杪 +杳 +枧 +杵 +枨 +枞 +枋 +杻 +杷 +杼 +矸 +砀 +刳 +奄 +瓯 +殁 +郏 +轭 +郅 +鸢 +盱 +昊 +昙 +杲 +昃 +咂 +呸 +昕 +昀 +旻 +昉 +炅 +咔 +畀 +虮 +咀 +呷 +黾 +呱 +呤 +咚 +咆 +咛 +呶 +呣 +呦 +咝 +岢 +岿 +岬 +岫 +帙 +岣 +峁 +刿 +迥 +岷 +剀 +帔 +峄 +沓 +囹 +罔 +钍 +钎 +钏 +钒 +钕 +钗 +邾 +迮 +牦 +竺 +迤 +佶 +佬 +佰 +侑 +侉 +臾 +岱 +侗 +侃 +侏 +侩 +佻 +佾 +侪 +佼 +佯 +侬 +帛 +阜 +侔 +徂 +刽 +郄 +怂 +籴 +瓮 +戗 +肼 +䏝 +肽 +肱 +肫 +剁 +迩 +郇 +狙 +狎 +狍 +狒 +咎 +炙 +枭 +饯 +饴 +冽 +冼 +庖 +疠 +疝 +疡 +兖 +妾 +劾 +炜 +𬉼 +炖 +炘 +炝 +炔 +泔 +沭 +泷 +泸 +泱 +泅 +泗 +泠 +泺 +泖 +泫 +泮 +沱 +泯 +泓 +泾 +怙 +怵 +怦 +怛 +怏 +怍 +㤘 +怩 +怫 +怿 +宕 +穹 +宓 +诓 +诔 +诖 +诘 +戾 +诙 +戽 +郓 +衩 +祆 +祎 +祉 +祇 +诛 +诜 +诟 +诠 +诣 +诤 +诧 +诨 +诩 +戕 +孢 +亟 +陔 +妲 +妯 +姗 +帑 +弩 +孥 +驽 +虱 +迦 +迨 +绀 +绁 +绂 +驷 +驸 +绉 +绌 +驿 +骀 +甾 +珏 +珐 +珂 +珑 +玳 +珀 +顸 +珉 +珈 +拮 +垭 +挝 +垣 +挞 +垤 +赳 +贲 +垱 +垌 +郝 +垧 +垓 +挦 +垠 +茜 +荚 +荑 +贳 +荜 +莒 +茼 +茴 +茱 +莛 +荞 +茯 +荏 +荇 +荃 +荟 +荀 +茗 +荠 +茭 +茨 +垩 +荥 +荦 +荨 +荩 +剋 +荪 +茹 +荬 +荮 +柰 +栉 +柯 +柘 +栊 +柩 +枰 +栌 +柙 +枵 +柚 +枳 +柞 +柝 +栀 +柢 +栎 +枸 +柈 +柁 +枷 +柽 +剌 +酊 +郦 +甭 +砗 +砘 +砒 +斫 +砭 +砜 +奎 +耷 +虺 +殂 +殇 +殄 +殆 +轱 +轲 +轳 +轶 +轸 +虿 +毖 +觇 +尜 +哐 +眄 +眍 +𠳐 +郢 +眇 +眊 +眈 +禺 +哂 +咴 +曷 +昴 +昱 +昵 +咦 +哓 +哔 +畎 +毗 +呲 +胄 +畋 +畈 +虼 +虻 +盅 +咣 +哕 +剐 +郧 +咻 +囿 +咿 +哌 +哙 +哚 +咯 +咩 +咤 +哝 +哏 +哞 +峙 +峣 +罘 +帧 +峒 +峤 +峋 +峥 +贶 +钚 +钛 +钡 +钣 +钤 +钨 +钫 +钯 +氡 +氟 +牯 +郜 +秕 +秭 +竽 +笈 +笃 +俦 +俨 +俅 +俪 +叟 +垡 +牮 +俣 +俚 +皈 +俑 +俟 +逅 +徇 +徉 +舢 +俞 +郗 +俎 +郤 +爰 +郛 +瓴 +胨 +胪 +胛 +胂 +胙 +胍 +胗 +胝 +朐 +胫 +鸨 +匍 +狨 +狯 +飑 +狩 +狲 +訇 +逄 +昝 +饷 +饸 +饹 +胤 +孪 +娈 +弈 +奕 +庥 +疬 +疣 +疥 +疭 +庠 +竑 +彦 +飒 +闼 +闾 +闿 +阂 +羑 +迸 +籼 +酋 +炳 +炻 +炽 +炯 +烀 +炷 +烃 +洱 +洹 +洧 +洌 +浃 +洇 +洄 +洙 +涎 +洎 +洫 +浍 +洮 +洵 +浒 +浔 +浕 +洳 +恸 +恓 +恹 +恫 +恺 +恻 +恂 +恪 +恽 +宥 +扃 +衲 +衽 +衿 +袂 +祛 +祜 +祓 +祚 +诮 +祗 +祢 +诰 +诳 +鸩 +昶 +郡 +咫 +弭 +牁 +胥 +陛 +陟 +娅 +姮 +娆 +姝 +姣 +姘 +姹 +怼 +羿 +炱 +矜 +绔 +骁 +骅 +绗 +绛 +骈 +耖 +挈 +珥 +珙 +顼 +珰 +珩 +珧 +珣 +珞 +琤 +珲 +敖 +恚 +埔 +埕 +埘 +埙 +埚 +挹 +耆 +耄 +埒 +捋 +贽 +垸 +捃 +盍 +荸 +莆 +莳 +莴 +莪 +莠 +莓 +莜 +莅 +荼 +莩 +荽 +莸 +荻 +莘 +莎 +莞 +莨 +鸪 +莼 +栲 +栳 +郴 +桓 +桡 +桎 +桢 +桤 +梃 +栝 +桕 +桁 +桧 +桅 +栟 +桉 +栩 +逑 +逋 +彧 +鬲 +豇 +酐 +逦 +厝 +孬 +砝 +砹 +砺 +砧 +砷 +砟 +砼 +砥 +砣 +剞 +砻 +轼 +轾 +辂 +鸫 +趸 +龀 +鸬 +虔 +逍 +眬 +唛 +晟 +眩 +眙 +哧 +哽 +唔 +晁 +晏 +鸮 +趵 +趿 +畛 +蚨 +蚜 +蚍 +蚋 +蚬 +蚝 +蚧 +唢 +圄 +唣 +唏 +盎 +唑 +崂 +崃 +罡 +罟 +峪 +觊 +赅 +钰 +钲 +钴 +钵 +钹 +钺 +钽 +钼 +钿 +铀 +铂 +铄 +铆 +铈 +铉 +铊 +铋 +铌 +铍 +䥽 +铎 +氩 +氤 +氦 +毪 +舐 +秣 +秫 +盉 +笄 +笕 +笊 +笏 +笆 +俸 +倩 +俵 +偌 +俳 +俶 +倬 +倏 +恁 +倭 +倪 +俾 +倜 +隼 +隽 +倌 +倥 +臬 +皋 +郫 +倨 +衄 +颀 +徕 +舫 +釜 +奚 +衾 +胯 +胱 +胴 +胭 +脍 +胼 +朕 +脒 +胺 +鸱 +玺 +鸲 +狷 +猁 +狳 +猃 +狺 +逖 +桀 +袅 +饽 +凇 +栾 +挛 +亳 +疳 +疴 +疸 +疽 +痈 +疱 +痂 +痉 +衮 +凋 +颃 +恣 +旆 +旄 +旃 +阃 +阄 +訚 +阆 +恙 +粑 +朔 +郸 +烜 +烨 +烩 +烊 +剡 +郯 +烬 +涑 +浯 +涞 +涟 +娑 +涅 +涠 +浞 +涓 +浥 +涔 +浜 +浠 +浣 +浚 +悚 +悭 +悝 +悒 +悌 +悛 +宸 +窈 +剜 +诹 +冢 +诼 +袒 +袢 +祯 +诿 +谀 +谂 +谄 +谇 +屐 +屙 +陬 +勐 +奘 +牂 +蚩 +陲 +姬 +娠 +娌 +娉 +娲 +娩 +娴 +娣 +娓 +婀 +畚 +逡 +绠 +骊 +绡 +骋 +绥 +绦 +绨 +骎 +邕 +鸶 +彗 +耜 +焘 +舂 +琏 +琇 +麸 +揶 +埴 +埯 +捯 +掳 +掴 +埸 +埵 +赧 +埤 +捭 +逵 +埝 +堋 +堍 +掬 +鸷 +掖 +捽 +掊 +堉 +掸 +捩 +掮 +悫 +埭 +埽 +掇 +掼 +聃 +菁 +萁 +菘 +堇 +萘 +萋 +菽 +菖 +萜 +萸 +萑 +棻 +菔 +菟 +萏 +萃 +菏 +菹 +菪 +菅 +菀 +萦 +菰 +菡 +梵 +梿 +梏 +觋 +桴 +桷 +梓 +棁 +桫 +棂 +啬 +郾 +匮 +敕 +豉 +鄄 +酞 +酚 +戛 +硎 +硭 +硒 +硖 +硗 +硐 +硇 +硌 +鸸 +瓠 +匏 +厩 +龚 +殒 +殓 +殍 +赉 +雩 +辄 +堑 +眭 +眦 +啧 +晡 +晤 +眺 +眵 +眸 +圊 +喏 +喵 +啉 +勖 +晞 +唵 +晗 +冕 +啭 +畦 +趺 +啮 +跄 +蚶 +蛄 +蛎 +蛆 +蚰 +蛊 +圉 +蚱 +蛉 +蛏 +蚴 +啁 +啕 +唿 +啐 +唼 +唷 +啖 +啵 +啶 +啷 +唳 +唰 +啜 +帻 +崚 +崦 +帼 +崮 +崤 +崆 +赇 +赈 +赊 +铑 +铒 +铗 +铙 +铟 +铠 +铡 +铢 +铣 +铤 +铧 +铨 +铩 +铪 +铫 +铬 +铮 +铯 +铰 +铱 +铳 +铵 +铷 +氪 +牾 +鸹 +秾 +逶 +笺 +筇 +笸 +笪 +笮 +笠 +笥 +笤 +笳 +笾 +笞 +偾 +偃 +偕 +偈 +傀 +偬 +偻 +皑 +皎 +鸻 +徜 +舸 +舻 +舴 +舷 +龛 +翎 +脬 +脘 +脲 +匐 +猗 +猡 +猞 +猝 +斛 +猕 +馗 +馃 +馄 +鸾 +孰 +庹 +庾 +痔 +痍 +疵 +翊 +旌 +旎 +袤 +阇 +阈 +阉 +阊 +阋 +阍 +阏 +羟 +粝 +粕 +敝 +焐 +烯 +焓 +烽 +焖 +烷 +焗 +渍 +渚 +淇 +淅 +淞 +渎 +涿 +淖 +挲 +淠 +涸 +渑 +淦 +淝 +淬 +涪 +淙 +涫 +渌 +淄 +惬 +悻 +悱 +惝 +惘 +悸 +惆 +惚 +惇 +惮 +窕 +谌 +谏 +扈 +皲 +谑 +裆 +袷 +裉 +谒 +谔 +谕 +谖 +谗 +谙 +谛 +谝 +逯 +郿 +隈 +粜 +隍 +隗 +婧 +婊 +婕 +娼 +婢 +婵 +胬 +袈 +翌 +恿 +欸 +绫 +骐 +绮 +绯 +绱 +骒 +绲 +骓 +绶 +绺 +绻 +绾 +骖 +缁 +耠 +琫 +琵 +琶 +琪 +瑛 +琦 +琥 +琨 +靓 +琰 +琮 +琯 +琬 +琛 +琚 +辇 +鼋 +揳 +堞 +搽 +揸 +揠 +堙 +趄 +揖 +颉 +塄 +揿 +耋 +揄 +蛩 +蛰 +塆 +摒 +揆 +掾 +聒 +葑 +葚 +靰 +靸 +葳 +葺 +葸 +萼 +葆 +葩 +葶 +蒌 +萱 +戟 +葭 +楮 +棼 +椟 +棹 +椤 +棰 +赍 +椋 +椁 +椪 +棣 +椐 +鹁 +覃 +酤 +酢 +酡 +鹂 +厥 +殚 +殛 +雯 +雱 +辊 +辋 +椠 +辍 +辎 +斐 +睄 +睑 +睇 +睃 +戢 +喋 +嗒 +喃 +喱 +喹 +晷 +喈 +跖 +跗 +跞 +跚 +跎 +跏 +跆 +蛱 +蛲 +蛭 +蛳 +蛐 +蛔 +蛞 +蛴 +蛟 +蛘 +喁 +喟 +啾 +嗖 +喑 +嗟 +喽 +嗞 +喀 +喔 +喙 +嵘 +嵖 +崴 +遄 +詈 +嵎 +崽 +嵬 +嵛 +嵯 +嵝 +嵫 +幄 +嵋 +赕 +铻 +铼 +铿 +锃 +锂 +锆 +锇 +锉 +锏 +锑 +锒 +锔 +锕 +掣 +矬 +氰 +毳 +毽 +犊 +犄 +犋 +鹄 +犍 +嵇 +黍 +稃 +稂 +筚 +筵 +筌 +傣 +傈 +舄 +牍 +傥 +傧 +遑 +傩 +遁 +徨 +媭 +畲 +弑 +颌 +翕 +釉 +鹆 +舜 +貂 +腈 +腌 +腓 +腆 +腴 +腑 +腚 +腱 +鱿 +鲀 +鲂 +颍 +猢 +猹 +猥 +飓 +觞 +觚 +猱 +颎 +飧 +馇 +馊 +亵 +脔 +裒 +痣 +痨 +痦 +痞 +痤 +痫 +痧 +赓 +竦 +瓿 +啻 +颏 +鹇 +阑 +阒 +阕 +粞 +遒 +孳 +焯 +焜 +焙 +焱 +鹈 +湛 +渫 +湮 +湎 +湜 +渭 +湍 +湫 +溲 +湟 +溆 +湲 +湔 +湉 +渥 +湄 +滁 +愠 +惺 +愦 +惴 +愀 +愎 +愔 +喾 +寐 +谟 +扉 +裢 +裎 +裥 +祾 +祺 +谠 +幂 +谡 +谥 +谧 +遐 +孱 +弼 +巽 +骘 +媪 +媛 +婷 +巯 +翚 +皴 +婺 +骛 +缂 +缃 +缄 +彘 +缇 +缈 +缌 +缑 +缒 +缗 +飨 +耢 +瑚 +瑁 +瑜 +瑗 +瑄 +瑕 +遨 +骜 +韫 +髡 +塬 +鄢 +趔 +趑 +摅 +摁 +蜇 +搋 +搪 +搐 +搛 +搠 +摈 +彀 +毂 +搦 +搡 +蓁 +戡 +蓍 +鄞 +靳 +蓐 +蓦 +鹋 +蒽 +蓓 +蓖 +蓊 +蒯 +蓟 +蓑 +蒿 +蒺 +蓠 +蒟 +蒡 +蒹 +蒴 +蒗 +蓥 +颐 +楔 +楠 +楂 +楝 +楫 +楸 +椴 +槌 +楯 +皙 +榈 +槎 +榉 +楦 +楣 +楹 +椽 +裘 +剽 +甄 +酮 +酰 +酯 +酩 +蜃 +碛 +碓 +硼 +碉 +碚 +碇 +碜 +鹌 +辏 +龃 +龅 +訾 +粲 +虞 +睚 +嗪 +韪 +嗷 +嗉 +睨 +睢 +雎 +睥 +嘟 +嗑 +嗫 +嗬 +嗔 +嗝 +戥 +嗄 +煦 +暄 +遢 +暌 +跬 +跶 +跸 +跐 +跣 +跹 +跻 +蛸 +蜊 +蜍 +蜉 +蜣 +畹 +蛹 +嗣 +嗯 +嗥 +嗲 +嗳 +嗌 +嗍 +嗨 +嗐 +嗤 +嗵 +罨 +嵊 +嵩 +嵴 +骰 +锗 +锛 +锜 +锝 +锞 +锟 +锢 +锨 +锩 +锭 +锱 +雉 +氲 +犏 +歃 +稞 +稗 +稔 +筠 +筢 +筮 +筲 +筱 +牒 +煲 +敫 +徭 +愆 +艄 +觎 +毹 +貊 +貅 +貉 +颔 +腠 +腩 +腼 +腭 +腧 +塍 +媵 +詹 +鲅 +鲆 +鲇 +鲈 +稣 +鲋 +鲐 +肄 +鹐 +飕 +觥 +遛 +馐 +鹑 +亶 +瘃 +痱 +痼 +痿 +瘐 +瘁 +瘆 +麂 +裔 +歆 +旒 +雍 +阖 +阗 +阙 +羧 +豢 +粳 +猷 +煳 +煜 +煨 +煅 +煊 +煸 +煺 +滟 +溱 +溘 +漭 +滢 +溥 +溧 +溽 +裟 +溻 +溷 +滗 +滫 +溴 +滏 +滃 +滦 +溏 +滂 +滓 +溟 +滪 +愫 +慑 +慊 +鲎 +骞 +窦 +窠 +窣 +裱 +褚 +裨 +裾 +裰 +禊 +谩 +谪 +媾 +嫫 +媲 +嫒 +嫔 +媸 +缙 +缜 +缛 +辔 +骝 +缟 +缡 +缢 +缣 +骟 +耥 +璈 +瑶 +瑭 +獒 +觏 +慝 +嫠 +韬 +叆 +髦 +摽 +墁 +撂 +摞 +撄 +翥 +踅 +摭 +墉 +墒 +榖 +綦 +蔫 +蔷 +靺 +靼 +鞅 +靿 +甍 +蔸 +蔟 +蔺 +戬 +蕖 +蔻 +蓿 +斡 +鹕 +蓼 +榛 +榧 +榻 +榫 +榭 +槔 +榱 +槁 +槟 +槠 +榷 +僰 +酽 +酶 +酹 +厮 +碡 +碴 +碣 +碲 +磋 +臧 +豨 +殡 +霆 +霁 +辕 +蜚 +裴 +翡 +龇 +龈 +睿 +䁖 +睽 +嘞 +嘈 +嘌 +嘁 +嘎 +暧 +暝 +踌 +踉 +蜞 +蜥 +蜮 +蝈 +蜴 +蜱 +蜩 +蜷 +蜿 +螂 +蜢 +嘘 +嘡 +鹗 +嘣 +嘤 +嘚 +嗾 +嘧 +罴 +罱 +幔 +嶂 +幛 +赙 +罂 +骷 +骶 +鹘 +锲 +锴 +锶 +锷 +锸 +锵 +镁 +镂 +犒 +箐 +箦 +箧 +箍 +箸 +箬 +箅 +箪 +箔 +箜 +箢 +箓 +毓 +僖 +儆 +僳 +僭 +劁 +僮 +魃 +魆 +睾 +艋 +鄱 +膈 +膑 +鲑 +鲔 +鲚 +鲛 +鲟 +獐 +觫 +雒 +夤 +馑 +銮 +塾 +麽 +瘌 +瘊 +瘘 +瘙 +廖 +韶 +旖 +膂 +阚 +鄯 +鲞 +粿 +粼 +粽 +糁 +槊 +鹚 +熘 +熥 +潢 +漕 +滹 +漯 +漶 +潋 +潴 +漪 +漉 +漳 +漩 +澉 +潍 +慵 +搴 +窨 +寤 +綮 +谮 +褡 +褙 +褓 +褛 +褊 +谯 +谰 +谲 +暨 +屣 +鹛 +嫣 +嫱 +嫖 +嫦 +嫚 +嫘 +嫡 +鼐 +翟 +瞀 +鹜 +骠 +缥 +缦 +缧 +缨 +骢 +缪 +缫 +耦 +耧 +瑾 +璜 +璀 +璎 +璁 +璋 +璇 +奭 +髯 +髫 +撷 +撅 +赭 +撸 +鋆 +撙 +撺 +墀 +聩 +觐 +鞑 +蕙 +鞒 +蕈 +蕨 +蕤 +蕞 +蕺 +瞢 +蕃 +蕲 +赜 +槿 +樯 +槭 +樗 +樘 +樊 +槲 +醌 +醅 +靥 +魇 +餍 +磔 +磙 +霈 +辘 +龉 +龊 +觑 +瞌 +瞋 +瞑 +嘭 +噎 +噶 +颙 +暹 +噘 +踔 +踝 +踟 +踒 +踬 +踮 +踯 +踺 +踞 +蝽 +蝾 +蝻 +蝰 +蝮 +螋 +蝓 +蝣 +蝼 +噗 +嘬 +颚 +噍 +噢 +噙 +噜 +噌 +噔 +颛 +幞 +幡 +嶙 +嶝 +骺 +骼 +骸 +镊 +镉 +镌 +镍 +镏 +镒 +镓 +镔 +稷 +箴 +篑 +篁 +篌 +篆 +牖 +儋 +徵 +磐 +虢 +鹞 +膘 +滕 +鲠 +鲡 +鲢 +鲣 +鲥 +鲧 +鲩 +獗 +獠 +觯 +馓 +馔 +麾 +廛 +瘛 +瘼 +瘢 +瘠 +齑 +羯 +羰 +𥻗 +遴 +糌 +糍 +糅 +熜 +熵 +熠 +澍 +澌 +潸 +潦 +潲 +鋈 +潟 +潼 +潺 +憬 +憧 +寮 +窳 +谳 +褴 +褟 +褫 +谵 +熨 +屦 +嬉 +勰 +戮 +蝥 +缬 +缮 +缯 +骣 +畿 +耩 +耨 +耪 +璞 +璟 +靛 +璠 +璘 +聱 +螯 +髻 +髭 +髹 +擀 +熹 +甏 +擞 +縠 +磬 +颞 +蕻 +鞘 +颟 +薤 +薨 +檠 +薏 +薮 +薜 +薅 +樾 +橛 +橇 +樵 +檎 +橹 +樽 +樨 +橼 +墼 +橐 +翮 +醛 +醐 +醍 +醚 +磲 +赝 +飙 +殪 +霖 +霏 +霓 +錾 +辚 +臻 +遽 +氅 +瞟 +瞠 +瞰 +嚄 +嚆 +噤 +暾 +蹀 +踹 +踵 +踽 +蹉 +蹁 +螨 +蟒 +螈 +螅 +螭 +螠 +螟 +噱 +噬 +噫 +噻 +噼 +罹 +圜 +䦃 +镖 +镗 +镘 +镚 +镛 +镝 +镞 +镠 +氇 +氆 +憩 +穑 +篝 +篥 +篦 +篪 +篙 +盥 +劓 +翱 +魉 +魈 +徼 +歙 +膳 +膦 +膙 +鲮 +鲱 +鲲 +鲳 +鲴 +鲵 +鲷 +鲻 +獴 +獭 +獬 +邂 +鹧 +廨 +赟 +瘰 +廪 +瘿 +瘵 +瘴 +癃 +瘳 +斓 +麇 +麈 +嬴 +壅 +羲 +糗 +瞥 +甑 +燎 +燠 +燔 +燧 +濑 +濉 +潞 +澧 +澹 +澥 +澶 +濂 +褰 +寰 +窸 +褶 +禧 +嬖 +犟 +隰 +嬗 +颡 +缱 +缲 +缳 +璨 +璩 +璐 +璪 +螫 +擤 +壕 +觳 +罄 +擢 +薹 +鞡 +鞬 +薷 +薰 +藓 +藁 +檄 +檩 +懋 +醢 +翳 +礅 +磴 +鹩 +龋 +龌 +豳 +壑 +黻 +嚏 +嚅 +蹑 +蹒 +蹊 +蟥 +螬 +螵 +疃 +螳 +蟑 +嚓 +羁 +罽 +罾 +嶷 +黜 +黝 +髁 +髀 +镡 +镢 +镣 +镦 +镧 +镩 +镪 +镫 +罅 +黏 +簌 +篾 +篼 +簖 +簋 +鼢 +黛 +儡 +鹪 +鼾 +皤 +魍 +龠 +繇 +貘 +邈 +貔 +臌 +膻 +臆 +臃 +鲼 +鲽 +鳀 +鳃 +鳅 +鳇 +鳊 +螽 +燮 +鹫 +襄 +糜 +縻 +膺 +癍 +麋 +懑 +濡 +濮 +濞 +濠 +濯 +蹇 +謇 +邃 +襁 +檗 +擘 +孺 +隳 +嬷 +蟊 +鹬 +鍪 +鏊 +鳌 +鬈 +鬃 +瞽 +鞯 +鞨 +鞫 +鞧 +鞣 +藜 +藠 +藩 +醪 +蹙 +礓 +燹 +餮 +瞿 +曛 +颢 +曜 +躇 +蹚 +鹭 +蟛 +蟪 +蟠 +蟮 +鹮 +黠 +黟 +髅 +髂 +镬 +镭 +镯 +馥 +簟 +簪 +鼬 +雠 +艟 +鳎 +鳏 +鳐 +癞 +癔 +癜 +癖 +糨 +蹩 +鎏 +懵 +彝 +邋 +鬏 +攉 +攒 +鞲 +鞴 +藿 +蘧 +蘅 +麓 +醮 +醯 +酃 +霪 +霭 +霨 +黼 +嚯 +蹰 +蹶 +蹽 +蹼 +蹴 +蹾 +蹿 +蠖 +蠓 +蟾 +蠊 +黢 +髋 +髌 +镲 +籀 +籁 +齁 +魑 +艨 +鳓 +鳔 +鳕 +鳗 +鳙 +麒 +鏖 +羸 +㸆 +瀚 +瀣 +瀛 +襦 +谶 +襞 +骥 +缵 +瓒 +攘 +蘩 +蘖 +醴 +霰 +酆 +矍 +曦 +躅 +鼍 +巉 +黩 +黥 +黪 +镳 +镴 +黧 +纂 +璺 +鼯 +臜 +鳜 +鳝 +鳟 +獾 +孀 +骧 +瓘 +鼙 +醺 +礴 +颦 +曩 +鳢 +癫 +麝 +夔 +爝 +灏 +禳 +鐾 +羼 +蠡 +耱 +懿 +蘸 +鹳 +霾 +氍 +饕 +躐 +髑 +镵 +穰 +饔 +鬻 +鬟 +趱 +攫 +攥 +颧 +躜 +鼹 +癯 +麟 +蠲 +蠹 +躞 +衢 +鑫 +灞 +襻 +纛 +鬣 +攮 +囔 +馕 +戆 +爨 +齉 +.0 +.1 +.2 +.3 +.4 +.5 +.6 +.7 +.8 +.9 +0% +0. +0.0 +0.00 +0.1 +0.2 +0.3 +0.4 +0.5 +0.6 +0.7 +0.8 +0.9 +00 +001 +002 +01 +010 +02 +03 +04 +05 +06 +07 +08 +09 +1% +1. +10 +100 +100% +101 +102 +103 +104 +105 +106 +107 +108 +109 +11 +110 +111 +112 +113 +114 +115 +116 +117 +118 +119 +12 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +13 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +14 +140 +144 +145 +15 +150 +155 +156 +158 +16 +160 +165 +168 +17 +170 +175 +178 +18 +180 +185 +188 +19 +190 +195 +196 +197 +198 +199 +2% +2. +20 +200 +201 +21 +210 +211 +22 +220 +225 +23 +230 +24 +240 +25 +250 +256 +26 +260 +27 +270 +28 +280 +29 +2o +3% +3. +30 +300 +31 +310 +32 +320 +33 +330 +34 +35 +350 +36 +360 +365 +37 +38 +380 +39 +3d +4% +4. +40 +400 +41 +42 +43 +44 +45 +46 +47 +48 +49 +4g +4k +4s店 +5% +5. +50 +500 +51 +52 +520 +53 +54 +55 +550 +56 +57 +58 +59 +5g +6% +6. +60 +61 +62 +63 +64 +65 +66 +67 +68 +69 +7% +7. +70 +71 +72 +73 +74 +75 +76 +77 +78 +79 +8% +8. +80 +800 +81 +82 +83 +84 +85 +86 +87 +88 +89 +9% +9. +90 +900 +91 +911 +92 +93 +94 +95 +96 +97 +98 +985 +99 +AP +AT +And +CA +CD +CP +CPU +Ch +Co +DV +ED +GB +Hz +ID +IP +IS +In +Int +LED +Li +MB +MP +NA +No +OS +PC +PS +Pro +SB +St +TV +The +USB +VIP +Win +You +a4 +aa +ab +abo +abs +ac +acc +ace +ach +ack +act +ad +add +ade +af +ag +age +ah +ai +ain +air +ak +al +alk +all +alt +am +ame +amp +an +and +ang +ank +ans +ant +any +ao +ap +api +app +ar +ard +are +ark +ars +art +ary +as +ase +ash +ass +ast +at +ata +ate +att +au +av +ava +ax +axy +ay +az +b1 +b2 +b2b +ba +bal +ban +bar +bb +bc +bd +be +bel +ber +bet +bi +big +bit +bl +ble +bmw +bo +box +bp +br +bre +bro +bs +bt +bu +by +ca +cal +can +cap +car +cba +cc +cct +cd +ce +cel +ceo +cer +ces +cg +ch +che +cho +ci +ck +cl +clo +cm +cn +co +col +com +con +cor +cos +cp +cpi +cpu +cr +cre +cro +cs +ct +cv +cy +da +day +dc +dd +de +des +di +dif +dis +diy +dj +dn +dna +do +don +dr +ds +dv +dy +ear +eb +ec +ed +edg +ef +el +ell +em +en +end +eng +ens +ent +ep +eq +er +ere +ers +es +ess +est +et +etc +etf +ev +ew +ex +fa +fc +fe +fer +ff +fi +fin +fir +fl +fo +for +fr +fri +ft +ga +gal +gb +gd +gdp +ge +gen +get +gg +gh +gl +go +got +gps +gr +gre +gro +gs +gt +h2 +h6 +ha +had +has +hd +he +hel +her +hi +his +ho +how +hpv +hr +ht +hy +hz +ia +ial +ian +ib +ibm +ic +ica +ice +ich +ick +ics +ict +id +ide +ie +ies +if +ife +ifi +ift +ig +igh +ign +ii +il +ild +ill +ily +im +ima +in +ina +ind +ine +ing +ink +ins +int +io +ion +ios +ip +ipo +ips +iq +ir +ire +irl +is +ise +ish +iso +iss +ist +it +ite +its +itt +ity +iv +ive +iz +jd +je +ji +jo +joh +ka +kdj +ke +key +kg +km +ko +ktv +kv +kw +la +lan +lar +ld +le +led +len +les +let +lex +lg +li +lin +ll +lo +log +lol +low +lpl +lt +lv +ly +lz +m2 +ma +mac +man +mar +mat +max +may +mb +mba +mc +me +mer +mg +mi +mic +min +ml +mm +mo +mod +mon +mor +mp +mp3 +mpv +mr +ms +mv +mvp +mw +my +na +nat +nba +nc +nd +ne +ner +net +new +ng +ni +nm +no +no. +not +now +ns +nt +nv +ny +o2o +ob +od +of +off +ofo +oft +og +oh +ok +ol +om +on +ond +one +ong +oo +op +ope +or +ord +ore +ort +ory +os +ot +ou +out +p2 +p2p +pa +par +pc +pd +pe +per +pet +pg +ph +php +pi +pk +pl +ple +pm +pm2.5 +pmi +po +pos +pp +ppt +pr +pre +pri +pro +ps +pt +pu +pv +pvc +px +py +qe +qq +ra +ram +re +red +ree +ref +ren +res +ret +ri +ris +rl +rn +rng +ro +rom +ron +rs +rsi +rt +ry +sa +sat +say +sci +sd +se +sed +see +sel +seo +ser +set +sh +she +sho +si +sim +sin +sk +skt +sl +sm +sn +so +son +sp +spe +sq +sql +ss +st +ste +str +su +suv +sw +sy +ta +td +te +ter +tf +th +the +tho +ti +tim +tm +to +ton +too +top +tp +tr +tra +try +ts +tt +tv +ty +uc +ue +ui +un +uni +up +ur +us +usb +ust +ut +uzi +va +val +van +var +vc +ve +vel +ven +ver +vi +vip +vis +vo +vol +vr +vs +wa +was +wat +way +we +web +wer +wh +who +wi +win +wn +wo +wor +wr +ws +x1 +x5 +xt +xx +ye +yes +yle +yo +you +ys +ze +zh +zi +一万 +一下 +一下子 +一不 +一不小心 +一世 +一丝 +一两 +一两个 +一个 +一个个 +一个多 +一个多月 +一个月 +一中 +一串 +一举 +一举一动 +一九 +一书 +一事 +一二 +一些 +一亮 +一人 +一亿 +一代 +一代人 +一件 +一份 +一众 +一伙 +一会 +一会儿 +一位 +一体 +一体化 +一例 +一侧 +一倍 +一元 +一共 +一再 +一出 +一击 +一刀 +一分 +一分钟 +一分钱 +一切 +一切众生 +一切都是 +一则 +一到 +一刹那 +一刻 +一剑 +一副 +一加 +一动 +一勺 +一包 +一匹 +一十 +一千 +一半 +一卷 +一去 +一双 +一反 +一发 +一口 +一口气 +一句 +一只 +一台 +一叶 +一号 +一同 +一名 +一向 +一听 +一员 +一周 +一味 +一品 +一回 +一团 +一国 +一圈 +一地 +一场 +一块 +一堂 +一堆 +一塌糊涂 +一声 +一壶 +一处 +一夜 +一夜之间 +一大 +一大堆 +一大批 +一大早 +一天 +一天天 +一头 +一套 +一女 +一如 +一如既往 +一字 +一季 +一季度 +一定 +一审 +一家 +一家人 +一寸 +一对 +一对一 +一封 +一封信 +一小 +一局 +一层 +一届 +一岁 +一帆 +一帆风顺 +一带 +一席 +一帮 +一幅 +一幕 +一幢 +一年 +一年一度 +一年四季 +一年级 +一并 +一应 +一应俱全 +一度 +一座 +一张 +一律 +一心 +一念 +一惊 +一想 +一意 +一成 +一战 +一户 +一所 +一扇 +一手 +一扫 +一批 +一把 +一抹 +一招 +一指 +一排 +一提 +一支 +一改 +一整天 +一整套 +一文 +一斑 +一斤 +一新 +一方 +一方面 +一旁 +一族 +一无 +一无所有 +一日 +一旦 +一早 +一时 +一时期 +一时间 +一是 +一晚 +一曲 +一月 +一有 +一望 +一朝 +一期 +一本 +一朵 +一束 +一条 +一来 +一杯 +一枚 +一枝 +一架 +一栋 +一株 +一样 +一根 +一格 +一案 +一桌 +一档 +一棵 +一棵树 +一楼 +一概 +一模一样 +一次 +一次性 +一次次 +一款 +一步 +一步步 +一段 +一段时间 +一段话 +一毛 +一气 +一池 +一汽 +一波 +一派 +一流 +一滴 +一点 +一点一点 +一点儿 +一点点 +一片 +一环 +一班 +一瓶 +一生 +一番 +一百 +一百年 +一盆 +一盏 +一盒 +一盘 +一目 +一目了然 +一直 +一看 +一眼 +一瞬间 +一碗 +一票 +一种 +一秒 +一站 +一站式 +一章 +一端 +一笑 +一笔 +一等 +一等奖 +一筹 +一篇 +一米 +一类 +一粒 +一系列 +一级 +一线 +一组 +一经 +一统 +一缕 +一群 +一股 +一脚 +一脸 +一至 +一致 +一致性 +一般 +一般来说 +一般而言 +一艘 +一节 +一节课 +一行 +一袋 +一见 +一见钟情 +一览 +一觉 +一角 +一言 +一词 +一试 +一语 +一说 +一课 +一谈 +一贯 +一起 +一趟 +一跃 +一路 +一路上 +一跳 +一身 +一轮 +一辆 +一辆车 +一辈 +一辈子 +一边 +一连 +一连串 +一通 +一遍 +一道 +一郎 +一部 +一部分 +一针 +一锅 +一键 +一长 +一门 +一闪 +一问 +一间 +一阵 +一阵子 +一集 +一面 +一页 +一项 +一顾 +一顿 +一颗 +一首 +一首歌 +丁丁 +丁克 +丁香 +七个 +七八 +七八糟 +七分 +七十 +七夕 +七大 +七天 +七年 +七彩 +七日 +七星 +七月 +七点 +万一 +万万 +万丈 +万个 +万事 +万亩 +万人 +万人次 +万亿 +万亿元 +万亿美元 +万众 +万余 +万余元 +万元 +万公 +万公里 +万分 +万千 +万历 +万变 +万台 +万名 +万吨 +万圣 +万多 +万字 +万家 +万寿 +万岁 +万左右 +万平方米 +万年 +万恶 +万户 +万桶 +万物 +万科 +万美元 +万股 +万能 +万象 +万辆 +万达 +万里 +万重 +丈夫 +丈母 +三七 +三万 +三世 +三两 +三个 +三五 +三亚 +三人 +三代 +三件 +三位 +三体 +三倍 +三元 +三八 +三农 +三分 +三分之一 +三分之二 +三分球 +三分钟 +三包 +三十 +三千 +三叉 +三口 +三只 +三号 +三合 +三名 +三四 +三国 +三国演义 +三大 +三天 +三季度 +三宝 +三家 +三尺 +三层 +三岁 +三峡 +三年 +三年级 +三张 +三改 +三文 +三方 +三日 +三明 +三星 +三昧 +三是 +三月 +三期 +三条 +三板 +三楼 +三次 +三款 +三步 +三段 +三毛 +三江 +三点 +三焦 +三环 +三生 +三甲 +三界 +三百 +三相 +三省 +三种 +三秒 +三章 +三等 +三等奖 +三类 +三级 +三线 +三维 +三者 +三联 +三菱 +三观 +三角 +三角形 +三轮 +三轮车 +三遍 +三道 +三部 +三里 +三重 +三门 +三面 +三项 +三餐 +上万 +上下 +上下游 +上下班 +上个 +上个世纪 +上个月 +上乘 +上交 +上任 +上传 +上位 +上前 +上加 +上千 +上升 +上午 +上半 +上半场 +上半年 +上半身 +上去 +上古 +上台 +上司 +上周 +上周五 +上图 +上场 +上天 +上学 +上官 +上将 +上层 +上山 +上岗 +上岸 +上市 +上市公司 +上师 +上帝 +上年 +上床 +上当 +上当受骗 +上扬 +上报 +上攻 +上方 +上旬 +上映 +上来 +上架 +上桌 +上榜 +上次 +上汽 +上浮 +上海 +上海市 +上涨 +上港 +上游 +上演 +上火 +上爬 +上班 +上班族 +上用 +上瘾 +上皮 +上空 +上级 +上线 +上网 +上肢 +上行 +上街 +上衣 +上要 +上角 +上证 +上证指数 +上诉 +上课 +上调 +上赛季 +上路 +上身 +上车 +上边 +上进 +上述 +上部 +上门 +上阵 +上限 +上面 +上风 +上饶 +下一代 +下个 +下乡 +下令 +下入 +下决心 +下列 +下功夫 +下午 +下午茶 +下半 +下半场 +下半年 +下半身 +下单 +下去 +下发 +下周 +下图 +下场 +下坡 +下垂 +下定 +下定决心 +下层 +下属 +下山 +下岗 +下巴 +下意识 +下手 +下挫 +下探 +下文 +下方 +下旬 +下来 +下楼 +下次 +下水 +下水道 +下沉 +下游 +下滑 +下班 +下级 +下线 +下肢 +下药 +下落 +下行 +下角 +下设 +下课 +下调 +下跌 +下路 +下身 +下车 +下载 +下辖 +下达 +下部 +下锅 +下降 +下雨 +下雪 +下面 +下颌 +不一 +不上 +不下 +不为 +不久 +不久前 +不乏 +不买 +不乱 +不了 +不予 +不争 +不二 +不亚于 +不亦 +不仅 +不仅仅 +不仅如此 +不以 +不以为 +不以为然 +不休 +不会 +不但 +不住 +不作 +不佳 +不便 +不俗 +不信 +不倒 +不值 +不假 +不停 +不偿 +不像 +不光 +不免 +不入 +不全 +不公 +不再 +不决 +不准 +不减 +不凡 +不出 +不分 +不利 +不利于 +不到 +不前 +不力 +不动 +不动产 +不单 +不厌 +不去 +不及 +不及待 +不发 +不受 +不变 +不可 +不可否认 +不可思议 +不可或缺 +不可避免 +不吃 +不合 +不合理 +不同 +不同于 +不听话 +不善 +不喜 +不在 +不在乎 +不在少数 +不在意 +不均 +不堪 +不复 +不多 +不够 +不大 +不太 +不太好 +不失 +不失为 +不好 +不好意思 +不如 +不如意 +不妥 +不妨 +不孕 +不安 +不完 +不定 +不定期 +不宜 +不容 +不容忽视 +不容易 +不对 +不小 +不少 +不尽 +不尽相同 +不屈 +不屑 +不差 +不已 +不带 +不干 +不平 +不幸 +不应 +不开 +不弃 +不当 +不得 +不得不 +不得了 +不得已 +不得而知 +不必 +不必要 +不忍 +不忍心 +不忘 +不快 +不怎么 +不怕 +不思 +不怪 +不息 +不惊 +不惜 +不想 +不愈 +不愧 +不愿 +不慎 +不懂 +不懈 +不懈努力 +不成 +不折不扣 +不拘 +不择 +不振 +不掉 +不提 +不改 +不放 +不敢 +不敢相信 +不料 +不断 +不断创新 +不断完善 +不断扩大 +不无 +不时 +不明 +不易 +不是 +不曾 +不服 +不朽 +不来 +不止 +不正 +不死 +不比 +不求 +不治 +不法 +不清 +不满 +不然 +不爱 +不爽 +不理 +不甘 +不生 +不用 +不用说 +不由 +不由得 +不由自主 +不畏 +不留 +不相 +不相同 +不看 +不着 +不知 +不知不觉 +不知所措 +不知道 +不破 +不确定性 +不禁 +不离 +不移 +不稳 +不穷 +不符 +不算 +不管 +不管怎么 +不管怎样 +不累 +不经 +不经意 +不绝 +不缺 +不羁 +不老 +不耐 +不耐烦 +不肯 +不育 +不胜 +不能 +不能不 +不腻 +不自 +不自禁 +不至 +不至于 +不舍 +不良 +不良反应 +不良影响 +不苟 +不菲 +不行 +不衰 +不要 +不要紧 +不见 +不见得 +不规则 +不觉 +不解 +不言 +不言而喻 +不计 +不认 +不让 +不许 +不论 +不论是 +不该 +不详 +不语 +不说 +不调 +不谈 +不负 +不负责 +不负责任 +不败 +不贵 +不起 +不起眼 +不足 +不足以 +不足道 +不输 +不过 +不进 +不远 +不远处 +不适 +不透 +不通 +不锈 +不锈钢 +不错 +不长 +不问 +不间断 +不难 +不难看出 +不需 +不靠 +不顺 +不顾 +不饱和 +不高 +不齐 +与众 +与众不同 +与会 +与其 +与否 +与时 +与时俱进 +与此 +与此同时 +与生 +与生俱来 +丑恶 +丑陋 +专一 +专业 +专业人士 +专业化 +专业性 +专业技能 +专业知识 +专业课 +专人 +专任 +专修 +专利 +专制 +专区 +专升 +专卖 +专卖店 +专员 +专场 +专家 +专家学者 +专家建议 +专家组 +专属 +专心 +专政 +专柜 +专栏 +专案 +专案组 +专注 +专用 +专科 +专科医院 +专线 +专职 +专营 +专访 +专车 +专辑 +专长 +专门 +专项 +专项斗争 +专项资金 +专题 +世上 +世事 +世人 +世代 +世俗 +世凯 +世博 +世博会 +世外 +世家 +世尊 +世界 +世界冠军 +世界卫生组织 +世界各地 +世界大战 +世界杯 +世界级 +世界观 +世纪 +世袭 +世贸 +世锦 +世锦赛 +世间 +丘陵 +丙烯 +丙酮 +业主 +业余 +业内 +业内人士 +业力 +业务 +业务员 +业态 +业界 +业绩 +业障 +丛书 +丛林 +丛生 +东东 +东亚 +东京 +东侧 +东北 +东北地区 +东北部 +东区 +东华 +东南 +东南亚 +东南部 +东吴 +东坡 +东山 +东岸 +东方 +东晋 +东欧 +东汉 +东海 +东湖 +东盟 +东站 +东至 +东芝 +东莞 +东莞市 +东营 +东西 +东路 +东边 +东邻 +东部 +东野 +东门 +东风 +丝丝 +丝毫 +丝瓜 +丝绸 +丝绸之路 +丝袜 +丞相 +丢人 +丢失 +丢弃 +丢掉 +两万 +两三 +两三个 +两个 +两人 +两件 +两份 +两会 +两位 +两侧 +两倍 +两分钟 +两千 +两半 +两口 +两句 +两只 +两名 +两周 +两国 +两地 +两场 +两块 +两大 +两天 +两头 +两套 +两委 +两家 +两层 +两岁 +两岸 +两市 +两年 +两座 +两张 +两性 +两手 +两把 +两支 +两旁 +两本 +两条 +两极 +两栖 +两样 +两根 +两次 +两款 +两步 +两段 +两点 +两片 +两百 +两眼 +两种 +两端 +两类 +两级 +两组 +两者 +两腿 +两边 +两部 +两门 +两队 +两面 +两项 +两颗 +严厉 +严厉打击 +严密 +严寒 +严峻 +严格 +严格执行 +严格控制 +严格要求 +严格遵守 +严禁 +严经 +严肃 +严苛 +严谨 +严重 +严重者 +严防 +丧失 +丧心 +个个 +个人 +个人信息 +个人所得税 +个体 +个别 +个头 +个子 +个性 +个性化 +个月 +个案 +个点 +个股 +丫头 +中专 +中世纪 +中东 +中亚 +中产 +中人 +中介 +中介机构 +中以 +中信 +中全会 +中共 +中共中央 +中共党员 +中关村 +中兴 +中医 +中医学 +中医药 +中医药大学 +中医院 +中午 +中华 +中华人民共和国 +中华文化 +中华民族 +中南 +中原 +中国 +中国人民 +中国人民大学 +中国人民解放军 +中国人民银行 +中国共产党 +中国女排 +中国式 +中国政府 +中国电信 +中国画 +中国科学院 +中国移动 +中国联通 +中国证监会 +中国银行 +中国队 +中场 +中型 +中外 +中天 +中央 +中央电视台 +中央空调 +中奖 +中学 +中学生 +中将 +中小 +中小企业 +中小型 +中小学 +中小学生 +中小板 +中层 +中山 +中山大学 +中年 +中年人 +中度 +中庸 +中式 +中心 +中性 +中成药 +中所 +中指 +中控 +中控台 +中文 +中文名 +中文网 +中断 +中新 +中新网 +中方 +中日 +中旬 +中是 +中暑 +中有 +中期 +中来 +中枢 +中枢神经 +中标 +中欧 +中止 +中段 +中毒 +中海 +中游 +中秋 +中秋节 +中科院 +中立 +中等 +中级 +中线 +中美 +中老年 +中老年人 +中考 +中耳 +中草药 +中药 +中药材 +中西 +中西医 +中西部 +中要 +中说 +中超 +中路 +中轨 +中转 +中途 +中部 +中铁 +中锋 +中长期 +中间 +中队 +中青年 +中风 +中餐 +丰厚 +丰台 +丰富 +丰富多彩 +丰收 +丰满 +丰田 +丰盛 +丰硕 +丰胸 +串串 +串口 +串联 +串行 +临场 +临安 +临床 +临床经验 +临床表现 +临床试验 +临摹 +临时 +临死 +临江 +临汾 +临沂 +临海 +临界 +临终 +临走 +临近 +丸子 +丹东 +丹参 +丹尼 +丹田 +丹阳 +丹青 +丹麦 +为一体 +为主 +为了 +为人 +为什么 +为何 +为例 +为先 +为准 +为力 +为名 +为啥 +为官 +为宜 +为数 +为数不多 +为期 +为本 +为止 +为此 +为民 +为生 +为辅 +为难 +为首 +主业 +主义 +主义者 +主产 +主人 +主人公 +主任 +主任医师 +主任委员 +主体 +主修 +主创 +主力 +主办 +主办方 +主动 +主动性 +主动权 +主卧 +主场 +主城区 +主妇 +主宰 +主导 +主屏 +主帅 +主席 +主干 +主干道 +主张 +主意 +主打 +主持 +主持人 +主播 +主攻 +主教练 +主料 +主旋律 +主旨 +主机 +主权 +主板 +主治 +主流 +主演 +主管 +主管部门 +主线 +主编 +主营 +主营业务 +主要 +主见 +主观 +主角 +主讲 +主轴 +主页 +主题 +主题曲 +主食 +丽丝 +丽人 +丽水 +丽江 +丽莎 +举世 +举个 +举例 +举办 +举动 +举手 +举报 +举措 +举止 +举行 +举证 +举起 +举足 +举重 +乃是 +乃至 +久久 +久之 +久坐 +久而 +久而久之 +久远 +久违 +义乌 +义务 +义务教育 +义勇 +义工 +义无 +义无反顾 +义气 +义词 +之一 +之三 +之上 +之下 +之中 +之为 +之久 +之乱 +之争 +之事 +之二 +之于 +之交 +之人 +之作 +之光 +之内 +之分 +之初 +之别 +之前 +之力 +之功 +之势 +之又 +之名 +之后 +之地 +之士 +之声 +之处 +之外 +之多 +之夜 +之大 +之女 +之子 +之家 +之年 +之徒 +之心 +之忧 +之急 +之恩 +之情 +之意 +之感 +之战 +之所以 +之手 +之旅 +之无愧 +之日起 +之时 +之星 +之歌 +之气 +之流 +之源 +之灾 +之物 +之王 +之用 +之神 +之称 +之类 +之美 +之苦 +之计 +之谜 +之象 +之路 +之道 +之重 +之间 +之际 +之风 +之高 +乌云 +乌克兰 +乌兰 +乌尔 +乌托 +乌拉 +乌斯 +乌鲁 +乌鲁木齐 +乌鸦 +乌龙 +乌龟 +乎乎 +乏力 +乏味 +乐乎 +乐乐 +乐于 +乐器 +乐团 +乐园 +乐坛 +乐天 +乐山 +乐府 +乐意 +乐曲 +乐福 +乐观 +乐视 +乐趣 +乐队 +乒乓 +乒乓球 +乔丹 +乔布斯 +乔木 +乔治 +乖乖 +乖巧 +乘务 +乘坐 +乘客 +乘用车 +乘车 +乙方 +乙烯 +乙肝 +乙醇 +九个 +九五 +九十 +九天 +九宫 +九寨 +九寨沟 +九州 +九年 +九月 +九江 +九点 +九章 +九龙 +乞丐 +乞讨 +也好 +也就是说 +也罢 +也许 +习以 +习俗 +习得 +习性 +习惯 +习惯于 +习惯性 +习气 +习近平 +习题 +乡下 +乡土 +乡村 +乡镇 +书上 +书中 +书信 +书写 +书刊 +书包 +书名 +书店 +书房 +书本 +书架 +书桌 +书法 +书法作品 +书法家 +书生 +书画 +书画家 +书目 +书籍 +书记 +书评 +书里 +书院 +书面 +书香 +买下 +买不到 +买不起 +买个 +买买 +买入 +买到 +买单 +买卖 +买回来 +买家 +买房 +买房子 +买方 +买来 +买点 +买菜 +买车 +买过 +买进 +乱七八糟 +乱世 +乱说 +乱象 +乳化 +乳头 +乳房 +乳汁 +乳液 +乳糖 +乳胶 +乳腺 +乳腺癌 +乳酸 +乾坤 +乾隆 +了不起 +了个 +了然 +了结 +了解 +予以 +争先 +争取 +争吵 +争夺 +争执 +争斗 +争相 +争端 +争议 +争论 +争霸 +事业 +事业单位 +事业部 +事事 +事件 +事例 +事儿 +事先 +事关 +事前 +事务 +事务所 +事半 +事半功倍 +事发 +事变 +事后 +事宜 +事实 +事实上 +事实证明 +事态 +事情 +事故 +事物 +事记 +事迹 +事项 +二三 +二世 +二个 +二中 +二人 +二代 +二元 +二十 +二十世纪 +二千 +二号 +二字 +二季度 +二审 +二层 +二年 +二年级 +二战 +二手 +二手房 +二手车 +二是 +二月 +二期 +二来 +二楼 +二次 +二氧化 +二氧化碳 +二流 +二环 +二百 +二等 +二等奖 +二级 +二线 +二维 +二维码 +二者 +二胎 +二胡 +二郎 +二醇 +于世 +于事 +于土 +于心 +于是 +亏损 +亏本 +云南 +云南省 +云海 +云狗 +云端 +云集 +云雾 +云龙 +互不 +互为 +互利 +互动 +互助 +互惠 +互感 +互换 +互生 +互相 +互联 +互联互通 +互联网 +互补 +互通 +五一 +五万 +五个 +五代 +五位 +五六 +五分 +五分钟 +五十 +五千 +五千年 +五味 +五四 +五大 +五天 +五官 +五岁 +五年 +五彩 +五指 +五日 +五星 +五星级 +五是 +五月 +五次 +五毛 +五点 +五百 +五种 +五笔 +五粮 +五粮液 +五脏 +五脏六腑 +五色 +五花 +五花八门 +五花肉 +五行 +五谷 +五通 +五金 +井上 +井水 +亚于 +亚健康 +亚军 +亚历 +亚太 +亚当 +亚洲 +亚热 +亚特 +亚瑟 +亚硝 +亚科 +亚运 +亚运会 +亚迪 +亚马 +亚马逊 +亚麻 +些许 +亡国 +亡灵 +交了 +交互 +交付 +交代 +交会 +交出 +交到 +交割 +交办 +交叉 +交叉口 +交友 +交响 +交大 +交往 +交战 +交房 +交所 +交手 +交换 +交接 +交易 +交易中心 +交易市场 +交易平台 +交易所 +交易日 +交易系统 +交易者 +交易量 +交替 +交朋友 +交汇 +交汇处 +交流 +交流会 +交界 +交界处 +交管 +交纳 +交织 +交给 +交融 +交警 +交警大队 +交谈 +交货 +交通 +交通事故 +交通大学 +交通安全 +交通工具 +交通管理 +交通运输 +交道 +交配 +交锋 +交错 +交际 +交集 +亦可 +产业 +产业化 +产业园 +产业结构 +产业链 +产值 +产出 +产区 +产卵 +产后 +产品 +产品开发 +产品线 +产品设计 +产品质量 +产地 +产妇 +产权 +产物 +产生 +产能 +产量 +产销 +产阶级 +亨利 +亩产 +享乐 +享受 +享年 +享有 +享用 +享誉 +京东 +京剧 +京华 +京城 +京师 +京报 +京沪 +京津 +京津冀 +京都 +亮丽 +亮度 +亮点 +亮相 +亮眼 +亲临 +亲亲 +亲人 +亲们 +亲切 +亲历 +亲友 +亲吻 +亲和 +亲和力 +亲子 +亲密 +亲属 +亲情 +亲戚 +亲戚朋友 +亲手 +亲朋 +亲朋好友 +亲民 +亲测 +亲爱 +亲王 +亲生 +亲眼 +亲笔 +亲自 +亲身 +亲身经历 +亲近 +人与 +人世 +人世间 +人中 +人为 +人事 +人人 +人以 +人们 +人会 +人体 +人像 +人到 +人力 +人力资源 +人厌 +人去 +人参 +人口 +人员 +人命 +人和 +人和事 +人品 +人均 +人士 +人多 +人大 +人大代表 +人大常委会 +人头 +人家 +人寿 +人居 +人工 +人工智能 +人心 +人性 +人性化 +人情 +人意 +人手 +人才 +人才培养 +人数 +人文 +人有 +人机 +人权 +人来 +人格 +人次 +人民 +人民代表大会 +人民共和国 +人民币 +人民政府 +人民日报 +人民检察院 +人民法院 +人民网 +人气 +人流 +人海 +人物 +人生 +人生观 +人用 +人知 +人称 +人类 +人缘 +人群 +人能 +人脉 +人脸 +人脸识别 +人行 +人行道 +人要 +人设 +人身 +人选 +人造 +人道 +人间 +人际 +人际交往 +人际关系 +人马 +亿万 +亿人 +亿元 +亿吨 +亿次 +亿立 +亿美元 +亿股 +什么 +什么样 +仁义 +仁慈 +仁波切 +仁爱 +仁者 +仅为 +仅仅 +仅供 +仅供参考 +仅次于 +仅此 +仅限 +仇人 +仇恨 +今后 +今夜 +今天 +今天上午 +今天下午 +今属 +今年 +今年以来 +今日 +今晚 +今朝 +今生 +介于 +介入 +介意 +介石 +介绍 +介质 +仍会 +仍旧 +仍然 +从不 +从业 +从业人员 +从业者 +从严 +从严治党 +从中 +从事 +从何 +从前 +从头 +从头到 +从容 +从小 +从小到大 +从心 +从未 +从来 +从来不 +从根本上 +从此 +从没 +从而 +从那 +仓位 +仓促 +仓储 +仓库 +仔细 +仔细观察 +他人 +他们 +他会 +他俩 +他命 +他妈 +他家 +付出 +付款 +付诸 +付费 +仙人 +仙剑 +仙境 +仙女 +仙子 +代之 +代代 +代价 +代办 +代号 +代名词 +代工 +代数 +代替 +代步 +代理 +代理人 +代理商 +代码 +代表 +代表作 +代表团 +代表大会 +代表性 +代言 +代言人 +代谢 +代购 +令人 +令营 +以上 +以上学历 +以下 +以下几点 +以东 +以为 +以人为本 +以便 +以免 +以内 +以前 +以北 +以南 +以及 +以后 +以土 +以备 +以外 +以太 +以往 +以待 +以期 +以来 +以此 +以求 +以示 +以至 +以至于 +以致 +以色列 +以西 +以赴 +以身 +以防 +以鼻 +仪器 +仪式 +仪表 +仪表盘 +仰卧 +仰慕 +仰望 +仲裁 +件事 +价位 +价值 +价值观 +价廉 +价格 +价款 +价比 +价钱 +任何 +任何人 +任免 +任凭 +任务 +任命 +任性 +任意 +任教 +任期 +任由 +任职 +份额 +仿佛 +仿制 +仿古 +仿真 +企业 +企业家 +企事 +企事业 +企划 +企图 +企稳 +企鹅 +伊利 +伊始 +伊拉克 +伊斯 +伊斯兰 +伊朗 +伊犁 +伍德 +伏羲 +休假 +休克 +休养 +休学 +休息 +休息时间 +休止 +休眠 +休闲 +众人 +众多 +众所 +众所周知 +众生 +众筹 +优于 +优先 +优劣 +优势 +优化 +优异 +优待 +优惠 +优惠券 +优惠政策 +优惠活动 +优步 +优点 +优秀 +优秀人才 +优秀教师 +优缺 +优美 +优胜 +优良 +优质 +优质服务 +优越 +优选 +优酷 +优雅 +伙伴 +伙食 +会上 +会为 +会以 +会儿 +会务 +会变 +会同 +会员 +会场 +会展 +会展中心 +会想 +会见 +会计 +会计师 +会议 +会议室 +会诊 +会谈 +会费 +会长 +会阴 +伟人 +伟大 +传世 +传人 +传入 +传出 +传到 +传动 +传奇 +传媒 +传导 +传感 +传感器 +传承 +传授 +传播 +传教 +传来 +传染 +传染性 +传染病 +传热 +传球 +传真 +传神 +传祺 +传给 +传统 +传言 +传记 +传说 +传输 +传达 +传送 +传递 +传销 +传闻 +伤亡 +伤人 +伤口 +伤官 +伤害 +伤寒 +伤心 +伤感 +伤残 +伤病 +伤痕 +伤痛 +伤者 +伦敦 +伦理 +伪劣 +伪装 +伪造 +伯克 +伯恩 +伯格 +伯爵 +伯特 +估价 +估值 +估算 +估计 +伴侣 +伴奏 +伴有 +伴着 +伴随 +伸出 +伸展 +伸手 +伸直 +伸缩 +伸长 +伺候 +伺服 +似乎 +似的 +但凡 +但愿 +但是 +位于 +位列 +位子 +位居 +位数 +位移 +位置 +低下 +低于 +低价 +低估 +低位 +低俗 +低保 +低分 +低压 +低吸 +低头 +低廉 +低开 +低成本 +低收入 +低温 +低点 +低碳 +低端 +低级 +低落 +低调 +低谷 +低迷 +低速 +低音 +低频 +住处 +住宅 +住宿 +住户 +住房 +住所 +住校 +住院 +佐助 +体会 +体内 +体制 +体制改革 +体力 +体味 +体型 +体外 +体式 +体弱 +体征 +体态 +体操 +体格 +体检 +体液 +体温 +体现 +体积 +体系 +体育 +体育场 +体育局 +体育运动 +体育锻炼 +体育馆 +体能 +体脂 +体裁 +体谅 +体质 +体贴 +体重 +体量 +体面 +体验 +何不 +何为 +何从 +何以 +何其 +何况 +何去 +何在 +何处 +何尝 +何必 +何方 +何时 +何种 +何等 +何苦 +何谓 +余万 +余万元 +余个 +余人 +余力 +余名 +余地 +余姚 +余家 +余年 +余生 +余篇 +余部 +余额 +佛像 +佛学 +佛家 +佛山 +佛山市 +佛教 +佛法 +佛祖 +佛经 +佛罗 +佛说 +佛门 +佛陀 +作业 +作为 +作假 +作出 +作品 +作坊 +作家 +作弊 +作息 +作恶 +作战 +作文 +作曲 +作案 +作死 +作法 +作物 +作用 +作画 +作者 +作者简介 +作词 +作风 +你们 +你好 +你家 +佣金 +佩戴 +佩服 +佳人 +佳作 +佳品 +佳肴 +佳能 +佳节 +佼佼 +佼佼者 +使人 +使出 +使劲 +使命 +使命感 +使得 +使用 +使用寿命 +使用权 +使者 +使馆 +侄子 +例外 +例如 +例子 +例行 +侍卫 +供不应求 +供养 +供奉 +供应 +供应商 +供应链 +供暖 +供水 +供求 +供热 +供电 +供给 +供血 +供试 +供货 +供销 +供需 +依依 +依恋 +依托 +依据 +依旧 +依次 +依法 +依然 +依照 +依稀 +依赖 +依赖于 +依赖性 +依附 +依靠 +侠客 +侥幸 +侦察 +侦探 +侦查 +侧脉 +侧重 +侧面 +侮辱 +侵入 +侵华 +侵占 +侵害 +侵权 +侵犯 +侵略 +侵蚀 +侵袭 +便于 +便会 +便利 +便利店 +便宜 +便当 +便捷 +便携 +便是 +便民 +便秘 +促使 +促成 +促进 +促销 +俄国 +俄罗斯 +俄语 +俊杰 +俏皮 +俗称 +俗话说 +俗语 +俘虏 +保住 +保佑 +保修 +保值 +保健 +保健品 +保健食品 +保全 +保养 +保利 +保卫 +保姆 +保存 +保守 +保安 +保定 +保密 +保护 +保护区 +保持 +保持良好 +保时 +保时捷 +保暖 +保有 +保洁 +保温 +保湿 +保留 +保监会 +保研 +保税 +保管 +保罗 +保证 +保证金 +保质 +保质期 +保费 +保送 +保险 +保险人 +保险公司 +保障 +保障局 +保驾 +保驾护航 +保鲜 +保鲜膜 +信仰 +信任 +信佛 +信号 +信噪 +信奉 +信封 +信徒 +信心 +信念 +信息 +信息化 +信息安全 +信息技术 +信息系统 +信托 +信教 +信服 +信条 +信用 +信用卡 +信誉 +信访 +信贷 +信赖 +信道 +信阳 +俨然 +修习 +修仙 +修养 +修剪 +修图 +修复 +修建 +修护 +修持 +修改 +修正 +修法 +修炼 +修理 +修真 +修筑 +修缮 +修罗 +修行 +修补 +修订 +修身 +修辞 +修道 +修长 +修饰 +俯卧 +俯瞰 +俯视 +俱乐部 +俱全 +俱来 +俱进 +倍受 +倍增 +倍感 +倍数 +倒下 +倒入 +倒卵 +倒地 +倒塌 +倒数 +倒置 +倒计时 +倒车 +倒闭 +倒霉 +倔强 +倘若 +候补 +候车 +候选 +候选人 +倚天 +借助 +借口 +借款 +借款人 +借此 +借用 +借着 +借给 +借贷 +借鉴 +借钱 +借阅 +倡导 +倡议 +债券 +债务 +债务人 +债权 +债权人 +值守 +值得 +值得一提 +值得一提的是 +值得注意 +值班 +值钱 +倾倒 +倾向 +倾听 +倾城 +倾心 +倾斜 +倾注 +倾诉 +假体 +假冒 +假如 +假定 +假日 +假期 +假装 +假设 +假象 +假货 +偏低 +偏偏 +偏僻 +偏向 +偏好 +偏差 +偏执 +偏方 +偏爱 +偏离 +偏移 +偏见 +偏远 +偏重 +偏高 +做个 +做主 +做事 +做些 +做人 +做作 +做作业 +做做 +做出 +做到 +做好 +做客 +做工 +做成 +做手术 +做梦 +做法 +做点 +做爱 +做生意 +做空 +做菜 +做起 +做过 +做错 +做题 +做饭 +停下 +停下来 +停产 +停放 +停机 +停止 +停滞 +停牌 +停电 +停留 +停药 +停车 +停车位 +停车场 +停靠 +停顿 +健全 +健壮 +健康 +健康成长 +健康状况 +健美 +健脾 +健身 +健身房 +偶像 +偶尔 +偶然 +偶遇 +偷偷 +偷懒 +偷拍 +偿还 +傀儡 +傍晚 +傣族 +储备 +储存 +储物 +储蓄 +储藏 +储量 +催促 +催化 +催化剂 +催生 +催眠 +傲娇 +傲慢 +傻傻 +傻子 +傻瓜 +像是 +像素 +僧人 +僵尸 +僵硬 +儒学 +儒家 +儿女 +儿媳 +儿子 +儿时 +儿科 +儿童 +儿童医院 +儿童节 +允许 +元人 +元代 +元件 +元化 +元吨 +元器 +元宝 +元宵 +元宵节 +元帅 +元年 +元斤 +元旦 +元月 +元朝 +元气 +元素 +元老 +兄妹 +兄弟 +兄弟姐妹 +充值 +充分 +充分体现 +充分利用 +充分发挥 +充分考虑 +充分肯定 +充实 +充当 +充斥 +充气 +充沛 +充满 +充满活力 +充电 +充电器 +充盈 +充血 +充裕 +充足 +先到 +先前 +先后 +先天 +先天性 +先导 +先是 +先机 +先河 +先生 +先用 +先看 +先知 +先秦 +先行 +先说 +先进 +先锋 +先驱 +光临 +光亮 +光伏 +光华 +光合 +光圈 +光大 +光头 +光子 +光学 +光度 +光彩 +光影 +光明 +光是 +光棍 +光泽 +光洁 +光源 +光滑 +光灯 +光照 +光环 +光电 +光盘 +光纤 +光线 +光绪 +光芒 +光荣 +光谱 +光辉 +光阴 +光顾 +光驱 +光鲜 +克兰 +克制 +克力 +克劳 +克尔 +克拉 +克斯 +克服 +克林 +克洛 +克罗 +克莱 +克萨斯 +克里 +克里斯 +克隆 +克雷 +克鲁 +免不了 +免去 +免得 +免提 +免疫 +免疫力 +免疫系统 +免税 +免责 +免费 +免除 +兑换 +兑现 +兔子 +党中央 +党内 +党务 +党参 +党史 +党员 +党员干部 +党和国家 +党委 +党委书记 +党委委员 +党工委 +党建 +党支 +党支部 +党政 +党校 +党派 +党的 +党组 +党组书记 +党组织 +党风 +党风廉政 +兢兢 +入伍 +入会 +入住 +入侵 +入党 +入口 +入味 +入园 +入围 +入场 +入境 +入学 +入市 +入库 +入户 +入手 +入睡 +入股 +入选 +入锅 +入门 +入院 +入驻 +全世界 +全书 +全会 +全体 +全全 +全军 +全力 +全力以赴 +全区 +全县 +全员 +全国 +全国性 +全国政协 +全场 +全城 +全域 +全天 +全天候 +全套 +全家 +全家人 +全局 +全市 +全年 +全心 +全心全意 +全息 +全文 +全新 +全方位 +全日 +全日制 +全明星 +全是 +全景 +全村 +全校 +全株 +全民 +全然 +全班 +全球 +全球化 +全盘 +全省 +全真 +全程 +全系 +全线 +全缘 +全网 +全美 +全职 +全能 +全自动 +全资 +全身 +全身心 +全过程 +全部 +全都 +全镇 +全长 +全队 +全集 +全靠 +全面 +全额 +八一 +八个 +八九 +八五 +八仙 +八分 +八十 +八千 +八卦 +八大 +八字 +八宝 +八岁 +八年 +八戒 +八方 +八月 +八点 +八百 +八角 +八路 +八达 +八道 +八门 +公主 +公交 +公交线路 +公交车 +公众 +公会 +公信 +公元 +公元前 +公克 +公公 +公共 +公共交通 +公共卫生 +公共场所 +公共服务 +公关 +公分 +公办 +公务 +公务员 +公募 +公司 +公司股票 +公告 +公园 +公墓 +公婆 +公子 +公孙 +公安 +公安分局 +公安厅 +公安局 +公安机关 +公安部 +公害 +公寓 +公尺 +公布 +公平 +公开 +公开课 +公开赛 +公式 +公投 +公报 +公文 +公斤 +公有 +公款 +公正 +公民 +公然 +公爵 +公牛 +公理 +公用 +公益 +公益事业 +公益性 +公益活动 +公知 +公示 +公社 +公私 +公积 +公积金 +公立 +公立医院 +公章 +公约 +公职 +公英 +公认 +公证 +公诉 +公路 +公车 +公道 +公里 +公顷 +公馆 +公鸡 +六一 +六个 +六个月 +六十 +六千 +六合 +六大 +六安 +六年 +六月 +六点 +六百 +六种 +六级 +六腑 +六角 +六道 +兰亭 +兰卡 +兰州 +兰德 +兰特 +兰花 +兰芳 +共产 +共产主义 +共产党 +共产党员 +共享 +共分 +共创 +共同 +共同体 +共同努力 +共和 +共和国 +共存 +共建 +共性 +共振 +共有 +共生 +共用 +共睹 +共计 +共识 +共赢 +共青团 +共鸣 +关上 +关东 +关中 +关乎 +关于 +关切 +关卡 +关口 +关头 +关心 +关怀 +关押 +关掉 +关晓彤 +关机 +关村 +关注 +关注度 +关火 +关照 +关爱 +关税 +关系 +关联 +关节 +关节炎 +关键 +关键在于 +关键字 +关键技术 +关键时刻 +关键点 +关键词 +关门 +关闭 +兴业 +兴办 +兴国 +兴奋 +兴建 +兴旺 +兴盛 +兴致 +兴衰 +兴起 +兴趣 +兴趣爱好 +兴隆 +兵力 +兵器 +兵团 +兵役 +兵法 +兵种 +兵马 +其一 +其三 +其中 +其为 +其乐 +其事 +其二 +其人 +其他 +其他人 +其余 +其后 +其境 +其妙 +其它 +其实 +其实不然 +其所 +其次 +其父 +其自然 +其身 +其道 +其间 +具体 +具体情况 +具体操作 +具备 +具有 +具足 +典型 +典故 +典礼 +典范 +典藏 +典雅 +养分 +养家 +养心 +养成 +养护 +养殖 +养殖业 +养殖场 +养殖户 +养活 +养狗 +养猪 +养生 +养老 +养老保险 +养老金 +养育 +养胃 +养血 +养颜 +养鱼 +养鸡 +兼任 +兼具 +兼备 +兼容 +兼并 +兼职 +兼顾 +兽医 +兽药 +内侧 +内分泌 +内功 +内向 +内含 +内在 +内地 +内壁 +内外 +内存 +内容 +内容简介 +内幕 +内心 +内心世界 +内心深处 +内战 +内政 +内敛 +内有 +内服 +内核 +内涵 +内疚 +内科 +内线 +内经 +内置 +内脏 +内膜 +内蒙 +内蒙古 +内蒙古自治区 +内衣 +内裤 +内设 +内部 +内阁 +内陆 +内障 +内需 +内饰 +再三 +再也 +再也不会 +再度 +再有 +再来 +再次 +再现 +再生 +再用 +再者 +再见 +再说 +再造 +冒充 +冒出 +冒犯 +冒险 +冗余 +写下 +写作 +写信 +写入 +写写 +写出 +写字 +写字楼 +写实 +写得 +写意 +写成 +写文章 +写法 +写照 +写生 +写真 +写给 +写诗 +写过 +写道 +军中 +军事 +军人 +军分 +军区 +军医 +军团 +军委 +军官 +军工 +军政 +军方 +军校 +军民 +军用 +军舰 +军营 +军衔 +军训 +军阀 +军队 +农业 +农业大学 +农业部 +农产品 +农作 +农作物 +农副 +农历 +农场 +农夫 +农学 +农家 +农家乐 +农工 +农户 +农机 +农村 +农林 +农民 +农民工 +农民收入 +农牧 +农用 +农田 +农科 +农耕 +农药 +农贸 +农贸市场 +冠军 +冠名 +冠心病 +冠状 +冠状病毒 +冤家 +冤枉 +冥冥 +冥想 +冬天 +冬季 +冬日 +冬瓜 +冬至 +冰冷 +冰冻 +冰凉 +冰块 +冰山 +冰岛 +冰川 +冰河 +冰淇淋 +冰激 +冰箱 +冰糖 +冰雪 +冲上 +冲冲 +冲出 +冲击 +冲击力 +冲刷 +冲刺 +冲动 +冲压 +冲天 +冲泡 +冲洗 +冲浪 +冲着 +冲破 +冲突 +冲绳 +冲锋 +冲高 +决不 +决定 +决定性 +决心 +决战 +决斗 +决断 +决策 +决算 +决胜 +决议 +决赛 +况且 +冶炼 +冶金 +冷冷 +冷冻 +冷凝 +冷却 +冷战 +冷暖 +冷水 +冷汗 +冷淡 +冷清 +冷漠 +冷热 +冷空气 +冷笑 +冷落 +冷藏 +冷血 +冷酷 +冷门 +冷静 +冷饮 +冻结 +净值 +净净 +净利润 +净化 +净化器 +净土 +净水 +净水器 +净流入 +净资产 +凄凉 +准入 +准则 +准备 +准备金 +准妈 +准妈妈 +准字 +准时 +准确 +准确性 +准考证 +凉拌 +凉水 +凉爽 +凌乱 +凌云 +凌晨 +减产 +减仓 +减低 +减免 +减压 +减去 +减小 +减少 +减弱 +减持 +减排 +减税 +减缓 +减肥 +减脂 +减轻 +减退 +减速 +减震 +减龄 +凑合 +凝固 +凝结 +凝聚 +凝聚力 +凝胶 +凝血 +凝视 +几万 +几下 +几个 +几乎 +几代 +几件 +几位 +几何 +几分 +几分钟 +几十 +几十万 +几十个 +几千 +几千年 +几句 +几句话 +几只 +几名 +几周 +几块 +几天 +几家 +几岁 +几年 +几张 +几日 +几本 +几条 +几次 +几款 +几步 +几点 +几率 +几百 +几百年 +几种 +几秒 +几经 +几部 +几首 +凡事 +凡人 +凡夫 +凡是 +凤凰 +凭借 +凭借着 +凭着 +凭空 +凭证 +凯尔特 +凯尔特人 +凯文 +凯旋 +凯特 +凯迪 +凯迪拉克 +凳子 +凶手 +凶残 +凶猛 +凸显 +凸起 +凹凸 +凹陷 +出世 +出乎 +出了 +出事 +出于 +出产 +出人 +出任 +出众 +出入 +出入口 +出入境 +出具 +出击 +出动 +出卖 +出厂 +出去 +出去玩 +出发 +出发点 +出口 +出台 +出名 +出品 +出售 +出国 +出土 +出场 +出境 +出处 +出头 +出奇 +出家 +出局 +出差 +出席 +出席会议 +出彩 +出征 +出息 +出战 +出手 +出新 +出来 +出柜 +出栏 +出水 +出汗 +出没 +出海 +出游 +出演 +出炉 +出版 +出版社 +出现 +出生 +出示 +出租 +出租车 +出线 +出自 +出色 +出血 +出行 +出让 +出货 +出货量 +出资 +出走 +出路 +出身 +出轨 +出过 +出道 +出钱 +出锅 +出错 +出门 +出院 +出面 +击中 +击杀 +击球 +击败 +函授 +函数 +刀具 +刀切 +刀剑 +刀子 +刀片 +分不开 +分不清 +分为 +分之一 +分享 +分会 +分公司 +分别 +分割 +分化 +分区 +分发 +分型 +分外 +分娩 +分子 +分寸 +分局 +分层 +分工 +分布 +分布式 +分开 +分成 +分手 +分批 +分担 +分支 +分支机构 +分散 +分数 +分数线 +分时 +分明 +分期 +分机 +分析 +分析师 +分枝 +分校 +分歧 +分段 +分水 +分泌 +分泌物 +分流 +分清 +分界 +分离 +分管 +分类 +分红 +分级 +分组 +分给 +分行 +分裂 +分解 +分辨 +分辨率 +分部 +分配 +分量 +分钟 +分销 +分队 +分院 +分隔 +切不可 +切丝 +切入 +切入点 +切切 +切削 +切割 +切勿 +切口 +切合 +切块 +切实 +切尔 +切尔西 +切开 +切忌 +切成 +切换 +切断 +切片 +切碎 +切磋 +切记 +切身 +切除 +刊物 +刊登 +刑事 +刑事拘留 +刑事案件 +刑事责任 +刑法 +刑罚 +刑警 +刑部 +划入 +划分 +划定 +划算 +划线 +列为 +列举 +列入 +列出 +列宁 +列表 +列车 +列颠 +刘海 +刚上 +刚买 +刚入 +刚出 +刚刚 +刚刚开始 +刚好 +刚性 +刚才 +创下 +创世 +创业 +创业板 +创业者 +创伤 +创作 +创作者 +创出 +创刊 +创办 +创始 +创始人 +创客 +创建 +创意 +创投 +创新 +创新型 +创新性 +创新能力 +创立 +创设 +创造 +创造力 +创造性 +初一 +初三 +初中 +初二 +初入 +初具 +初创 +初始 +初学 +初学者 +初审 +初心 +初恋 +初期 +初次 +初步 +初级 +初衷 +初见 +初试 +删除 +判决 +判别 +判处 +判定 +判断 +利于 +利亚 +利器 +利好 +利害 +利尿 +利己 +利息 +利斯 +利比亚 +利润 +利润率 +利物 +利物浦 +利率 +利用 +利用率 +利益 +利税 +利空 +利落 +别人 +别克 +别具 +别名 +别墅 +别太 +别忘了 +别想 +别扭 +别提 +别有 +别样 +别的 +别看 +别离 +别致 +别说 +到位 +到场 +到处 +到大 +到头 +到头来 +到家 +到底 +到手 +到时候 +到晚 +到期 +到来 +到达 +制作 +制作方法 +制冷 +制剂 +制动 +制品 +制图 +制备 +制定 +制宜 +制导 +制度 +制式 +制成 +制服 +制止 +制法 +制热 +制片 +制片人 +制程 +制约 +制胜 +制药 +制裁 +制订 +制造 +制造业 +制造商 +制高 +刷刷 +刷卡 +刷子 +刷屏 +刷新 +刷牙 +刷票 +券商 +刹车 +刹那 +刺史 +刺客 +刺杀 +刺激 +刺激性 +刺猬 +刺痛 +刺绣 +刻刻 +刻度 +刻录 +刻意 +刻画 +刻苦 +刻骨 +刻骨铭心 +剂量 +削减 +削弱 +前一天 +前三 +前不久 +前世 +前两天 +前些 +前人 +前任 +前值 +前列 +前列腺 +前列腺炎 +前十 +前卫 +前去 +前台 +前后 +前夕 +前天 +前奏 +前女 +前妻 +前年 +前往 +前所 +前所未有 +前排 +前提 +前方 +前景 +前期 +前来 +前段 +前段时间 +前沿 +前瞻 +前瞻性 +前程 +前端 +前线 +前置 +前者 +前脸 +前茅 +前行 +前要 +前言 +前身 +前辈 +前进 +前途 +前锋 +前面 +剑桥 +剔透 +剔除 +剖析 +剖腹 +剖腹产 +剖面 +剥削 +剥夺 +剥皮 +剥离 +剧中 +剧作 +剧场 +剧情 +剧本 +剧烈 +剧照 +剧目 +剧组 +剧透 +剧院 +剧集 +剩下 +剩余 +剩女 +剪刀 +剪切 +剪纸 +剪裁 +剪辑 +副主 +副产品 +副会 +副会长 +副作用 +副厅 +副县 +副县长 +副处 +副局 +副局长 +副总 +副总经理 +副总裁 +副教授 +副本 +副校 +副校长 +副省 +副科 +副组 +割据 +力争 +力图 +力学 +力宏 +力度 +力所能及 +力气 +力求 +力矩 +力量 +劝导 +劝说 +劝阻 +办事 +办事处 +办公 +办公厅 +办公室 +办公桌 +办公楼 +办好 +办学 +办案 +办法 +办理 +功利 +功力 +功劳 +功勋 +功名 +功夫 +功底 +功德 +功效 +功法 +功率 +功用 +功耗 +功能 +功能性 +功能障碍 +功臣 +功课 +加上 +加之 +加仓 +加以 +加价 +加倍 +加入 +加入适量 +加减 +加分 +加利 +加剧 +加压 +加厚 +加固 +加大 +加密 +加州 +加工 +加工厂 +加床 +加强 +加快 +加息 +加成 +加拉 +加拿大 +加持 +加收 +加斯 +加水 +加油 +加油站 +加深 +加湿 +加点 +加热 +加班 +加盐 +加盖 +加盟 +加盟商 +加盟店 +加码 +加索 +加紧 +加薪 +加装 +加载 +加速 +加重 +加长 +务实 +务局 +务工 +务必 +劣势 +劣质 +动不动 +动乱 +动人 +动作 +动力 +动力学 +动力电池 +动力系统 +动向 +动听 +动员 +动工 +动心 +动态 +动感 +动手 +动摇 +动机 +动植物 +动漫 +动物 +动物园 +动用 +动画 +动画片 +动症 +动能 +动脉 +动脉硬化 +动脑 +动荡 +动词 +动车 +动辄 +动静 +动魄 +助人 +助力 +助听 +助听器 +助学 +助手 +助推 +助攻 +助理 +助长 +努力 +努力学习 +劫持 +励志 +劲儿 +劳作 +劳力 +劳务 +劳动 +劳动力 +劳动合同 +劳动者 +劳工 +劳斯 +劳累 +劳资 +势利 +势力 +势头 +势必 +势必会 +勃勃 +勃起 +勇于 +勇士 +勇士队 +勇敢 +勇气 +勇猛 +勇者 +勉强 +勋章 +勒布 +勒索 +勘察 +勘探 +勘查 +勘测 +募捐 +募集 +勤俭 +勤劳 +勤奋 +勤工 +勺子 +勾勒 +勾搭 +勿扰 +包养 +包包 +包含 +包围 +包头 +包子 +包容 +包括 +包皮 +包袱 +包装 +包裹 +包邮 +匆匆 +匆忙 +匈奴 +匈牙利 +化为 +化作 +化合 +化合物 +化名 +化妆 +化妆品 +化学 +化学反应 +化学品 +化学物质 +化工 +化成 +化疗 +化痰 +化石 +化纤 +化肥 +化脓 +化解 +化身 +化验 +北上 +北京 +北京大学 +北京市 +北伐 +北侧 +北区 +北大 +北宋 +北平 +北斗 +北方 +北极 +北欧 +北汽 +北洋 +北海 +北站 +北约 +北纬 +北美 +北航 +北路 +北邻 +北部 +北面 +北风 +北魏 +北齐 +匠心 +匪夷 +匮乏 +匹林 +匹配 +区位 +区内 +区分 +区划 +区别 +区县 +区块 +区域 +区域性 +区委 +区政府 +区长 +区间 +医保 +医务 +医务人员 +医嘱 +医学 +医学会 +医学院 +医师 +医护 +医护人员 +医术 +医治 +医生 +医用 +医疗 +医疗保险 +医疗卫生 +医疗器械 +医疗机构 +医科 +医科大学 +医药 +医闹 +医院 +匿名 +十一 +十一月 +十一条 +十七 +十七条 +十七章 +十万 +十三 +十三条 +十三章 +十个 +十九 +十九章 +十二 +十二五 +十二月 +十二条 +十二章 +十五 +十五章 +十位 +十余 +十佳 +十倍 +十八 +十八章 +十六 +十六章 +十几 +十几个 +十几年 +十分 +十分钟 +十四 +十四条 +十四章 +十块 +十堰 +十多 +十多年 +十大 +十天 +十字 +十字路口 +十岁 +十年 +十年代 +十方 +十日 +十月 +十条 +十来 +十点 +十章 +十足 +十里 +千万 +千万别 +千人 +千亿 +千余 +千元 +千兆 +千克 +千千万 +千古 +千叶 +千字 +千家 +千寻 +千山 +千年 +千方 +千方百计 +千玺 +千瓦 +千百 +千秋 +千篇 +千米 +千里 +千金 +升值 +升华 +升学 +升温 +升级 +升职 +升至 +升起 +升降 +升高 +午后 +午夜 +午睡 +午餐 +午饭 +半个 +半决赛 +半句 +半场 +半夏 +半夜 +半天 +半导体 +半小 +半小时 +半山 +半岛 +半年 +半径 +半数 +半斤 +半月 +半死 +半点 +半球 +半生 +半路 +半身 +半部 +华东 +华中 +华为 +华丽 +华人 +华侨 +华北 +华南 +华夏 +华尔 +华尔街 +华山 +华文 +华润 +华盛顿 +华硕 +华纳 +华美 +华西 +华语 +协会 +协会会员 +协作 +协力 +协办 +协助 +协同 +协和 +协商 +协奏 +协定 +协议 +协议书 +协调 +卑微 +卑鄙 +卓著 +卓越 +单一 +单个 +单亲 +单人 +单从 +单价 +单位 +单体 +单元 +单击 +单列 +单单 +单反 +单向 +单品 +单子 +单手 +单打 +单日 +单曲 +单机 +单板 +单核 +单片 +单独 +单相 +单纯 +单薄 +单行 +单词 +单调 +单身 +单车 +单边 +单项 +卖出 +卖场 +卖家 +卖掉 +卖方 +卖淫 +卖点 +卖萌 +南下 +南亚 +南京 +南京市 +南侧 +南充 +南北 +南北朝 +南区 +南县 +南宁 +南宁市 +南安 +南宋 +南宫 +南山 +南岸 +南开 +南方 +南无 +南昌 +南朝 +南极 +南沙 +南洋 +南海 +南湖 +南瓜 +南站 +南美 +南航 +南路 +南通 +南邻 +南部 +南门 +南阳 +南非 +南面 +博主 +博会 +博士 +博士后 +博士学位 +博士生 +博大 +博大精深 +博学 +博客 +博弈 +博彩 +博文 +博爱 +博物 +博物院 +博物馆 +博览 +博览会 +占卜 +占地 +占地面积 +占据 +占有 +占有率 +占用 +占领 +卡尔 +卡拉 +卡斯 +卡片 +卡牌 +卡特 +卡组 +卡罗 +卡路里 +卡车 +卡通 +卡顿 +卢卡 +卧室 +卧床 +卧槽 +卧铺 +卧龙 +卫健 +卫国 +卫士 +卫星 +卫浴 +卫生 +卫生组织 +卫生部 +卫生间 +卫生院 +卫衣 +卫视 +印制 +印刷 +印发 +印尼 +印度 +印章 +印花 +印记 +印证 +印象 +危及 +危害 +危急 +危机 +危重 +危险 +危险性 +即位 +即使 +即便 +即兴 +即刻 +即可 +即将 +即日起 +即时 +即用 +却是 +却说 +卵圆 +卵子 +卵巢 +卵形 +卵用 +卵管 +卷入 +卷发 +卷子 +卷烟 +卷轴 +卸妆 +卸载 +厂商 +厂家 +厂房 +厂长 +厅长 +历代 +历任 +历史 +历史性 +历史悠久 +历年 +历时 +历来 +历程 +历练 +历经 +历险 +厉害 +压倒 +压制 +压力 +压器 +压实 +压抑 +压根 +压榨 +压缩 +压缩机 +压迫 +厌倦 +厌恶 +厌烦 +厕所 +厘头 +厘米 +厚厚 +厚厚的 +厚实 +厚度 +厚道 +厚重 +厚非 +原产 +原产地 +原件 +原价 +原住 +原作 +原作者 +原先 +原则 +原则上 +原创 +原发 +原名 +原告 +原味 +原唱 +原因 +原地 +原型 +原声 +原始 +原子 +原文 +原料 +原有 +原本 +原材料 +原来 +原汁 +原汁原味 +原油 +原版 +原理 +原生 +原生态 +原画 +原装 +原谅 +厢情 +厦门 +厦门市 +厨具 +厨师 +厨房 +去世 +去向 +去处 +去年 +去年同期 +去往 +去掉 +去皮 +去过 +去除 +县城 +县域 +县委 +县委书记 +县市 +县政府 +县级 +县长 +参与 +参与者 +参会 +参保 +参军 +参加 +参展 +参差 +参战 +参政 +参数 +参演 +参照 +参考 +参考书 +参考文献 +参考资料 +参见 +参观 +参议 +参谋 +参谋长 +参赛 +参选 +又名 +又称 +及其 +及待 +及早 +及时 +及时发现 +及格 +友人 +友们 +友善 +友好 +友情 +友爱 +友谊 +双人 +双休 +双侧 +双击 +双十 +双双 +双向 +双子 +双层 +双床 +双手 +双打 +双方 +双核 +双眼 +双眼皮 +双肩 +双胞 +双胞胎 +双脚 +双腿 +双臂 +双色 +双语 +双赢 +双边 +双重 +双面 +双鱼 +双龙 +反之 +反倒 +反光 +反击 +反动 +反反 +反反复复 +反叛 +反向 +反响 +反复 +反对 +反射 +反差 +反应 +反弹 +反思 +反恐 +反感 +反抗 +反攻 +反映 +反智 +反正 +反派 +反省 +反而 +反腐 +反观 +反转 +反过来 +反问 +反面 +反顾 +反馈 +反驳 +发了 +发传 +发作 +发光 +发出 +发到 +发力 +发动 +发动机 +发呆 +发售 +发型 +发声 +发射 +发小 +发展 +发展中国家 +发展前景 +发展史 +发展壮大 +发展潜力 +发展观 +发展趋势 +发布 +发布会 +发布公告 +发帖 +发心 +发性 +发愁 +发愿 +发扬 +发抖 +发指 +发挥 +发挥作用 +发掘 +发改 +发改委 +发放 +发散 +发文 +发明 +发明专利 +发来 +发泄 +发泡 +发源 +发源地 +发火 +发炎 +发烧 +发热 +发现 +发现自己 +发球 +发生 +发生变化 +发生率 +发电 +发电机 +发病 +发病率 +发短信 +发票 +发红 +发给 +发育 +发胖 +发脾气 +发自 +发自内心 +发芽 +发行 +发行人 +发表 +发觉 +发言 +发言人 +发誓 +发证 +发财 +发货 +发起 +发车 +发达 +发达国家 +发送 +发酵 +发问 +发霉 +发音 +发黄 +发黑 +叔叔 +取代 +取其 +取决 +取决于 +取出 +取名 +取向 +取得 +取得成功 +取悦 +取景 +取暖 +取材 +取消 +取经 +取缔 +取胜 +取舍 +取证 +受不了 +受人 +受众 +受伤 +受体 +受到 +受制 +受力 +受命 +受孕 +受宠 +受害 +受害人 +受害者 +受损 +受欢迎 +受灾 +受理 +受用 +受益 +受精 +受聘 +受苦 +受访 +受访者 +受贿 +受过 +受邀 +受阻 +受限 +受骗 +变为 +变动 +变化 +变压器 +变小 +变幻 +变异 +变形 +变得 +变态 +变性 +变成 +变换 +变更 +变焦 +变现 +变电 +变白 +变相 +变美 +变色 +变质 +变迁 +变速 +变速器 +变速箱 +变量 +变革 +变频 +变黑 +叙事 +叙利亚 +叙述 +叛乱 +叛逆 +叠加 +口中 +口令 +口号 +口吃 +口吻 +口味 +口头 +口岸 +口径 +口感 +口才 +口服 +口气 +口水 +口渴 +口碑 +口红 +口罩 +口腔 +口腔溃疡 +口臭 +口舌 +口袋 +口诀 +口译 +口语 +古人 +古今 +古代 +古兰 +古典 +古城 +古墓 +古巴 +古希腊 +古建筑 +古怪 +古文 +古时 +古朴 +古树 +古玩 +古琴 +古筝 +古籍 +古老 +古董 +古装 +古诗 +古迹 +古道 +古镇 +古风 +古龙 +句型 +句子 +句话 +另一 +另一半 +另一只 +另一方 +另一方面 +另一边 +另一面 +另外 +另有 +另类 +另行 +只不过 +只为 +只会 +只好 +只得 +只怕 +只想 +只是 +只有 +只求 +只用 +只管 +只能 +只能靠 +只要 +只见 +只顾 +叫做 +叫声 +叫好 +叫醒 +召唤 +召回 +召开 +召集 +叮嘱 +可不 +可不可以 +可不是 +可为 +可乐 +可以 +可供 +可信 +可厚 +可取 +可变 +可口 +可口可乐 +可可 +可否 +可喜 +可忍 +可怕 +可怜 +可思 +可恶 +可悲 +可惜 +可想而知 +可控 +可数 +可是 +可比 +可治 +可溶 +可爱 +可玩 +可用 +可疑 +可知 +可笑 +可耻 +可能 +可能性 +可行 +可行性 +可见 +可观 +可视 +可言 +可读 +可调 +可谓 +可贵 +可达 +可逆 +可选 +可靠 +可靠性 +台上 +台下 +台北 +台南 +台山 +台州 +台湾 +台灯 +台独 +台球 +台词 +台阶 +台面 +台风 +史上 +史书 +史学 +史密 +史料 +史蒂 +史记 +史诗 +右上 +右上角 +右下 +右侧 +右手 +右派 +右翼 +右脚 +右腿 +右转 +右边 +右铭 +叶子 +叶片 +叶轮 +叶酸 +号召 +号楼 +号牌 +号码 +号称 +号线 +司令 +司令部 +司徒 +司机 +司法 +司法机关 +司空 +司考 +司职 +司长 +司马 +叹息 +吃不 +吃亏 +吃些 +吃力 +吃喝 +吃喝玩乐 +吃完饭 +吃惊 +吃掉 +吃水果 +吃法 +吃点 +吃瓜 +吃素 +吃苦 +吃苦耐劳 +吃药 +吃货 +吃过 +吃醋 +吃零 +吃饭 +吃饱 +各不相同 +各业 +各个 +各个方面 +各个环节 +各人 +各位 +各区 +各县 +各国 +各地 +各处 +各大 +各家 +各异 +各式 +各式各样 +各方 +各族 +各有 +各样 +各界 +各省 +各种 +各种各样 +各科 +各类 +各级 +各自 +各色 +各行 +各行各业 +各路 +各项 +合一 +合乎 +合伙 +合伙人 +合体 +合作 +合作伙伴 +合作社 +合力 +合同 +合同法 +合唱 +合并 +合影 +合成 +合拍 +合格 +合法 +合法性 +合法权益 +合照 +合物 +合理 +合理安排 +合理性 +合租 +合约 +合群 +合肥 +合肥市 +合规 +合计 +合资 +合适 +合金 +吉他 +吉凶 +吉利 +吉安 +吉尔 +吉林 +吉林省 +吉祥 +吉祥物 +吊带 +吊销 +吊顶 +同一 +同一个 +同业 +同乡 +同事 +同仁 +同伴 +同名 +同城 +同姓 +同学 +同居 +同年 +同心 +同志 +同性 +同性恋 +同情 +同意 +同日 +同时 +同期 +同样 +同桌 +同款 +同步 +同比 +同治 +同济 +同源 +同班 +同理 +同盟 +同等 +同类 +同级 +同胞 +同行 +同行业 +同质 +同道 +同食 +同龄 +同龄人 +名下 +名为 +名义 +名人 +名优 +名作 +名列 +名列前茅 +名利 +名副 +名副其实 +名医 +名单 +名叫 +名号 +名城 +名声 +名字 +名家 +名将 +名师 +名录 +名曰 +名校 +名次 +名气 +名片 +名牌 +名称 +名胜 +名胜区 +名著 +名言 +名誉 +名词 +名贵 +名额 +后世 +后人 +后代 +后任 +后会 +后勤 +后半 +后卫 +后台 +后备 +后备箱 +后天 +后宫 +后市 +后悔 +后排 +后方 +后期 +后来 +后果 +后汉 +后盾 +后继 +后续 +后缀 +后置 +后者 +后背 +后裔 +后要 +后视 +后视镜 +后记 +后退 +后遗 +后遗症 +后门 +后面 +后顾 +后顾之忧 +吐槽 +吐蕃 +向上 +向下 +向东 +向前 +向南 +向外 +向导 +向往 +向日葵 +向来 +向着 +向西 +向量 +向阳 +吓人 +吓唬 +吓死 +君主 +君子 +君王 +君臣 +吝啬 +吝赐 +吞吐 +吞咽 +吞噬 +否决 +否则 +否定 +否认 +含义 +含有 +含水 +含糊 +含糖 +含蓄 +含量 +含金量 +听不懂 +听从 +听众 +听到 +听力 +听取 +听听 +听得 +听懂 +听歌 +听见 +听觉 +听讲 +听证 +听话 +听说 +听课 +听过 +听途 +听闻 +启动 +启发 +启用 +启示 +启蒙 +启超 +启迪 +吴国 +吴江 +吴越 +吵吵 +吵架 +吵闹 +吸入 +吸取 +吸尘 +吸引 +吸引力 +吸收 +吸毒 +吸气 +吸水 +吸烟 +吸纳 +吸脂 +吸血 +吸血鬼 +吸附 +吸食 +吹牛 +吹风 +吹风机 +吻合 +呆呆 +呈现 +呈现出 +告别 +告白 +告知 +告终 +告诉 +告诫 +呐喊 +呕吐 +员外 +员工 +周一 +周三 +周二 +周五 +周公 +周六 +周刊 +周到 +周四 +周围 +周密 +周岁 +周年 +周恩来 +周旋 +周日 +周易 +周期 +周期性 +周末 +周炎 +周知 +周礼 +周线 +周身 +周转 +周边 +周遭 +味儿 +味甘 +味精 +味蕾 +味觉 +味道 +呵呵 +呵护 +呻吟 +呼叫 +呼吁 +呼吸 +呼吸困难 +呼吸系统 +呼吸道 +呼呼 +呼和浩特 +呼唤 +呼啸 +呼喊 +呼噜 +呼声 +呼应 +呼气 +命中 +命中率 +命令 +命名 +命局 +命理 +命运 +命门 +命题 +咀嚼 +咆哮 +和尚 +和平 +和弦 +和田 +和田玉 +和睦 +和蔼 +和解 +和谐 +和谐社会 +咒语 +咕噜 +咖啡 +咖啡因 +咖啡馆 +咖喱 +咨询 +咨询服务 +咨询电话 +咫尺 +咬牙 +咱们 +咳嗽 +咳崩 +咸丰 +咸阳 +咸鱼 +咽下 +咽喉 +咽炎 +哀乐 +品位 +品味 +品尝 +品德 +品格 +品牌 +品牌形象 +品种 +品类 +品行 +品质 +哈佛 +哈佛大学 +哈利 +哈哈 +哈哈大笑 +哈密 +哈尔 +哈尔滨 +哈尔滨市 +哈工 +哈弗 +哈特 +哈登 +哈萨克 +哈里 +响亮 +响应 +响起 +哎呀 +哑巴 +哔哩 +哗众 +哗啦 +哥们 +哥伦 +哥伦比亚 +哥哥 +哥特 +哪一 +哪个 +哪些 +哪位 +哪儿 +哪天 +哪家 +哪怕 +哪有 +哪来 +哪种 +哪里 +哭声 +哭泣 +哭笑 +哭闹 +哮喘 +哲学 +哲学家 +哲理 +哺乳 +哺乳期 +哼哼 +唇膏 +唏嘘 +唐书 +唐人 +唐代 +唐僧 +唐宋 +唐山 +唐朝 +唐诗 +唠叨 +唤起 +唤醒 +售价 +售出 +售卖 +售后 +售后服务 +售楼 +售票 +售货 +唯一 +唯个 +唯有 +唯物 +唯独 +唯美 +唱歌 +唱片 +唾液 +商丘 +商业 +商业化 +商业地产 +商业模式 +商业街 +商业银行 +商人 +商会 +商务 +商务部 +商品 +商品房 +商圈 +商场 +商城 +商学院 +商家 +商店 +商户 +商报 +商机 +商标 +商用 +商界 +商行 +商贸 +商量 +商铺 +啊啊 +啤酒 +啧啧 +啰嗦 +喀什 +喂养 +喂奶 +喂食 +善事 +善于 +善待 +善恶 +善意 +善根 +善用 +善良 +喇叭 +喇嘛 +喉咙 +喜事 +喜剧 +喜好 +喜庆 +喜怒 +喜怒哀乐 +喜悦 +喜欢 +喜爱 +喝一杯 +喝咖啡 +喝多 +喝完 +喝彩 +喝水 +喝汤 +喝点 +喝茶 +喝酒 +喝醉 +喧嚣 +喧闹 +喷出 +喷剂 +喷嚏 +喷射 +喷气 +喷水 +喷泉 +喷洒 +喷涂 +喷雾 +嗅觉 +嗓子 +嗓音 +嗜血 +嗷嗷 +嘈杂 +嘉兴 +嘉奖 +嘉定 +嘉宾 +嘉年华 +嘉庆 +嘉靖 +嘟嘟 +嘱咐 +嘲笑 +嘲讽 +嘴上 +嘴唇 +嘴巴 +嘴炮 +嘴角 +嘴边 +嘴里 +嘻嘻 +嘿嘿 +器乐 +器件 +器具 +器官 +器材 +器械 +器物 +器皿 +噩梦 +噪声 +噪音 +嚣张 +囊括 +囊肿 +四个 +四五 +四人 +四位 +四军 +四分 +四分之一 +四十 +四千 +四合 +四周 +四处 +四大 +四天 +四季 +四季度 +四射 +四川 +四川省 +四年 +四库 +四方 +四时 +四是 +四月 +四条 +四柱 +四次 +四海 +四点 +四环 +四百 +四种 +四章 +四级 +四维 +四肢 +四角 +四通 +四面 +四项 +四驱 +回乡 +回事 +回到 +回升 +回去 +回合 +回味 +回国 +回复 +回头 +回家 +回应 +回归 +回忆 +回忆起 +回想 +回报 +回报率 +回撤 +回收 +回旋 +回族 +回暖 +回来 +回民 +回流 +回眸 +回答 +回老 +回老家 +回落 +回访 +回调 +回购 +回路 +回转 +回过 +回避 +回顾 +回馈 +回首 +因为 +因人 +因人而异 +因其 +因地 +因地制宜 +因子 +因果 +因此 +因素 +因缘 +因而 +团伙 +团体 +团员 +团团 +团圆 +团委 +团战 +团支 +团结 +团聚 +团购 +团长 +团队 +园内 +园区 +园林 +园艺 +困境 +困惑 +困扰 +困难 +围城 +围墙 +围巾 +围攻 +围棋 +围着 +围绕 +围观 +固体 +固化 +固定 +固定资产 +固态 +固执 +固有 +固然 +固醇 +国产 +国人 +国企 +国会 +国债 +国共 +国内 +国内外 +国军 +国力 +国务 +国务院 +国土 +国土资源 +国外 +国学 +国安 +国宝 +国家 +国家一级 +国家有关 +国家机关 +国家标准 +国家税务 +国家税务总局 +国家级 +国家统计局 +国家队 +国庆 +国庆节 +国库 +国度 +国情 +国旗 +国有 +国有企业 +国有资产 +国服 +国标 +国歌 +国民 +国民党 +国民经济 +国泰 +国王 +国画 +国税 +国立 +国籍 +国美 +国荣 +国药 +国营 +国藩 +国语 +国货 +国贸 +国资 +国资委 +国足 +国道 +国门 +国防 +国防部 +国际 +国际化 +国际标准 +国际贸易 +图上 +图中 +图为 +图书 +图书馆 +图像 +图册 +图层 +图形 +图文 +图标 +图案 +图片 +图画 +图纸 +图腾 +图表 +图解 +圆圆 +圆弧 +圆形 +圆柱 +圆桌 +圆润 +圆满 +圆筒 +圆锥 +圈内 +圈子 +圈里 +土司 +土地 +土壤 +土建 +土星 +土木 +土耳其 +土著 +土豆 +土豪 +圣人 +圣地 +圣殿 +圣母 +圣经 +圣诞 +圣诞节 +圣贤 +在一起 +在世 +在世界上 +在乎 +在于 +在位 +在内 +在历史上 +在场 +在外 +在实践中 +在家 +在座 +在建 +在意 +在我 +在我看来 +在所 +在手 +在校 +在校学生 +在此 +在此之前 +在此期间 +在生活 +在生活中 +在线 +在职 +在读 +地上 +地下 +地下室 +地下水 +地中海 +地为 +地主 +地久 +地亚 +地产 +地价 +地位 +地利 +地势 +地区 +地去 +地名 +地向 +地图 +地址 +地块 +地坪 +地域 +地基 +地壳 +地处 +地委 +地层 +地带 +地形 +地想 +地摊 +地支 +地方 +地板 +地标 +地步 +地段 +地毯 +地点 +地狱 +地球 +地理 +地理位置 +地瓜 +地用 +地盘 +地砖 +地税 +地笑 +地级 +地缘 +地藏 +地表 +地被 +地说 +地貌 +地质 +地道 +地里 +地铁 +地铁站 +地问 +地震 +地面 +地黄 +场上 +场内 +场合 +场地 +场均 +场外 +场所 +场景 +场面 +场馆 +均价 +均匀 +均线 +均衡 +坎坷 +坏习惯 +坏事 +坏人 +坏死 +坐下 +坐在 +坐姿 +坐标 +坐落 +坐落于 +坐车 +块状 +块钱 +坚信 +坚决 +坚固 +坚守 +坚定 +坚定不移 +坚实 +坚强 +坚持 +坚持下去 +坚持不懈 +坚挺 +坚果 +坚硬 +坚韧 +坟墓 +坠落 +坦克 +坦然 +坦白 +坦言 +坦诚 +垂体 +垂直 +垂钓 +垃圾 +垃圾桶 +垄断 +型号 +型材 +垫底 +埃及 +埃尔 +埃斯 +埋头 +埋怨 +埋没 +埋葬 +埋藏 +城东 +城中 +城乡 +城乡居民 +城关 +城内 +城北 +城区 +城南 +城县 +城堡 +城墙 +城市 +城市化 +城市群 +城建 +城管 +城西 +城郊 +城里 +城镇 +城镇化 +城门 +城际 +城隍 +域名 +培养 +培植 +培育 +培训 +培训中心 +培训班 +基于 +基准 +基友 +基因 +基地 +基尼 +基层 +基建 +基数 +基本 +基本上 +基本功 +基本知识 +基本面 +基点 +基督 +基督徒 +基督教 +基石 +基础 +基础教育 +基础知识 +基础设施 +基站 +基调 +基质 +基部 +基金 +基金会 +堂堂 +堂皇 +堆放 +堆积 +堕胎 +堕落 +堡垒 +堪比 +堪称 +堵住 +堵塞 +堵车 +塑性 +塑料 +塑料袋 +塑胶 +塑造 +塔尔 +塞尔 +塞罗 +塞进 +填充 +填写 +填报 +填料 +填满 +填空 +填补 +填词 +境内 +境地 +境外 +境界 +墓地 +墓志 +墓葬 +墙上 +墙体 +墙壁 +墙角 +墙面 +增产 +增值 +增值税 +增减 +增加 +增加值 +增压 +增发 +增多 +增大 +增幅 +增强 +增持 +增收 +增效 +增殖 +增添 +增生 +增益 +增肌 +增至 +增补 +增设 +增资 +增进 +增速 +增量 +增长 +增长率 +增长速度 +增高 +墨尔本 +墨水 +墨西哥 +墨迹 +墨镜 +壁垒 +壁挂 +壁画 +壁纸 +士兵 +士官 +士气 +士顿 +壮丽 +壮大 +壮志 +壮族 +壮烈 +壮观 +壮阔 +壮阳 +声乐 +声名 +声响 +声器 +声声 +声学 +声明 +声望 +声波 +声称 +声色 +声誉 +声音 +处世 +处事 +处于 +处以 +处分 +处在 +处境 +处处 +处女 +处方 +处死 +处理 +处理器 +处级 +处罚 +处置 +处长 +备份 +备受 +备忘 +备战 +备案 +备注 +备用 +备考 +备课 +复习 +复产 +复仇 +复位 +复兴 +复出 +复制 +复印 +复印件 +复原 +复发 +复古 +复合 +复合材料 +复复 +复审 +复工 +复旦 +复旦大学 +复杂 +复杂性 +复查 +复核 +复活 +复苏 +复议 +复试 +复读 +夏令 +夏令营 +夏侯 +夏天 +夏威夷 +夏季 +夏日 +夏普 +夏洛 +夏秋 +夕阳 +外乎 +外事 +外交 +外交部 +外人 +外企 +外伤 +外侧 +外公 +外出 +外力 +外加 +外包 +外卖 +外号 +外向 +外商 +外围 +外国 +外国人 +外国语 +外地 +外墙 +外壳 +外套 +外婆 +外媒 +外层 +外币 +外延 +外形 +外挂 +外接 +外援 +外文 +外星 +外星人 +外景 +外来 +外汇 +外汇储备 +外甥 +外用 +外界 +外皮 +外省 +外科 +外籍 +外线 +外置 +外行 +外衣 +外表 +外观 +外观设计 +外语 +外貌 +外贸 +外资 +外边 +外部 +外链 +外阴 +外面 +多一些 +多一点 +多万 +多万元 +多个 +多为 +多久 +多么 +多于 +多云 +多些 +多人 +多以 +多伦 +多位 +多余 +多元 +多元化 +多公里 +多出 +多功能 +多加 +多半 +多单 +多发 +多变 +多名 +多喝水 +多地 +多坏 +多处 +多多 +多大 +多天 +多头 +多好 +多姿 +多媒体 +多家 +多少 +多层 +多层次 +多岁 +多年 +多彩 +多得 +多情 +多想 +多才 +多数 +多方 +多方面 +多月 +多有 +多条 +多样 +多样化 +多样性 +多次 +多款 +多点 +多用 +多种 +多种多样 +多种形式 +多空 +多篇 +多米 +多糖 +多维 +多芬 +多见 +多见于 +多谢 +多达 +多远 +多部 +多重 +多钱 +多长 +多长时间 +多难 +多项 +多食 +夜夜 +夜市 +夜幕 +夜店 +夜总 +夜晚 +夜景 +夜深 +夜班 +夜空 +夜色 +夜里 +夜间 +大一 +大三 +大不了 +大专 +大业 +大中型 +大为 +大乘 +大事 +大二 +大于 +大人 +大件 +大众 +大伙 +大会 +大体 +大佛 +大作 +大佬 +大使 +大使馆 +大侠 +大便 +大全 +大关 +大兴 +大写 +大军 +大减 +大到 +大力 +大力发展 +大力支持 +大势 +大匙 +大千 +大半 +大卫 +大厅 +大厦 +大发 +大叔 +大叫 +大可 +大吃 +大吉 +大同 +大名 +大吵 +大哥 +大哭 +大唐 +大喊 +大喜 +大四 +大国 +大圣 +大地 +大块 +大坝 +大型 +大城市 +大堂 +大堆 +大增 +大声 +大多 +大多数 +大大 +大大增加 +大大小小 +大大提高 +大大的 +大大降低 +大夫 +大头 +大奖 +大好 +大妈 +大姐 +大姨 +大字 +大学 +大学毕业 +大学生 +大宗 +大宝 +大家 +大家庭 +大将 +大将军 +大小 +大局 +大屏 +大展 +大山 +大峡谷 +大巴 +大师 +大帝 +大幅 +大幅度 +大庆 +大度 +大开 +大德 +大怒 +大悟 +大悲 +大意 +大戏 +大成 +大战 +大户 +大手 +大打 +大打折扣 +大批 +大把 +大拇指 +大招 +大方 +大明 +大有 +大本 +大本营 +大杯 +大林 +大枣 +大树 +大桥 +大棚 +大楼 +大概 +大殿 +大气 +大气污染 +大汉 +大汗 +大江 +大河 +大法 +大洋 +大海 +大涨 +大清 +大湾 +大潮 +大火 +大灯 +大炮 +大热 +大熊猫 +大爱 +大爷 +大片 +大牌 +大牛 +大王 +大理 +大理石 +大病 +大白 +大白菜 +大盘 +大碗 +大神 +大笑 +大米 +大红 +大约 +大纲 +大肆 +大肠 +大胆 +大胜 +大脑 +大腿 +大臣 +大自然 +大致 +大葱 +大蒜 +大街 +大街小巷 +大衣 +大西洋 +大观 +大规模 +大豆 +大象 +大财 +大赏 +大赛 +大跌 +大路 +大辞 +大运 +大连 +大连市 +大选 +大通 +大道 +大部 +大部分 +大都 +大都市 +大酒店 +大量 +大钱 +大长 +大门 +大闸 +大闹 +大队 +大阪 +大陆 +大院 +大雨 +大雪 +大面积 +大革命 +大额 +大风 +大餐 +大骂 +大鱼 +大鹏 +大麦 +大麻 +大黄 +大龄 +天一 +天上 +天下 +天下第一 +天主 +天亮 +天人 +天价 +天体 +天使 +天候 +天内 +天台 +天后 +天启 +天呐 +天命 +天国 +天地 +天堂 +天大 +天天 +天子 +天安门 +天宝 +天宫 +天山 +天干 +天平 +天府 +天性 +天成 +天才 +天数 +天文 +天时 +天桥 +天气 +天气预报 +天水 +天池 +天河 +天津 +天津市 +天涯 +天灾 +天然 +天然气 +天猫 +天王 +天生 +天皇 +天盖 +天真 +天神 +天秤 +天空 +天窗 +天竺 +天籁 +天线 +天花 +天花板 +天蝎 +天赋 +天边 +天道 +天长 +天门 +天际 +天马 +天鹅 +天黑 +天龙 +太久 +太低 +太冷 +太原 +太原市 +太后 +太多 +太大 +太太 +太好 +太好了 +太子 +太守 +太宗 +太小 +太少 +太岁 +太差 +太平 +太平洋 +太强 +太快 +太晚 +太极 +太极拳 +太深 +太湖 +太白 +太监 +太祖 +太空 +太累 +太行 +太贵 +太过 +太远 +太重 +太阳 +太阳能 +太阴 +太难 +太高 +夫人 +夫妇 +夫妻 +夫妻俩 +夫子 +央企 +央行 +央视 +夯实 +失业 +失业率 +失传 +失信 +失利 +失去 +失常 +失忆 +失恋 +失意 +失控 +失效 +失明 +失望 +失真 +失眠 +失落 +失衡 +失误 +失调 +失败 +失踪 +头上 +头像 +头发 +头号 +头孢 +头戴 +头昏 +头晕 +头条 +头来 +头疼 +头痛 +头皮 +头盔 +头禅 +头等 +头脑 +头衔 +头部 +头顶 +夸大 +夸奖 +夸张 +夹克 +夹杂 +夺冠 +夺取 +夺回 +夺得 +夺目 +奇侠 +奇奇 +奇妙 +奇幻 +奇异 +奇怪 +奇想 +奇才 +奇特 +奇瑞 +奇石 +奇艺 +奇葩 +奇观 +奇迹 +奇遇 +奈何 +奉命 +奉献 +奉行 +奋力 +奋勇 +奋发 +奋战 +奋斗 +奋进 +契丹 +契合 +契机 +契约 +奔放 +奔波 +奔腾 +奔走 +奔赴 +奔跑 +奔驰 +奖励 +奖品 +奖学 +奖学金 +奖惩 +奖杯 +奖牌 +奖赛 +奖金 +奖项 +套利 +套房 +套牢 +套装 +套路 +套餐 +奠基 +奠定 +奢侈 +奢侈品 +奢华 +奢望 +奥会 +奥利 +奥地利 +奥尔 +奥尼 +奥巴 +奥巴马 +奥数 +奥斯 +奥斯卡 +奥林匹克 +奥特 +奥特曼 +奥秘 +奥运 +奥运会 +奥迪 +女主 +女主角 +女人 +女人味 +女儿 +女友 +女同 +女士 +女娲 +女婿 +女子 +女孩 +女孩儿 +女孩子 +女巫 +女性 +女性朋友 +女排 +女方 +女明星 +女星 +女朋友 +女权 +女演员 +女王 +女生 +女真 +女神 +女票 +女童 +女装 +女足 +女郎 +奴隶 +奶奶 +奶油 +奶牛 +奶瓶 +奶粉 +奶茶 +奶酪 +她们 +她家 +好不 +好不好 +好不容易 +好久 +好书 +好了 +好事 +好些 +好人 +好似 +好像 +好几 +好几个 +好几年 +好友 +好吃 +好听 +好喝 +好坏 +好处 +好多 +好奇 +好奇心 +好好 +好好学习 +好学 +好孩子 +好心 +好感 +好日子 +好歹 +好比 +好汉 +好消息 +好点 +好玩 +好用 +好看 +好笑 +好莱 +好莱坞 +好评 +好象 +好转 +好运 +如下 +如今 +如何 +如其 +如初 +如同 +如图 +如实 +如意 +如愿 +如是 +如有 +如期 +如来 +如来佛 +如果 +如果说 +如梦 +如此 +如此一来 +如水 +如火 +如火如荼 +如玉 +如生 +如画 +如花 +如若 +如荼 +如需 +妄想 +妆容 +妇产 +妇产科 +妇人 +妇女 +妇幼 +妇幼保健 +妇科 +妇科医院 +妇科疾病 +妇联 +妈咪 +妈妈 +妈祖 +妈蛋 +妊娠 +妓女 +妖怪 +妖魔 +妙用 +妥协 +妥善 +妥妥 +妥当 +妨碍 +妩媚 +妹妹 +妹子 +妹纸 +妻子 +姆斯 +姊妹 +始于 +始发 +始建 +始皇 +始祖 +始终 +始终保持 +始至 +姐妹 +姐姐 +姐弟 +姑且 +姑姑 +姑娘 +姓名 +姓氏 +委任 +委会 +委内瑞拉 +委员 +委员会 +委屈 +委托 +委托人 +委派 +姚明 +姜末 +姜片 +姥姥 +姥爷 +姨妈 +姻缘 +姿势 +姿态 +威严 +威信 +威力 +威尔 +威尼斯 +威廉 +威慑 +威斯 +威望 +威武 +威海 +威胁 +威风 +娃娃 +娇小 +娓娓 +娘娘 +娘子 +娘家 +娜娜 +娱乐 +娱乐圈 +娴熟 +婆婆 +婆媳 +婉转 +婚前 +婚后 +婚外 +婚姻 +婚庆 +婚恋 +婚礼 +婚纱 +婚纱照 +婴儿 +婴幼 +婴幼儿 +婷婷 +媒介 +媒体 +媒体报道 +媲美 +媳妇 +嫁妆 +嫁接 +嫁给 +嫂子 +嫉妒 +嫌弃 +嫌疑 +嫌疑人 +嫖娼 +嫦娥 +嬉戏 +子公司 +子女 +子孙 +子宫 +子弟 +子弹 +子房 +子里 +孔子 +孔明 +孔雀 +孕妇 +孕妈 +孕期 +孕育 +字体 +字典 +字号 +字帖 +字幕 +字形 +字数 +字样 +字母 +字画 +字眼 +字符 +字节 +字面 +存储 +存取 +存在 +存放 +存有 +存栏 +存款 +存活 +存货 +存量 +存钱 +孙中山 +孙女 +孙子 +孙悟空 +孝感 +孝敬 +孝顺 +孟子 +孢子 +季军 +季后 +季后赛 +季度 +季报 +季节 +季节性 +季风 +孤僻 +孤儿 +孤单 +孤寂 +孤独 +孤立 +孤身 +学业 +学习 +学习成绩 +学习者 +学了 +学会 +学位 +学佛 +学分 +学刊 +学到 +学前 +学前教育 +学区 +学医 +学历 +学名 +学员 +学园 +学堂 +学士 +学士学位 +学大 +学好 +学姐 +学子 +学学 +学家 +学年 +学府 +学徒 +学得 +学报 +学期 +学术 +学校 +学派 +学渣 +学生 +学生会 +学界 +学着 +学研 +学科 +学籍 +学系 +学者 +学识 +学说 +学费 +学过 +学金 +学长 +学问 +学院 +学霸 +学风 +孩子 +孩童 +孩纸 +孵化 +宁县 +宁可 +宁夏 +宁市 +宁德 +宁愿 +宁波 +宁波市 +宁静 +它们 +它会 +宅基地 +宇宙 +宇文 +宇航 +守住 +守信 +守候 +守卫 +守备 +守护 +守望 +守法 +守门 +安东 +安东尼 +安乐 +安保 +安倍 +安全 +安全事故 +安全可靠 +安全带 +安全性 +安全意识 +安全感 +安全法 +安全隐患 +安利 +安区 +安卓 +安县 +安吉 +安妮 +安娜 +安宁 +安安 +安定 +安居 +安市 +安庆 +安康 +安德 +安徽 +安徽省 +安心 +安息 +安慰 +安抚 +安排 +安放 +安检 +安溪 +安然 +安眠 +安石 +安神 +安稳 +安置 +安葬 +安装 +安详 +安迪 +安逸 +安防 +安阳 +安静 +宋代 +宋元 +宋史 +宋朝 +宋江 +完事 +完全 +完后 +完善 +完备 +完好 +完完 +完工 +完成 +完整 +完整性 +完毕 +完爆 +完结 +完美 +完颜 +完饭 +宏伟 +宏大 +宏观 +宏观经济 +宗教 +宗族 +宗旨 +官僚 +官兵 +官司 +官员 +官场 +官府 +官方 +官方网站 +官网 +官至 +定下 +定为 +定义 +定价 +定位 +定做 +定制 +定力 +定向 +定型 +定居 +定律 +定性 +定时 +定期 +定格 +定点 +定理 +定论 +定量 +定金 +定额 +宛如 +宜人 +宜兴 +宜家 +宜宾 +宜居 +宜昌 +宜春 +宝剑 +宝塔 +宝妈 +宝安 +宝宝 +宝山 +宝座 +宝强 +宝玉 +宝石 +宝藏 +宝贝 +宝贵 +宝钗 +宝马 +宝骏 +宝鸡 +实业 +实为 +实习 +实习生 +实事 +实事求是 +实体 +实体店 +实例 +实况 +实则 +实力 +实务 +实名 +实在 +实地 +实处 +实实 +实实在在 +实属 +实干 +实惠 +实战 +实操 +实效 +实施 +实施方案 +实施细则 +实时 +实景 +实木 +实物 +实现 +实用 +实用性 +实行 +实训 +实证 +实话 +实质 +实质性 +实践 +实践经验 +实际 +实际上 +实际操作 +实际行动 +实验 +实验学校 +实验室 +宠爱 +宠物 +审判 +审定 +审慎 +审批 +审查 +审核 +审理 +审稿 +审美 +审视 +审计 +审议 +客串 +客人 +客体 +客厅 +客场 +客家 +客座 +客户 +客户服务 +客户端 +客房 +客服 +客机 +客栈 +客气 +客流 +客源 +客票 +客观 +客车 +客运 +客运站 +宣传 +宣传教育 +宣传片 +宣传部 +宣判 +宣告 +宣布 +宣战 +宣扬 +宣泄 +宣称 +宣统 +宣言 +宣誓 +宣讲 +室内 +室友 +室外 +室温 +宪政 +宪法 +宫中 +宫廷 +宫殿 +宫腔 +宫颈 +宫颈癌 +宰相 +害人 +害怕 +害羞 +宴会 +家中 +家乡 +家人 +家伙 +家住 +家具 +家务 +家喻户晓 +家园 +家境 +家家 +家家户户 +家居 +家属 +家常 +家庄 +家庭 +家庭成员 +家庭教育 +家政 +家教 +家族 +家有 +家用 +家电 +家禽 +家装 +家里 +家里人 +家长 +家长会 +家门 +家门口 +家风 +容器 +容忍 +容易 +容积 +容积率 +容纳 +容貌 +容量 +容错 +容颜 +宽大 +宽容 +宽屏 +宽带 +宽广 +宽度 +宽恕 +宽敞 +宽松 +宽约 +宽阔 +宾客 +宾馆 +宿主 +宿命 +宿舍 +宿迁 +寂寞 +寂静 +寄主 +寄存 +寄宿 +寄托 +寄生 +寄生虫 +密不可分 +密切 +密切相关 +密宗 +密密 +密封 +密度 +密斯 +密码 +密钥 +密闭 +密集 +富于 +富人 +富养 +富力 +富含 +富士 +富士康 +富帅 +富强 +富有 +富民 +富翁 +富裕 +富豪 +富贵 +富足 +寒假 +寒冬 +寒冷 +寒凉 +寒气 +寒风 +寓意 +寓言 +寝室 +察觉 +寥寥 +对不起 +对于 +对人 +对付 +对冲 +对决 +对准 +对劲 +对华 +对口 +对外 +对外开放 +对峙 +对应 +对待 +对得起 +对战 +对手 +对抗 +对接 +对方 +对标 +对此 +对比 +对流 +对焦 +对照 +对症 +对称 +对立 +对策 +对联 +对话 +对话框 +对象 +对错 +对阵 +对面 +寺庙 +寺院 +寻常 +寻找 +寻求 +寻觅 +导体 +导入 +导出 +导向 +导图 +导师 +导引 +导弹 +导数 +导游 +导演 +导火 +导热 +导电 +导管 +导线 +导致 +导航 +导语 +导读 +导购 +寿司 +寿命 +寿险 +封为 +封信 +封印 +封口 +封建 +封杀 +封神 +封装 +封锁 +封闭 +封闭式 +封面 +封顶 +射击 +射手 +射程 +射箭 +射精 +射线 +射门 +射频 +将会 +将军 +将士 +将来 +将要 +将近 +将领 +尊严 +尊敬 +尊者 +尊贵 +尊重 +小三 +小丑 +小事 +小于 +小人 +小众 +小伙 +小伙伴 +小伙子 +小便 +小偷 +小儿 +小兔 +小刚 +小到 +小勺 +小包 +小区 +小卖 +小叶 +小号 +小吃 +小品 +小哥 +小块 +小型 +小城 +小声 +小女孩 +小妹 +小姐 +小姐姐 +小姑娘 +小姨 +小子 +小学 +小学生 +小孩 +小孩子 +小宝 +小宝宝 +小家 +小家伙 +小将 +小小 +小小的 +小屋 +小岛 +小巧 +小巷 +小幅 +小店 +小康 +小康社会 +小异 +小弟 +小微 +小心 +小心翼翼 +小户型 +小手 +小提琴 +小时 +小时候 +小明 +小朋友 +小板 +小林 +小桥 +小气 +小河 +小波 +小游戏 +小溪 +小火 +小熊 +小牛 +小狗 +小猪 +小猫 +小王 +小班 +小球 +小男孩 +小白 +小盘 +小看 +小窍门 +小米 +小红 +小组 +小组赛 +小结 +小编 +小肠 +小腹 +小腿 +小船 +小花 +小草 +小菜 +小虎 +小规模 +小觑 +小说 +小贴士 +小路 +小车 +小道 +小镇 +小雨 +小雪 +小额 +小马 +小鱼 +小鸟 +小鸡 +小麦 +小黑 +小鼠 +小龙 +小龙虾 +少不了 +少于 +少儿 +少女 +少将 +少少 +少年 +少年儿童 +少数 +少数民族 +少有 +少林 +少爷 +少见 +少许 +少量 +尔德 +尔斯 +尔曼 +尔盖 +尔蒙 +尔雅 +尔顿 +尖叫 +尖头 +尖端 +尖锐 +尖锐湿疣 +尘世 +尘土 +尘埃 +尚书 +尚可 +尚存 +尚无 +尚有 +尚未 +尝到 +尝尝 +尝试 +尤为 +尤为重要 +尤其 +就业 +就会 +就别 +就医 +就够 +就是 +就是指 +就是说 +就此 +就算 +就职 +就行 +就行了 +就要 +就让 +就诊 +就读 +就象 +就近 +就餐 +尴尬 +尸体 +尺寸 +尺度 +尼亚 +尼克 +尼古 +尼奥 +尼尔 +尼康 +尼斯 +尼泊尔 +尼玛 +尼采 +尼龙 +尽全 +尽全力 +尽力 +尽可能 +尽头 +尽心 +尽快 +尽情 +尽早 +尽显 +尽有 +尽管 +尽管如此 +尽职 +尽致 +尽责 +尽量 +尽量减少 +尽量少 +尽量避免 +尾声 +尾巴 +尾灯 +尾盘 +尾部 +尿液 +尿素 +尿道 +尿酸 +尿频 +局势 +局局 +局部 +局长 +局限 +局限于 +局限性 +局面 +屁孩 +屁股 +屁颠 +层出 +层出不穷 +层层 +层次 +层次感 +层级 +层面 +居于 +居住 +居住地 +居住证 +居士 +居多 +居委 +居委会 +居室 +居家 +居易 +居民 +居然 +居高 +屈原 +屈指 +屈服 +屈辱 +届时 +屋内 +屋子 +屋子里 +屋檐 +屋里 +屋面 +屋顶 +屏幕 +屏蔽 +屏障 +屏风 +展会 +展位 +展出 +展区 +展厅 +展品 +展开 +展望 +展现 +展现出 +展示 +展览 +展览会 +展馆 +属于 +属地 +属实 +属性 +属相 +屠宰 +屠杀 +屠龙 +屡屡 +屡次 +履历 +履职 +履行 +山上 +山下 +山丘 +山东 +山东省 +山中 +山体 +山区 +山南 +山县 +山口 +山地 +山坡 +山大 +山头 +山寨 +山峰 +山川 +山市 +山庄 +山村 +山林 +山楂 +山水 +山水画 +山河 +山泉 +山海 +山石 +山羊 +山脉 +山脚 +山药 +山西 +山西省 +山谷 +山路 +山里 +山镇 +山间 +山顶 +山麓 +屹立 +岁月 +岗位 +岗位职责 +岛上 +岛屿 +岩浆 +岩石 +岭南 +岳父 +岳阳 +岳飞 +岸边 +峡谷 +峨眉 +峰会 +峰值 +崇尚 +崇拜 +崇高 +崔永 +崛起 +崩溃 +崩盘 +崭新 +嵌入 +嵌入式 +巅峰 +川县 +川普 +川芎 +川菜 +州区 +州市 +州立 +巡回 +巡察 +巡查 +巡洋 +巡航 +巡视 +巡逻 +工业 +工业化 +工业园 +工业园区 +工业大学 +工人 +工件 +工会 +工伤 +工伤保险 +工作 +工作人员 +工作室 +工作岗位 +工作效率 +工作日 +工作站 +工作组 +工作者 +工作量 +工作队 +工信 +工信部 +工具 +工农 +工匠 +工厂 +工商 +工商局 +工商管理 +工商行政 +工商银行 +工地 +工大 +工夫 +工委 +工学 +工序 +工期 +工矿 +工科 +工程 +工程学院 +工程师 +工程建设 +工程技术 +工程施工 +工程质量 +工程项目 +工笔 +工艺 +工艺品 +工艺美术 +工行 +工装 +工贸 +工资 +工部 +左上 +左下 +左侧 +左右 +左手 +左旋 +左派 +左翼 +左脚 +左腿 +左转 +左边 +巧克力 +巧合 +巧妙 +巨人 +巨型 +巨大 +巨头 +巨星 +巨石 +巨蟹 +巨资 +巨量 +巨额 +巩固 +差不多 +差价 +差别 +差异 +差异化 +差点 +差距 +差错 +差阳 +差额 +已久 +已于 +已婚 +已成 +已有 +已然 +已知 +已经 +已达 +巴克 +巴哈 +巴基 +巴基斯坦 +巴塞罗那 +巴士 +巴尔 +巴巴 +巴拉 +巴掌 +巴斯 +巴特 +巴萨 +巴西 +巴马 +巴黎 +巷子 +巾帼 +市中心 +市井 +市值 +市内 +市区 +市县 +市场 +市场主体 +市场份额 +市场化 +市场占有率 +市场监管 +市场竞争 +市场经济 +市场营销 +市场需求 +市委 +市委书记 +市委常委 +市容 +市政 +市政府 +市民 +市盈 +市盈率 +市级 +市长 +市面 +市面上 +布兰 +布尔 +布局 +布拉 +布政 +布料 +布斯 +布施 +布朗 +布满 +布线 +布置 +布莱 +布衣 +布袋 +布里 +布雷 +布鲁 +帅哥 +帅气 +帆布 +帆船 +师专 +师傅 +师兄 +师团 +师大 +师弟 +师徒 +师德 +师父 +师生 +师范 +师范大学 +师范学院 +师资 +师长 +希尔 +希拉 +希望 +希特勒 +希腊 +帐号 +帐户 +帐篷 +帕克 +帖子 +帝国 +帝国主义 +帝王 +带上 +带入 +带到 +带动 +带回 +带回家 +带头 +带头人 +带宽 +带有 +带来 +带给 +带走 +带队 +带领 +席位 +席卷 +帮到 +帮助 +帮帮 +帮忙 +帮手 +帮扶 +帮派 +帷幕 +常人 +常会 +常住 +常务 +常务副 +常务委员会 +常务理事 +常吃 +常在 +常委 +常委会 +常州 +常常 +常年 +常德 +常态 +常数 +常有 +常温 +常熟 +常用 +常绿 +常见 +常见病 +常规 +常规赛 +常识 +常说 +常青 +常驻 +帽子 +幅度 +幕后 +幕墙 +幕府 +干事 +干什么 +干净 +干嘛 +干干 +干干净净 +干扰 +干支 +干旱 +干洗 +干活 +干涉 +干涩 +干燥 +干线 +干练 +干细胞 +干脆 +干警 +干货 +干辣椒 +干过 +干部 +干部职工 +干预 +平仄 +平仓 +平价 +平凡 +平原 +平县 +平反 +平台 +平和 +平地 +平均 +平均工资 +平均气温 +平均水平 +平均线 +平坦 +平复 +平安 +平定 +平常 +平常心 +平平 +平底 +平庸 +平息 +平整 +平方 +平方公里 +平方米 +平日 +平时 +平权 +平板 +平民 +平淡 +平滑 +平生 +平稳 +平等 +平米 +平行 +平衡 +平装 +平阳 +平静 +平面 +年产 +年仅 +年代 +年份 +年会 +年内 +年初 +年前 +年化 +年华 +年均 +年夜 +年头 +年少 +年幼 +年底 +年度 +年报 +年收入 +年春 +年期 +年末 +年来 +年版 +年生 +年秋 +年级 +年纪 +年终 +年老 +年薪 +年货 +年轻 +年轻人 +年轻化 +年里 +年间 +年限 +年青 +年龄 +年龄段 +并不 +并不一定 +并不多 +并不大 +并不知道 +并不需要 +并且 +并举 +并于 +并入 +并列 +并发 +并发症 +并处 +并存 +并未 +并用 +并称 +并肩 +并行 +并论 +并购 +并重 +并非 +幸亏 +幸好 +幸存 +幸福 +幸福感 +幸福生活 +幸运 +幻影 +幻想 +幻灯 +幻觉 +幼儿 +幼儿园 +幼小 +幼年 +幼稚 +幼虫 +幽灵 +幽默 +广东 +广东省 +广为 +广义 +广受 +广告 +广场 +广大 +广大群众 +广州 +广州市 +广度 +广播 +广播电视 +广汽 +广泛 +广泛应用 +广深 +广电 +广西 +广角 +广阔 +庄严 +庄园 +庄子 +庄家 +庄村 +庄稼 +庄重 +庄镇 +庆典 +庆幸 +庆祝 +庇护 +床上 +床位 +床单 +床垫 +床头 +床边 +序列 +序号 +序幕 +庐山 +库克 +库存 +库容 +库尔 +库房 +库里 +应为 +应付 +应力 +应变 +应对 +应届 +应当 +应急 +应手 +应援 +应收 +应是 +应有 +应有尽有 +应激 +应用 +应立即 +应聘 +应聘者 +应试 +应该 +应运 +应运而生 +应邀 +应酬 +底下 +底子 +底层 +底座 +底气 +底盘 +底线 +底蕴 +底部 +店主 +店内 +店员 +店家 +店里 +店铺 +店长 +店面 +庙会 +庞大 +废墟 +废弃 +废弃物 +废旧 +废气 +废水 +废物 +废话 +废除 +度假 +度假区 +度假村 +度日 +度过 +度量 +座位 +座标 +座椅 +座落 +座谈 +座谈会 +庭审 +庭院 +康乐 +康复 +康德 +康熙 +庸俗 +廉价 +廉政 +廉洁 +廊坊 +延伸 +延安 +延展 +延年 +延时 +延期 +延续 +延缓 +延误 +延边 +延迟 +延长 +建于 +建仓 +建军 +建制 +建华 +建国 +建安 +建成 +建房 +建有 +建材 +建构 +建档 +建档立卡 +建模 +建立 +建筑 +建筑业 +建筑工程 +建筑师 +建筑材料 +建筑物 +建筑设计 +建筑面积 +建议 +建设 +建设工程 +建设银行 +建设项目 +建造 +开业 +开个 +开会 +开元 +开光 +开关 +开具 +开出 +开刀 +开创 +开办 +开发 +开发区 +开发商 +开发者 +开口 +开启 +开国 +开场 +开复 +开头 +开始 +开学 +开封 +开局 +开展 +开工 +开幕 +开幕式 +开店 +开庭 +开张 +开心 +开悟 +开战 +开户 +开房 +开拓 +开拓者 +开挂 +开挖 +开播 +开支 +开放 +开放式 +开明 +开朗 +开机 +开来 +开水 +开源 +开玩笑 +开盘 +开眼 +开着 +开示 +开窗 +开端 +开篇 +开胃 +开花 +开裂 +开设 +开车 +开辟 +开通 +开采 +开销 +开门 +开阔 +开除 +开黑 +异乡 +异味 +异国 +异地 +异域 +异常 +异形 +异性 +异样 +异步 +异物 +异类 +异议 +弄清 +弄清楚 +弊端 +弓箭 +引人 +引人注目 +引以为 +引入 +引出 +引力 +引发 +引导 +引战 +引擎 +引来 +引水 +引流 +引爆 +引玉 +引用 +引申 +引诱 +引资 +引起 +引进 +引领 +弗兰 +弗拉 +弗里 +弘扬 +弟兄 +弟子 +弟弟 +张力 +张口 +张嘴 +张家口 +张家界 +张开 +张扬 +张贴 +弥勒 +弥漫 +弥补 +弦乐 +弧度 +弧形 +弯曲 +弯腰 +弯路 +弯道 +弱势 +弱化 +弱小 +弱弱 +弱智 +弱点 +弱者 +弹出 +弹力 +弹奏 +弹性 +弹簧 +弹药 +弹跳 +弹道 +强人 +强健 +强制 +强制性 +强力 +强加 +强劲 +强势 +强化 +强国 +强壮 +强大 +强奸 +强度 +强弱 +强强 +强悍 +强有 +强有力 +强求 +强烈 +强盗 +强盛 +强硬 +强者 +强行 +强调 +强身 +强迫 +归于 +归功 +归国 +归宿 +归属 +归属于 +归来 +归根 +归根结底 +归档 +归类 +归纳 +归结 +归还 +当上 +当下 +当中 +当之无愧 +当事 +当事人 +当今 +当代 +当众 +当作 +当做 +当初 +当前 +当即 +当地 +当地人 +当地政府 +当场 +当天 +当头 +当官 +当家 +当局 +当属 +当年 +当归 +当当 +当心 +当成 +当户 +当日 +当时 +当晚 +当月 +当有 +当期 +当然 +当真 +当着 +当红 +当选 +当面 +录像 +录入 +录制 +录取 +录影 +录用 +录音 +彝族 +形似 +形体 +形势 +形容 +形式 +形形 +形态 +形成 +形状 +形码 +形色 +形象 +彩信 +彩妆 +彩电 +彩礼 +彩票 +彩绘 +彩色 +彩虹 +彰显 +影业 +影像 +影响 +影响力 +影城 +影子 +影片 +影线 +影视 +影视作品 +影视剧 +影评 +影迷 +影院 +影音 +彷徨 +彻底 +彼岸 +彼得 +彼时 +彼此 +往上 +往下 +往事 +往前 +往前走 +往后 +往外 +往常 +往年 +往往 +往日 +往来 +往生 +往返 +征信 +征兆 +征地 +征战 +征收 +征文 +征服 +征求 +征求意见 +征用 +征程 +征税 +征管 +征集 +待人 +待机 +待用 +待遇 +很久 +很久没 +很会 +很多 +很多年 +很多很多 +很大 +很小 +很少 +很差 +很帅 +很强 +很快 +很想 +很早 +很棒 +很深 +很漂亮 +很火 +很累 +很长 +很难 +律师 +徐州 +徐徐 +徐志 +徒刑 +徒劳 +徒弟 +徒手 +徒步 +得上 +得不偿失 +得不到 +得主 +得以 +得住 +得体 +得其 +得出 +得分 +得到 +得力 +得名 +得多 +得天 +得天独厚 +得失 +得好 +得当 +得心 +得心应手 +得意 +得来 +得益 +得益于 +得知 +得罪 +得起 +得过 +徘徊 +御史 +循序 +循序渐进 +循环 +微不足道 +微信 +微信群 +微分 +微创 +微博 +微博上 +微商 +微型 +微妙 +微小 +微弱 +微循环 +微微 +微机 +微波 +微波炉 +微生物 +微笑 +微粒 +微薄 +微观 +微软 +微量 +微量元素 +微风 +德军 +德勒 +德华 +德国 +德国人 +德尔 +德川 +德州 +德拉 +德斯 +德曼 +德甲 +德罗 +德育 +德行 +德语 +德里 +徽州 +徽章 +心上 +心中 +心事 +心仪 +心力 +心动 +心包 +心地 +心塞 +心境 +心声 +心头 +心如 +心存 +心安 +心寒 +心平 +心平气和 +心底 +心律 +心得 +心念 +心怀 +心态 +心思 +心急 +心性 +心悦 +心悸 +心情 +心惊 +心想 +心意 +心愿 +心慌 +心扉 +心智 +心有 +心机 +心来 +心灵 +心烦 +心爱 +心率 +心理 +心理健康 +心理准备 +心理压力 +心理咨询 +心理学 +心理学家 +心理素质 +心甘 +心甘情愿 +心生 +心疼 +心病 +心痛 +心目 +心眼 +心神 +心绞痛 +心肌 +心肌梗 +心肺 +心胸 +心脏 +心脏病 +心脑血管 +心血 +心血管 +心路 +心跳 +心酸 +心里 +心静 +必不 +必不可少 +必修 +必备 +必定 +必定会 +必将 +必有 +必杀 +必然 +必经 +必胜 +必要 +必要性 +必读 +必需 +必需品 +必须 +忌讳 +忍不住 +忍受 +忍心 +忍无 +忍者 +忍耐 +忏悔 +忐忑 +志同道合 +志向 +志在 +志强 +志愿 +志愿者 +志玲 +忘不了 +忘却 +忘掉 +忘记 +忙于 +忙忙 +忙碌 +忠义 +忠于 +忠告 +忠实 +忠心 +忠诚 +忧伤 +忧愁 +忧虑 +忧郁 +快乐 +快到 +快快 +快感 +快慢 +快手 +快报 +快捷 +快播 +快活 +快点 +快船 +快要 +快车 +快递 +快速 +快速增长 +快门 +快餐 +念佛 +念头 +念念 +念念不忘 +念诵 +忽悠 +忽然 +忽略 +忽视 +怀孕 +怀念 +怀抱 +怀旧 +怀疑 +怀着 +怀里 +态势 +态度 +怎么 +怎么办 +怎么回事 +怎么样 +怎样 +怎样才能 +怎能 +怒气 +怒火 +怕冷 +怕死 +怜悯 +思义 +思念 +思想 +思想家 +思潮 +思索 +思绪 +思维 +思维能力 +思考 +思聪 +思议 +思路 +思辨 +急于 +急切 +急剧 +急功 +急忙 +急性 +急救 +急诊 +急躁 +急速 +急需 +怦然 +性交 +性价比 +性侵 +性别 +性功能 +性化 +性命 +性子 +性情 +性感 +性格 +性欲 +性激素 +性爱 +性生活 +性病 +性能 +性行 +性质 +怨恨 +怨气 +怨言 +怪不得 +怪兽 +怪异 +怪怪 +怪物 +总之 +总书记 +总产 +总产值 +总人口 +总价 +总会 +总体 +总体规划 +总值 +总公司 +总共 +总冠军 +总决赛 +总分 +总和 +总在 +总局 +总工 +总工会 +总归 +总得 +总想 +总成 +总成绩 +总支 +总收入 +总数 +总是 +总有 +总有一天 +总理 +总的 +总的来说 +总监 +总督 +总算 +总线 +总经理 +总结 +总统 +总而 +总而言之 +总能 +总裁 +总要 +总计 +总论 +总资产 +总部 +总量 +总队 +总面 +总面积 +总额 +恋人 +恋情 +恋爱 +恍惚 +恍然 +恍然大悟 +恐吓 +恐怕 +恐怖 +恐惧 +恐慌 +恐极 +恐龙 +恒大 +恒定 +恒星 +恒温 +恢复 +恢复正常 +恨不得 +恩怨 +恩施 +恩格 +恩爱 +恪守 +恭喜 +恭敬 +恭维 +息息 +息息相关 +息肉 +恰到 +恰到好处 +恰好 +恰巧 +恰当 +恰恰 +恶业 +恶人 +恶劣 +恶势力 +恶化 +恶心 +恶性 +恶性循环 +恶性肿瘤 +恶意 +恶毒 +恶魔 +悄悄 +悄悄地 +悄然 +悉尼 +悉心 +悔恨 +悟空 +悠久 +悠悠 +悠然 +悠闲 +患上 +患儿 +患处 +患失 +患有 +患病 +患者 +您好 +悬崖 +悬念 +悬挂 +悬架 +悬浮 +悬疑 +悲伤 +悲剧 +悲哀 +悲惨 +悲痛 +悲观 +情不自禁 +情书 +情人 +情人节 +情侣 +情况 +情商 +情境 +情形 +情怀 +情感 +情愿 +情报 +情操 +情景 +情有 +情欲 +情歌 +情深 +情理 +情结 +情绪 +情缘 +情节 +情节严重 +情话 +情调 +情谊 +情趣 +惆怅 +惊人 +惊叹 +惊吓 +惊呆 +惊喜 +惊天 +惊奇 +惊心 +惊恐 +惊悚 +惊慌 +惊艳 +惊讶 +惊醒 +惊险 +惊魂 +惋惜 +惟一 +惟有 +惠及 +惠州 +惠普 +惠民 +惧怕 +惨淡 +惨烈 +惨遭 +惨重 +惩戒 +惩罚 +惬意 +惭愧 +惯例 +惯性 +惯用 +惰性 +想一想 +想不到 +想像 +想到 +想学 +想尽 +想当 +想得 +想得到 +想必 +想念 +想想 +想换 +想方设法 +想来 +想法 +想着 +想要 +想象 +想象力 +想起 +想过 +惶恐 +愈加 +愈发 +愈合 +愈来 +愈来愈 +愉快 +愉悦 +意义 +意向 +意味 +意味着 +意图 +意在 +意境 +意外 +意大利 +意志 +意志力 +意念 +意思 +意想 +意想不到 +意愿 +意料 +意气 +意见 +意识 +意识形态 +意象 +愚昧 +愚蠢 +感人 +感光 +感兴趣 +感冒 +感到 +感动 +感受 +感受一下 +感叹 +感官 +感应 +感性 +感恩 +感悟 +感情 +感想 +感慨 +感染 +感染力 +感染者 +感激 +感知 +感觉 +感触 +感谢 +感谢您 +愤怒 +愧疚 +愿意 +愿景 +愿望 +慈善 +慈悲 +慈溪 +慈禧 +慌乱 +慌张 +慎用 +慎重 +慕名 +慕容 +慢性 +慢性病 +慢慢 +慢跑 +慰问 +慵懒 +慷慨 +憔悴 +憧憬 +懂事 +懂得 +懈怠 +懒人 +懒得 +懒惰 +懒散 +懦弱 +懵懂 +戈壁 +戈尔 +戏剧 +戏曲 +成为 +成交 +成交价 +成交量 +成交额 +成人 +成份 +成佛 +成像 +成全 +成分 +成功 +成功人士 +成功率 +成功者 +成千 +成千上万 +成吉思汗 +成名 +成员 +成员国 +成品 +成品油 +成因 +成型 +成天 +成套 +成家 +成就 +成就感 +成年 +成年人 +成形 +成才 +成指 +成效 +成本 +成本低 +成果 +成正 +成正比 +成活 +成熟 +成片 +成瘾 +成立 +成章 +成绩 +成绩单 +成群 +成荫 +成药 +成语 +成败 +成都 +成都市 +成长 +成长性 +成龙 +我们 +我会 +我俩 +我军 +我刚 +我区 +我县 +我发 +我国 +我太 +我妈 +我家 +我市 +我心 +我怕 +我想 +我所 +我方 +我来 +我校 +我爱 +我爱你 +我爸 +我用 +我省 +我能 +我行 +我要 +我见 +我该 +我院 +戒律 +戒指 +戒毒 +戒烟 +或多或少 +或是 +或用 +或缺 +或者 +或者说 +或许 +战乱 +战争 +战列 +战功 +战友 +战后 +战国 +战地 +战场 +战士 +战役 +战斗 +战斗力 +战斗机 +战时 +战术 +战机 +战死 +战法 +战火 +战略 +战略性 +战神 +战线 +战绩 +战胜 +战败 +战车 +战队 +截图 +截止 +截然 +截然不同 +截至 +截面 +戴上 +戴尔 +戴着 +戴维 +戴维斯 +户口 +户型 +户外 +户外活动 +户户 +户数 +户籍 +户通 +户部 +房东 +房产 +房产税 +房产证 +房价 +房企 +房地产 +房型 +房子 +房屋 +房源 +房租 +房贷 +房车 +房里 +房门 +房间 +所为 +所以 +所作 +所含 +所在 +所在地 +所处 +所学 +所属 +所幸 +所得 +所得税 +所思 +所指 +所措 +所有 +所有人 +所有制 +所有权 +所欲 +所用 +所知 +所示 +所能 +所致 +所见 +所言 +所说 +所谓 +所述 +所长 +所闻 +扁平 +扁桃 +扁桃体 +手上 +手下 +手中 +手册 +手写 +手动 +手势 +手可 +手头 +手套 +手工 +手心 +手感 +手把 +手持 +手指 +手掌 +手提 +手握 +手术 +手术室 +手机 +手机号 +手机号码 +手枪 +手柄 +手段 +手法 +手游 +手电 +手绘 +手续 +手续费 +手脚 +手腕 +手臂 +手艺 +手表 +手足 +手部 +手里 +手镯 +才华 +才子 +才干 +才智 +才能 +才艺 +才行 +扎实 +扎根 +扑克 +打下 +打个 +打乱 +打交 +打交道 +打人 +打仗 +打倒 +打入 +打出 +打击 +打分 +打动 +打包 +打卡 +打印 +打印机 +打压 +打发 +打听 +打响 +打坐 +打字 +打工 +打底 +打开 +打得 +打成 +打打 +打扫 +打扮 +打扰 +打折 +打招呼 +打拼 +打斗 +打断 +打来 +打架 +打死 +打法 +打游戏 +打滚 +打火 +打牌 +打猎 +打球 +打理 +打电话 +打着 +打破 +打碎 +打磨 +打算 +打败 +打赢 +打车 +打过 +打进 +打通 +打造 +打造出 +打野 +打针 +打闹 +打骂 +打麻将 +扔掉 +托付 +托尼 +托盘 +托福 +托管 +托运 +托马斯 +扣分 +扣押 +扣篮 +扣除 +执业 +执勤 +执导 +执念 +执意 +执政 +执教 +执法 +执法人员 +执照 +执着 +执笔 +执行 +执行力 +执行官 +扩充 +扩大 +扩容 +扩展 +扩建 +扩张 +扩散 +扫地 +扫描 +扫码 +扫荡 +扫除 +扫黑 +扬声 +扬声器 +扬州 +扬扬 +扭曲 +扭矩 +扭转 +扮演 +扰乱 +扶手 +扶持 +扶贫 +批准 +批判 +批发 +批发市场 +批复 +批斗 +批评 +批量 +扼杀 +扼要 +找个 +找人 +找准 +找出 +找到 +找回 +找寻 +找茬 +承办 +承包 +承压 +承受 +承受能力 +承德 +承担 +承担责任 +承接 +承租 +承认 +承诺 +承载 +承运 +承重 +技工 +技巧 +技师 +技术 +技术人员 +技术创新 +技术开发区 +技术性 +技术指标 +技术支持 +技术水平 +技校 +技法 +技能 +技艺 +抄底 +抄袭 +抉择 +把关 +把手 +把握 +把握住 +抑制 +抑或 +抑郁 +抑郁症 +抒发 +抒情 +抓住 +抓好 +抓手 +抓拍 +抓捕 +抓紧 +抓紧时间 +抓获 +抓起 +投产 +投保 +投入 +投入使用 +投向 +投奔 +投射 +投影 +投放 +投机 +投标 +投标人 +投档 +投注 +投票 +投稿 +投篮 +投融资 +投行 +投诉 +投资 +投资人 +投资收益 +投资者 +投身 +投递 +投降 +抖动 +抖音 +抗争 +抗体 +抗击 +抗压 +抗原 +抗性 +抗战 +抗拒 +抗日 +抗日战争 +抗旱 +抗氧化 +抗生 +抗生素 +抗疫 +抗病 +抗病毒 +抗癌 +抗菌 +抗衡 +抗衰老 +抗议 +抗震 +折叠 +折合 +折射 +折扣 +折旧 +折磨 +折腾 +抚养 +抚慰 +抚摸 +抚顺 +抛光 +抛出 +抛售 +抛开 +抛弃 +抛砖 +抠门 +抢先 +抢劫 +抢占 +抢夺 +抢救 +抢断 +抢眼 +抢购 +抢走 +抢险 +护卫 +护发 +护士 +护栏 +护法 +护照 +护理 +护肤 +护肤品 +护航 +护送 +报业 +报仇 +报价 +报关 +报刊 +报名 +报告 +报复 +报导 +报应 +报废 +报批 +报收 +报案 +报班 +报社 +报答 +报纸 +报考 +报表 +报警 +报送 +报道 +报酬 +报销 +披萨 +披针 +披露 +抬头 +抬起 +抬高 +抱住 +抱团 +抱怨 +抱抱 +抱歉 +抱负 +抵制 +抵御 +抵扣 +抵抗 +抵抗力 +抵押 +抵挡 +抵消 +抵触 +抵达 +抹布 +押运 +押金 +抽出 +抽取 +抽奖 +抽屉 +抽搐 +抽查 +抽样 +抽检 +抽烟 +抽空 +抽筋 +抽象 +担任 +担保 +担当 +担心 +担忧 +担负 +拆卸 +拆开 +拆装 +拆迁 +拆除 +拇指 +拉丁 +拉伸 +拉克 +拉力 +拉动 +拉升 +拉姆 +拉开 +拉德 +拉扯 +拉拉 +拉拢 +拉斯 +拉着 +拉票 +拉美 +拉肚 +拉萨 +拉近 +拉里 +拉链 +拉长 +拉面 +拉高 +拌匀 +拍出 +拍卖 +拍戏 +拍成 +拍打 +拍拍 +拍摄 +拍案 +拍照 +拍片 +拐卖 +拐弯 +拐点 +拒绝 +拓宽 +拓展 +拓扑 +拓跋 +拔出 +拔尖 +拔牙 +拔萃 +拖延 +拖拉 +拖拉机 +拖欠 +拖累 +拖鞋 +拘泥 +拘留 +招人 +招募 +招呼 +招商 +招商引资 +招商银行 +招式 +招录 +招待 +招惹 +招收 +招来 +招标 +招牌 +招生 +招考 +招聘 +招聘会 +招股 +招致 +拜仁 +拜师 +拜年 +拜托 +拜访 +拜金 +拟定 +拟订 +拥堵 +拥护 +拥抱 +拥挤 +拥有 +拦截 +拨号 +拨打 +拨款 +择业 +择优 +择偶 +拭目 +拭目以待 +拯救 +拳击 +拳头 +拷贝 +拼写 +拼命 +拼图 +拼接 +拼搏 +拼死 +拼音 +拿下 +拿出 +拿到 +拿手 +拿来 +拿破 +拿破仑 +拿走 +持久 +持之以恒 +持仓 +持平 +持有 +持枪 +持续 +持续性 +持续时间 +持股 +挂号 +挂机 +挂牌 +挂科 +挂钩 +指令 +指出 +指南 +指取 +指向 +指定 +指导 +指导价 +指尖 +指引 +指指 +指挥 +指挥部 +指控 +指数 +指明 +指望 +指标 +指点 +指环 +指甲 +指着 +指示 +指纹 +指责 +指针 +按下 +按住 +按压 +按揭 +按摩 +按时 +按照 +按理 +按规定 +按钮 +按键 +挑出 +挑剔 +挑战 +挑拨 +挑衅 +挑起 +挑选 +挑逗 +挖出 +挖掘 +挖掘机 +挚友 +挡住 +挣扎 +挣钱 +挤出 +挤压 +挥之 +挥发 +挥手 +挥洒 +挥霍 +挨打 +挪威 +挪用 +挫折 +振作 +振兴 +振动 +振奋 +振荡 +挺大 +挺拔 +挺直 +挺身 +挽回 +挽救 +挽留 +捂着 +捆绑 +捍卫 +捐助 +捐款 +捐献 +捐资 +捐赠 +捕捉 +捕捞 +捕获 +捕食 +捕鱼 +捞出 +捞起 +损伤 +损坏 +损失 +损害 +损益 +损耗 +换上 +换个 +换乘 +换代 +换位 +换取 +换句 +换句话说 +换届 +换成 +换手 +换手率 +换挡 +换来 +换热 +换算 +换装 +换言之 +捣乱 +捣烂 +据介绍 +据传 +据悉 +据此 +据统计 +据说 +捷克 +捷径 +掀起 +授予 +授权 +授课 +掉下 +掉下来 +掉头 +掉落 +掌声 +掌心 +掌控 +掌握 +掌柜 +掌管 +掌门 +掏出 +排位 +排便 +排出 +排列 +排卵 +排名 +排名第 +排尿 +排序 +排挤 +排放 +排斥 +排查 +排毒 +排气 +排水 +排污 +排泄 +排版 +排球 +排练 +排行 +排行榜 +排解 +排量 +排长 +排队 +排除 +排骨 +掘金 +掠夺 +探头 +探寻 +探底 +探望 +探测 +探测器 +探究 +探索 +探讨 +探访 +探险 +接下 +接下来 +接入 +接到 +接力 +接受 +接口 +接听 +接吻 +接地 +接壤 +接头 +接待 +接手 +接收 +接班 +接班人 +接着 +接种 +接管 +接纳 +接线 +接见 +接触 +接踵 +接轨 +接过 +接近 +接连 +接送 +接通 +控制 +控制器 +控制系统 +控告 +控油 +控球 +控股 +控诉 +推介 +推倒 +推出 +推力 +推动 +推卸 +推向 +推崇 +推广 +推开 +推手 +推拿 +推断 +推测 +推理 +推移 +推算 +推翻 +推荐 +推行 +推论 +推车 +推进 +推迟 +推送 +推销 +掩护 +掩盖 +掩饰 +措施 +措辞 +掺杂 +揉搓 +描写 +描绘 +描述 +提了 +提交 +提供 +提供商 +提倡 +提出 +提到 +提前 +提升 +提及 +提取 +提取物 +提名 +提成 +提拔 +提振 +提早 +提案 +提炼 +提琴 +提督 +提示 +提神 +提纲 +提要 +提议 +提起 +提过 +提速 +提醒 +提问 +提高 +插件 +插入 +插图 +插头 +插座 +插曲 +插槽 +插队 +握住 +握手 +揣测 +揭发 +揭开 +揭晓 +揭牌 +揭示 +揭秘 +揭阳 +揭露 +援助 +搅匀 +搅拌 +搏击 +搜到 +搜寻 +搜查 +搜狐 +搜索 +搜索引擎 +搜集 +搞好 +搞定 +搞清楚 +搞笑 +搬出 +搬家 +搬迁 +搬运 +搭乘 +搭建 +搭档 +搭讪 +搭载 +搭配 +携带 +携手 +携程 +摄像 +摄像头 +摄像机 +摄入 +摄入量 +摄制 +摄取 +摄影 +摄影师 +摄氏 +摄氏度 +摆动 +摆在 +摆摊 +摆放 +摆渡 +摆脱 +摆设 +摇头 +摇摆 +摇摇 +摇晃 +摇曳 +摇滚 +摇篮 +摊位 +摊手 +摒弃 +摔倒 +摔跤 +摘下 +摘录 +摘自 +摘要 +摧残 +摧毁 +摩天 +摩尔 +摩托 +摩托车 +摩擦 +摩根 +摩登 +摩羯 +摩诃 +摩诃萨 +摸摸 +摸索 +撇开 +撑起 +撒上 +撒娇 +撒谎 +撕裂 +撞击 +撞死 +撤回 +撤离 +撤退 +撤销 +撩汉 +播出 +播放 +播种 +播音 +撰写 +撰稿 +擂台 +擅自 +擅长 +操作 +操作性 +操作方法 +操作系统 +操场 +操心 +操控 +操盘 +操纵 +擦干 +擦拭 +擦肩 +擦肩而过 +攀升 +攀比 +攀登 +攒钱 +支付 +支付宝 +支出 +支原 +支招 +支持 +支援 +支撑 +支撑位 +支教 +支架 +支柱 +支气管 +支气管炎 +支行 +支那 +支部 +支配 +支队 +收买 +收于 +收入 +收养 +收到 +收割 +收发 +收取 +收受 +收听 +收回 +收复 +收官 +收容 +收录 +收拾 +收支 +收敛 +收款 +收益 +收益率 +收盘 +收盘价 +收看 +收紧 +收纳 +收缩 +收获 +收藏 +收视 +收视率 +收货 +收购 +收费 +收费站 +收起 +收银 +收集 +收音 +收音机 +改为 +改了 +改写 +改制 +改动 +改变 +改变命运 +改名 +改善 +改委 +改建 +改成 +改掉 +改正 +改版 +改用 +改编 +改良 +改装 +改进 +改造 +改革 +改革开放 +攻克 +攻关 +攻击 +攻击力 +攻势 +攻坚 +攻坚战 +攻城 +攻打 +攻略 +攻读 +攻防 +放下 +放不下 +放任 +放假 +放入 +放养 +放出 +放到 +放在 +放大 +放学 +放宽 +放射 +放射性 +放屁 +放开 +放弃 +放得 +放心 +放慢 +放手 +放映 +放松 +放水 +放火 +放生 +放电 +放疗 +放眼 +放着 +放纵 +放缓 +放置 +放荡 +放过 +放进 +放进去 +放量 +放飞 +政事 +政党 +政务 +政协 +政协委员 +政委 +政工 +政府 +政府部门 +政教 +政权 +政治 +政治家 +政治局 +政法 +政法委 +政策 +政策性 +政策措施 +政绩 +故乡 +故事 +故作 +故宫 +故宫博物院 +故居 +故意 +故而 +故里 +故障 +效仿 +效力 +效应 +效果 +效果图 +效率 +效率高 +效用 +效益 +效能 +敌人 +敌对 +敌意 +敌方 +敏感 +敏感性 +敏捷 +敏锐 +救世 +救人 +救出 +救助 +救命 +救护 +救护车 +救援 +救治 +救济 +救灾 +救赎 +教主 +教书 +教会 +教养 +教务 +教化 +教区 +教堂 +教委 +教学 +教学内容 +教学方法 +教学楼 +教学模式 +教学质量 +教官 +教室 +教导 +教师 +教师节 +教师队伍 +教授 +教改 +教材 +教法 +教派 +教皇 +教研 +教研室 +教科 +教科书 +教程 +教练 +教给 +教职工 +教育 +教育厅 +教育学 +教育家 +教育局 +教育资源 +教育部 +教训 +教诲 +教辅 +敞开 +敢于 +散乱 +散发 +散发出 +散射 +散布 +散户 +散散 +散文 +散步 +散热 +散落 +敦煌 +敬业 +敬佩 +敬意 +敬畏 +敬老 +敬请 +数万 +数以 +数值 +数十 +数十年 +数千 +数字 +数字化 +数学 +数年 +数据 +数据中心 +数据分析 +数据库 +数控 +数日 +数月 +数次 +数理 +数百 +数目 +数码 +数组 +数量 +数额 +敲击 +敲打 +敲门 +整个 +整体 +整合 +整天 +整套 +整容 +整形 +整改 +整数 +整整 +整日 +整机 +整治 +整洁 +整流 +整理 +整车 +整部 +整顿 +整齐 +敷衍 +文中 +文书 +文人 +文件 +文体 +文具 +文凭 +文创 +文化 +文化交流 +文化产业 +文化底蕴 +文化节 +文化遗产 +文华 +文史 +文坛 +文娱 +文字 +文学 +文学作品 +文学奖 +文学家 +文山 +文帝 +文并 +文庙 +文摘 +文教 +文旅 +文昌 +文明 +文明城市 +文本 +文案 +文档 +文武 +文殊 +文火 +文物 +文物保护 +文献 +文理 +文盲 +文科 +文秘 +文稿 +文章 +文章内容 +文笔 +文联 +文艺 +文言 +文身 +文选 +文采 +文集 +文静 +文革 +文风 +斑斓 +斑点 +斑病 +斑马 +斑驳 +斗争 +斗志 +斗牛 +斗鱼 +料理 +料酒 +斟酌 +斤斤 +斩获 +断定 +断层 +断开 +断断 +断电 +断绝 +断续 +断裂 +断路 +断面 +斯克 +斯坦 +斯坦福 +斯基 +斯堡 +斯塔 +斯大林 +斯托 +斯拉 +斯特 +斯科 +斯诺 +斯顿 +新一 +新一代 +新一轮 +新世界 +新世纪 +新东方 +新乡 +新书 +新人 +新低 +新作 +新兴 +新兴产业 +新冠 +新加 +新加坡 +新区 +新华 +新华社 +新华网 +新品 +新型 +新城 +新增 +新奇 +新娘 +新婚 +新安 +新年 +新建 +新意 +新房 +新手 +新政 +新旧 +新星 +新春 +新月 +新机 +新村 +新款 +新浪 +新浪网 +新潮 +新版 +新生 +新生代 +新生儿 +新疆 +新秀 +新股 +新能源 +新药 +新西兰 +新规 +新课 +新车 +新郎 +新闻 +新闻出版 +新闻网 +新陈代谢 +新颖 +新风 +新高 +新鲜 +新鲜感 +方丈 +方位 +方便 +方便快捷 +方便面 +方剂 +方可 +方向 +方向盘 +方圆 +方块 +方子 +方式 +方形 +方方面面 +方案 +方正 +方法 +方法论 +方程 +方舟 +方言 +方针 +方面 +施加 +施展 +施工 +施工单位 +施工现场 +施暴 +施用 +施肥 +施行 +旁人 +旁观 +旁边 +旅客 +旅游 +旅游业 +旅游区 +旅游局 +旅游景点 +旅程 +旅行 +旅行社 +旅途 +旅馆 +旋律 +旋转 +旋风 +族人 +族群 +旗下 +旗帜 +旗舰 +旗舰店 +旗袍 +无一 +无上 +无不 +无为 +无事 +无产阶级 +无人 +无人机 +无人驾驶 +无从 +无以 +无休 +无偿 +无关 +无几 +无前 +无力 +无功 +无助 +无双 +无可 +无可奈何 +无名 +无名指 +无味 +无声 +无处 +无处不在 +无奈 +无害 +无尽 +无常 +无异 +无形 +无形中 +无心 +无忌 +无忧 +无忧无虑 +无怨 +无悔 +无情 +无意 +无意识 +无意间 +无愧 +无成 +无所 +无所谓 +无损 +无故 +无效 +无敌 +无数 +无数个 +无数次 +无时 +无明 +无暇 +无有 +有期 +无期 +无机 +无权 +无条件 +无极 +无止境 +无毒 +无比 +无毛 +无法 +无理 +无用 +无畏 +无疑 +无痛 +无益 +无知 +无碍 +无私 +无穷 +无穷的 +无端 +无线 +无线电 +无缘 +无缝 +无罪 +无耻 +无聊 +无能 +无能为力 +无脑 +无良 +无色 +无菌 +无虑 +无视 +无言 +无论 +无论如何 +无论是 +无证 +无话 +无语 +无误 +无赖 +无路 +无辜 +无边 +无量 +无锡 +无间 +无闻 +无际 +无限 +无需 +无霜 +无非 +无须 +既定 +既往 +既有 +既然 +日久 +日产 +日元 +日光 +日内 +日军 +日出 +日到 +日制 +日前 +日历 +日后 +日均 +日复 +日复一日 +日夜 +日子 +日常 +日常生活 +日式 +日志 +日报 +日文 +日日 +日晒 +日晚 +日月 +日期 +日本 +日渐 +日漫 +日照 +日电 +日益 +日程 +日系 +日线 +日至 +日落 +日讯 +日记 +日语 +日起 +日趋 +日间 +旧书 +旧址 +旧时 +旧金山 +旨在 +早上 +早些 +早产 +早前 +早就 +早已 +早年 +早教 +早日 +早早 +早晚 +早晨 +早期 +早泄 +早点 +早熟 +早盘 +早睡 +早起 +早餐 +早饭 +旱地 +时不时 +时事 +时代 +时任 +时会 +时俱 +时候 +时光 +时分 +时刻 +时尚 +时常 +时应 +时才 +时报 +时效 +时日 +时时 +时时刻刻 +时有 +时期 +时机 +时段 +时用 +时空 +时而 +时至 +时至今日 +时节 +时装 +时要 +时许 +时说 +时辰 +时速 +时针 +时钟 +时长 +时间 +时间段 +时间表 +时限 +时隔 +时髦 +旺季 +旺盛 +昂贵 +昆仑 +昆山 +昆明 +昆明市 +昆虫 +昌平 +明亮 +明代 +明信 +明史 +明天 +明媚 +明年 +明快 +明日 +明明 +明星 +明显 +明晰 +明智 +明暗 +明月 +明朗 +明朝 +明末 +明治 +明清 +明珠 +明白 +明目 +明知 +明确 +明确提出 +明确规定 +明细 +昏暗 +昏迷 +易于 +易学 +易懂 +易燃 +易用 +易经 +昔日 +星光 +星巴克 +星座 +星星 +星期 +星期一 +星期五 +星期六 +星期天 +星火 +星爷 +星球 +星空 +星系 +星级 +星辰 +星际 +映像 +映射 +映衬 +春光 +春夏 +春天 +春季 +春日 +春晚 +春暖 +春暖花开 +春树 +春秋 +春联 +春节 +春运 +春雨 +春风 +昨夜 +昨天 +昨天晚上 +昨日 +昨晚 +昭和 +是不是 +是从 +是否 +是否是 +是因为 +是非 +昼夜 +显出 +显卡 +显存 +显得 +显微 +显微镜 +显然 +显现 +显瘦 +显眼 +显着 +显示 +显示器 +显示屏 +显而易见 +显著 +显赫 +显露 +晃动 +晋升 +晋江 +晋级 +晒出 +晒太阳 +晒干 +晓彤 +晓得 +晓松 +晕倒 +晕车 +晚上 +晚会 +晚安 +晚宴 +晚年 +晚报 +晚期 +晚清 +晚间 +晚餐 +晚饭 +普世 +普京 +普利 +普勒 +普及 +普惠 +普拉 +普斯 +普普 +普林 +普查 +普森 +普法 +普洱 +普洱茶 +普罗 +普贤 +普通 +普通人 +普通话 +普通高中 +普通高校 +普遍 +普遍存在 +普陀 +景区 +景德 +景德镇 +景气 +景点 +景物 +景致 +景色 +景观 +景象 +晴天 +晴朗 +晶体 +晶晶 +晶莹 +智利 +智力 +智商 +智库 +智慧 +智者 +智能 +智能化 +智能家居 +智能手机 +智障 +晾干 +晾晒 +暂且 +暂停 +暂时 +暂行 +暑假 +暑期 +暖和 +暖心 +暖暖 +暖暖的 +暖气 +暗中 +暗夜 +暗恋 +暗暗 +暗杀 +暗淡 +暗示 +暗自 +暧昧 +暨南 +暴击 +暴利 +暴力 +暴发 +暴富 +暴晒 +暴涨 +暴走 +暴跌 +暴躁 +暴雨 +暴露 +暴风 +暴食 +曙光 +曝光 +曝出 +曲子 +曲张 +曲折 +曲目 +曲线 +曲艺 +曲阜 +曲面 +曲风 +更为 +更为重要 +更会 +更何 +更何况 +更佳 +更具 +更加 +更名 +更多 +更多人 +更多地 +更大 +更好 +更年期 +更强 +更快 +更换 +更改 +更新 +更是 +更显 +更有 +更正 +更深 +更能 +更要 +更让人 +更进一步 +更长 +更难 +更高 +曼城 +曼联 +曼谷 +曾任 +曾多次 +曾经 +替代 +替换 +替补 +替身 +最为 +最优 +最低 +最低价 +最低点 +最佳 +最先 +最具 +最初 +最受 +最合适 +最后 +最坏 +最多 +最大 +最大化 +最好 +最小 +最少 +最差 +最强 +最快 +最怕 +最想 +最新 +最早 +最最 +最有 +最深 +最爱 +最终 +最美 +最能 +最让人 +最贵 +最起码 +最近 +最长 +最难 +最高 +最高人民法院 +最高点 +月亮 +月份 +月光 +月入 +月初 +月台 +月子 +月季 +月底 +月末 +月球 +月生 +月经 +月经不调 +月薪 +月饼 +有个 +有事 +有些 +有人 +有位 +有何 +有余 +有偿 +有关 +有利 +有利于 +有别 +有力 +有加 +有助 +有助于 +有句 +有史 +有史以来 +有名 +有名气 +有味 +有声 +有多 +有大 +有太多 +有如 +有害 +有害物质 +有幸 +有序 +有形 +有待 +有心 +有心人 +有志 +有情 +有意 +有意思 +有意识 +有感 +有成 +有所 +有所不同 +有效 +有效性 +有效成分 +有效期 +有效率 +有数 +有无 +有时 +有时候 +有望 +有期徒刑 +有机 +有机物 +有机肥 +有权 +有条 +有次 +有毒 +有氧 +有没有 +有点 +有点像 +有点儿 +有理 +有生 +有用 +有病 +有的是 +有益 +有益于 +有目共睹 +有着 +有神 +有种 +有空 +有线 +有线电视 +有缘 +有罪 +有色 +有色金属 +有趣 +有车 +有过 +有道 +有钱 +有钱人 +有限 +有限公司 +有限责任 +朋友 +朋友圈 +服从 +服务 +服务业 +服务中心 +服务体系 +服务区 +服务员 +服务商 +服务器 +服务平台 +服务态度 +服务水平 +服务站 +服务质量 +服务费 +服务项目 +服役 +服气 +服用 +服药 +服装 +服饰 +朗诵 +朗读 +望去 +望远 +望远镜 +朝代 +朝伟 +朝向 +朝圣 +朝夕 +朝天 +朝廷 +朝气 +朝着 +朝阳 +朝阳区 +朝鲜 +期内 +期刊 +期待 +期望 +期末 +期权 +期满 +期盼 +期货 +期货市场 +期间 +期限 +朦胧 +木偶 +木兰 +木刻 +木地板 +木头 +木工 +木星 +木有 +木材 +木板 +木瓜 +木结构 +木耳 +木讷 +木质 +木门 +木雕 +木马 +未免 +未婚 +未尝 +未尽 +未必 +未成年 +未成年人 +未曾 +未有 +未来 +未满 +未知 +未经 +未能 +未见 +末尾 +末年 +末日 +末期 +末端 +本上 +本书 +本事 +本人 +本会 +本体 +本作 +本分 +本周 +本命 +本品 +本国 +本土 +本地 +本场 +本子 +本届 +本山 +本市 +本性 +本意 +本报 +本报记者 +本文 +本方 +本是 +本月 +本期 +本来 +本校 +本案 +本次 +本法 +本田 +本病 +本着 +本科 +本科生 +本站 +本章 +本职 +本能 +本色 +本草 +本营 +本该 +本质 +本赛季 +本身 +本轮 +本部 +本金 +本钱 +本院 +本领 +术后 +术语 +朱雀 +朴实 +朴素 +朵朵 +机会 +机体 +机关 +机制 +机动 +机动车 +机器 +机器人 +机场 +机型 +机密 +机床 +机房 +机智 +机构 +机枪 +机架 +机械 +机械化 +机械设备 +机油 +机灵 +机率 +机理 +机电 +机票 +机箱 +机组 +机缘 +机能 +机芯 +机要 +机身 +机车 +机遇 +杀人 +杀伤 +杀伤力 +杀害 +杀戮 +杀手 +杀掉 +杀敌 +杀死 +杀毒 +杀灭 +杀生 +杀菌 +杀虫 +杀跌 +杂乱 +杂交 +杂志 +杂念 +杂技 +杂文 +杂物 +杂粮 +杂草 +杂货 +杂质 +权利 +权力 +权势 +权威 +权属 +权益 +权衡 +权证 +权贵 +权重 +权限 +杆菌 +李世民 +李克强 +李嘉诚 +李子 +李宁 +李家 +李白 +杏仁 +杏花 +材料 +材质 +村内 +村务 +村委 +村委会 +村子 +村干部 +村庄 +村情 +村村 +村民 +村级 +村落 +村里 +村镇 +杜兰 +杜兰特 +杜甫 +杜绝 +杜鹃 +束缚 +杠杆 +条件 +条例 +条形 +条形码 +条文 +条款 +条码 +条约 +条纹 +来不及 +来世 +来个 +来临 +来之 +来信 +来到 +来历 +来回 +来宾 +来往 +来得 +来得及 +来时 +来来 +来源 +来源于 +来电 +来看 +来答 +来自 +来袭 +来讲 +来访 +来说 +杨柳 +杨梅 +杭州 +杭州市 +杯子 +杯水 +杯酒 +杰伦 +杰作 +杰克 +杰出 +松下 +松动 +松子 +松开 +松弛 +松懈 +松散 +松松 +松柏 +松树 +松江 +松软 +松鼠 +板凳 +板块 +板型 +板式 +板材 +板栗 +板桥 +板载 +极为 +极乐 +极乐世界 +极了 +极佳 +极其 +极具 +极力 +极品 +极大 +极好 +极富 +极小 +极少 +极少数 +极度 +极强 +极拳 +极易 +极有 +极点 +极端 +极管 +极致 +极速 +极限 +极高 +构件 +构图 +构建 +构思 +构想 +构成 +构筑 +构造 +枇杷 +枕头 +林业 +林中 +林丹 +林地 +林夕 +林寺 +林志 +林木 +林立 +林肯 +枚举 +果仁 +果品 +果园 +果地 +果壳 +果子 +果实 +果报 +果断 +果期 +果树 +果汁 +果然 +果皮 +果真 +果肉 +果蔬 +枝叶 +枝头 +枝条 +枢密 +枢纽 +枣庄 +枪击 +枪支 +枪械 +枪管 +枫叶 +枯燥 +枯萎 +架子 +架构 +架空 +枸杞 +枸杞子 +柏拉 +柏林 +柏油 +某一 +某个 +某些 +某人 +某天 +某某 +某种 +某种意义 +某种程度 +柑橘 +染上 +染发 +染料 +染色 +染色体 +柔和 +柔弱 +柔性 +柔情 +柔美 +柔软 +柔韧 +柚子 +柜台 +柜员 +柜子 +柠檬 +柠檬汁 +查出 +查处 +查封 +查找 +查明 +查理 +查看 +查获 +查询 +查阅 +查验 +柬埔寨 +柯南 +柱头 +柱子 +柳州 +柳树 +柴油 +柴胡 +柴静 +柿子 +栀子 +栅栏 +标价 +标准 +标准化 +标志 +标志性 +标明 +标本 +标杆 +标榜 +标注 +标点 +标的 +标示 +标签 +标致 +标记 +标识 +标语 +标配 +标题 +栈道 +栏杆 +栏目 +树上 +树叶 +树干 +树木 +树林 +树枝 +树皮 +树种 +树立 +树脂 +栓塞 +栖息 +栗子 +校内 +校准 +校区 +校友 +校园 +校外 +校招 +校服 +校本 +校正 +校生 +校车 +校长 +校门 +校风 +校验 +栩栩 +株式 +株洲 +样品 +样子 +样式 +样本 +样板 +样样 +核准 +核发 +核定 +核实 +核对 +核弹 +核心 +核查 +核桃 +核武器 +核电 +核磁 +核算 +核酸 +根基 +根据 +根据地 +根本 +根本原因 +根本无法 +根治 +根源 +根系 +根茎 +根部 +格兰 +格力 +格勒 +格外 +格子 +格尔 +格局 +格式 +格律 +格拉 +格斗 +格斯 +格林 +格栅 +格格 +格罗 +格调 +格里 +格雷 +栽培 +栽植 +桂园 +桂圆 +桂林 +桂枝 +桂皮 +桂花 +桃园 +桃子 +桃李 +桃源 +桃花 +框架 +案件 +案例 +案发 +案子 +案情 +桌上 +桌子 +桌椅 +桌游 +桌面 +桑德 +桑拿 +桓公 +桔梗 +档案 +档次 +桥头 +桥梁 +梁山 +梅尔 +梅州 +梅花 +梅西 +梗阻 +梦中 +梦到 +梦境 +梦寐 +梦幻 +梦想 +梦游 +梦见 +梦里 +梧州 +梧桐 +梨花 +梯度 +梯形 +梯田 +梯队 +梳理 +检举 +检修 +检出 +检定 +检察 +检察官 +检察机关 +检察院 +检查 +检查和 +检测 +检疫 +检索 +检讨 +检验 +棉花 +棋子 +棋牌 +棋盘 +棍子 +棒子 +棒棒 +棒球 +棕榈 +棕色 +森林 +森林公园 +棱角 +棺材 +椅子 +植入 +植发 +植树 +植株 +植物 +植物园 +植物油 +植被 +椎间 +椭圆 +椭圆形 +椰子 +楔形 +楚国 +楚楚 +楚王 +楷书 +楼上 +楼下 +楼主 +楼宇 +楼层 +楼市 +楼房 +楼梦 +楼梯 +楼盘 +楼道 +楼顶 +楼高 +概况 +概念 +概念股 +概括 +概率 +概述 +榆林 +榜单 +榜样 +榜首 +榨汁 +榴弹 +榴莲 +槐蚀 +槟榔 +模仿 +模具 +模块 +模型 +模式 +模拟 +模板 +模样 +模特 +模糊 +模组 +模联 +模范 +横向 +横幅 +横扫 +横盘 +横行 +横贯 +横跨 +樱桃 +樱花 +橄榄 +橄榄油 +橘子 +橙子 +橙色 +橡皮 +橡胶 +橱柜 +欠缺 +次于 +次年 +次序 +次数 +次新股 +次日 +次第 +次要 +欢乐 +欢呼 +欢喜 +欢声 +欢快 +欢笑 +欢迎 +欢迎您 +欣喜 +欣慰 +欣然 +欣赏 +欧亚 +欧元 +欧元区 +欧冠 +欧式 +欧文 +欧洲 +欧洲央行 +欧盟 +欧美 +欧阳 +欧雅 +欲望 +欺人 +欺诈 +欺负 +欺骗 +款式 +款项 +歇尔 +歌剧 +歌唱 +歌坛 +歌声 +歌手 +歌曲 +歌舞 +歌词 +歌迷 +歌颂 +止咳 +止损 +止步 +止渴 +止痛 +止盈 +止血 +止跌 +正中 +正义 +正值 +正反 +正向 +正品 +正在 +正大 +正好 +正如 +正宗 +正常 +正常人 +正式 +正弦 +正当 +正文 +正是 +正月 +正月初 +正比 +正气 +正法 +正版 +正畸 +正直 +正确 +正经 +正统 +正装 +正规 +正视 +正道 +正面 +此举 +此书 +此事 +此人 +此刻 +此前 +此后 +此地 +此基础 +此处 +此外 +此文 +此方 +此时 +此时此刻 +此期 +此案 +此次 +此法 +此生 +此病 +此种 +此类 +此行 +此项 +步伐 +步入 +步兵 +步枪 +步步 +步行 +步行街 +步骤 +武侠 +武则天 +武力 +武功 +武器 +武士 +武大 +武夷 +武学 +武将 +武帝 +武当 +武昌 +武术 +武林 +武汉 +武汉大学 +武汉市 +武王 +武艺 +武装 +武警 +武陵 +歧视 +歪曲 +死亡 +死亡率 +死刑 +死去 +死叉 +死后 +死心 +死板 +死神 +死者 +死角 +歼灭 +殆尽 +殊不 +殊不知 +殊胜 +殊荣 +残余 +残存 +残忍 +残暴 +残渣 +残留 +残疾 +残疾人 +残粉 +残缺 +残酷 +殖民 +殖民地 +殴打 +段位 +段子 +段时间 +段落 +段话 +殿堂 +毁坏 +毁掉 +毁灭 +毅力 +毅然 +毋庸 +毋庸置疑 +母上 +母乳 +母乳喂养 +母亲 +母亲节 +母体 +母公司 +母女 +母婴 +母子 +母校 +母爱 +母猪 +母鸡 +每一 +每一件 +每个 +每亩 +每人 +每位 +每分钟 +每到 +每周 +每天 +每天晚上 +每家 +每年 +每当 +每户 +每日 +每晚 +每月 +每次 +每每 +每种 +每秒 +每股 +每逢 +每隔 +毒副作用 +毒品 +毒害 +毒性 +毒理 +毒素 +毒药 +毒蛇 +比上 +比不上 +比丘 +比亚 +比亚迪 +比例 +比分 +比利 +比利时 +比喻 +比如 +比如说 +比尔 +比拟 +比拼 +比方 +比比 +比比皆是 +比特 +比率 +比肩 +比赛 +比起 +比较 +比较严重 +比较复杂 +比较简单 +比重 +毕业 +毕业生 +毕业证 +毕生 +毕竟 +毗邻 +毛主席 +毛利 +毛利率 +毛发 +毛囊 +毛坯 +毛孔 +毛巾 +毛毛 +毛泽东 +毛病 +毛笔 +毛线 +毛细 +毛细血管 +毛衣 +毛钱 +毫不 +毫不犹豫 +毫克 +毫升 +毫安 +毫无 +毫无疑问 +毫米 +氏族 +民主 +民主党 +民乐 +民事 +民众 +民俗 +民兵 +民办 +民商 +民国 +民宿 +民居 +民工 +民币 +民心 +民意 +民政 +民政局 +民族 +民歌 +民法 +民生 +民生银行 +民用 +民科 +民粹 +民航 +民营 +民营企业 +民警 +民谣 +民进 +民间 +民风 +气体 +气候 +气候变化 +气功 +气动 +气势 +气化 +气压 +气味 +气和 +气喘 +气囊 +气场 +气垫 +气度 +气息 +气愤 +气机 +气死 +气氛 +气泡 +气流 +气温 +气滞 +气球 +气短 +气管 +气缸 +气色 +气虚 +气血 +气象 +气象台 +气质 +氛围 +氧化 +氧化物 +氧气 +氨基 +氨基酸 +氨酸 +氯化 +水上 +水下 +水中 +水乡 +水产 +水仙 +水份 +水位 +水体 +水准 +水分 +水利 +水务 +水县 +水土 +水域 +水墨 +水壶 +水平 +水库 +水彩 +水性 +水文 +水族 +水星 +水晶 +水果 +水池 +水泡 +水泥 +水泵 +水洗 +水流 +水浒 +水润 +水深 +水温 +水源 +水溶 +水滴 +水火 +水煎服 +水瓶 +水生 +水田 +水电 +水稻 +水管 +水箱 +水系 +水肿 +水草 +水解 +水质 +水资源 +水路 +水里 +水量 +水银 +水陆 +水面 +水龙头 +永不 +永久 +永乐 +永嘉 +永安 +永定 +永恒 +永无 +永浩 +永生 +永远 +永远都是 +求助 +求婚 +求学 +求实 +求得 +求救 +求教 +求是 +求生 +求真 +求知 +求精 +求职 +求职者 +求证 +汇合 +汇总 +汇报 +汇款 +汇率 +汇编 +汇聚 +汇集 +汉中 +汉书 +汉代 +汉化 +汉口 +汉堡 +汉奸 +汉姆 +汉子 +汉字 +汉族 +汉朝 +汉武帝 +汉语 +汕头 +汗水 +汗液 +汗腺 +江东 +江北 +江区 +江南 +江县 +江宁 +江山 +江市 +江户 +江水 +江汉 +江河 +江泽 +江泽民 +江浙 +江淮 +江湖 +江苏 +江苏省 +江西 +江西省 +江门 +江阴 +江陵 +池塘 +污垢 +污染 +污染物 +污水 +污水处理 +污泥 +污渍 +汤匙 +汤圆 +汤姆 +汤汁 +汪峰 +汲取 +汶川 +汹涌 +汽水 +汽油 +汽车 +汽车产业 +汽车站 +汽车行业 +沃尔 +沃尔沃 +沃尔玛 +沈阳 +沈阳市 +沉寂 +沉思 +沉沉 +沉浸 +沉淀 +沉溺 +沉着 +沉睡 +沉积 +沉稳 +沉迷 +沉醉 +沉重 +沉闷 +沉降 +沉静 +沉香 +沉默 +沐浴 +沙发 +沙子 +沙拉 +沙河 +沙滩 +沙漠 +沙特 +沙龙 +沟通 +没事 +没人 +没什么 +没关系 +没准 +没到 +没多 +没多久 +没完 +没带 +没得 +没想 +没想到 +没收 +没敢 +没有 +没法 +没用 +没考 +没能 +没落 +没见 +没说 +没过 +没钱 +没错 +沥干 +沥青 +沦为 +沦落 +沦陷 +沧州 +沧桑 +沧海 +沮丧 +河东 +河北 +河北省 +河南 +河南省 +河口 +河床 +河水 +河流 +河西 +河谷 +河边 +河道 +河里 +沸水 +沸点 +沸腾 +油价 +油墨 +油性 +油条 +油气 +油污 +油漆 +油炸 +油烟 +油烟机 +油田 +油画 +油盐 +油管 +油箱 +油耗 +油脂 +油腻 +油菜 +油菜花 +油锅 +油门 +治党 +治国 +治好 +治学 +治安 +治愈 +治理 +治疗 +治病 +沼气 +沼泽 +沿岸 +沿江 +沿海 +沿用 +沿着 +沿线 +沿途 +沿革 +泄漏 +泄露 +泉州 +泉水 +泊尔 +泌尿 +法人 +法令 +法兰 +法则 +法制 +法务 +法医 +法国 +法学 +法学院 +法官 +法定 +法定代表 +法宝 +法师 +法庭 +法式 +法律 +法律法规 +法律责任 +法拉 +法拉利 +法术 +法案 +法治 +法王 +法理 +法界 +法规 +法语 +法身 +法门 +法院 +泛泛 +泛滥 +泡水 +泡沫 +泡泡 +泡脚 +泡茶 +泡菜 +泡面 +波兰 +波切 +波动 +波及 +波士顿 +波尔 +波形 +波折 +波斯 +波段 +波波 +波浪 +波澜 +波特 +波纹 +波罗 +波罗蜜 +波音 +泥土 +泥沙 +注入 +注册 +注册资本 +注塑 +注定 +注射 +注意 +注意事项 +注意力 +注意安全 +注明 +注目 +注视 +注释 +注重 +注销 +注音 +泪水 +泪流 +泪流满面 +泪点 +泪盈 +泰勒 +泰国 +泰坦 +泰安 +泰山 +泰州 +泰迪 +泳池 +泸州 +泼水 +洁净 +洁白 +洁面 +洋房 +洋洋 +洋溢 +洋溢着 +洋芋 +洋葱 +洒脱 +洗净 +洗发 +洗发水 +洗头 +洗完 +洗手 +洗手间 +洗洗 +洗浴 +洗涤 +洗漱 +洗澡 +洗白 +洗盘 +洗碗 +洗礼 +洗脑 +洗脚 +洗脸 +洗衣 +洗衣服 +洗衣机 +洗车 +洗面 +洗面奶 +洛克 +洛夫 +洛杉矶 +洛阳 +洞口 +洞察 +洞穴 +津津 +津液 +津贴 +洪水 +洪荒 +洲际 +活下去 +活佛 +活儿 +活力 +活动 +活化 +活在 +活塞 +活得 +活性 +活性炭 +活泼 +活活 +活生 +活生生 +活着 +活血 +活跃 +洽谈 +派出 +派出所 +派对 +派遣 +流下 +流亡 +流产 +流传 +流体 +流入 +流出 +流利 +流动 +流动性 +流向 +流域 +流失 +流年 +流感 +流放 +流星 +流氓 +流水 +流水线 +流泪 +流派 +流浪 +流淌 +流畅 +流离 +流程 +流经 +流血 +流行 +流行病 +流言 +流转 +流过 +流连 +流通 +流逝 +流速 +流量 +流长 +流露 +流露出 +浅入 +浅显 +浅浅 +浅色 +浇水 +测定 +测控 +测算 +测绘 +测评 +测试 +测量 +测验 +济南 +济南市 +济宁 +浏览 +浏览器 +浏阳 +浑浊 +浑浑 +浑然 +浑身 +浓厚 +浓度 +浓浓 +浓浓的 +浓烈 +浓缩 +浓郁 +浓重 +浙江 +浙江大学 +浙江省 +浦东 +浩瀚 +浩然 +浩特 +浪漫 +浪潮 +浪花 +浪费 +浪费时间 +浮云 +浮出 +浮动 +浮夸 +浮游 +浮现 +浮肿 +浮躁 +浮雕 +浴室 +浴缸 +海上 +海事 +海伦 +海关 +海内 +海内外 +海军 +海南 +海南省 +海参 +海口 +海啸 +海域 +海外 +海宁 +海尔 +海岛 +海岸 +海峡 +海带 +海底 +海归 +海战 +海报 +海拔 +海景 +海棠 +海水 +海洋 +海浪 +海涛 +海淀 +海淀区 +海淘 +海湾 +海滨 +海滩 +海盗 +海绵 +海藻 +海豚 +海贼 +海边 +海运 +海里 +海量 +海面 +海风 +海马 +海鲜 +浸泡 +浸润 +涂上 +涂层 +涂抹 +涂改 +涂料 +涂装 +涂鸦 +消化 +消化不良 +消化吸收 +消化系统 +消化道 +消失 +消息 +消息面 +消散 +消极 +消毒 +消灭 +消炎 +消瘦 +消磨 +消耗 +消肿 +消融 +消费 +消费品 +消费市场 +消费者 +消退 +消逝 +消遣 +消防 +消防员 +消防安全 +消除 +消食 +涉及 +涉外 +涉嫌 +涉案 +涉足 +涌入 +涌动 +涌泉 +涌现 +涟漪 +涡轮 +润发 +润泽 +润滑 +润滑油 +润肠 +润肤 +润肺 +涨价 +涨停 +涨停板 +涨势 +涨幅 +涨跌 +液体 +液化 +液压 +液态 +液晶 +涵养 +涵盖 +淀粉 +淄博 +淋巴 +淋巴结 +淋浴 +淋漓 +淋漓尽致 +淑女 +淘宝 +淘宝网 +淘气 +淘汰 +淡化 +淡季 +淡定 +淡水 +淡泊 +淡淡 +淡淡的 +淡然 +淡雅 +淡黄 +淫秽 +淬火 +淮北 +淮南 +淮安 +淮河 +淮海 +深信 +深入 +深入人心 +深入开展 +深入研究 +深切 +深刻 +深化 +深化改革 +深厚 +深受 +深呼吸 +深圳 +深圳市 +深处 +深夜 +深奥 +深层 +深层次 +深山 +深度 +深得 +深思 +深情 +深感 +深有 +深水 +深沉 +深沟 +深浅 +深海 +深深 +深深地 +深渊 +深爱 +深知 +深耕 +深色 +深蓝 +深藏 +深远 +深造 +深邃 +深陷 +淳朴 +混乱 +混凝 +混凝土 +混动 +混合 +混日 +混杂 +混沌 +混淆 +混混 +混血 +混迹 +混音 +淹没 +添加 +添加剂 +添花 +清仓 +清代 +清净 +清凉 +清北 +清华 +清华大学 +清单 +清扫 +清新 +清明 +清明节 +清晨 +清晰 +清朝 +清末 +清楚 +清气 +清水 +清汤 +清河 +清泉 +清洁 +清洗 +清淡 +清清 +清清楚楚 +清澈 +清热 +清热解毒 +清爽 +清理 +清白 +清真 +清秀 +清算 +清纯 +清脆 +清蒸 +清远 +清醒 +清闲 +清除 +清静 +清风 +清香 +清高 +渊明 +渊源 +渐变 +渐渐 +渐进 +渔业 +渔民 +渔船 +渗入 +渗出 +渗透 +渗透到 +渠道 +渡口 +渡江 +渡河 +渡边 +渡过 +渤海 +温和 +温哥 +温婉 +温室 +温州 +温差 +温带 +温度 +温开水 +温情 +温控 +温暖 +温柔 +温水 +温泉 +温润 +温热 +温饱 +温馨 +港元 +港口 +港台 +港币 +港湾 +港澳 +港澳台 +港股 +渲染 +渴望 +游乐 +游人 +游侠 +游击 +游刃 +游客 +游戏 +游戏规则 +游泳 +游泳池 +游牧 +游玩 +游离 +游艇 +游荡 +游行 +游览 +游记 +游走 +渺小 +渺茫 +湄公 +湖人 +湖人队 +湖北 +湖北省 +湖区 +湖南 +湖南卫视 +湖南省 +湖州 +湖水 +湖泊 +湖畔 +湖边 +湖面 +湘潭 +湘西 +湛江 +湿地 +湿度 +湿气 +湿润 +湿热 +湿疣 +湿疹 +溃疡 +源于 +源头 +源性 +源氏 +源泉 +源源 +源源不断 +源码 +源自 +源远 +溜溜 +溜达 +溢价 +溢出 +溪水 +溪流 +溯源 +溶于 +溶剂 +溶性 +溶洞 +溶液 +溶血 +溶解 +溺水 +溺爱 +滋养 +滋味 +滋润 +滋生 +滋补 +滋阴 +滑冰 +滑动 +滑坡 +滑板 +滑稽 +滑落 +滑行 +滑雪 +滔滔 +滚动 +滚滚 +滚筒 +滞后 +滞留 +满分 +满嘴 +满地 +满天 +满心 +满怀 +满意 +满意度 +满族 +满是 +满月 +满洲 +满清 +满满 +满满的 +满目 +满脑 +满脸 +满足 +满身 +满载 +满面 +滤波 +滤芯 +滤过 +滤镜 +滥用 +滨州 +滨江 +滨海 +滴定 +滴水 +滴滴 +漂亮 +漂泊 +漂流 +漂浮 +漂白 +漂移 +漆黑 +漏斗 +漏水 +漏洞 +演义 +演习 +演出 +演化 +演变 +演员 +演唱 +演唱会 +演奏 +演戏 +演技 +演示 +演练 +演绎 +演职 +演艺 +演艺圈 +演讲 +演说 +演进 +漩涡 +漫天 +漫威 +漫步 +漫游 +漫漫 +漫画 +漫长 +漱口 +漳州 +潇洒 +潍坊 +潜伏 +潜入 +潜力 +潜在 +潜心 +潜意识 +潜水 +潜移 +潜移默化 +潜能 +潜艇 +潜规则 +潮州 +潮汐 +潮汕 +潮流 +潮湿 +澄清 +澎湃 +澡堂 +澳元 +澳大利亚 +澳洲 +澳门 +激光 +激动 +激励 +激发 +激情 +激活 +激烈 +激素 +激起 +激进 +濒临 +濒危 +濮阳 +瀑布 +灌木 +灌水 +灌注 +灌溉 +灌篮 +灌装 +灌输 +灌顶 +火候 +火力 +火山 +火影 +火星 +火柴 +火火 +火灾 +火炬 +火烧 +火热 +火焰 +火爆 +火箭 +火箭队 +火腿 +火花 +火药 +火车 +火车票 +火车站 +火锅 +火龙 +灭亡 +灭火 +灭绝 +灭菌 +灯光 +灯具 +灯塔 +灯泡 +灯火 +灯笼 +灯饰 +灰尘 +灰暗 +灰白 +灰色 +灰褐 +灵光 +灵动 +灵山 +灵异 +灵性 +灵感 +灵敏 +灵气 +灵活 +灵活性 +灵芝 +灵魂 +灾区 +灾害 +灾难 +灿烂 +炉石 +炎帝 +炎炎 +炎热 +炎症 +炎黄 +炒作 +炒股 +炒菜 +炒锅 +炒鸡 +炫富 +炫耀 +炮儿 +炮兵 +炮友 +炮台 +炮弹 +炮火 +炸弹 +炸药 +炸鸡 +点亮 +点位 +点儿 +点击 +点到 +点半 +点名 +点多 +点头 +点子 +点心 +点数 +点滴 +点火 +点点 +点点头 +点点滴滴 +点燃 +点球 +点睛 +点缀 +点评 +点赞 +点钟 +炼成 +炼油 +烈士 +烈日 +烈火 +烈烈 +烘干 +烘托 +烘烤 +烘焙 +烟台 +烟囱 +烟火 +烟花 +烟草 +烟酒 +烟雨 +烟雾 +烤箱 +烤肉 +烦恼 +烦躁 +烧伤 +烧开 +烧烤 +烧热 +烧结 +烧饼 +烫伤 +热力 +热巴 +热带 +热度 +热心 +热忱 +热情 +热播 +热敷 +热气 +热水 +热水器 +热潮 +热火 +热点 +热烈 +热爱 +热电 +热线 +热能 +热血 +热衷 +热衷于 +热议 +热身 +热量 +热销 +热门 +热闹 +烹调 +烹饪 +烽火 +焊接 +焕发 +焕然 +焚烧 +焦作 +焦急 +焦点 +焦虑 +焦躁 +然后 +然而 +煎服 +煎熬 +煎饼 +煤气 +煤炭 +煤矿 +照亮 +照射 +照常 +照搬 +照料 +照明 +照样 +照片 +照相 +照着 +照耀 +照镜 +照顾 +煮开 +煮沸 +煮熟 +煮粥 +煸炒 +熄火 +熄灭 +熄灯 +熊市 +熊猫 +熏陶 +熔断 +熟人 +熟悉 +熟知 +熟练 +熟能 +熬夜 +熬过 +燃放 +燃料 +燃气 +燃油 +燃烧 +燕京 +燕子 +燕山 +燕窝 +燕麦 +爆出 +爆发 +爆料 +爆棚 +爆款 +爆炸 +爆照 +爆破 +爆竹 +爆笑 +爆米 +爆表 +爆香 +爬山 +爬虫 +爬行 +爬起来 +爱上 +爱人 +爱因斯坦 +爱国 +爱国主义 +爱奇艺 +爱好 +爱好者 +爱尔 +爱尔兰 +爱心 +爱恋 +爱恨 +爱情 +爱情故事 +爱惜 +爱意 +爱慕 +爱护 +爱是 +爱玩 +爱玲 +爱看 +爱着 +爱美 +爱车 +爱过 +爵士 +父亲 +父女 +父子 +父母 +爷们 +爷爷 +爷爷奶奶 +爸妈 +爸爸 +爸爸妈妈 +爽口 +爽快 +爽肤 +片中 +片刻 +片区 +片名 +片头 +片子 +片尾 +片段 +片长 +片面 +版主 +版图 +版块 +版型 +版本 +版权 +版画 +版面 +牌号 +牌坊 +牌子 +牌照 +牙刷 +牙医 +牙周 +牙痛 +牙科 +牙膏 +牙齿 +牙龈 +牛仔 +牛仔裤 +牛头 +牛奶 +牛市 +牛排 +牛油 +牛津 +牛皮 +牛皮癣 +牛羊 +牛肉 +牛股 +牛角 +牛顿 +牡丹 +牡蛎 +牢固 +牢牢 +牢记 +牢骚 +牧场 +牧师 +牧羊 +牧草 +物业 +物业公司 +物业管理 +物产 +物价 +物体 +物力 +物化 +物品 +物料 +物权 +物流 +物理 +物理学 +物种 +物美 +物质 +物资 +牵制 +牵动 +牵头 +牵引 +牵强 +牵手 +牵扯 +牵挂 +牵涉 +牵着 +牵连 +特么 +特产 +特价 +特写 +特别 +特制 +特务 +特勒 +特区 +特地 +特大 +特定 +特尔 +特工 +特异 +特异性 +特征 +特性 +特惠 +特意 +特技 +特效 +特斯拉 +特有 +特朗 +特权 +特此 +特殊 +特殊性 +特点 +特种 +特约 +特级 +特色 +特色产业 +特警 +特许 +特质 +特邀 +特长 +牺牲 +犀利 +犀牛 +犯人 +犯法 +犯罪 +犯罪分子 +犯规 +犯错 +犯错误 +状元 +状况 +状态 +犹太 +犹太人 +犹如 +犹豫 +狂妄 +狂欢 +狂热 +狂风 +狐狸 +狐臭 +狗屁 +狗屎 +狗狗 +狗肉 +狗血 +狙击 +狠心 +狠抓 +狠狠 +狡猾 +狩猎 +独一无二 +独具 +独创 +独到 +独占 +独厚 +独唱 +独处 +独奏 +独孤 +独家 +独居 +独往 +独有 +独特 +独生 +独生子女 +独秀 +独立 +独立思考 +独立性 +独自 +独著 +独行 +独裁 +独角 +独角兽 +独资 +狭义 +狭小 +狭窄 +狭长 +狭隘 +狮子 +狼狈 +猎人 +猎头 +猎物 +猕猴 +猕猴桃 +猛烈 +猛然 +猛进 +猛龙 +猜想 +猜测 +猜猜 +猝死 +猥亵 +猥琐 +猩猩 +猪八戒 +猪油 +猪肉 +猪肝 +猪脚 +猪蹄 +猫咪 +献上 +献出 +献给 +献血 +献身 +猴子 +玄关 +玄学 +玄宗 +玄幻 +玄武 +率为 +率先 +率达 +率领 +玉兰 +玉器 +玉林 +玉树 +玉皇 +玉石 +玉米 +玉龙 +王八 +王国 +王妃 +王子 +王宝强 +王室 +王家 +王府 +王朝 +王母 +王氏 +王爷 +王牌 +王老师 +王者 +王道 +玛丽 +玛瑙 +玛雅 +玩乐 +玩儿 +玩具 +玩家 +玩弄 +玩得 +玩意 +玩法 +玩游戏 +玩玩 +玩笑 +玩耍 +玩过 +玫瑰 +玫瑰花 +环保 +环卫 +环境 +环境优美 +环境保护 +环境卫生 +环境污染 +环形 +环比 +环氧 +环游 +环状 +环球 +环绕 +环节 +环路 +现为 +现今 +现代 +现代人 +现代农业 +现代化 +现代医学 +现任 +现在 +现场 +现如今 +现存 +现实 +现实主义 +现实生活 +现将 +现已 +现役 +现成 +现时 +现有 +现款 +现状 +现行 +现象 +现货 +现身 +现金 +现金流 +现阶段 +玲珑 +玻璃 +玻璃钢 +珊瑚 +珍品 +珍宝 +珍惜 +珍爱 +珍珠 +珍稀 +珍藏 +珍贵 +珠三 +珠三角 +珠子 +珠宝 +珠江 +珠海 +班上 +班主任 +班子 +班族 +班级 +班组 +班车 +班里 +班长 +球员 +球场 +球形 +球星 +球菌 +球衣 +球迷 +球队 +理事 +理事会 +理事长 +理会 +理化 +理发 +理取 +理学 +理工 +理工大学 +理工学院 +理应 +理念 +理性 +理想 +理所 +理所当然 +理智 +理气 +理清 +理由 +理疗 +理科 +理解 +理论 +理论知识 +理财 +理财产品 +理赔 +琉球 +琉璃 +琐事 +琐碎 +琢磨 +琥珀 +琳琅 +琵琶 +琼斯 +琼脂 +瑕疵 +瑜伽 +瑞典 +瑞士 +瑞安 +瑞拉 +瑶族 +璀璨 +瓜子 +瓜果 +瓦斯 +瓦解 +瓶子 +瓶盖 +瓶装 +瓶颈 +瓷器 +瓷砖 +甘心 +甘愿 +甘油 +甘甜 +甘肃 +甘肃省 +甘草 +甘蔗 +甘露 +甚者 +甚至 +甚至于 +甜味 +甜品 +甜点 +甜甜 +甜甜的 +甜美 +甜蜜 +甜食 +生下 +生二 +生于 +生产 +生产力 +生产厂家 +生产商 +生产工艺 +生产总值 +生产线 +生产者 +生产能力 +生人 +生出 +生前 +生动 +生化 +生发 +生命 +生命力 +生命危险 +生命周期 +生命安全 +生命线 +生姜 +生子 +生存 +生平 +生态 +生态环境 +生态系统 +生怕 +生性 +生意 +生成 +生抽 +生效 +生日 +生日快乐 +生机 +生来 +生根 +生死 +生殖 +生殖器 +生气 +生水 +生津 +生活 +生活习惯 +生活品质 +生活态度 +生活用品 +生活费 +生涯 +生源 +生灵 +生物 +生物学 +生猪 +生理 +生生 +生病 +生硬 +生素 +生肖 +生育 +生菜 +生计 +生锈 +生长 +生长发育 +生鲜 +用于 +用人 +用人单位 +用以 +用作 +用具 +用到 +用力 +用功 +用品 +用地 +用处 +用尽 +用工 +用心 +用意 +用户 +用房 +用手 +用料 +用来 +用水 +用法 +用狗 +用电 +用笔 +用药 +用词 +用语 +用车 +用过 +用途 +用量 +用钱 +用餐 +田中 +田园 +田地 +田径 +田野 +田间 +由于 +由此 +由此可见 +由衷 +甲亢 +甲午 +甲基 +甲壳 +甲子 +甲方 +甲烷 +甲状腺 +甲等 +甲级 +甲醇 +甲醛 +甲骨 +申办 +申报 +申明 +申花 +申论 +申诉 +申请 +申请人 +申请者 +申请表 +申购 +申领 +电位 +电信 +电力 +电动 +电动机 +电动汽车 +电动车 +电厂 +电压 +电台 +电商 +电器 +电图 +电场 +电子 +电子产品 +电子信息 +电子商务 +电子客票 +电子邮件 +电容 +电工 +电影 +电影节 +电影院 +电报 +电控 +电教 +电机 +电极 +电梯 +电气 +电池 +电流 +电源 +电热 +电瓶 +电磁 +电离 +电站 +电竞 +电算 +电线 +电缆 +电网 +电能 +电脑 +电荷 +电视 +电视剧 +电视台 +电视机 +电视节目 +电解 +电讯 +电话 +电话号码 +电路 +电车 +电量 +电镀 +电阻 +电饭 +男主 +男主角 +男人 +男儿 +男友 +男同 +男士 +男女 +男子 +男孩 +男孩子 +男性 +男方 +男朋友 +男演员 +男生 +男神 +男票 +男科 +男篮 +男装 +画作 +画像 +画册 +画出 +画卷 +画图 +画家 +画廊 +画报 +画法 +画画 +画笔 +画质 +画面 +画风 +畅快 +畅游 +畅通 +畅销 +界定 +界线 +界限 +界面 +畏惧 +留下 +留下来 +留住 +留出 +留在 +留存 +留学 +留学生 +留守 +留心 +留念 +留恋 +留情 +留意 +留有 +留校 +留着 +留给 +留言 +畜牧 +畜牧业 +畜禽 +略带 +略微 +略显 +略有 +番外 +番禺 +番茄 +畸形 +疏导 +疏忽 +疏散 +疏松 +疏远 +疏通 +疑似 +疑惑 +疑虑 +疑问 +疑难 +疗养 +疗效 +疗法 +疗程 +疙瘩 +疤痕 +疫情 +疫病 +疫苗 +疯子 +疯狂 +疱疹 +疲乏 +疲倦 +疲劳 +疲惫 +疲软 +疼爱 +疼痛 +疾患 +疾控 +疾病 +病人 +病例 +病历 +病原 +病变 +病史 +病因 +病害 +病床 +病态 +病患 +病情 +病房 +病死 +病毒 +病灶 +病理 +病症 +病痛 +病程 +病菌 +病虫 +病虫害 +症状 +痉挛 +痊愈 +痔疮 +痕迹 +痘痘 +痛哭 +痛心 +痛快 +痛点 +痛经 +痛苦 +痛风 +痤疮 +痴呆 +痴情 +痴迷 +瘙痒 +瘟疫 +瘦下 +瘦弱 +瘦肉 +瘦身 +瘫痪 +癌症 +癌细胞 +癫痫 +癫痫病 +登上 +登台 +登场 +登山 +登录 +登机 +登记 +登陆 +登高 +白云 +白人 +白兔 +白内障 +白发 +白天 +白头 +白宫 +白富 +白居 +白居易 +白山 +白带 +白开水 +白斑 +白族 +白日 +白术 +白沙 +白玉 +白痴 +白癜 +白癜风 +白白 +白皙 +白眼 +白石 +白砂 +白糖 +白纸 +白细胞 +白羊 +白色 +白芍 +白花 +白莲 +白菜 +白萝卜 +白虎 +白血病 +白衣 +白酒 +白醋 +白金 +白银 +白雪 +白领 +白马 +白骨 +百万 +百事 +百亿 +百余 +百倍 +百公里 +百分 +百分之 +百分之百 +百分比 +百分点 +百分百 +百合 +百姓 +百家 +百岁 +百川 +百年 +百度 +百强 +百态 +百搭 +百日 +百病 +百科 +百般 +百花 +百草 +百计 +百货 +百里 +百零 +的是 +的确 +的话 +皆是 +皆知 +皇上 +皇冠 +皇后 +皇太 +皇子 +皇室 +皇宫 +皇家 +皇帝 +皇族 +皇马 +皈依 +皮下 +皮书 +皮具 +皮卡 +皮尔 +皮带 +皮毛 +皮炎 +皮肤 +皮肤病 +皮肤科 +皮脂 +皮质 +皮革 +皮鞋 +皱纹 +盆友 +盆地 +盆景 +盆栽 +盆腔 +盆腔炎 +盈亏 +盈利 +益匪 +益处 +益寿 +益智 +益气 +益生 +益生菌 +益阳 +盎司 +盎然 +盐城 +盐水 +盐酸 +监事 +监会 +监听 +监委 +监察 +监局 +监护 +监护人 +监控 +监测 +监狱 +监理 +监督 +监督管理 +监管 +监管局 +监管部门 +监考 +监视 +盒子 +盖上 +盖子 +盖章 +盖茨 +盗墓 +盗版 +盗窃 +盗贼 +盘中 +盘古 +盘子 +盘整 +盘旋 +盘点 +盘面 +盛世 +盛会 +盛典 +盛名 +盛夏 +盛大 +盛宴 +盛开 +盛行 +盟军 +盟友 +目光 +目击 +目前 +目前为止 +目录 +目标 +目测 +目的 +目的地 +目睹 +盲人 +盲目 +直击 +直到 +直到现在 +直升 +直升机 +直奔 +直属 +直径 +直接 +直播 +直播间 +直板 +直流 +直立 +直系 +直线 +直肠 +直至 +直营 +直观 +直觉 +直角 +直言 +直辖 +直辖市 +直达 +直通 +直通车 +直销 +直隶 +直面 +相互 +相互作用 +相交 +相亲 +相传 +相伴 +相似 +相位 +相依 +相信 +相克 +相关 +相关性 +相册 +相助 +相反 +相合 +相同 +相声 +相处 +相容 +相对 +相对来说 +相差 +相干 +相应 +相当 +相当于 +相思 +相恋 +相成 +相接 +相提 +相望 +相机 +相比 +相比之下 +相爱 +相片 +相看 +相知 +相符 +相等 +相约 +相结合 +相继 +相聚 +相见 +相识 +相貌 +相距 +相较 +相辅 +相辅相成 +相近 +相连 +相适应 +相通 +相逢 +相遇 +相邻 +相间 +相随 +相隔 +盼望 +省事 +省份 +省会 +省内 +省力 +省去 +省委 +省市 +省心 +省政府 +省油 +省略 +省级 +省道 +省部 +省钱 +省长 +眉头 +眉毛 +看一看 +看上 +看上去 +看不 +看不出 +看不出来 +看不到 +看不懂 +看不清 +看不见 +看不起 +看中 +看书 +看似 +看作 +看做 +看出 +看到 +看吧 +看多 +看好 +看守 +看待 +看得 +看得出 +看得出来 +看得见 +看成 +看房 +看望 +看来 +看法 +看涨 +看清 +看清楚 +看点 +看电视 +看病 +看看 +看着 +看见 +看起来 +看过 +看透 +看重 +真主 +真事 +真人 +真伪 +真假 +真切 +真实 +真实性 +真心 +真情 +真想 +真挚 +真是 +真是太 +真有 +真正 +真爱 +真理 +真的 +真皮 +真相 +真真 +真空 +真能 +真菌 +真言 +真诚 +真话 +真谛 +真题 +眨眼 +眩晕 +眷属 +眷恋 +眺望 +眼下 +眼中 +眼光 +眼前 +眼前一亮 +眼周 +眼圈 +眼帘 +眼底 +眼影 +眼泪 +眼球 +眼界 +眼皮 +眼看 +眼眶 +眼睁 +眼睑 +眼睛 +眼神 +眼科 +眼线 +眼花 +眼花缭乱 +眼袋 +眼见 +眼角 +眼部 +眼里 +眼镜 +眼霜 +着力 +着头 +着实 +着急 +着想 +着手 +着火 +着眼 +着眼于 +着色 +着装 +着迷 +着重 +着陆 +睁开 +睁眼 +睡不着 +睡前 +睡得 +睡眠 +睡眠不足 +睡着 +睡衣 +睡觉 +睡觉时 +督促 +督办 +督察 +督导 +督查 +睫毛 +睾丸 +睿智 +瞄准 +瞌睡 +瞩目 +瞬时 +瞬间 +瞳孔 +矛盾 +矢量 +知乎 +知名 +知名企业 +知名品牌 +知名度 +知己 +知心 +知性 +知情 +知晓 +知觉 +知识 +知识产权 +知识分子 +知识点 +知足 +知道 +知青 +知音 +矩形 +矩阵 +矫情 +矫正 +短信 +短发 +短小 +短时间 +短暂 +短期 +短期内 +短板 +短片 +短短 +短短的 +短篇 +短线 +短缺 +短袖 +短裙 +短裤 +短路 +石刻 +石化 +石器 +石墨 +石头 +石子 +石家庄 +石材 +石板 +石柱 +石桥 +石榴 +石油 +石灰 +石狮 +石窟 +石膏 +石英 +石门 +石雕 +矿业 +矿井 +矿产 +矿产资源 +矿山 +矿工 +矿泉水 +矿物 +矿物质 +矿石 +码农 +码头 +砂浆 +砂糖 +砂锅 +研习 +研修 +研判 +研制 +研发 +研磨 +研究 +研究会 +研究员 +研究室 +研究成果 +研究所 +研究生 +研究者 +研究院 +研讨 +研讨会 +砖头 +砖木 +破产 +破位 +破坏 +破损 +破旧 +破案 +破灭 +破烂 +破碎 +破裂 +破解 +破财 +破门 +破除 +硅胶 +硅藻 +硅谷 +硅酸 +硕士 +硕士学位 +硕果 +硝烟 +硝酸 +硫化 +硫酸 +硬件 +硬化 +硬币 +硬度 +硬座 +硬性 +硬是 +硬朗 +硬盘 +硬笔 +硬质 +确保 +确信 +确切 +确定 +确定性 +确实 +确有 +确确 +确立 +确认 +确诊 +碍于 +碎片 +碎石 +碑林 +碧桂园 +碰上 +碰到 +碰巧 +碰撞 +碰碰 +碱性 +碳化 +碳水 +碳水化合物 +碳纤维 +碳酸 +碾压 +磁力 +磁场 +磁带 +磁性 +磁盘 +磁铁 +磅礴 +磨合 +磨损 +磨砂 +磨砺 +磨练 +磨难 +磷脂 +磷酸 +示例 +示威 +示意 +示意图 +示范 +示范区 +礼仪 +礼包 +礼品 +礼堂 +礼拜 +礼服 +礼物 +礼盒 +礼节 +礼貌 +社交 +社会 +社会主义 +社会保险 +社会保障 +社会化 +社会各界 +社会学 +社会科学 +社保 +社区 +社团 +社科 +社稷 +社群 +社长 +祈求 +祈祷 +祈福 +祖传 +祖先 +祖国 +祖宗 +祖师 +祖母 +祖父 +祖辈 +祛斑 +祛痘 +祛除 +祛风 +祝你 +祝好 +祝您 +祝愿 +祝福 +祝贺 +神仙 +神作 +神农 +神功 +神器 +神圣 +神奇 +神学 +神州 +神态 +神情 +神明 +神灵 +神父 +神秘 +神经 +神经系统 +神经衰弱 +神舟 +神话 +神通 +神采 +神韵 +神马 +神龙 +祥和 +票价 +票务 +票房 +票据 +祭祀 +祷告 +禁令 +禁区 +禁忌 +禁止 +禁毒 +禁用 +禅宗 +禅定 +禅师 +福利 +福州 +福建 +福建省 +福德 +福报 +福气 +福特 +福田 +福音 +离不开 +离世 +离休 +离别 +离去 +离合 +离场 +离奇 +离婚 +离子 +离家 +离开 +离异 +离心 +离散 +离职 +离谱 +禽兽 +秀丽 +秀才 +秀美 +私下 +私人 +私信 +私募 +私家 +私家车 +私密 +私心 +私房 +私教 +私有 +私立 +私自 +私营 +秉承 +秉持 +秋冬 +秋天 +秋季 +秋水 +秋雨 +秋风 +种子 +种族 +种植 +种植业 +种种 +种类 +科举 +科创 +科员 +科大 +科夫 +科学 +科学家 +科学性 +科学技术 +科学研究 +科学院 +科室 +科尔 +科属 +科幻 +科技 +科技大学 +科技成果 +科技进步 +科教 +科普 +科比 +科特 +科班 +科目 +科研 +科研成果 +科科 +科长 +科院 +秒杀 +秒钟 +秘书 +秘书长 +秘密 +秘方 +秘籍 +秘诀 +秘鲁 +租借 +租房 +租用 +租赁 +租车 +租金 +秦国 +秦始皇 +秦岭 +秦朝 +秦汉 +秦皇 +秦皇岛 +秩序 +积分 +积攒 +积木 +积极 +积极主动 +积极参与 +积极向上 +积极开展 +积极性 +积极探索 +积水 +积淀 +积累 +积聚 +积蓄 +积雪 +称为 +称之 +称之为 +称作 +称其 +称号 +称呼 +称得 +称得上 +称谓 +称赞 +称霸 +秸秆 +移交 +移位 +移动 +移动电话 +移居 +移植 +移步 +移民 +移送 +稀土 +稀奇 +稀少 +稀有 +稀疏 +稀缺 +稀罕 +稀释 +稀里 +程序 +程度 +程式 +程控 +稍微 +稍稍 +税前 +税务 +税务局 +税务机关 +税收 +税款 +税法 +税率 +税费 +税额 +稚嫩 +稳产 +稳健 +稳压 +稳固 +稳妥 +稳定 +稳定性 +稳步 +稳稳 +稳重 +稻田 +稻草 +稽查 +稿件 +穆斯 +穴位 +究其 +究其原因 +究竟 +穷人 +穷养 +穷困 +穷游 +空中 +空余 +空军 +空前 +空单 +空地 +空头 +空心 +空性 +空手 +空旷 +空格 +空气 +空气污染 +空气质量 +空洞 +空白 +空空 +空缺 +空腹 +空虚 +空袭 +空调 +空运 +空闲 +空间 +空降 +空隙 +穿刺 +穿孔 +穿戴 +穿插 +穿梭 +穿着 +穿行 +穿衣 +穿衣服 +穿越 +穿过 +穿透 +突出 +突击 +突厥 +突发 +突发事件 +突变 +突围 +突如其来 +突然 +突破 +突破口 +突袭 +突起 +窍门 +窒息 +窗前 +窗口 +窗台 +窗外 +窗帘 +窗户 +立下 +立业 +立于 +立交 +立体 +立体感 +立刻 +立功 +立即 +立场 +立志 +立意 +立方 +立方米 +立柱 +立案 +立法 +立足 +立足于 +立面 +立项 +立马 +竖起 +站上 +站位 +站台 +站点 +站稳 +站立 +站长 +竞争 +竞争力 +竞争对手 +竞价 +竞技 +竞赛 +竞选 +竟是 +竟然 +章法 +章程 +章节 +章鱼 +竣工 +童子 +童年 +童心 +童装 +童话 +童鞋 +竭力 +竭尽 +竭诚 +端午 +端午节 +端口 +端子 +端庄 +端正 +端的 +竹子 +竹林 +竹笋 +竹马 +笑了 +笑了笑 +笑傲 +笑声 +笑容 +笑点 +笑笑 +笑脸 +笑话 +笑语 +笔下 +笔墨 +笔法 +笔画 +笔直 +笔者 +笔记 +笔记本 +笔记本电脑 +笔试 +笛子 +符号 +符合 +符合条件 +第一 +第七 +第三 +第三方 +第三者 +第九 +第二 +第五 +第八 +第六 +第十 +第四 +笼子 +笼统 +笼罩 +等于 +等价 +等候 +等到 +等同 +等同于 +等国 +等奖 +等待 +等离子 +等等 +等级 +筋骨 +答主 +答复 +答应 +答案 +答疑 +答辩 +答题 +策划 +策略 +筛查 +筛选 +筷子 +筹划 +筹备 +筹建 +筹码 +筹资 +筹集 +签下 +签到 +签名 +签字 +签约 +签署 +签订 +签证 +简书 +简介 +简体 +简便 +简化 +简单 +简历 +简明 +简易 +简洁 +简直 +简短 +简称 +简简 +简简单单 +简约 +简而 +简要 +简陋 +算不上 +算了 +算命 +算得 +算是 +算术 +算法 +算计 +管制 +管委 +管委会 +管子 +管家 +管控 +管教 +管理 +管理中心 +管理人员 +管理体系 +管理制度 +管理员 +管理处 +管理学 +管理局 +管理层 +管理工作 +管理机构 +管理条例 +管理模式 +管理水平 +管理系统 +管理者 +管理费 +管用 +管线 +管网 +管辖 +管道 +箭头 +箱体 +箱包 +箱子 +篆刻 +篇幅 +篇文章 +篇章 +篮子 +篮板 +篮球 +籍贯 +米亚 +米兰 +米勒 +米尔 +米左右 +米粉 +米饭 +类产品 +类人 +类似 +类别 +类型 +类推 +类比 +类药物 +粉丝 +粉刺 +粉尘 +粉底 +粉末 +粉碎 +粉笔 +粉红 +粉红色 +粉色 +粒子 +粗壮 +粗大 +粗心 +粗暴 +粗略 +粗粮 +粗糙 +粗细 +粘合 +粘土 +粘度 +粘性 +粘液 +粘稠 +粘结 +粘膜 +粘贴 +粘连 +粤港澳 +粤菜 +粤语 +粥样 +粪便 +粮油 +粮食 +粳米 +粽子 +精准 +精制 +精力 +精华 +精品 +精妙 +精子 +精密 +精巧 +精度 +精彩 +精彩内容 +精心 +精明 +精气 +精油 +精液 +精深 +精湛 +精灵 +精炼 +精益 +精益求精 +精盐 +精确 +精神 +精神分裂 +精神文明 +精神状态 +精神疾病 +精神病 +精简 +精算 +精细 +精细化 +精美 +精致 +精良 +精英 +精装 +精诚 +精进 +精选 +精通 +精锐 +精髓 +糊口 +糊弄 +糊涂 +糊状 +糊糊 +糕点 +糖分 +糖尿 +糖尿病 +糖果 +糖浆 +糖类 +糖醋 +糜烂 +糟糕 +糯米 +系列 +系列产品 +系数 +系统 +系统化 +系统性 +紊乱 +素养 +素描 +素有 +素材 +素质 +素质教育 +素颜 +素食 +索取 +索尼 +索引 +索性 +索要 +索赔 +紧凑 +紧凑型 +紧固 +紧密 +紧密结合 +紧张 +紧急 +紧扣 +紧接 +紧接着 +紧紧 +紧紧围绕 +紧绷 +紧缩 +紧缺 +紧致 +紧要 +紧贴 +紧跟 +紧身 +紧迫 +紧邻 +紧随 +紧靠 +紫外 +紫外线 +紫檀 +紫砂 +紫红 +紫色 +紫菜 +紫金 +累死 +累积 +累累 +累计 +繁体 +繁华 +繁多 +繁忙 +繁杂 +繁殖 +繁琐 +繁育 +繁荣 +繁衍 +繁重 +纠正 +纠纷 +纠结 +纠缠 +纠错 +红军 +红利 +红包 +红十字 +红十字会 +红卫兵 +红叶 +红外 +红尘 +红斑 +红旗 +红星 +红木 +红枣 +红柿 +红椒 +红楼 +红楼梦 +红润 +红火 +红灯 +红烧 +红白 +红米 +红糖 +红素 +红红 +红线 +红细胞 +红肿 +红色 +红花 +红茶 +红薯 +红血 +红血丝 +红衣 +红褐 +红豆 +红辣椒 +红酒 +红颜 +纤细 +纤维 +纤维素 +约会 +约合 +约定 +约束 +约翰 +约车 +级别 +级数 +纪元 +纪委 +纪实 +纪录 +纪录片 +纪律 +纪念 +纪念日 +纪念碑 +纪念馆 +纪检 +纪检监察 +纪要 +纬度 +纯中药 +纯净 +纯天然 +纯属 +纯度 +纯收入 +纯棉 +纯正 +纯洁 +纯真 +纯粹 +纱布 +纲上 +纲目 +纲要 +纲领 +纳什 +纳入 +纳德 +纳斯 +纳税 +纳税人 +纳米 +纳粹 +纳闷 +纵使 +纵向 +纵容 +纵横 +纵深 +纵然 +纵观 +纵队 +纷争 +纷呈 +纷纷 +纷纷表示 +纸上 +纸巾 +纸币 +纸张 +纸条 +纸牌 +纸盒 +纸箱 +纸质 +纹理 +纹路 +纹身 +纹饰 +纺织 +纽带 +纽约 +线上 +线下 +线图 +线圈 +线形 +线性 +线条 +线段 +线程 +线索 +线路 +练习 +练功 +练就 +练琴 +练练 +组件 +组分 +组合 +组团 +组委 +组委会 +组建 +组成 +组成部分 +组组 +组织 +组织部 +组装 +组长 +组队 +绅士 +细分 +细则 +细化 +细嫩 +细小 +细微 +细心 +细思 +细末 +细砂糖 +细纹 +细细 +细胞 +细腻 +细致 +细节 +细菌 +细长 +细雨 +织物 +织田 +终于 +终将 +终归 +终日 +终极 +终止 +终点 +终生 +终究 +终端 +终结 +终老 +终身 +绍兴 +经不起 +经久 +经典 +经办 +经历 +经受 +经商 +经常 +经常出现 +经常性 +经开 +经得起 +经文 +经期 +经济 +经济体 +经济区 +经济危机 +经济学 +经济学家 +经济性 +经济损失 +经济效益 +经济社会 +经理 +经理人 +经由 +经管 +经纪 +经纪人 +经纬 +经络 +经脉 +经营 +经营性 +经营权 +经营者 +经营范围 +经贸 +经费 +经过 +经销 +经销商 +经验 +经验丰富 +绑定 +绑架 +绒毛 +结业 +结了 +结交 +结伴 +结印 +结合 +结婚 +结实 +结尾 +结局 +结成 +结晶 +结束 +结构 +结构性 +结构调整 +结果 +结果显示 +结核 +结盟 +结石 +结算 +结缘 +结肠 +结膜 +结节 +结论 +结识 +结语 +绕行 +绕过 +绘制 +绘图 +绘本 +绘画 +给予 +给人以 +给付 +给出 +给力 +给我 +绚丽 +绚烂 +绝不 +绝不会 +绝不能 +绝世 +绝伦 +绝佳 +绝地 +绝大多数 +绝大部分 +绝妙 +绝对 +绝技 +绝招 +绝无 +绝望 +绝缘 +绝育 +绝非 +绞痛 +统一 +统帅 +统战 +统治 +统治者 +统称 +统筹 +统统 +统考 +统计 +统计局 +统计数据 +统领 +绣花 +继任 +继发 +继承 +继承人 +继电 +继续 +继而 +绩效 +绩点 +续约 +续续 +续航 +绯闻 +绳子 +维也纳 +维亚 +维他命 +维修 +维克 +维吾尔 +维基 +维多 +维多利亚 +维奇 +维尔 +维度 +维护 +维拉 +维持 +维斯 +维新 +维权 +维特 +维生素 +维系 +绵延 +绵绵 +绵阳 +综上 +综上所述 +综合 +综合体 +综合征 +综合性 +综合治理 +综合症 +综艺 +综艺节目 +绽放 +绿化 +绿叶 +绿地 +绿树 +绿水 +绿洲 +绿灯 +绿绿 +绿色 +绿茶 +绿豆 +缅怀 +缅甸 +缓冲 +缓和 +缓存 +缓慢 +缓缓 +缓解 +缔约 +缔造 +编为 +编入 +编写 +编制 +编剧 +编号 +编委 +编审 +编导 +编排 +编撰 +编曲 +编码 +编程 +编纂 +编织 +编者 +编著 +编译 +编辑 +编造 +缘分 +缘故 +缘由 +缘起 +缝合 +缝纫 +缝隙 +缠绕 +缠绵 +缤纷 +缩写 +缩减 +缩小 +缩影 +缩放 +缩水 +缩短 +缩量 +缭乱 +缭绕 +缴存 +缴纳 +缴费 +缺乏 +缺口 +缺失 +缺少 +缺席 +缺氧 +缺水 +缺点 +缺爱 +缺血 +缺陷 +罐头 +网上 +网卡 +网友 +网吧 +网址 +网店 +网易 +网格 +网民 +网游 +网点 +网状 +网球 +网盘 +网站 +网红 +网约车 +网络 +网络安全 +网络小说 +网络平台 +网络游戏 +网络营销 +网购 +网贷 +网路 +网速 +网银 +网页 +罕见 +罗伯 +罗伯特 +罗兰 +罗列 +罗宾 +罗拉 +罗斯 +罗曼 +罗杰 +罗汉 +罗湖 +罗辑 +罗马 +罚款 +罚球 +罢了 +罢免 +罢工 +罪名 +罪恶 +罪犯 +罪行 +置业 +置于 +置信 +置换 +置疑 +置身 +置身于 +署名 +羁绊 +羊毛 +羊绒 +羊羊 +羊肉 +美丽 +美人 +美元 +美军 +美剧 +美化 +美协 +美发 +美味 +美团 +美国 +美国政府 +美女 +美好 +美好生活 +美妆 +美妙 +美学 +美容 +美容院 +美工 +美帝 +美式 +美德 +美感 +美方 +美景 +美术 +美术学院 +美术家 +美术馆 +美洲 +美满 +美甲 +美男 +美白 +美的 +美美 +美联储 +美股 +美观 +美誉 +美貌 +美酒 +美金 +美院 +美颜 +美食 +羞愧 +羞涩 +羞耻 +羞辱 +羡慕 +群众 +群体 +群发 +群山 +群岛 +群星 +群落 +群里 +群雄 +羽毛 +羽毛球 +羽状 +羽绒 +羽绒服 +翅膀 +翠绿 +翡翠 +翩翩 +翰林 +翻倍 +翻出 +翻唱 +翻墙 +翻天 +翻开 +翻新 +翻滚 +翻炒 +翻看 +翻翻 +翻脸 +翻译 +翻身 +翻转 +翻过 +翻阅 +翼翼 +耀眼 +老一辈 +老中医 +老乡 +老二 +老人 +老人家 +老伴 +老先 +老公 +老兵 +老化 +老去 +老友 +老君 +老和尚 +老外 +老大 +老天 +老太 +老太太 +老夫 +老头 +老奶奶 +老妈 +老娘 +老婆 +老子 +老字号 +老实 +老客户 +老家 +老将 +老少 +老师 +老年 +老年人 +老总 +老抽 +老挝 +老旧 +老是 +老朋友 +老板 +老板娘 +老爷 +老爸 +老牌 +老王 +老百姓 +老罗 +老老 +老老实实 +老者 +老虎 +老街 +老鹰 +老鼠 +老龄 +老龄化 +考上 +考了 +考入 +考前 +考勤 +考取 +考古 +考场 +考完 +考官 +考察 +考得 +考查 +考核 +考点 +考生 +考研 +考究 +考网 +考虑 +考证 +考评 +考试 +考试成绩 +考过 +考量 +考题 +考验 +而上 +而下 +而不 +而且 +而为 +而今 +而入 +而出 +而后 +而复 +而外 +而定 +而已 +而异 +而成 +而是 +而来 +而然 +而生 +而知 +而立 +而至 +而行 +而言 +而言之 +而论 +而过 +耐久 +耐克 +耐力 +耐劳 +耐压 +耐受 +耐寒 +耐心 +耐火 +耐热 +耐用 +耐磨 +耐药 +耕作 +耕地 +耕耘 +耗尽 +耗时 +耗材 +耗电 +耗费 +耦合 +耳塞 +耳朵 +耳机 +耳熟 +耳环 +耳目 +耳聋 +耳边 +耳鸣 +耳鼻 +耳鼻喉 +耶夫 +耶律 +耶稣 +耶鲁 +耻辱 +耽美 +耽误 +聆听 +聊城 +聊天 +聊斋 +聊聊 +聊起 +职业 +职业技能 +职业生涯 +职业道德 +职位 +职务 +职员 +职场 +职工 +职权 +职称 +职能 +职能部门 +职责 +职高 +联保 +联储 +联军 +联动 +联合 +联合会 +联合国 +联名 +联席 +联想 +联手 +联播 +联机 +联欢 +联盟 +联系 +联系人 +联系方式 +联系电话 +联结 +联络 +联网 +联考 +联谊 +联赛 +联通 +联邦 +聘任 +聘用 +聘请 +聚乙烯 +聚会 +聚合 +聚在 +聚在一起 +聚居 +聚氨 +聚氨酯 +聚焦 +聚酯 +聚集 +聚餐 +聪慧 +聪明 +聪明人 +肆意 +肆无忌惮 +肇事 +肇庆 +肉丝 +肉体 +肉桂 +肉片 +肉眼 +肉类 +肉质 +肉食 +肉馅 +肋骨 +肌瘤 +肌肉 +肌肤 +肖像 +肚子 +肚皮 +肚脐 +肛肠 +肛门 +肝功能 +肝气 +肝火 +肝炎 +肝病 +肝癌 +肝硬 +肝硬化 +肝经 +肝肾 +肝胆 +肝脏 +肠胃 +肠道 +股东 +股东大会 +股价 +股份 +股份制 +股市 +股指 +股本 +股权 +股民 +股票 +股票市场 +股骨 +股骨头 +肢体 +肤浅 +肤色 +肤质 +肥厚 +肥大 +肥料 +肥沃 +肥皂 +肥肉 +肥胖 +肩上 +肩膀 +肩负 +肩部 +肯定 +肯德 +肯德基 +育人 +育儿 +育才 +育种 +肺气 +肺炎 +肺癌 +肺腑 +肺部 +肾上 +肾上腺 +肾功能 +肾小 +肾气 +肾炎 +肾病 +肾脏 +肾虚 +肿块 +肿大 +肿痛 +肿瘤 +肿胀 +胀痛 +胃口 +胃溃疡 +胃炎 +胃病 +胃痛 +胃癌 +胃肠 +胃肠道 +胃部 +胃酸 +胆囊 +胆固醇 +胆子 +胆小 +胆怯 +胆汁 +胆碱 +胆结石 +背上 +背书 +背光 +背包 +背叛 +背后 +背影 +背心 +背景 +背着 +背离 +背诵 +背负 +背部 +背锅 +背靠 +背面 +背驰 +胎儿 +胎盘 +胖子 +胖胖 +胚胎 +胜任 +胜出 +胜利 +胜地 +胜率 +胜负 +胜过 +胞胎 +胡乱 +胡同 +胡子 +胡椒 +胡椒粉 +胡歌 +胡萝卜 +胡萝卜素 +胡说 +胡适 +胡锦涛 +胰岛 +胰岛素 +胰腺 +胳膊 +胶东 +胶体 +胶原 +胶原蛋白 +胶囊 +胶带 +胶水 +胶片 +胶版 +胶粘 +胶质 +胸前 +胸口 +胸大 +胸怀 +胸腔 +胸部 +胸闷 +能为 +能力 +能动 +能否 +能够 +能干 +能手 +能源 +能耐 +能耗 +能量 +脂肪 +脂肪肝 +脂肪酸 +脆弱 +脉冲 +脉搏 +脉络 +脊柱 +脊椎 +脊髓 +脍炙 +脏器 +脏腑 +脑中 +脑力 +脑子 +脑子里 +脑残 +脑洞 +脑海 +脑海中 +脑筋 +脑细胞 +脑膜 +脑血管 +脑补 +脑袋 +脑部 +脖子 +脚下 +脚印 +脚尖 +脚底 +脚本 +脚步 +脚气 +脚趾 +脚跟 +脚踏 +脚踏实地 +脚踝 +脱俗 +脱发 +脱口 +脱欧 +脱毛 +脱水 +脱硫 +脱离 +脱胎 +脱落 +脱衣 +脱贫 +脱颖 +脱颖而出 +脸上 +脸型 +脸庞 +脸皮 +脸红 +脸色 +脸蛋 +脸部 +脸颊 +脾气 +脾胃 +脾虚 +腊八 +腊月 +腊肉 +腋下 +腋臭 +腌制 +腐女 +腐朽 +腐烂 +腐蚀 +腐蚀性 +腐败 +腥味 +腰带 +腰椎 +腰椎间盘 +腰痛 +腰线 +腰部 +腰酸 +腹中 +腹泻 +腹痛 +腹肌 +腹胀 +腹腔 +腹膜 +腹部 +腾腾 +腾讯 +腾飞 +腿部 +膀胱 +膜炎 +膝关 +膝关节 +膝盖 +膨胀 +膳食 +臀部 +自个 +自主 +自习 +自从 +自以为 +自以为是 +自传 +自体 +自作 +自保 +自信 +自信心 +自创 +自制 +自动 +自动化 +自助 +自卑 +自卫 +自发 +自古 +自古以来 +自在 +自大 +自如 +自娱 +自学 +自定义 +自家 +自尊 +自尊心 +自己 +自已 +自带 +自干 +自幼 +自弃 +自强 +自律 +自得 +自性 +自恋 +自愈 +自愿 +自慰 +自成 +自我 +自我介绍 +自拍 +自拔 +自控 +自救 +自暴 +自有 +自杀 +自来 +自来水 +自查 +自欺 +自此 +自治 +自治区 +自治县 +自治州 +自测 +自然 +自然人 +自然保护区 +自然村 +自然灾害 +自然环境 +自然界 +自然科学 +自然而然 +自然资源 +自理 +自由 +自由基 +自由自在 +自由贸易 +自知 +自私 +自称 +自立 +自答 +自考 +自营 +自行 +自行车 +自觉 +自认 +自诩 +自语 +自豪 +自负 +自责 +自贸 +自贸区 +自费 +自足 +自身 +自转 +自达 +自选 +自重 +自闭 +自闭症 +自问 +自驾 +自驾游 +臭味 +臭氧 +至上 +至于 +至今 +至关 +至关重要 +至尊 +至少 +至极 +至此 +至终 +致使 +致力 +致力于 +致命 +致富 +致敬 +致歉 +致死 +致电 +致病 +致癌 +致谢 +致辞 +致远 +舅舅 +舆论 +舌头 +舌尖 +舌苔 +舍不得 +舍利 +舍弃 +舍得 +舒展 +舒心 +舒服 +舒畅 +舒缓 +舒适 +舒适度 +舒适性 +舞台 +舞弊 +舞曲 +舞蹈 +舟山 +航天 +航母 +航海 +航班 +航空 +航空公司 +航空航天 +航线 +航行 +航运 +航道 +般的 +般若 +舰艇 +舰队 +船上 +船只 +船舶 +船长 +良多 +良好 +良心 +良性 +良机 +良知 +良种 +良药 +艰巨 +艰苦 +艰辛 +艰难 +色彩 +色情 +色拉 +色斑 +色泽 +色狼 +色系 +色素 +色色 +色调 +色谱 +艳丽 +艺人 +艺兴 +艺术 +艺术品 +艺术家 +艺术类 +艺术节 +艺考 +艾伦 +艾尔 +艾斯 +艾滋 +艾滋病 +艾灸 +艾米 +节俭 +节假 +节假日 +节制 +节后 +节奏 +节度 +节拍 +节日 +节气 +节水 +节点 +节目 +节省 +节约 +节能 +节节 +节课 +节选 +节食 +芋头 +芒果 +芙蓉 +芜湖 +芝加哥 +芝士 +芝麻 +芦笋 +芦苇 +芦荟 +芬兰 +芬芳 +芭比 +芭蕾 +芯片 +花儿 +花卉 +花园 +花岗 +花开 +花心 +花期 +花木 +花朵 +花果 +花柱 +花样 +花椒 +花海 +花瓣 +花瓶 +花生 +花生油 +花生米 +花盆 +花粉 +花纹 +花肉 +花色 +花花 +花草 +花药 +花萼 +花费 +花都 +花钱 +花香 +花鸟 +花鸟画 +芳草 +芳香 +芹菜 +苍白 +苍蝇 +苏东坡 +苏丹 +苏区 +苏南 +苏宁 +苏州 +苏州市 +苏打 +苏格兰 +苏联 +苏轼 +苏醒 +苗族 +苗木 +苗条 +苛刻 +苟且 +若干 +若惊 +若有 +若能 +苦心 +苦恼 +苦涩 +苦瓜 +苦练 +苦苦 +苦难 +英伦 +英俊 +英勇 +英国 +英国人 +英寸 +英尺 +英才 +英文 +英文名 +英格兰 +英汉 +英特尔 +英美 +英语 +英超 +英里 +英镑 +英雄 +苹果 +苹果公司 +茁壮 +茂名 +茂密 +茂盛 +范例 +范儿 +范围 +范式 +范畴 +茄子 +茅台 +茅台酒 +茉莉 +茫然 +茫茫 +茯苓 +茴香 +茶几 +茶匙 +茶叶 +茶园 +茶树 +茶水 +茶艺 +荆州 +荆棘 +荆门 +草丛 +草书 +草原 +草地 +草坪 +草木 +草本 +草根 +草案 +草泥 +草稿 +草药 +草莓 +荒凉 +荒唐 +荒地 +荒山 +荒废 +荒漠 +荒诞 +荒谬 +荔枝 +荞麦 +荟萃 +荡漾 +荡荡 +荣华 +荣威 +荣幸 +荣耀 +荣获 +荣誉 +荣誉称号 +荧光 +荨麻疹 +药业 +药丸 +药剂 +药品 +药学 +药师 +药店 +药性 +药房 +药效 +药方 +药材 +药水 +药物 +药理 +药用 +荷兰 +荷叶 +荷尔 +荷尔蒙 +荷花 +莆田 +莉亚 +莎士 +莫及 +莫名 +莫名其妙 +莫大 +莫扎 +莫斯科 +莫测 +莫过于 +莱坞 +莱德 +莱斯 +莱昂 +莱特 +莱茵 +莲子 +莲花 +莲藕 +获利 +获取 +获奖 +获得 +获得成功 +获得者 +获悉 +获批 +获益 +获省 +获胜 +菊花 +菌丝 +菏泽 +菜单 +菜品 +菜市场 +菜肴 +菜谱 +菜鸟 +菠菜 +菠萝 +菩提 +菩萨 +菱形 +菲律宾 +萃取 +萌芽 +萌萌 +萎缩 +萝卜 +萝莉 +营业 +营业执照 +营业税 +营业部 +营业额 +营养 +营养不良 +营养价值 +营养成分 +营养物质 +营养素 +营利 +营口 +营商 +营地 +营收 +营救 +营运 +营造 +营销 +萧山 +萧条 +萨克 +萨尔 +萨拉 +萨摩 +萨斯 +萨满 +落下 +落入 +落到 +落到实处 +落叶 +落后 +落地 +落实 +落寞 +落差 +落幕 +落成 +落户 +落日 +落水 +落泪 +落脚 +落花 +落落 +落魄 +著作 +著作权 +著名 +著述 +葛根 +葡萄 +葡萄干 +葡萄牙 +葡萄糖 +葡萄酒 +董事 +董事会 +董事长 +葫芦 +葱姜 +葱白 +葱花 +葵花 +蒂斯 +蒋介石 +蒙古 +蒙德 +蒙特 +蒙蒙 +蒜末 +蒲公英 +蒸发 +蒸气 +蒸汽 +蒸熟 +蒸馏 +蓄势 +蓄水 +蓄电 +蓄电池 +蓄积 +蓝光 +蓝图 +蓝天 +蓝牙 +蓝筹 +蓝筹股 +蓝色 +蓝莓 +蓬勃 +蓬勃发展 +蓬松 +蓬莱 +蔓延 +蔚来 +蔬果 +蔬菜 +蔷薇 +蕴含 +蕴藏 +蕾丝 +薄弱 +薄片 +薄膜 +薄荷 +薄薄 +薄薄的 +薏米 +薪水 +薪资 +薪酬 +薪金 +薪阶 +薰衣 +薰衣草 +藏书 +藏传 +藏品 +藏族 +藤原 +藻类 +蘑菇 +虎牙 +虐待 +虐狗 +虔诚 +虚伪 +虚假 +虚实 +虚寒 +虚幻 +虚弱 +虚心 +虚拟 +虚拟现实 +虚无 +虚构 +虚空 +虚荣 +虫子 +虫害 +虫草 +虹桥 +虽然 +虽说 +虾仁 +蚂蚁 +蚊子 +蚕丝 +蚕桑 +蚕豆 +蚝油 +蚩尤 +蛋液 +蛋清 +蛋白 +蛋白质 +蛋糕 +蛋黄 +蛤蟆 +蜀山 +蜂窝 +蜂蜜 +蜕变 +蜗牛 +蜘蛛 +蜜月 +蜜蜂 +蜡烛 +蜥蜴 +蜻蜓 +蜿蜒 +蝙蝠 +蝴蝶 +螃蟹 +融为 +融为一体 +融会 +融入 +融化 +融合 +融洽 +融融 +融资 +螳螂 +螺丝 +螺旋 +螺杆 +螺栓 +螺纹 +蟑螂 +蠕动 +血丝 +血压 +血型 +血小板 +血战 +血栓 +血气 +血汗 +血泪 +血流 +血浆 +血液 +血液循环 +血淋 +血清 +血症 +血管 +血糖 +血红 +血统 +血缘 +血肉 +血脂 +血脉 +血腥 +血色 +血虚 +血量 +行不 +行业 +行业协会 +行业标准 +行为 +行之 +行书 +行事 +行人 +行使 +行军 +行列 +行动 +行动计划 +行医 +行各 +行善 +行情 +行政 +行政区 +行政区域 +行政处罚 +行政村 +行政部门 +行文 +行星 +行李 +行李箱 +行榜 +行程 +行者 +行至 +行走 +行车 +行进 +行销 +行长 +行驶 +衍生 +衔接 +街上 +街区 +街头 +街机 +街边 +街道 +衙门 +衡水 +衡量 +衡阳 +衢州 +衣冠 +衣库 +衣服 +衣柜 +衣物 +衣着 +衣草 +衣裙 +衣裳 +衣食 +补习 +补偿 +补充 +补刀 +补助 +补救 +补气 +补水 +补益 +补短 +补给 +补考 +补肾 +补血 +补课 +补贴 +补钙 +表决 +表哥 +表妹 +表姐 +表层 +表彰 +表态 +表情 +表扬 +表明 +表格 +表演 +表现 +表现力 +表现形式 +表白 +表皮 +表示 +表示感谢 +表象 +表达 +表达能力 +表述 +表里 +表露 +表面 +衬托 +衬衣 +衬衫 +衰减 +衰弱 +衰竭 +衰老 +衰落 +衰退 +衷心 +袋子 +袖子 +袜子 +被判 +被动 +被告 +被告人 +被困 +被套 +被子 +被害 +被害人 +被执行人 +被捕 +被盗 +被称作 +被窝 +被誉为 +被迫 +袭击 +裁决 +裁判 +裁剪 +裁员 +裁定 +裁缝 +裂纹 +裂缝 +装作 +装修 +装入 +装卸 +装在 +装备 +装扮 +装有 +装机 +装满 +装潢 +装甲 +装盘 +装箱 +装置 +装订 +装载 +装逼 +装配 +装饰 +裙子 +裤子 +裸体 +裸裸 +裸露 +褐斑 +褐色 +褒贬 +褪色 +褶皱 +襄樊 +襄阳 +西亚 +西侧 +西兰花 +西北 +西北部 +西区 +西医 +西南 +西周 +西哥 +西域 +西夏 +西宁 +西安 +西安市 +西山 +西斯 +西方 +西方人 +西晋 +西服 +西林 +西汉 +西洋 +西游 +西游记 +西湖 +西班牙 +西瓜 +西甲 +西站 +西红 +西红柿 +西药 +西蒙 +西藏 +西装 +西路 +西边 +西邻 +西部 +西门 +西风 +西餐 +要不是 +要不然 +要么 +要命 +要塞 +要学 +要少 +要是 +要死 +要求 +要点 +要用 +要素 +要紧 +要花 +要说 +要钱 +要领 +覆去 +覆盖 +覆盖率 +见习 +见于 +见到 +见底 +见得 +见效 +见解 +见证 +见识 +见过 +见闻 +见面 +观世 +观世音 +观众 +观众们 +观光 +观察 +观影 +观念 +观想 +观摩 +观景 +观望 +观止 +观测 +观点 +观看 +观赏 +观音 +规划 +规则 +规定 +规律 +规格 +规模 +规模化 +规矩 +规程 +规章 +规章制度 +规范 +规范化 +规范性 +规避 +视为 +视力 +视听 +视察 +视线 +视网 +视网膜 +视而 +视觉 +视觉效果 +视角 +视野 +视镜 +视频 +觉察 +觉得 +觉悟 +觉醒 +角度 +角度看 +角膜 +角色 +角落 +角落里 +角质 +角质层 +角逐 +解体 +解决 +解决不了 +解决办法 +解决方案 +解决问题 +解冻 +解剖 +解套 +解密 +解开 +解惑 +解放 +解放军 +解救 +解散 +解构 +解析 +解毒 +解码 +解禁 +解答 +解脱 +解说 +解读 +解调 +解释 +解锁 +解除 +解题 +触动 +触及 +触发 +触手 +触控 +触摸 +触摸屏 +触犯 +触电 +触碰 +触觉 +言之 +言情 +言行 +言论 +言语 +言谈 +言辞 +詹姆斯 +誉为 +誓言 +警力 +警务 +警卫 +警告 +警备 +警官 +警察 +警惕 +警戒 +警报 +警方 +警示 +警觉 +警车 +譬如 +计价 +计入 +计划 +计划生育 +计数 +计时 +计生 +计算 +计算机 +计费 +计较 +计量 +订单 +订婚 +订票 +订立 +订货 +订购 +订阅 +认为 +认出 +认可 +认同 +认定 +认清 +认真 +认真学习 +认知 +认认 +认证 +认识 +认购 +认错 +讨厌 +讨好 +讨论 +让人 +让你 +让你在 +让步 +训练 +训练营 +议事 +议会 +议员 +议案 +议论 +议院 +议题 +讯息 +记下 +记事 +记住 +记录 +记录下来 +记得 +记忆 +记忆力 +记忆里 +记着 +记者 +记账 +记载 +记述 +讲义 +讲台 +讲堂 +讲学 +讲师 +讲座 +讲授 +讲故事 +讲求 +讲真 +讲究 +讲解 +讲讲 +讲话 +讲课 +讲过 +讲述 +讲道理 +许久 +许可 +许可证 +许多 +许愿 +许昌 +许许多多 +论事 +论坛 +论据 +论文 +论点 +论证 +论语 +论述 +讽刺 +设在 +设备 +设定 +设想 +设施 +设有 +设法 +设立 +设置 +设计 +设计师 +设计方案 +访谈 +访问 +诀窍 +证书 +证人 +证件 +证券 +证券交易所 +证券公司 +证券市场 +证实 +证据 +证明 +证监 +证监会 +评为 +评价 +评估 +评分 +评判 +评奖 +评委 +评定 +评审 +评析 +评比 +评测 +评级 +评议 +评论 +评选 +诅咒 +识别 +识字 +识破 +诈骗 +诉求 +诉讼 +诉讼法 +诉说 +诊所 +诊断 +诊治 +诊疗 +词典 +词曲 +词条 +词汇 +词组 +词语 +译文 +译者 +试一试 +试剂 +试卷 +试图 +试客 +试想 +试探 +试点 +试玩 +试用 +试用期 +试着 +试管 +试行 +试试 +试试看 +试过 +试问 +试题 +试驾 +试验 +试验区 +诗人 +诗作 +诗句 +诗意 +诗文 +诗歌 +诗经 +诗词 +诗集 +诚信 +诚实 +诚心 +诚恳 +诚意 +诚挚 +诚然 +话剧 +话语 +话语权 +话说 +话费 +话题 +诞生 +诞辰 +诟病 +诠释 +诡异 +诡计 +询问 +该书 +该剧 +该县 +该国 +该是 +该机 +该村 +该校 +该案 +该片 +该股 +该车 +该院 +该项 +详尽 +详情 +详细 +详见 +详解 +诧异 +语义 +语句 +语境 +语录 +语文 +语气 +语法 +语种 +语言 +语言表达 +语调 +语音 +误以为 +误会 +误区 +误导 +误差 +误解 +误认 +误诊 +诱人 +诱发 +诱因 +诱导 +诱惑 +说不 +说不出 +说不定 +说了 +说了算 +说些 +说出 +说到 +说到底 +说卡 +说句 +说唱 +说好 +说完 +说实话 +说明 +说明书 +说是 +说服 +说服力 +说来 +说法 +说白 +说白了 +说话 +说谎 +说起 +说道 +诵读 +请假 +请勿 +请客 +请教 +请来 +请求 +请示 +请问 +诸位 +诸佛 +诸侯 +诸君 +诸多 +诸如 +诸法 +诸神 +诸葛 +诺基 +诺基亚 +诺夫 +诺曼 +诺言 +诺贝尔 +诺贝尔奖 +读书 +读写 +读出 +读博 +读卡 +读取 +读完 +读懂 +读物 +读者 +读读 +读过 +读音 +诽谤 +课上 +课件 +课余 +课后 +课堂 +课堂教学 +课外 +课文 +课时 +课本 +课标 +课程 +课间 +课题 +谁家 +谁知 +谁谁 +调侃 +调入 +调养 +调制 +调剂 +调动 +调匀 +调味 +调味品 +调味料 +调和 +调度 +调戏 +调换 +调控 +调整 +调料 +调查 +调理 +调用 +调皮 +调研 +调色 +调节 +调解 +调试 +调速 +调配 +调频 +谅解 +谈不上 +谈判 +谈到 +谈及 +谈吐 +谈心 +谈恋 +谈恋爱 +谈论 +谈话 +谈谈 +谈起 +谈过 +谋划 +谋取 +谋杀 +谋求 +谋生 +谋略 +谎言 +谐振 +谐波 +谓之 +谓语 +谢绝 +谢谢 +谣言 +谦虚 +谦逊 +谨慎 +谨防 +谩骂 +谱写 +谴责 +谷子 +谷歌 +谷物 +豁然 +豁达 +豆制品 +豆子 +豆沙 +豆浆 +豆瓣 +豆类 +豆腐 +豆芽 +豆角 +豆豆 +豌豆 +象形 +象征 +象棋 +象牙 +豪华 +豪宅 +豪情 +豪放 +豪杰 +豪车 +豪迈 +豪门 +貌似 +貌美 +贝克 +贝壳 +贝尔 +贝拉 +贝斯 +贞观 +负债 +负担 +负有 +负离 +负罪 +负荷 +负责 +负责人 +负责同志 +负载 +负重 +负面 +负面影响 +贡献 +财产 +财会 +财力 +财务 +财务管理 +财团 +财富 +财报 +财政 +财政局 +财政收入 +财政部 +财星 +财源 +财物 +财神 +财税 +财经 +财经大学 +财运 +责令 +责任 +责任人 +责任制 +责任心 +责任感 +责备 +责怪 +败坏 +账单 +账号 +账户 +账款 +账面 +货品 +货币 +货币政策 +货架 +货款 +货源 +货物 +货车 +货运 +质保 +质地 +质感 +质押 +质朴 +质检 +质疑 +质量 +质问 +贩卖 +贪图 +贪婪 +贪官 +贪心 +贪污 +贫困 +贫困人口 +贫困地区 +贫困户 +贫困村 +贫富 +贫民 +贫穷 +贫苦 +贫血 +贬义 +贬低 +贬值 +购买 +购房 +购房者 +购物 +购物中心 +购票 +购置 +购车 +贮存 +贮藏 +贯彻 +贯彻执行 +贯彻落实 +贯穿 +贯通 +贴合 +贴图 +贴士 +贴心 +贴着 +贴纸 +贴身 +贴近 +贵人 +贵在 +贵妃 +贵宾 +贵州 +贵州省 +贵族 +贵重 +贵金属 +贵阳 +贷款 +贸市场 +贸易 +费力 +费劲 +费城 +费尔 +费率 +费用 +贿赂 +资产 +资产重组 +资产阶级 +资助 +资料 +资本 +资本主义 +资格 +资格证书 +资深 +资源 +资源优势 +资源共享 +资源管理 +资源配置 +资管 +资讯 +资质 +资费 +资金 +赋予 +赋能 +赌博 +赌场 +赎回 +赏析 +赏花 +赏识 +赐予 +赔付 +赔偿 +赔钱 +赖以 +赘肉 +赚取 +赚点 +赚钱 +赛事 +赛前 +赛区 +赛后 +赛场 +赛季 +赛车 +赛道 +赛马 +赞助 +赞叹 +赞同 +赞成 +赞扬 +赞美 +赞誉 +赞赏 +赠与 +赠送 +赡养 +赢利 +赢家 +赢得 +赣州 +赤壁 +赤字 +赤峰 +赤裸 +赤道 +赫尔 +赫赫 +走上 +走低 +走入 +走出 +走到 +走动 +走势 +走向 +走廊 +走开 +走弯 +走强 +走得 +走心 +走来 +走神 +走私 +走红 +走访 +走走 +走路 +走过 +走近 +走进 +走遍 +走高 +赴美 +赵国 +赶上 +赶到 +赶回 +赶往 +赶快 +赶来 +赶紧 +赶走 +赶赴 +起义 +起伏 +起初 +起到 +起动 +起因 +起头 +起始 +起家 +起居 +起床 +起来 +起步 +起源 +起源于 +起火 +起点 +起着 +起码 +起舞 +起草 +起诉 +起跑 +起跑线 +起身 +起重 +起锅 +起飞 +趁早 +趁机 +趁热 +趁着 +超乎 +超人 +超低 +超值 +超凡 +超出 +超前 +超声 +超声波 +超大 +超导 +超市 +超强 +超标 +超级 +超群 +超薄 +超越 +超跌 +超车 +超载 +超过 +超额 +超高 +越冬 +越剧 +越南 +越发 +越多 +越大 +越好 +越小 +越是 +越来 +越来越 +越来越少 +越狱 +越过 +越远 +越野 +越野车 +越长 +越高 +趋于 +趋势 +趋向 +趣事 +趣味 +趣味性 +足三里 +足以 +足协 +足够 +足球 +足足 +足轻重 +足迹 +足部 +趴在 +跃进 +跌倒 +跌停 +跌势 +跌宕 +跌幅 +跌破 +跌至 +跌落 +跑步 +跑跑 +跑车 +跑道 +距今 +距离 +跟上 +跟不上 +跟前 +跟着 +跟踪 +跟进 +跟随 +跟鞋 +跟风 +跨国 +跨境 +跨年 +跨度 +跨界 +跨越 +路上 +路人 +路况 +路口 +路基 +路子 +路径 +路易 +路段 +路灯 +路由 +路由器 +路程 +路线 +路边 +路过 +路透 +路途 +路里 +路面 +跳下 +跳出 +跳动 +跳楼 +跳槽 +跳水 +跳空 +跳绳 +跳舞 +跳跃 +跳过 +践行 +践踏 +跻身 +踊跃 +踏上 +踏入 +踏实 +踏板 +踏踏 +踏踏实实 +踢球 +踪影 +踪迹 +蹦出 +蹲下 +身上 +身世 +身临 +身为 +身亡 +身价 +身份 +身份证 +身体 +身体健康 +身体状况 +身体素质 +身后 +身在 +身处 +身子 +身形 +身影 +身心 +身心健康 +身手 +身旁 +身材 +身着 +身穿 +身躯 +身边 +身高 +躯体 +躯干 +躲过 +躲避 +车上 +车主 +车企 +车位 +车内 +车厢 +车友 +车型 +车头 +车子 +车尾 +车展 +车床 +车库 +车灯 +车牌 +车票 +车祸 +车程 +车窗 +车站 +车身 +车轮 +车载 +车辆 +车速 +车道 +车里 +车门 +车间 +车队 +车顶 +轨迹 +轨道 +轨道交通 +轩辕 +转世 +转业 +转为 +转会 +转入 +转到 +转动 +转化 +转化成 +转化率 +转发 +转变 +转向 +转型 +转基因 +转头 +转弯 +转战 +转折 +转折点 +转换 +转接 +转机 +转眼 +转移 +转而 +转自 +转让 +转账 +转身 +转转 +转载 +转过 +转运 +转速 +轮回 +轮子 +轮廓 +轮换 +轮机 +轮椅 +轮毂 +轮流 +轮胎 +软件 +软体 +软化 +软弱 +软文 +软硬 +软组织 +软软 +软骨 +轰动 +轰炸 +轰炸机 +轰轰烈烈 +轴承 +轴线 +轴距 +轶事 +轻便 +轻型 +轻声 +轻工 +轻巧 +轻度 +轻微 +轻快 +轻易 +轻松 +轻柔 +轻盈 +轻而易举 +轻薄 +轻视 +轻轨 +轻轻 +轻轻地 +轻轻松松 +轻重 +载体 +载入 +载荷 +轿车 +较为 +较大 +较差 +较弱 +较强 +较真 +较量 +较长 +辅以 +辅佐 +辅助 +辅导 +辅导员 +辅导班 +辅料 +辅食 +辆车 +辈子 +辉煌 +辍学 +辐射 +输入 +输出 +输卵管 +输掉 +输液 +输电 +输给 +输血 +输赢 +输送 +辖区 +辖市 +辗转 +辛亥 +辛劳 +辛勤 +辛苦 +辛辛 +辛辛苦苦 +辛辣 +辛酸 +辜负 +辞书 +辞去 +辞职 +辣椒 +辣鸡 +辨别 +辨析 +辨认 +辨证 +辨识 +辩护 +辩解 +辩论 +辩证 +辫子 +辱骂 +边上 +边境 +边形 +边框 +边界 +边疆 +边缘 +边远 +边防 +边际 +辽东 +辽宁 +辽宁省 +辽阔 +辽阳 +达克 +达到 +达尔 +达成 +达拉 +达摩 +达标 +迁入 +迁就 +迁徙 +迁移 +迂回 +迄今 +迄今为止 +迅猛 +迅速 +迅雷 +过不去 +过世 +过于 +过低 +过关 +过分 +过剩 +过半 +过去 +过后 +过境 +过多 +过夜 +过大 +过失 +过头 +过客 +过年 +过度 +过往 +过得 +过快 +过户 +过敏 +过敏性 +过日 +过日子 +过早 +过时 +过期 +过来 +过渡 +过滤 +过滤器 +过激 +过热 +过生日 +过瘾 +过硬 +过程 +过节 +过重 +过量 +过错 +过长 +过高 +迈克 +迈克尔 +迈出 +迈向 +迈进 +迎合 +迎宾 +迎战 +迎接 +迎春 +迎来 +迎面 +运会 +运作 +运动 +运动会 +运动员 +运动量 +运动鞋 +运势 +运气 +运河 +运球 +运用 +运筹 +运算 +运维 +运营 +运营商 +运行 +运费 +运转 +运载 +运输 +运送 +近三 +近两年 +近乎 +近些 +近些年 +近亲 +近代 +近似 +近几年 +近几年来 +近十 +近年 +近年来 +近日 +近期 +近来 +近百 +近视 +近距离 +返乡 +返利 +返回 +返校 +返程 +返还 +还会 +还原 +还好 +还应 +还是 +还有 +还款 +还给 +还要 +这一 +这下 +这个 +这么 +这么久 +这事 +这些 +这些年来 +这件 +这份 +这会 +这位 +这儿 +这句 +这台 +这名 +这回 +这场 +这块 +这天 +这套 +这家 +这幅 +这座 +这张 +这才 +这批 +这支 +这方面 +这时 +这时候 +这是 +这本 +这条 +这样 +这样一来 +这样的话 +这次 +这款 +这段 +这波 +这点 +这片 +这种 +这笔 +这篇 +这类 +这群 +这般 +这要 +这话 +这货 +这起 +这辆 +这辈子 +这边 +这道 +这部 +这部分 +这里 +这门 +这项 +这颗 +这首 +进一步 +进一步提高 +进京 +进修 +进入 +进军 +进出 +进出口 +进制 +进化 +进去 +进取 +进口 +进场 +进城 +进士 +进展 +进度 +进心 +进攻 +进来 +进步 +进气 +进水 +进球 +进程 +进而 +进行 +进补 +进货 +进退 +进门 +进阶 +进食 +进驻 +远东 +远去 +远古 +远处 +远大 +远征 +远方 +远景 +远比 +远洋 +远眺 +远离 +远程 +远见 +远超 +远近 +远远 +远远不够 +远高 +远高于 +违反 +违法 +违法犯罪 +违法行为 +违章 +违约 +违约金 +违纪 +违背 +违规 +违规行为 +连串 +连云 +连云港 +连同 +连夜 +连带 +连年 +连忙 +连接 +连日 +连服 +连环 +连用 +连线 +连结 +连续 +连绵 +连胜 +连衣裙 +连贯 +连载 +连连 +连通 +连锁 +迟到 +迟早 +迟缓 +迟迟 +迟钝 +迪丽 +迪亚 +迪士 +迪士尼 +迪奥 +迪拜 +迪斯 +迫不及待 +迫于 +迫使 +迫切 +迫害 +迭代 +迷人 +迷你 +迷信 +迷失 +迷宫 +迷恋 +迷惑 +迷离 +迷糊 +迷茫 +迷路 +迷迷 +迷雾 +迹象 +追上 +追击 +追到 +追加 +追回 +追寻 +追忆 +追捧 +追星 +追杀 +追求 +追涨 +追溯 +追究 +追赶 +追踪 +追逐 +追问 +追随 +退伍 +退休 +退出 +退化 +退回 +退学 +退市 +退役 +退换 +退款 +退步 +退税 +退缩 +退货 +退还 +送上 +送入 +送到 +送回 +送往 +送来 +送礼 +送给 +送货 +送达 +适中 +适于 +适合 +适宜 +适应 +适应性 +适度 +适当 +适时 +适用 +适用范围 +适配 +适量 +适龄 +逃亡 +逃出 +逃生 +逃离 +逃脱 +逃课 +逃走 +逃跑 +逃避 +逆势 +逆变 +逆向 +逆境 +逆天 +逆流 +逆行 +逆袭 +逆转 +选中 +选为 +选举 +选修 +选出 +选取 +选址 +选定 +选手 +选拔 +选择 +选择性 +选择题 +选材 +选民 +选用 +选秀 +选股 +选课 +选购 +选项 +选题 +逊色 +逍遥 +透光 +透出 +透彻 +透支 +透明 +透明度 +透析 +透气 +透视 +透过 +透镜 +透露 +逐一 +逐年 +逐步 +逐渐 +递交 +递减 +递增 +递归 +递给 +途中 +途径 +途经 +逗比 +通便 +通俗 +通信 +通关 +通勤 +通向 +通告 +通天 +通宵 +通州 +通常 +通往 +通报 +通气 +通水 +通用 +通电 +通畅 +通知 +通知书 +通缉 +通胀 +通行 +通行证 +通讯 +通讯员 +通话 +通货 +通货膨胀 +通路 +通车 +通达 +通过 +通透 +通通 +通道 +通风 +逛街 +逛逛 +逝世 +逝去 +速写 +速度 +速成 +速率 +造价 +造作 +造假 +造化 +造反 +造型 +造就 +造影 +造成 +造林 +造物 +造福 +造纸 +造船 +造血 +造诣 +造谣 +造车 +逮捕 +逻辑 +逼人 +逼格 +逼真 +逼近 +逼迫 +逾期 +逾越 +遇上 +遇事 +遇到 +遇到困难 +遇见 +遇过 +遇难 +遍及 +遍地 +遍布 +遏制 +道义 +道光 +道具 +道场 +道士 +道家 +道德 +道教 +道歉 +道理 +道菜 +道路 +道路交通 +遗产 +遗传 +遗体 +遗嘱 +遗址 +遗失 +遗弃 +遗忘 +遗憾 +遗漏 +遗物 +遗留 +遗迹 +遥感 +遥控 +遥控器 +遥远 +遥遥 +遭到 +遭受 +遭遇 +遮住 +遮挡 +遮掩 +遮盖 +遮阳 +遴选 +遵义 +遵守 +遵循 +遵照 +避免 +避孕 +避孕药 +避开 +避暑 +避险 +避难 +邀请 +邂逅 +邓丽 +邓小平 +邓超 +邢台 +那一刻 +那个 +那么 +那些 +那份 +那会 +那位 +那儿 +那句 +那句话 +那场 +那天 +那年 +那才 +那时 +那条 +那样 +那次 +那段 +那点 +那片 +那种 +那群 +那般 +那边 +那里 +那颗 +邪恶 +邪教 +邪淫 +邮件 +邮寄 +邮政 +邮电 +邮票 +邮箱 +邮编 +邮轮 +邯郸 +邵阳 +邻居 +邻近 +邻里 +郁郁 +郁金 +郁闷 +郊区 +郑州 +郑州市 +郑重 +部下 +部件 +部份 +部位 +部分 +部委 +部曲 +部级 +部署 +部落 +部长 +部门 +部队 +郴州 +都市 +都市报 +都是 +都督 +鄂尔多斯 +鄙夷 +鄙视 +酋长 +酌情 +配上 +配乐 +配以 +配件 +配偶 +配制 +配合 +配图 +配备 +配套 +配对 +配料 +配方 +配有 +配比 +配电 +配置 +配色 +配角 +配资 +配送 +配音 +配饰 +酒业 +酒后 +酒吧 +酒庄 +酒店 +酒杯 +酒类 +酒精 +酗酒 +酝酿 +酣畅 +酥脆 +酰胺 +酱汁 +酱油 +酱紫 +酵母 +酵素 +酷暑 +酷炫 +酷睿 +酸奶 +酸性 +酸爽 +酸甜 +酸痛 +酸盐 +酸碱 +酸菜 +酸辣 +酸酸 +酿成 +酿造 +酿酒 +醇厚 +醉酒 +醒悟 +醒来 +醒目 +醒过 +采光 +采取 +采取措施 +采摘 +采收 +采暖 +采样 +采用 +采矿 +采纳 +采自 +采访 +采购 +采集 +释义 +释放 +释迦 +释迦牟尼 +里亚 +里克 +里头 +里奥 +里斯 +里昂 +里程 +里程碑 +里边 +里面 +重中 +重中之重 +重于 +重仓 +重任 +重伤 +重修 +重力 +重叠 +重合 +重启 +重回 +重在 +重型 +重塑 +重复 +重大 +重大项目 +重工 +重庆 +重庆市 +重度 +重建 +重心 +重担 +重新 +重来 +重构 +重温 +重点 +重点项目 +重现 +重生 +重用 +重申 +重疾 +重病 +重症 +重磅 +重约 +重组 +重置 +重者 +重装 +重要 +重要性 +重要讲话 +重视 +重返 +重逢 +重重 +重量 +重量级 +重金属 +重阳 +野兽 +野外 +野心 +野战 +野猪 +野生 +野生动物 +野菜 +野蛮 +野马 +野鸡 +量产 +量化 +量子 +量身 +金丝 +金价 +金光 +金刚 +金华 +金叉 +金城 +金奖 +金子 +金字 +金字塔 +金属 +金山 +金币 +金庸 +金星 +金曲 +金沙 +金牌 +金牛 +金石 +金矿 +金秋 +金色 +金莲 +金融 +金融业 +金融危机 +金融市场 +金融服务 +金融机构 +金针 +金钟 +金钱 +金银 +金银花 +金陵 +金额 +金马 +金鱼 +金鸡 +金鹰 +金黄 +金黄色 +鉴于 +鉴别 +鉴定 +鉴赏 +针刺 +针对 +针对性 +针形 +针灸 +针织 +钉子 +钓鱼 +钓鱼岛 +钙质 +钞票 +钟声 +钟情 +钟楼 +钟爱 +钟表 +钢丝 +钢厂 +钢材 +钢板 +钢琴 +钢笔 +钢筋 +钢管 +钢结构 +钢铁 +钥匙 +钱包 +钱币 +钱财 +钻井 +钻孔 +钻石 +钻研 +钻进 +铁板 +铁矿石 +铁血 +铁路 +铁道 +铃声 +铃木 +铅笔 +铜陵 +铝合 +铝合金 +铭心 +铭文 +铭记 +银子 +银屑 +银屑病 +银川 +银幕 +银杏 +银河 +银牌 +银耳 +银联 +银色 +银行 +银行业 +银行卡 +银行贷款 +银魂 +铸就 +铸造 +铸铁 +铺垫 +铺设 +链接 +链条 +链路 +销售 +销售收入 +销售量 +销售额 +销毁 +销量 +锁定 +锂电 +锂电池 +锅中 +锅内 +锅底 +锅炉 +锅盖 +锅里 +锋利 +锋芒 +锐意 +错位 +错失 +错综 +错落 +错觉 +错误 +错过 +错题 +锤子 +锦上 +锦州 +锦标 +锦标赛 +锦江 +锦绣 +锦赛 +键盘 +锯齿 +锻炼 +锻炼身体 +锻造 +镀金 +镀铬 +镀锌 +镂空 +镇上 +镇压 +镇守 +镇定 +镇政府 +镇江 +镇痛 +镇长 +镇静 +镜像 +镜头 +镜子 +镜片 +镜面 +镶嵌 +长三角 +长久 +长假 +长出 +长发 +长城 +长处 +长大 +长子 +长安 +长官 +长宽 +长寿 +长年 +长度 +长廊 +长征 +长得 +长成 +长效 +长效机制 +长方形 +长时间 +长春 +长春市 +长期 +长期以来 +长款 +长此 +长毛 +长江 +长沙 +长沙市 +长河 +长治 +长生 +长痘 +长相 +长短 +长篇 +长篇小说 +长约 +长线 +长老 +长者 +长虹 +长袖 +长裙 +长裤 +长足 +长跑 +长辈 +长达 +长远 +长途 +长长 +长长的 +长颈 +门下 +门人 +门前 +门口 +门外 +门将 +门市 +门店 +门户 +门户网站 +门槛 +门票 +门禁 +门窗 +门类 +门诊 +门锁 +门面 +闪亮 +闪光 +闪光灯 +闪存 +闪烁 +闪现 +闪电 +闪耀 +闪闪 +闭上 +闭上眼睛 +闭关 +闭口 +闭合 +闭塞 +闭幕 +闭环 +闭路 +闭门 +问下 +问世 +问候 +问到 +问卷 +问及 +问答 +问责 +问过 +问道 +问问 +问题 +闯入 +闯关 +闯荡 +闲暇 +闲着 +闲置 +闲聊 +间接 +间断 +间歇 +间盘 +间谍 +间距 +间隔 +间隙 +闵行 +闷声 +闷热 +闹事 +闹钟 +闹闹 +闺女 +闺密 +闺蜜 +闻乐 +闻到 +闻名 +闽东 +闽南 +阀门 +阅兵 +阅卷 +阅历 +阅览 +阅读 +阈值 +阐明 +阐述 +阐释 +阑尾 +阔叶 +阜阳 +队伍 +队列 +队友 +队员 +队长 +防伪 +防卫 +防备 +防守 +防尘 +防弹 +防御 +防抖 +防护 +防控 +防晒 +防晒霜 +防止 +防水 +防汛 +防治 +防洪 +防滑 +防潮 +防火 +防爆 +防疫 +防病 +防癌 +防盗 +防空 +防线 +防腐 +防腐剂 +防范 +防锈 +防雷 +防震 +防风 +阳光 +阳区 +阳县 +阳台 +阳市 +阳性 +阳明 +阳春 +阳气 +阳痿 +阳线 +阳虚 +阴凉 +阴天 +阴影 +阴性 +阴暗 +阴极 +阴线 +阴茎 +阴虚 +阴谋 +阴道 +阴道炎 +阴阳 +阴险 +阴雨 +阴霾 +阵亡 +阵地 +阵容 +阵营 +阵阵 +阵雨 +阶层 +阶梯 +阶段 +阶段性 +阶级 +阻击 +阻力 +阻力位 +阻塞 +阻抗 +阻拦 +阻挡 +阻断 +阻止 +阻滞 +阻燃 +阻碍 +阿克 +阿卡 +阿姨 +阿富汗 +阿尔 +阿布 +阿弥陀佛 +阿德 +阿拉 +阿拉伯 +阿斯 +阿根廷 +阿森 +阿森纳 +阿联 +阿胶 +阿迪 +阿里 +阿里巴巴 +阿难 +陀罗 +陀螺 +附上 +附中 +附件 +附加 +附加值 +附图 +附子 +附属 +附带 +附录 +附着 +附赠 +附近 +陆军 +陆地 +陆游 +陆续 +陆陆 +陈列 +陈旧 +陈氏 +陈皮 +陈述 +陌生 +陌生人 +降临 +降价 +降低 +降低成本 +降准 +降到 +降压 +降噪 +降幅 +降息 +降水 +降水量 +降温 +降级 +降至 +降落 +降血 +降血压 +降解 +降雨 +限于 +限价 +限制 +限定 +限度 +限时 +限期 +限行 +限购 +限速 +限量 +限额 +陕北 +陕西 +陕西省 +院内 +院士 +院子 +院校 +院系 +院落 +院长 +除了 +除去 +除夕 +除外 +除夜 +除尘 +除恶 +除掉 +除此 +除此之外 +除湿 +除皱 +除草 +除非 +陨石 +险恶 +陪伴 +陪同 +陵园 +陶冶 +陶器 +陶瓷 +陶艺 +陶醉 +陷入 +陷害 +陷阱 +隆胸 +隆起 +隆重 +隆鼻 +隋唐 +隋朝 +随之 +随之而来 +随从 +随便 +随即 +随口 +随后 +随地 +随处 +随处可见 +随心 +随心所欲 +随性 +随意 +随手 +随时 +随时随地 +随机 +随着 +随笔 +随缘 +随访 +随身 +随身携带 +随风 +隐匿 +隐含 +隐居 +隐形 +隐忍 +隐性 +隐患 +隐瞒 +隐私 +隐秘 +隐约 +隐蔽 +隐藏 +隐身 +隐隐 +隔壁 +隔夜 +隔断 +隔热 +隔离 +隔绝 +隔膜 +隔音 +障碍 +隧道 +隶书 +隶属 +隶属于 +难事 +难于 +难以 +难免 +难免会 +难关 +难受 +难度 +难得 +难忘 +难怪 +难易 +难民 +难点 +难看 +难过 +难道 +难题 +雀斑 +雄伟 +雄厚 +雄安 +雄心 +雄性 +雅典 +雅思 +雅致 +雅虎 +集中 +集会 +集体 +集体经济 +集合 +集团 +集团公司 +集市 +集成 +集成电路 +集散 +集权 +集约 +集结 +集群 +集聚 +集装箱 +集训 +集贸 +集资 +集邮 +集锦 +雇主 +雇佣 +雇员 +雌性 +雌激 +雌激素 +雌雄 +雍正 +雕像 +雕刻 +雕塑 +雕琢 +雨伞 +雨天 +雨季 +雨林 +雨水 +雨量 +雨雪 +雪地 +雪山 +雪白 +雪花 +雪芹 +零下 +零件 +零售 +零售业 +零售商 +零散 +零星 +零点 +零碎 +零花 +零部 +零部件 +零零 +零食 +雷克萨斯 +雷军 +雷斯 +雷电 +雷神 +雷诺 +雷达 +雷锋 +雷雨 +雷霆 +需求 +需求量 +需要 +需谨慎 +震动 +震惊 +震慑 +震撼 +震荡 +霉素 +霉菌 +霍华 +霍夫 +霍尔 +霍金 +霓虹 +露出 +露天 +露水 +露面 +霸主 +霸占 +霸权 +霸气 +霸王 +霸道 +霹雳 +青云 +青光 +青团 +青少年 +青山 +青岛 +青岛市 +青年 +青年人 +青旅 +青春 +青春期 +青春痘 +青梅 +青椒 +青海 +青海省 +青涩 +青睐 +青石 +青色 +青花 +青菜 +青藏 +青蛙 +青铜 +青青 +青鸟 +青龙 +靓丽 +静下 +静下心来 +静坐 +静安 +静心 +静态 +静止 +静物 +静电 +静脉 +静脉曲张 +静谧 +静静 +静静地 +静静的 +静音 +非但 +非典 +非农 +非凡 +非常 +非常容易 +非常感谢 +非常明显 +非常简单 +非常适合 +非常重视 +非得 +非所 +非机动车 +非正 +非法 +非洲 +非要 +非诚 +非遗 +靠山 +靠拢 +靠着 +靠近 +面上 +面临 +面具 +面前 +面包 +面向 +面团 +面子 +面孔 +面容 +面对 +面对面 +面料 +面条 +面板 +面瘫 +面皮 +面目 +面相 +面积 +面粉 +面糊 +面纱 +面而来 +面膜 +面色 +面议 +面试 +面试官 +面貌 +面部 +面霜 +面食 +革命 +革新 +革质 +靶向 +鞋子 +鞋底 +鞍山 +鞠躬 +鞭炮 +韦博 +韦德 +韧带 +韧性 +韩剧 +韩国 +韩式 +韩文 +韩版 +韩语 +韭菜 +音乐 +音乐会 +音乐学院 +音乐家 +音乐节 +音像 +音响 +音效 +音符 +音箱 +音色 +音节 +音译 +音调 +音质 +音量 +音频 +韵味 +韵律 +韶关 +页岩 +页数 +页面 +顶上 +顶多 +顶尖 +顶层 +顶峰 +顶点 +顶礼 +顶端 +顶级 +顶部 +项目 +项目管理 +项目经理 +项链 +顺丰 +顺义 +顺从 +顺便 +顺其自然 +顺利 +顺利完成 +顺势 +顺序 +顺应 +顺德 +顺手 +顺时针 +顺水 +顺畅 +顺眼 +顺着 +顺风 +须知 +顽固 +顽强 +顾及 +顾名 +顾名思义 +顾客 +顾忌 +顾虑 +顾问 +顿悟 +顿时 +顿饭 +颁发 +颁奖 +颁奖典礼 +颁布 +预习 +预付 +预估 +预先 +预判 +预后 +预告 +预售 +预备 +预定 +预应 +预想 +预报 +预料 +预期 +预案 +预测 +预热 +预留 +预知 +预示 +预科 +预算 +预约 +预见 +预览 +预言 +预警 +预计 +预订 +预设 +预选 +预防 +领事 +领会 +领先 +领先地位 +领克 +领养 +领军 +领取 +领土 +领地 +领域 +领头 +领官 +领导 +领导人 +领导力 +领导小组 +领导班子 +领导者 +领带 +领悟 +领略 +领航 +领衔 +领袖 +领跑 +领队 +颇为 +颇具 +颇受 +颈椎 +颈椎病 +颈部 +频发 +频率 +频繁 +频谱 +频道 +频频 +颓废 +颗粒 +题为 +题主 +题型 +题外 +题库 +题材 +题目 +题词 +颜值 +颜料 +颜色 +额外 +额头 +额定 +额度 +颠倒 +颠簸 +颠覆 +颤抖 +风中 +风云 +风俗 +风光 +风力 +风华 +风口 +风向 +风吹 +风味 +风土 +风声 +风头 +风寒 +风尘 +风尚 +风度 +风情 +风扇 +风控 +风景 +风景区 +风景线 +风暴 +风机 +风格 +风气 +风水 +风波 +风流 +风湿 +风湿性 +风琴 +风电 +风筝 +风范 +风行 +风衣 +风貌 +风趣 +风速 +风采 +风险 +风险管理 +风雨 +风靡 +风顺 +飘过 +飘逸 +飘飘 +飘香 +飙升 +飞天 +飞奔 +飞快 +飞扬 +飞机 +飞来 +飞猛 +飞翔 +飞舞 +飞船 +飞行 +飞行员 +飞起 +飞跃 +飞车 +飞轮 +飞过 +飞速 +飞速发展 +飞鸟 +飞龙 +食品 +食品安全 +食品药品 +食堂 +食客 +食指 +食材 +食欲 +食欲不振 +食物 +食用 +食用油 +食疗 +食盐 +食神 +食管 +食肉 +食谱 +食道 +餐具 +餐厅 +餐桌 +餐桌上 +餐饮 +餐馆 +饥渴 +饥饿 +饭前 +饭后 +饭店 +饭桌 +饭碗 +饭菜 +饭馆 +饮品 +饮料 +饮水 +饮用 +饮用水 +饮茶 +饮酒 +饮食 +饮食习惯 +饰品 +饰演 +饱受 +饱含 +饱和 +饱满 +饲养 +饲料 +饺子 +饼干 +饿死 +馄饨 +馅料 +馅饼 +馆藏 +馈赠 +馒头 +首个 +首乌 +首付 +首位 +首先 +首创 +首发 +首家 +首富 +首尔 +首屈 +首届 +首席 +首批 +首播 +首日 +首映 +首次 +首款 +首歌 +首相 +首答 +首要 +首轮 +首选 +首都 +首页 +首领 +首饰 +香味 +香山 +香料 +香格里拉 +香槟 +香气 +香水 +香油 +香港 +香火 +香烟 +香甜 +香精 +香肠 +香草 +香菇 +香菜 +香葱 +香蕉 +香辣 +马丁 +马上 +马克 +马克思 +马克思主义 +马列 +马刺 +马力 +马哈 +马尔 +马尾 +马屁 +马德里 +马拉 +马拉松 +马斯 +马来 +马来西亚 +马桶 +马特 +马甲 +马自达 +马蓉 +马虎 +马赛 +马赫 +马路 +马路上 +马蹄 +马车 +马达 +马里 +马铃薯 +马龙 +驰名 +驰骋 +驱使 +驱动 +驱车 +驱逐 +驱逐舰 +驳回 +驻地 +驻扎 +驻村 +驻足 +驾乘 +驾校 +驾照 +驾车 +驾驭 +驾驶 +驾驶员 +驾驶证 +驿站 +骁龙 +骂人 +骄傲 +骆驼 +验光 +验收 +验方 +验证 +骑兵 +骑士 +骑着 +骑行 +骑车 +骑马 +骗人 +骗取 +骗子 +骗局 +骚年 +骚扰 +骨头 +骨子里 +骨干 +骨干教师 +骨折 +骨架 +骨灰 +骨盆 +骨科 +骨肉 +骨质 +骨骼 +骨髓 +骷髅 +高一 +高三 +高下 +高中 +高中生 +高丽 +高举 +高二 +高于 +高产 +高人 +高价 +高估 +高位 +高低 +高傲 +高僧 +高光 +高兴 +高冷 +高出 +高分 +高分子 +高危 +高压 +高原 +高发 +高品质 +高地 +高处 +高大 +高宗 +高官 +高尔 +高尔夫 +高尚 +高层 +高层次 +高山 +高峰 +高峰期 +高峰论坛 +高度 +高度重视 +高开 +高强度 +高性能 +高手 +高效 +高效率 +高数 +高新 +高新区 +高新技术 +高昂 +高明 +高架 +高标准 +高校 +高档 +高桥 +高楼 +高歌 +高水平 +高汤 +高涨 +高深 +高清 +高温 +高潮 +高点 +高烧 +高热 +高盛 +高祖 +高票 +高科技 +高空 +高端 +高等 +高等学校 +高等教育 +高等院校 +高管 +高粱 +高素质 +高级 +高考 +高耸 +高职 +高能 +高薪 +高蛋白 +高血压 +高血脂 +高调 +高质量 +高贵 +高超 +高跟 +高跟鞋 +高达 +高远 +高通 +高速 +高速公路 +高铁 +高阶 +高雄 +高雅 +高音 +高频 +高颜值 +高额 +高风险 +高高的 +高龄 +鬼吹 +鬼子 +鬼怪 +鬼神 +魅力 +魅族 +魏则 +魏国 +魏晋 +魔兽 +魔力 +魔女 +魔幻 +魔术 +魔法 +魔王 +魔界 +魔鬼 +鱼儿 +鱼头 +鱼尾 +鱼尾纹 +鱼类 +鱼肉 +鱼雷 +鱼龙 +鱿鱼 +鲁国 +鲁斯 +鲁班 +鲁能 +鲁迅 +鲍鱼 +鲜为 +鲜卑 +鲜奶 +鲜嫩 +鲜明 +鲜活 +鲜红 +鲜美 +鲜肉 +鲜艳 +鲜花 +鲜血 +鲤鱼 +鲨鱼 +鲫鱼 +鳄鱼 +鳞片 +鸟儿 +鸟巢 +鸟类 +鸡毛 +鸡汤 +鸡皮 +鸡精 +鸡翅 +鸡肉 +鸡胸 +鸡腿 +鸡蛋 +鸡血 +鸦片 +鸭子 +鸳鸯 +鸽子 +鹦鹉 +鹿晗 +麒麟 +麦克 +麦克风 +麦当劳 +麦田 +麦芽 +麦迪 +麻将 +麻木 +麻油 +麻烦 +麻疹 +麻痹 +麻花 +麻辣 +麻醉 +麻雀 +麻麻 +麻黄 +黄冈 +黄土 +黄埔 +黄山 +黄帝 +黄昏 +黄梅 +黄河 +黄油 +黄浦 +黄海 +黄牛 +黄瓜 +黄疸 +黄石 +黄色 +黄芪 +黄花 +黄蜂 +黄褐 +黄褐斑 +黄豆 +黄连 +黄酒 +黄酮 +黄金 +黄金周 +黄鹤 +黄龙 +黎明 +黏膜 +黑人 +黑名单 +黑夜 +黑头 +黑客 +黑帮 +黑幕 +黑恶势力 +黑暗 +黑木耳 +黑板 +黑洞 +黑点 +黑白 +黑眼圈 +黑社会 +黑色 +黑色素 +黑芝麻 +黑衣 +黑褐 +黑豆 +黑马 +黑龙江 +黑龙江省 +默化 +默契 +默认 +默默 +默默地 +默默无闻 +黛玉 +黯然 +鼎盛 +鼎鼎 +鼓动 +鼓励 +鼓吹 +鼓掌 +鼓楼 +鼓浪 +鼓舞 +鼓起 +鼠标 +鼻咽 +鼻塞 +鼻子 +鼻孔 +鼻梁 +鼻涕 +鼻炎 +鼻翼 +鼻腔 +齐全 +齐国 +齐心 +齐放 +齐白石 +齐聚 +齐鲁 +齐齐 +齿轮 +龙井 +龙凤 +龙华 +龙头 +龙头企业 +龙女 +龙山 +龙岗 +龙岩 +龙族 +龙江 +龙泉 +龙湖 +龙湾 +龙潭 +龙王 +龙珠 +龙眼 +龙舟 +龙虎 +龙虾 +龙门 +龙骨 + Azure + mobil + There + crops + solution +alonso +flat + sour +grimaced +rouch +sized +wingspan + irresponsible +Marginal + cooled +uga + willing +laura + Goldman + females + morale + posed + traction + Exxon +woolf +hindu + Rob +masovian +中文系 + combining + Minion + Princ +jected + Fant +igl +convicts + nuanced +華 +JSON + diseases +voc + Scandinav + TODAY + chancellor + Thomas + Chow + refurb + shed + Flo +Cop +anan + fif + Judy +clara + Dept +appa + flex + courtyard + subreddit + Tid + Wast +irements +doing +localhost + Bad +kelvin +thritis + genus +bailey + reconsider +一见钟 + Bone + Autism + rainbow +www +NSA + vulnerability +iac +yl + facade +livingstone + nitrogen + socialist + intellect +tools +rediscovered +trump + TAMADRA + conservatism + license + Coalition + post +zane + incom + Robertson + Francis + register + enormous + Moves +mortem + BBQ +odes + mistakenly +Unit + expansion +alks +lauderdale + Franz + ratios + snatch +urion + handgun + ke + cooks +审美观 +vent + Shiite + Pie + Composite +udic +Capt +raint +frontiers + redu + Holocaust + Seas + deeds + shortly + insured +potter +reading + Trend + subtitles +hazel + Ot + falsehood + Fist +gay +cyl + churn + Twilight + Chero +平顶山 +Tro + pools + welding + insults + centuries + Franken + laboratory +bulldogs + Wand +ERROR +donnell + Cort +bison +Mom +cision + AJ + USE +Protect +sufi + weren + MER +新闻报 + earthquake + hindered + Scholarship + forwarding + rating +cheat + afterlife + purpose + Handler + Million +celaena + Zy +ced +iev +Vice + Responsibility + Lex +Mexico + Scientific + CAP + deceit +inous +denton + program + reported +数十万 +ology +Sounds + Stur + Northern + able + semic + unfamiliar +reds + enforcing + Forest + L + pul + atom + chall + indications + fraternity + Collector + & +drug + Patterns + dram + blog + Zone +Apr +Organ + chose +olk + Opening +players + stove + booster +leo +plays +elines + Dart + TBD + PO +requency +MAX +steve +regnancy +bane +Microsoft +omon + wasting + mechanisms + async + alphabet + studio + sampling + Triangle + winners +Types +ayer + hallucinations + staff + agricultural + OW +爭 + cheated +取而代 + camouflage + Monteneg + compr + pastor + offset + adept + enhances +uscript + Angle +mclaughlin +telecommunication + gradient +athan +meta +bangalore + complement + Tul + cubes + Alec + WCS + Ukrainians + Pom + continually +quer +Lead + puck +omnibus + franc +head + Gau + erosion + recreation + cran + manpower + Moder +FOR + decipher + exodus + therap + reacting +marlins +lotus + Class +semiconductor + leveraging + manner +nel +IFE + Nixon +astonishment + faction +ophical +hopper + globally + With + scattering + Flood + understanding + devast + LinkedIn + Mahmoud +Insp + schizophren + abst +reb + hires + Cong +babu + Fer +urnal +Cole + tank + certification +capacity + interoper + http +acists +johansson +BA +alloc +statement + companies + GER + Amb +brance + freshly + Glory +lea +hendricks +philharmonic + Pear + Mobile +ortion + FIRST + Hoover +LI +cadillac + Kart + Caption +iever +pox + earthquakes + greeted +mario + CFR + tweaking +ollie +dana + Gaal + quarterbacks +Ward + Desktop + doorway +lik + Ahead + designs +YN + summer + ever + cures + embargo +rador +aterasu +Common +ochet +malabar +frazier + cloves + elig +STAT + Keith +attribute +PLC + persist +orian +rael +upp + soils + iPads +Jack + Campbell + Solar + devices + hint + utilizing + crispy + rac + loan +fuel +Cool + accumulated + Lua +Lastly +berlin + waited + equival + fleeting + heterogeneity + exiled +ARY + clothes +zz + lively + almond +arf + nause + Metroid + lawmaker + AN + Martial + Bab + Kabul +fern +leon +abel +fighter + roast + fingertips + arbitration + Apostles + roasted +ki +brace +Instruct + Penguins + good + Means + Swansea + Fit + Hour + hunted + 285 + Rebirth +Yet + sincerely + warped +enza + homeowners +ppel +uterte + Spells + Brav +aday +patel +Minimum + repe + undefined + Atl + authority +spokane + declines + Conc +源源不 + Mail + Haas +redesignated +souza + Polly + continuation +romancer + Prix + impetus + bombardment +MIUI +claus +called +Ber + durability + Militia + incoming +第三十四 + employ +Normal +shapiro + Schwarzenegger + gasped + justifying +similar + GCC + rhetoric +Attack + Modes + Goes +ergic + deems +opot + pal +aza + Raspberry +Princ +ESPN + coincides +Armor +OOD +inyl + mentioning + bullying + dates +ACY +Mil + disciplinary + harms + [- + Options +quay + manufactured +appy +jaenelle + reservation +communications + senator +side +template +eureka +ACTED +sun + molded +send +期徒 +ebert + outfit +irgin + setbacks + beforehand +sticks + synd + Spider +rament + souls + trumpet +xa +nico +rench +otrop +Cod +ingram + Guide +Pick +painters +efer + agrees + bash + suppliers +orphanage + Hanna + hunter + trillion +chilean + Config +apses +dona +rich +shaped +arkin + outdoor +alogy + nightclub + surfaces +ducked +Bind + Mitch + clad + Accountability + video +decker +bla +Tan +finder +gency + meshes + Crypt +estadio + And + Buddhism + Straw +Name + lives + s + invaluable +Beast +remixed + writer +airplay +gener +NER +AK +anners +checked +neville + erase + economics + Growth + wonderfully +rail + Osama +barrow + Cab + Concepts + Donovan +ission + Homs +biographies + Huang + natives + Tele +burton + fourteen + Clim +bosnia +mcc + mimic + Erik + ADA +ActionCode + appealed + Flu +Iran +roads + Hos +bles + inher +wild + Applications + link + fantasies + brilliance + Hust + recurring +script + member + concent + reperto +aroo + womb + Shea +kin + inadequate +appropri +ablished + Champions + Cloud + wicked + Fore +degree +unker + doesn +hymn +ulin + Lamb + defunct + Enable + emerges + altar + assassinated + BUS + isot + conservative + UK + Zoom + 360 + 191 +pherd + incompetence + Koz +]] + murky + thinly +国际金融 + trash +區 +ThumbnailImage + warranties + develops + outl +icating + manufacturers + wrestling + railroad +District +gram + plead + Rodrig +aji + silently +wan +karlsruhe + Arrest + PTS + afraid + voluntarily +Natural + Maj +battalions +eor +應 + MPEG + creeping + Belfast +YR + somet + tired + identity + Reward + Kirby +ilton +fourier +area + reiterated +archment +linguistics + Riding +princeton + stri +llor +ANCE + LGBTQ + unfortunately + mortals +rpg + Egyptian + using +Module +hold + Platinum +aire +bremen +timor + pedals + torch + perplex +roying + Bravo +holloway +alta + activist +winnie + unanim + Muk + CHAR + Sonny + monster + professionalism +arya + stood +Ident + Fle + quo +hur +中下游 +zone + Mining + polluted + Paula + extremes +ascot +igating +baike +halle + expose +vaudeville + FANTASY + fem + Francisco + censored +tatar + dramatically + pants +raising +chal +writhing + We +collier +itialized + Left + inherited + bowl +gillespie + obnoxious + Savannah +girlfriend +rehearsals +client +iowa + worn + Motorola + transit + traps +Sham + [[ +xin + pretended + Already + hung + franchise +VGA + Dogs +swirled + chop + disposable + value +oug + LEG +pause +admiral + neg + Design + high + Izan + scorer + lipstick + shaman +before + Heck +remixes + reson + veteran + mega +Alexander +TIT + Cass +aan +ONE +Current +Other + 28 + Roh + not + Examiner + Crimson +porting + Sonia + pleaded + explosive +ills + Cook + awaited + symm + Licensed + pesticides +Sea + Chronicle +Fort + proud +norris + Grape +estine + buf +嚴 + shy +jose + clam +apse + parody + thank + Starr +prima + released + seaw + smir +anything + Fare +ilan + Alert + Register +ona +Were + pool + hor + stranger +alle + reflex + Scotland +pps + focal + loc + Disk + gunshot + Johnston +Technology + puts +kate + axe +cochrane + scratching + Ack +uish + Bry + aerobic +JD +farm +xbox + Trade + Dark + Bundes + John +luence + Sche + Ober + pept + imperialist + citations +Mand + ? + Kevin + Aside + retarded +stan + ask +short + respective + clout + Toro + settling +guerrero +middlesbrough + Mark +duane + boasts +swans + cruelty + bite +不匿 +zion + subt + PLEASE +GHz +htar +reality + bolted +combat + McH +ene + assess +depend +johanna +laughter + Pump +gestured + Cornell + torso + mirror +botanist + Boris + Cheese + routers + ardent +guiana +Ever + unemployment + 1923 + maths + olive + Kaf +Arcade + Loud + LOL + accomplishment + Ethan + distributions +Deploy +arroyo +hartman + achie +istrates + fearless + converts +amon + Equ +uff + apple +setting +睡不 +Detailed +inav + dopamine +Director + TER +shea + Witcher + segregation + deductions + casino + probation +lists +killing + doctrine +疒 + defeat + Provides +priced +ceiver + Thankfully +Regular +fig + Card + certify + Pf + Tacoma +Revolution + Trials +cfg +Speaking + Mort + Burgess + Adjust +一技之 +research + Rolls + enc + prepare + marine +Meet +mani + RTX + exclaimed + legislators + vari + Proud +umerous + interven +kolkata + aid +nellie + penis +joice + baseline +RT + disadvantages + satirical + Eco +reporting + salient +erness + glitter +flaps + Oculus +queensland +essential +ller +brahms +portation +CON + Caller + artists +unitary + utilizes +生理学 + Hollande +cruz + Levels + Bat + jugg +churchyard + Warn + Emanuel +emetery +Ok + perhaps + Ident + variety + navig +第三十七 + os + Severus +olo +练习曲 +carpets + inexper +rae +帶 + innovations + prophet + Artifact + favoring + scanning + extracts + Build + itiner +arantine + Vince + enters +hibit +America + 261 +nosis + pelvic + Hearth +Device + published +lest + dad +emigration +verve + exams +austrians +sinclair +ling + Hawth +royals + Chevy +ueless +isaiah +becca + Sands + opium +alvaro + Branch +erential + differentiation + staple +ility +asin + Bachelor + folk + Revenge + inherent +digy +Items + astonished +worth + thirty + 318 + Celebration + say +hesis + Fundamental +pmid + collaborating + Marty + finite + offending + commencement +DVI + dynamic +Around +wrinkled +irt + Com + mystical + smuggled + Leaves +LA + forceful +aghan + Shane + smashing +Northern + Cod + staffed +democratic +atonin + Jackets + Cornel + Pigs + executives + ton + arrests + Temple + stabbed + adequ + Returns + 308 + SPACE + disemb +rim + backed + capit + generously + pestic +ratulations +tall +poli +kendall + Articles + Goddard + gent +atis + Vet + Centers +EST + Edgar + left +mont + convincing + Gro + Reporter + joins +HOW +arding +porto +Nature + investor +awaru +creat + Functions +教学设 + Reps +darby +nant + Correspond +zo + digestion + Holdings + pint + Panel + crop + burgers +ventions + excruciating + shelf + partial + restaurant +oS +arterial +boomed + de + Vista + Hammer +WS + spiritual + subordinates + Sharma +CHAR + Squadron +aval +atu +oven +breeding +dowager +\"> + ambiguity + undermining + guiding +aegean + reckon +staff +bishopric + tsp + attainment +Statistics +igans + af +auditioned +Process + dudes +nicole +myers + exceeds + sol +kson +ograms + dental +cession + Sut + overload +eme + Santa + falsely + comprise +ocations + windows +paler + Real +escal +speed + frog + organise +ordes +judas +avalon + known +tten + Rhod +yt + comrade + scarce + surpassed + costs +ciplinary + Temporary + approaches + reserves + Ferguson +recited + proportional + CT + collabor + athleticism + Profile +bable + unrelated + Puppet + outright + gluten + deny +dyer + calibrated +respect + foreground +param + hijacked +Brand + 385 + Swedish + humiliated + sulfur + Introdu + Tempest + Maze + slump + Important + tempting +Yep + lots +confederates +ool +filly +sox + 147 + magn + Bru + incarceration + Quadro + pushing +mn + Commentary + summoning + Geneva +rowe + Beat + Micro +observatory + lux + therapeutic + brave +prefix +bazaar +steel +okingly + DL + Fern +ilers + MOD + chap + gazed + MySQL +At +houses + after + UCS +ogeneity + consult + specifications + schemes + (). + Yard +daily + Chinatown + 193 +phan + Tasmania + subversive +grid +\' + recognizes +esses + Kits +steele + securely + circus + disruptive +xes + Sky + Kentucky + system + warships + fiercely +bavaria +ornia +[' + Ther + fiscal + Fisher + JFK + Cavaliers + cho + Kosovo + lemon +gra +Light +itic +manent +mop + catcher +hn + Help + noble + avocado + depths + Leave + striker +nunez +evi + discretionary + logistical + chapel + migrated + erased + lift + Drops +稱 + Do + Gree +agustin + geometric + hobby + VAL + ranges + warmth + characterization + bra + precious +good +via +alsa + Domin + Pri +Loader + mmol + bullpen + deduction + Minimum + moves + Phase + CES + chim +ategy + Requ +微生物学 +Draw + broadcasters +facebook +banco +southampton + malf + exciting +ithmetic + Quiet + Clinic +fauna + musicians +戰 + Navigation + graphical +iliation +reek + Gaw + simultaneous +Es +enced +agically +aturdays + Pak + construed +open +reverend + veterin + Upgrade +ino + glob +deafening +mes +SHARE + travelled + speak +poses + Lauren + artifact + 698 + Gr +Aug +enstein +oyle + Chairman + sat + pessimistic + filib + Taiwanese + LAT +lehman + conductor + doubtless + Barack +pass + Children +Clinton + harass +riel + Smithsonian + liberals +ooming +never + Sik + Elim + sacrifice + posters + Gandhi + Sturgeon + piss +clasped + Jab +francesca + Lod +isp +提不起 +adrian + Winc + confessions +ETHOD +colm + painstaking +darren +harrow + architectural + Thought +ogenesis + Tehran +書 +貴 +阿姆斯特 +lished +weaver + DLC + penetrated + Clair +Depths +elsa + remembrance + wakes + Mermaid + teacher +ugg + clones +ctica + interview +auditorium +sparkle + cables +Enabled +chip + Arcade +]+ + guru + diapers + Sims +BBS + vigorously + sufficiently + trimmed + Stew + prehistoric +oubtedly + carts +reed + 287 + Browse +许嵩 +pilgrim + complainant + swapping + screens + ours +asher + suitable + stool + sparking +ENSE + shaved + WWF +puters + experienced +Flying + Bie +hematically +germanic +TT + manners + longing + dealings + added + Forced + specific + pressured + Hearts + Shift + proclaim + hip + macro + Defense +pga +henri + senses +riage + exhaust +wick + Extensions + Astros + SSH +dog + induces +korean + said + GPIO +meritorious + additive +agan +Find +Nitrome +vae + fingerprints + absorbing +hiv + controversy +供不应 +第四十一 + 1924 + focus + Xbox + Rockets +itud +ions +該 +classed + required + Rend +float +latin + Highlands +matthews + intern + ft +_____ + put + threads + DHCP + pact + beating + Smoking +ghost +anos + Array +daphne + evapor +regency + Ik +master + Zhu + merciless + Contrast +sienna +hades + tackled +uria + moved + Runes + Elliot +invest +linked + Pillar + Dav +omer + Nigel + bark +esa +gregation + rive +abytes + Surely +aqueduct + disl +exasperated + boss + wildfire + INTO + maneuver +maia + tag + simulated +gina +務 + suffice + Spears +sharif + 183 + parole + 1988 +Shock + autonom +IRE +inker + Drunk +cairo + psychopath + Duncan + rav + ovarian +Department + widespread + Mile + FG +Ready +Async + protocols + necessities + islands + operates + eth +stylized + fingert +minas + eff + worsening +goth + accumulate +osal + existence +lis + 276 + :: + landscapes +XM + payday + calendars + THREE +alted + outset +tatum +ensions +represent + cardio + coils + Buenos + cholesterol +udd +arnold + facets +ork +nobel +Experts + planting +morgan + Shows + conditional +tina +eleanor + mailbox +martinez +oubted + staffing + readily + Curve + essentially + DEL +ardon + 248 + functioning + dealership +陳 +mendoza +first + executable +concacaf + BART + dying +dauphin +決 + digitally +zl +florence + regulates +rones +khz +carry +chall +absor + are + Love + sorrow +ocratic + relate +uda +sociological +castillo + assistants +rel + Funny + Shawn + Brit + Petersburg + reply +maddox +arium +hirsch + 38 + Tanner +former + creativity + rings + drawn +vidal + 499 +rade +accompaniment +harry + firearms + Sark +rations + emblem +因為 +stirling + planner + sacked +aleb + NEO +dies + swath +reements + poly + Vall +speedway + brood +cellence + independent +liner + Logic + Scouting + likelihood + consultations + spor + treason +olated +ashley + refining + signifies + immigrant +anka + Syrians + Minority +fernandez + Ending + unveil + searching +fts + CC + controversial + Wiggins +sky + severity + rinse +_> + Sea +andro + acclaim + mashed + views + recal + splash + artificially + curtail +lincolnshire + legality + subjective +bps + donate + certainly +umenthal +pam + avoid +chelsea +istically + perpetrated +othing + clown + trustee + programming + Murphy +ney + Decay +benji + cum +etz + 1927 + hereditary + Jackson + Gene + spans + fortified + redund + united + Ars + scams +理所应 + packets + mi + powers +cream + deferred + Armory + piles +oswald +avis + Irvine + fulfill +average +arro +董卓 + accurately +shrugs + Dub +greco + Hans +hetamine + namesake + glimpse +Synopsis +ICH + uninstall +figure + Household +Lo + NEED + tuple + ethanol + election + authors +andreas + lighter +bors + head + Ind + Campaign +盈盈 + }, +CE +amped + Sicily + crime + boxing +Step +Off + army + Rub +Oct + AUTHOR + leaned + advers +uart +arrett + generating +ones +abuse +nbl + swings + resumes +Built +OUGH + Border +ioc +ggies + save + void + cyl + ingredients +qus + vendors +renches + Duchess +werewolf + Similarly + startling + track + annihilation +ventral + licensee + ambig + Elon + anew + Got +neil +anz +cific + faint + Moto + CPC +External + HTTP + stressful +NR + forge +ustomed +werewolves + Darth + Optional + nutrit + popular + refresh + 265 +ignment + analogue + SCH + negotiators +Philipp +mas + labels +ritch + championships +aska + reg + transpired +accompanied +istance + longer + Daredevil +picking + reciproc + 279 + wipes + greedy +\) + redef + Optimus +ritten + favor +grassland + Armed + Lah + Nasa + vibrations + Service + ital +bis + Intelligent + compassionate +panel +kilometre + Emil +shit + outburst +naacp +armistice +asturias + nerves +ames + professional + interception + randomly + burn +iannopoulos + gamers +EX +cat + cess +oran +Chicago + explode +horne +Panel +evidence +Freedom +troy +elgin + wishes + sever + rou + offences + rejo + tails + Catalan + cha +isition + coached + juveniles +agree + Chevron +Tips +obi +Adam + satisfaction + New +calcutta + Pru +alys + scrut +aram + Ab + plotted + reversal + chooses +barak +stroke + subs +hussein + productive + Gibson + Superintendent + metropolitan +highlands + Pinterest +stuttgart + org +trainee + shouting +ateral + obligated + Develop + injure +ocalyptic +amazement + tricky +Sign +founded + Mc + Ob +fritz +mohammed +renault +ecake + Wireless + Latino + tearing +siren + Lilith +quote + agony + workings + proxies + Loop +民政部 +refit +foo + interrupts + incarnation +barney + parameters + repatri +trial + beautifully +beverley +banged +Object + voters +ST + facebook +tennessee +GD + filed + punching + cracked + Marx + vitro + Gallery +verse +goo + Millennials + Fallen + ML + Jakarta +twitching +reasonable + north +Success + prominence +borough + handwritten + Seb +synonym +fx + Prediction + Lap + Feeling + ot +always +ume + diluted +sinai +newcastle +dt + lapt + Approximately +come + professionally + Greece +storm + drains +pausing + anticipating +dung + visibly + 0 + Does + international + pont +melanie + tyr + obtaining +skinned +wordpress + Gonz + coat +ounge +acles + Jess +一百多 + Cyborg + soaking +change +中央党 + Mn + Bush +bul + blown + net + negligible + misinformation +quickShip + hopeful + earn + nost + CBD +Lt + Lions + preferably +webster + Business +Ra + extradition + feathers + Rangers +education + tutorials +ttes + ES + ID +finch +中国美术 + princ + Conver +illet +Er +izophren + Channel + rip + Simpson + Emergency + Twelve +gdansk + Sharks + quizz +revue + disguised +ardo + Earth +urther +百八十 + Mang + competing + ok +BOOK + Coliseum +Sat +emin +mary +andum + equivalent +edded +newark + Bash + Trin + hypot +tackle + try + schools +otton + deep +October + halls + Carolyn + boxed +urse + keyboards +cochran +albanian + meaningful +福克斯 + JUST + farm + ABV +tk + finances + subsection + Offensive +consuming +beat + Toronto + LOW +freighter + Fil + aph +AUT + scholarship +atra + Patients +lear + Flor + darker + Sept +shawn + Olympic +unicode + Graphics +scranton + Percent +gazette +awk + Ung + PDT +raid + regulate + Oman +accept +Mal + tim + + Egypt +kade + registrations +ackle + delayed + Copenhagen + Kris + shaving + Pluto +]" +EU + Kus + creamy + detached + biomark + Nutr + flowers + div +imeter + angular + Rox + Christine + undercover + Provision +moreno + reward +growing +LOC + ranking + attitude +Blake +schild +ishes + rejected + extrad + expressly +paisley +Song + tattoos +uds +esthesia + Cinem + false +heartbreak + legislator + municipal + Benef +Ven +theme +materialized +lucan + insisting +intention + flow +Du + reproduce + Teams +aran + smiles + Celeb +OPPO + BBC + manipulative +兼容性 + iter + damning +gregorian +Better + Archdemon + vetted + Gamergate + therapist +Upon +bianca + Capitals + fairly + shut +Gas + painter + undertook +Fake +Nazi + unexpectedly + apologies +langdon +barbara +visors + IPM + remote +sil +Advertisement +ewing + menstrual + accelerator + immature + cows + Abstract +olympics +八达岭 + urgent +anguage +liga + toug +rz +taylor + Alter + rooted +YY +\\ +Quora +hya +EB +ually + Control + directors +igr + cultivating + Investigation + Reporting + archae + fucked +kathmandu +NOT +ventures +chains + Refer + ironically +Create + fanbase + quantities + candle + nursing + Barg +truss + Sweden + Endurance +fortable +advertisement + restricts + regiment + al + Glob +AY + tremendously + Klu + fal +allah +slovenian + complexes + recipients + Pinball +Buffer + hurt +bows +Redd +三四十 +tabitha +Details + GO +hounds + repealing + disciple + insurance +straits + assisting + holds +raised + Rarity + pant +sound + disenfranch +ND +juliette + confiscated + graffiti + demonstrators + Neutral + Stru + Crimea + then + 1909 + orbit +afi +agic + Centre + Conv +gabrielle + synchronization + tighten +War + undermines + appetite +apa +myles + Beauty +trunks + paper +CH +marcus + turret + '/ + UFOs +fore + doesnt + treats +gloucestershire + Ply +motorway + accessed + /** + dunk + Oriental + prevailed +Atlanta +owan + Visa +許 +Michael + incurred + TW + Grade +abraham + cocoa +Error + popping +地理学 + sectors +EXT +asing +restarted + Caval +Example + freezer + suggestions + Scan +Includes + discriminated + Noel + mundane +歇斯底 + Manziel +kochi + Taiwan + holding + SATA +ilant + Engine + figuring + Aires + valued + continual +raiders +motorsports + awarding + TV +papyrus +ochemical + physicists +Semitic + launched + enhancements + Kuro + Bye + ACLU + strugg + D + reversing + impacting +perate + Catalog + (+ +galveston + manufacturing +Lew + sluggish + Removal + Elvis +Learning +wav + coupons + Blazers +scents + $ + informed +eus + Hann +ohio + twenties + stigma +genre +latvian + combine + Warning + hands +Actor +libertadores + protagonists +ON +crew +變 + Specific + corrupt + standards + bends + counter +faa + ear +ulkan +Remove +IRT + wors +izes +Wonder +edition + estimate +native + Spirits + Epidem + lust + ultraviolet +ihad + Tav + Prosper + Cambodia + Candidate +meter + knees +irvine +ober + disarm + dock + MM + varied + parcels +Scale +琊 + lun + occurrences + Water + appointment +enses +adia + Patriarch +Customer + Genetics + QC + Settings + comparatively + NTS + eld +reared + indulge +udeau +howe + Fra + Dozens + crowded + narrow +secure + herself +elijah +marissa +ARDS +Wide +niagara + usable + describe + activism +史以 + personal +統 + implying +eder +editor + munitions + Areas +jace + bush +Select + drawback + Illum +swaying + fountain +Query + blend + desire + Expansion + probability + shack +hurling + An +imating + co + Inc + exhausting +italics +第四十七 + baptized + expression +producing + philosophical + Pod +filmfare + Conservative + seas +trident + persuade + residency + Tang + bask + hospital +inance + Released +stals + Storage +sarawak + treating +visible +estonian + mayhem +irtual +asts +alla +pascal + desolate + Connection +push +Clark + predominantly +fort + narrowly + kWh + politely + Turtles + hooks + awkward +earle +cience + Dell +olin +bars + absence + throws + em +full + mutations +manhattan +一百九十 +isen + amy +DF +utsu +Fra + BCE + autonomous + diligent + Ric + Ashton + homeowner +eagles + Hollow + processing +ked +ORD + Compact + telephone +OLOG + memorial +split + pokemon + MSM +rax +DN + ultra +blooded + flashbacks + Clinton + Feature +coni +silas +confederate + Fried + Coil + unt +dong + EVs + Worm + Driving + rap + capture +othes + battered +ificent + Farm +server +Six +vg + condensed + lumber +ylum + rightful +mount + hath + lacked + Agriculture + z + FN + Newsp + Jedi +pantry + conclusions + grapp + fluorescent + Milk +waters + Cabrera + Welfare +stones +rector +oleg +hesitantly +cyclones +NY +imeo + unthinkable + lived + reliant +mind +osterone +海峡两岸 +Dash +感覺 + aforementioned +johnny +going +krit + communion + committees +judaism + relieve +ripley +pers +cambrian + gay +ultane +ueller +oggles + superior + Balt + income + Pres +itta +itone +waterloo +basilica +highest +Disc +Tele +ernel +onies + jihadist + psi + tac + filler + disks + habitual + riddled + catalogue + Breach + Ran + occupancy + Serv +仙人掌 + Bomb + begging + potentially + gestation + FACE +find + Ut + deadline + coaches +anguard +keep + unnoticed + cried + LG + everlasting + Fidel + eco + taco + Diary +joanne + np +chandler +document + gener +stick + BR +dalton + doubling +普鲁士 + relax +christian +Vaults + monog + Patel + Bild + underway + Levine + Greenland + textual + DEFENSE + pulling +exemplified + tow +ATOR + Tate + Palmer + rapid +urable +刘翔 + Selected + sem + resistor + Drum +nets + danger +fp +atta + AMERICA +whirled + mediated + soaring + clot + Knowing + Touch + Updates + Federal + Forget + Nag +rav + Viv +tommy + complaining + Cel + locked + Wings + binary +hawker + Cameron +salamanca +flinders + arming + 61 + db +確 +jockey + terrible + route + investigative + sales +smyth +montrose +cigarettes + Championships + antagon + puzzle +estern + dancers + compliance + exactly +umblr +Tex + devoid +pointe + avail + Seasons + contradictory + Myers +gow + sci + splend +OFF + Eve +vinnie +carver +thood + subsid +Too + rebirth + booths + unmarked +过氧化 + Op +place + sw + LT +aren + coupled +合格证 + posting + terrified +omm +fastened +richmond + chapter + wish + worms + Lane + leveled + arisen + statistic + chiefs + proponents +vh1 +Commission + cel + Berry + bright +marx +ocard +dem + Prosecut + Huffington + Seym + denies + township +ids + mars +hitchcock +zig +Yo +Psych +ZI +igger + Cure + pictured + random +std +Writer +Person + Products + sewing +uki + Korean + Reply + drought + Tracks + Costs +announced + buys +cupboard + sleeping + Success + 36 +intelligence +ally + gra +larry +Mech + stip + flexible + undergrad + 768 + Bu + salts + idiot +usc + Jenny +Creat +ographically +uebl + Dempsey +謝 + transformed +refurbished + reference +IPS + Puzzle +gustav + Dele + (= +lor +pub +終 + Mysterious + average + vib + Literary +changing + sinister + Tall +jimi +mcgregor + savior + Sparks +binary + halt + unsuccessfully + alternatives + RM + traditions + Alberto +assemb +Ton +Ins +Prep +hess + truthful + Oval + grasped + fellows +roup +oried + Launcher +buy + salon + 1926 +notre +octopus + Indonesia + responding +buf + Wein +message +iscons + GNOME +IED + promoters + regulars + Ely + delusions +archive + Stud + innovate +興 + FW + stripes + insanity + fared + themes + activists + lib + rib +escorts +kat + programmed + free + Chimera + ones + Chambers +ators +apostolic + interruption + Deadpool + bully +spartans + doping + Anthem +alec + Churches +loretta + abandoning + endlessly +Sarah + Kenny + dolls +Count + Anthropology +分析仪 + Morales + separated + seminars + Zimbabwe +ription + girlfriends +nh +YE +LM + Dum +Math + curls +edmonton +mused + bending + tweet + municipalities +Magikarp + wouldn + clients + trees + COMMUN + Turbo + wheel +barron + inquiry +cumberland + epile + databases +pathic +pha + mammal +maximus +ether + succ +write + capability +cakes + denim +Paris +amples +bulgaria + Indeed + 2400 + Bengal +ivery + Q + Xavier + XII +Latin + Frankenstein + undergraduate +Dep +racers +watch +fan +現 + envelop +ffe + reverted + Antar + blogger + gelatin +hastings + ign +icy + Shelby + Modest + Titan + millionaires +黑名 + borrow +Sim + caut +Please + J +Choose +constable +Ireland + Luo +LAB +brigham + Siberia + classrooms + insulting + flourishing +Utah +Hart + Frame + multinational + patient + challenger +pennant +場 +incerity + Amit + Browne + outfits +CNN + cider +rain +aturation +AIN + Hol +ipop +ligue +unt + attribution +北京师范大学 + Chad +bottoms +持之以 + taxpayer + Kim +ovie +jured + Lever +hyper +healy +REF + parser +hare + Rockies +York +messiah +Cooldown + Phi + neat +laude +jing + practicable + Proper +fuscous + Ultimately + DX + Rept + Shay + HAR +hanson +views +uri + Tavern + incrim +atisf +vii +mandarin + cave + coasts + Honolulu + permissible +igrant +Files + airports + Burlington + court + bunch +bohemia +ascade +nell +itto +berwick + Molecular +venezuelan +Gb + 67 + sniper + intervening + Needless + potion + multiple + Limbaugh + Deus +因特网 + syrup + Gale + Flags + [& +PORT +inosaur +orph +XXXX + virtuous + LX +faso + gap +brendan +istered + undert + Ek + primarily +Names +socrates +mei +標 + Experiment + SUB + Killed +kuwait + accountant + surrounded +ranging +心甘情 + Empty + Crew + kernels +hooded +eatures + Bastard +Thom +ruff +ascension + germ + shoe + Flat + Family + Learned + Wilde + very + ambitious +局域网 + Tob +kazakhstan + reviewing +entary + Stark +atham +bewildered + continu +russell + No + thinker +engraving + order + SHE + Pasadena + stickers + newborn + Promotion + Teen + clips + MF + seizure +relative + Mine + buses +oter + structural +ilight +资企业 + adequately +Jump + 321 +aunders + Series +glen +請 + DAR + Care + imprisonment + inciting + Chance + Mohamed + Kik + ALSO +LinkedIn +holt + Cumber +premio + ss + adventurer + gifted +martian + Quest + feeling + Toledo + cent +warn +rem + champ +kenji + Ladies + Horses +children + suspended +Dec + Beaut + eyeb +chopra + sick +Hat + mayor +ocular + Flow +EXE + latency +default + DI +asy + fast +ination + fetal +CLOSE + paying +molina + exempted + dungeon + rest +planned +LY + fan +免疫学 + given +放在心 + humanoid +bare +urs + shitty +irms +leaders + Tel + upheld + Ahmed + Tenth +unc + reaction + snowball + compact + Contracts +bernatorial + savvy + trap + dams +ledge + Exodus +iggs + theories + fox +richter + Methodist +uth +Kid +Posts + Cognitive + Rouge +erebidae + Nicholas +Fail + Rookie + Lem + pomp +fold +_- + jihadists + collusion +ronic + outfielder + dilig + cleared +healthy + zip +hang +engine + shorten + Steven +GBT + genetically +uganda +ommel + MSG + contrasting + Tone + Everest + Cloak + Bridges + Show + Jo +KS + acre + Britt + misinterpret + Houses + obsessed +NBA + havens + insight +hart + oppress + Poster + Castro +RESULTS + exasper +louisiana +volkswagen +ulrich + Lumin + OpenGL +DIS + Spain + stitching + blessing + Exchange + chosen + Guitar + Cecil + Allows +aito + ladder + temperatures + slashing +americana + cler + bestowed +blossom +syllable + fluoride +practised + 344 +mandela + specified +roman +uez + regulations + Germans +Sold +left +affected +slow +eping + Sabres +STE + Himself + brains +beit + /* +San + correspondence + heroin +iche + Pell + harmful + Rutherford + recreate +operas +cavaliers + significantly + renovations + living + squid + gate + comics + Vikings + travellers +Bird + RSA + spring +owell + spells +thirst +Would + Bullets + economy + Have +maclean + For + Fraz +khan + Fei +Route + subtle + Lawrence + mirrors + WARRANT + Rays + sticker + suspense + snow + brig + Oy +ettel +durban + Byrd +James + crappy +hilly + attribute +mclean +OSH +ence +acon +vilnius + Paran + councill + ownership + actor + cherry + GMOs +chester + alterations + 357 +weber + expires +Money + brute + REST +English + Historic + indicted + fools +atinum + marginally + Kimberly + conquer + changed +finalist +sagged +ansion +arde + Hidden + downed +无话可 +athenian + refinement + tacit +plunging + unpleasant +soto + ushered + orphans +Brook + primal + skills + CTRL + Shirley + prompt + masculinity +concurrency + Tables +這些 +reads +cubs +ount +Amb + mechan + dash +elects + directories +oriental + barracks +ILS + Writers + Reck +ters +ops +cond + acquaintance +bellamy + 356 + 174 +punta + 408 + commemor +OPE +Gary + mere + Client +nyder + cherished + automobiles + wrest + Verizon +APH + CBC + praising + atheists + Bolivia + homophobia + MAR + Hiroshima + darkened +kirby +马鞍山 + Decker +ords + 133 + Grey + organizer + aff +atoes + cares + warehouses + Link + Expl +ource + touchdown + receiving + inconvenient +ender +represented +ett +aunts +icipated + Marks +definition + headsets + Droid +thening + National +avorable + Travel + betray + ideological + potent + Putting + Application + creed + upload + aids + supposed +Publisher + torn +Nob +eval + divid + prerequisite +fae + escalating + exhaustive +sco + Kira +mentioned +Canada +MRI + Cornwall + assassinate +olulu +nih + exhilar +runs + syndrome +multi + trough +sundance +minent + thin + Ne + Miche +straighten +surveyor +opolis + Treatment + Yor + hence + indecent +angling + Scrib + iterations + skirm + Ste +marquis +britannia + showers + scram +第五十六 + whisky + meticulously +rhineland +Nearly + mah + forestry +調 +onson + coerc + flashy + counterfe + quir + Valerie + elementary + Goose +distal + Crunch + Werewolf +lems + appe + SEAL + 2030 + Pyrrha +sidekick +large + sighed +outer +scots + 02 +escape + Hughes + cartridges +mode + postseason +slade + RIP +fey + Procedure + mindset +kincaid +ivated + summarize + overdose +realDonaldTrump + rogue + contemporaries + resurgence + voyage + 202 +renee +Camera + Watkins +trevor + elector + frozen + hassle + Lindsey +tibet + Troll + Argentine + nutrient +Assembly + Julius + SIM + spawning + Flight +Heart + Coinbase +istan +selage +urous + condesc +ship + coax + Fract + tariffs +break + unmatched + hikes + query +Ba +enfranch + Redmond + Anth +Key +ensis +heartedly + Ao +kos + Postal +libretto +arest +常务委员 + Particularly +ongevity +ATION + Difference +umph +marg + COUR + labyrinth +leslie + Roads +warning +undrum + concessions + Morrison +eastward + elements + Videos + reconnect + viewpoints + stealing + flames + Getting +clusion +overture + Assad +interrupting +reginald +有目共 +dol + Altern +huskies +fallon + fight + Overview +icelandic +angs + CCP +christchurch + accompl + Protective +groaned + Reagan + misleading + surgeon + behavi +stud +herr +ATED +Park + weekday +Indiana + SERVICES + Kat + Nab + bandwagon +vicente + norm +Your + Committee +yet +peed + infringing + nevertheless +azon + Moj +aground + LE +kern +idates +AB +salazar + 780 + steel + Township +guido + indemn + cartoon +hyun + encouraged +阿尔卑斯 + Runner +dei +lavender +Miller + Ul + acted + honored +dea + cheek + Pand +不如说 +atable +adal + Legion +Adventure + thumbnail +有机化学 +listed + Echo +actionDate + Brewery +tammy + theatre +african + Reilly + glyc +raider + Viet + ingredient + Tart + colleges + Anglo + Enh + Definitions +eline +josephine + Review + 1959 +fairfax + aggravated +ASED +conservancy +ballard + bre + applauded +libyan +speech + sincerity + Hold +rob + circumvent +madagascar +biochemistry + WTO +established + Stef + servic + indie + Barrier + Helena +tif + Ninja + merely +allegro +arry + transport + Brazil + Aid +Australian +Project + unfair +ikawa + evil + guideline + loophole + 190 +Works + noteworthy +grim + GU +白血 + cynical + flyers + indeed +ARR +restling + contenders + Jac +danzig + Apex +fated +见义勇为 + PR +aution + salute +mingham + McG + GM +gracefully +plot + Tears +Vs + Income + BASE + tid + rope + feral + sleeves + whats + masturb + tablet + Hor +MAL +augsburg + Croatia +Under +hate +ostic + thee + lookup +HD + STATES + Empress + keys + Fischer + reacts +qb + breweries +Ho +oler + Ruin + prevent +yiddish +idan + approval + snipp + Bringing + barriers + Deluxe +FA +closed + lawyer + Gur +paso +belarusian + unmist +}" +mobi +brows + virtually +avy + infrastructure +shakespeare + False + recons +update + behavior +bluegrass + restricting +孟加拉 + naked +spective + anniversary +midi + nurses +arrondissement +chel + additions + Tau +everest +Reviewed + nan + jailed +lowe + boarding + naturally + still +nis +Bs + Doomsday + assist +从小到 +Service + justification + embryonic + jackets + manages + bios +trondheim + wheat +胡思乱 +pled +ramsey + Wicked +limerick + MEP +signed + Kuh + cattle + laptops + conced +usive + Tweet + ` +doctorate + optimizing +Character + Sc +aund +Merit +ansen + TRANS +offic + roofs + misfortune + merging + Hampton + brilliantly + Strauss + consistency +organizes + 231 + currently + liaison + frag +OIL + sequest + assemblies + extravag +Enter + wrapping +Very + Theodore + Ken +Journal + collector +European + Kling +mounds + territorial + overshadowed +ourmet + tried + except +mitch +dm + knight +前所未 + worthwhile +MHz +Lear + genomic +韩寒 +idget + ancient + headphones + freshwater + cul + IN + Planet +orio +NASA +antiques +GL +artist + tracing + charities + compelling +Nev + stronghold + frustrating +vant +沒有 + developers +Dr +YP +Tesla + Amar + Calling + strateg + Taking + Daryl +orioles + cases + averaging +ickle +meng + intraven + Voters + SMS + raft + Mohammed + Pref + 93 + Pepe + glow +CTV + lett + EAR +enez + Forums +guangdong + Ramsay +robbins + societal + affecting + collide + repealed + persists + Priority + swamp +erer +創 +第四十六 + Rohingya + spacing + midterm + Prem + Leth + du + FEC +第二十五 + impat +lizzie +interlude + PNG + persecution + car +isable + https +vey + ATP +chiang + kHz + Tuls +judo + Sherlock + emphatically +Repeat + fiery + Recap + forging +Wood + Stephen + Ub + coolest + tubes + darts +onse + orient +shelton + posted + Mack +rie + performing + Conscious + bless + aer +tyler + Og + warranted +dorset + Scourge +commandant +atletico + Ogre +gpu + Phil +YC + negotiation + slave + teammates + RAW +Calling + 106 + hors + doctoral + firepower + feeding +convection + Hernandez +senegal +brotherhood +peat + reduce + SEA + seafood +stant + submerged +社交圈 + integer +ogether + offers +chao + spiritually + rural +ographic +dar + players +Queen +moderate + handheld +woody + capped + railing + flav + LET +bubba + Kirin +rwanda + monastery +cyrus + ignored + cortex + famine + inadvertently +rived +city +MON +graded +ISSN + WITH +msc +tolkien + defend +Republican + Mirage +los + counselors +ussr + loses + mandates + styl +islamic +urrency + joining + 162 + alternatively + dep + notions + hoard +agging + ammunition + achieve +报告文学 +Text + hugs +blue +rey +jen +abit +OOL + Halifax + relayed + obsessive +IPP +celer + himself +usha + ruined + majorities +athena + spree + FOR +MH + discouraging + Partners + creditor +lester + netted +Cas + wearer + Veg +spect + partic +第五十五 +aura + Orn +asonable +trl + Migration + SOME + Pi + Atlas +Honestly + vegetable + occupies +presenters + NR +FBI + UNDER + Call + Caps +Call +osaka + glucose + behave + latitude + Dollars + Darrell + handsome + entrenched + proble + assassin + Okawaru + aiding +Access +hawkins + fmt +iak +yan + crystall + Marriott + medicines +sapphire + phenomenal +Roaming +引人注 + Properties +fred + pensions + access + samurai + parasites +ichita + shudder + Lena + reviewer + neuron + unjust + OPS +nestor + Crosby + yes +anca + tags + worldly +onis +sz +doin +uxe + AS +Beg + propensity +diablo +ourney + refugees + triumphant + trains + plugin + 214 + prism + Dism + wishing + TPP + blasting +oyster + psychotic +geoff +beh + Poor + Hig +coon + 364 + inflicted +unionist + neoc + summ + past + ME +irmation + Doctrine + Speech + Leading +looking + masked +Temperature +Sher + Strip + Mig + Mod +CAN + immersive + Kear + BO + repositories +stephen + rhetorical + Stay +ythm +power +Prov +haynes +Andre + ousted + tit + Ir + wrongly + punt +barbarian + Residents + //[ +Investigators + Mast + Thrust +romani +ailability + veterinary + condom + cause + guarantee + initial +gospels +tau +apply + inverted + HOW +yers + prayer +orting + Same + forming + 138 + Welcome +phone + ultras + Sections + Shortly + transition + Player + echoed +oil + auction +chases +persia +rica + cockpit +kyushu + Nationwide +ursula +luciano + joined + downfall +NAT + Pole + dermat + 418 + Elections + undeniable +olympian + pree +eous + lo + REG + responsibility +Depending + Contact + activating + authorizing +reprise +leach + extensions + plots + Rodham + 4090 +whorls + Bird + True +iration + fibre + spores + assass + marketed + customary +utsch +ultural + Administrator + Garcia + Printed + Qian +nosed +airs +idency +intersects + ranc + enthusi +Neigh + indisc + accelerate + Buffalo + decoding + validity + Cambridge +Two +tto + idiosyncr + broth +pottery +Abyss +macau +anyahu +castes + Pavel +cds +alli + strang +gleam + Russo + abused +defaulted + motif + Rage + Bai + enz + molten +ining + contacts + slowed +eileen +ael +azores + Auburn + March +prototype +actively +palmer +RA +syn +mods + 241 + manslaughter + Pastebin +helen + Packs +nightingale + aerospace + resolving +Sk +Likewise + deduct +lasting + hashes +patty +ILY + Portland +ambers +Tre +bela +monasteries + unc + Vi +continue + HT + Netherlands + Vaj +mallow + steam +kers +phas + Black +hbo +agement + circumstance + Geek +started +rolf + broaden + primer +shut +rost +metacritic + lease + Healing + prototyp +trigger + carbohyd +sweetheart +ille +Further +estival +離 +pmwiki + upstairs +hostel + Cream + kindness +amous +abouts +ibu + synthesis +Header +remington + Opportun +River + lect + Brend + diver + bedroom +pleistocene + Cultural +germain + poor + defended + notably + Services + poison + handle + Talking +cas +bouquet + 1906 + Either + formed + huge + Kov +Luc +显现出 + yielding + alright +Attempt +que + Randall + configure +cheng +glint +Steam + supreme + redress + uncommon + legions +folk + php + tear + lighthouse + Vintage + 302 +omic +terms +include + trove + aspirations + PTSD +grupo + anthology + Shanahan +frog + keeper + spons +winner + trades + mind +aud +campbell +monica +barefoot +VA + je + dro + sm + skulls + Professional + Markets + recent + pred + farmland +cki + farmers +monastic + polar +cool +rosse + Sel +Todd + straightforward +Lens +遠 + arthritis + shatter +gill + Under + clearance + Davidson + Facebook +Js + Storm + eased +kh + Gent +tributary + threat +kato + kings +moffat +chemy + Dress +outing + Globe + starship +elliptical +hummed + mattress + Chemistry +lastly +ayette +ority + Pierce + playback +yucatan +ler + phen + lowering +IVES + Designed + Tomato +marathi +peters +bayou + stormed +sheath +ombat +Valid +addy + "# + 2014 + creep +picked + Animation +doug +richie + Ez + attire + Ho + Aadhaar +vanguard + Thermal + Ukrain +assisted + incomp + sets + closest +TPPStreamerBot +saddam + hints +ono +Brown +\", +yankee +Arc +util + Vim + proceeds + Tot +bos + ESL +gordon +Na +uate +bolton + rpm +自由职业 +ernesto + vocabulary + demonstrate + civilisation + prone + Saiyan + Oblivion +twitter +athom + suited + CARE +tip + underestimated + therapy + reside + boats + unfortunate + estimation + floods +qi +phoenix +Larry + 335 +icularly + creepy + grand + explosions +redients +imb +essor + Rabbi + RR +news + Congress + Fowler +orz +akespeare +inating +beard +flushing + militia + brush + Frederick +conn +shear +leith + magnification + ranked + sermon + Weight +Arch + Mahm + Course +oli + pitcher + Hers + PRODUCT + 157 +barrister + miscar +mfa +outube +anged + stomach +chieve +thomas + Anon + streaming + quickest +aed + 2017 + shine + Bears +裏 + status + Ultron +v8 +French +Euro +uala + synerg + defends + Dad + Ded + viable +lesh + gradual + queen + Tyr + storyline + wig +casimir + tribes + sauce + northeastern + accredited +Price +chewed + questioned + expertise + appearance +dunn +vfl + tonight + Codes +ovic + estate + Kee + gallery + calling + Roy + moder + bruises +cody +uer +anch + publications +ptic + Sac +Domin + dealt + Magnus + Tiny + font + civilians + URL +adelaide +WE + comply +熱 +izer + curled + Strike +odore +evie + 1930 + friction +brenda + familiar + [' + negotiating + Occ +SER +cube + exported + reliance +farrell +gliding +baha +station +akira +limit + Penet +reacher +Tag +unctions + Savings + Louise + manager +Ky + Haram +ressor +Bank +clapping + dismantled + maintained + pione + whereas + Try + vague + saddened +western +cox +sara + fake + Masquerade + NVIDIA + chlorine + waves +Leg + Jade + before +nicky +Conf + Planning +erence + perception + doubles + Operations + vision +PP + Sheet +tell + dart +Could +jeff +phabet + ALL + advantageous +HC + Telecom +william + Terra + publishers + kinetic +orious +Obama +dwarfs +leasing + induction + SO + (& + Demo +社会工作 + tangled + algorithm +nce + WHEN + Zuckerberg + spread +endars +total + Email + Iter + advisory + Balance +bwf + Ong +runways +horace +hunan +Fin + Barth +Ring + confrontation +castile + Pelicans +第五十二 + concludes +rison + carbs +Ari +iate + pes + influences + firms + plank +isky + electr +catholic + northwestern + films +ished +comedies + thoroughly + Allah + unfor + Inform +meet +ossip +igil + Nature + largest + oily +万立方 +townships + capitals +艺谋 +constance +host + snaps + Blooming + chants + directing +gasping + cache +carole +ashing +alaska +ago +tajikistan + flatt + Lock +hays +brazilian + LP +nard + facing + lenses + JD + observe + predis + 316 +AC + bul +prop + SHA +ruary +Similarly + Keeping +racehorse + subscribed +Patch + 1962 + haven +WO + Experts +ilaterally + deterrent + ecc + waters +harlan + Horizons + Pharm +NW + mutation + caster + Melee +title +apped + Garner + chuckle + Poly +Rev + scandals +ardless + Handling + WT + stare +bruins + Radio + Zhang +verona + intersect + culture +iker +umbledore + gripped + reminds +门户网 + turbulence + awa + peacefully + Caucasian +etter + participation +anhalt + available + Offense +hackett +cole +biographer + Dinner +coco + lob +marks +cour +KEN + footage +ena + caution +Har + rupture + der +served +atography +Pict + evenly +GS + Orders +Comp +电气工程 + premiums +第六十五 + ch + Chin +李自成 + na + Ct +aph +耐腐蚀 + whisk + leaving + biscuits + mailed +ched + surely +roofed +Individual +nuts +iants + Prim +NOW +setup + Tobacco + Silent + hr +sava +iper +vered + Our + minimalist + tested + Polaris + Innocent + locality + painful + Galaxy +hoe + season + Xu + Presbyterian + picking +odies + attention +constructions +ixie + Someone + Tibetan + spike +quilt +napoleon + Arist +phthal +FB +barnes + memorable + sovereignty +ATHER +fates +hack + vul + researchers +cambodia + thwarted +纠纷案 + Resp + libertarians + comprises + galvan + dividing +Released + budding + CPS +Cos + ambitions + surges +hun +见仁见 + secrecy + 1936 +barked +语文课 + sought + NAFTA +war + Fnatic + determines + certainty +robert +related +build + amph + KP +enne +三十余 + Tire + Ethiopian +incoln + ls +ORK +should + convention +ynt +egypt +mysore +spot + pages + deepen + Tent +ub +jj + Decl +almighty + Lav +tie + predator + repairing + bol +dating +pty + -- + Tooth + Favor +handball + auto + yen +michael + majority + graz + paperback + perfectly + favorites + feedback +postdoctoral +hawk +ulatory + Reg + broken +thatcher + astonishing +bequeathed +ursor + counts +clarinet + consumption +edy + Buddy +Bitcoin +grey + elic + resolve +flexed +algia + conviction +piring + plur +leaped + deterrence +wra + Phys +italianate + making +jolted +Torrent +iameter +enting + voice + refres +govern +crazy + Despair +washington +grown + disparity + preventing +port + concentrations + vetoed +keepers +odder + intervene +sponsored + wonder +oops + exposed + suspicions +vicar + Ame + encourage +瞧不起 +Action +Chall + bailout + paths +種 + fascinating +Monster + ASA + Jacob +ile + ._ + dictionary + improbable +lied +mexico + problematic + Saul +consul +ordon +indu + lad +roxy + Instr + Realm + purchases +ustainable + Twitter +LB + transporter + STE + Arc + Shore + Sue +tay + an +emeritus + summons +cling + evident +De + EN + hesitated + hearing +ciudad + describes + Memphis + Se + Ember + Hawaii + kicked +aming + Conserv +blanc +GV +abil +lanka +recorded + Sup +ancies +IA +Japan + Ai + bob + barn + murdering + algae + storing +moans + Dave + resp + Cullen + Taste + writers +lydia + Concept + incomes +iqbal + intelligent +Otherwise + mayors +ellar + resultant + criminal + trembling +工商业 +cgi + Dodge + MAT +nine + involvement +bernie + Jobs + 218 +laim + Les +inar +surrey +Kg +horst + fallen +zzy +iths +ptives +cebu + 1982 + Mavericks +uggish + shield + Prayer + Sett + objection +PA + Urug + Southern + stocking + asserts +OULD + appear +c2 +toe + inconsist + welcomed + Copper +ocation +atts +IE +urious +UT + highest + LONG + taller +SD + 330 +ochemistry + saints +caroline + Removed + Spell + anat +護 + Redskins +itter + coins +rc + fors + Blueprint + Delta + abduct + exhib + Voc + lake +不论怎 + Kush + DAM + Three + this + cz +overgrown + perv +Denver + extended +reformer +derrick + birds + compilation + officer +grades + Cynthia + absolute + unf +Cu + failed +家喻户 + dominion +christoph + Streets +xxx +unease +Different + Que +sequ +attle + javascript + sax + Ratings +Monitor +forum +qu + Promise + businessmen +hilltop + destroy + pornographic + Slug + primed +ocide + Sasha + depend + V +Internal + chars + Authority + capacities + squ +vinci + finger + Ticket + dominate + accents + Afric +internment + Case +lr + Abedin +ushi + openness + Small + unified + Protest + brass + Kathleen + 368 +archdiocese +ossus +Miss + proc +sharon + transsexual + Das + capturing +clarkson +antha +Par +states +ELY + cush + optional + concerns + newcomer + families + enhance +ruption +caribbean + NIH +poly +INAL + Imam +IVE +wait +Enough + forfe + Matte + adult +stefan + cookie + lipid + entirely + NEXT + english + Bolt + Believe +itor +maximilian + charg + unim + Rican +spectroscopy + sarcastic + Armenia + throb + dishonest + Mad +cain + rubbish + Raptors + Opt +guinea + dig +itv + artic +sunday +itamin + fs +friends + whine +conqueror + Saber + Gol +ymph +itism + stem + Tactical + avoids + timetable +andrey +ellis +belgarath + hapl +archibald + personality + scientific + ptr +ages +uld +mare + Weed + Qin + Forms +公共场 + thirsty + egregious + displ + deficits + colonies + skeletons + Stevens + Emir +Consumer + Giov + Rugby + entrepreneurship + lucrative +lowlands + grew +harrisburg +marriage + approvals +ofi +contact + Delicious + Bayer + trade + drinking + tor +igated + aspect +hitler + Quart +關 + Curtis + 270 + thrust +leen +nch + Aur +olved +uddy +randall +lights + digital + unconstitutional + markets + Taliban +randy + computing + labelled +iliate +ettes +deck +中共党 + invent + observes + stockp +NET + TOUR + num + outraged +hesive + harsher +teness + standardized + bind + psyche +Internet + 120 + Rh + Volunteer +rika + Cosby +glas + websites + sperm + Perception +crawley +ure + cooper +ucc +illon + Decoder + photograph +数百万 + exoner +enabled + Eaton +viva + mistake + Dul + rainy + MRI + Nare +nutrition + recovers +OP + breathed + biom +greenberg + Transcript +venus + Deaths + wars +PM +minute + Spanish +gorman + attempted + Buckingham + marijuana +rolet + 184 + nit + Yu +kernel + overw + attempts + Islamabad + television + sex + Saying +lds + docks + Shirt + congratulate +Simple +ogan + Analog + delighted +eper + altered +black +Controller +保加利亚 +erek + impover +valencia + Lara + kil + steadily + predominant +austen + persecut + crippled +cora + Zimmerman +opened + braking + Templar + Differences + Looks +ursday + somewhat + obvious + Trojan + lacks + embracing +zionist + Hunt +replace + Comet + Preservation + cared +Witness + inconsistent + teaching +jean +fluttering + Hoff +madhya +一切都 + Ug + Hindus + effic +intend + Illuminati + tha + DSM + actress +DERR + razor +vertising +巴洛克 + stick + tidy +QL +vpn + 1700 + import +rollins +Visitor + will + Springer + organizing +ayed + Morph + Wichita + Solitaire + primaries + Horn + canyon + Newcastle +leipzig +Setting +constabulary + CMS + COMM + exaggerated +tarzan +asty + something + polish +lao + Code + pocket + contracted + Shade +middle +molly + atten +专利权 +newton + lesions +hampstead + wel + ware +emery + 331 +naires + candidacy + Swarm + inverse + serpent + encomp +jagged +serge + 51 + backlash + renewable + antioxidant + wartime + Icon +Connection +alg + results + missiles +kawasaki + customs + contributes + irrational + Gallup + Center + menstru + Oak + Provincial + Output + soothing +katrina + Bound + publisher + lobster + discover +simone + revealing + owners + Kavanaugh + Bast + adversary + cheesy + capacitor + referencing +UGH + bids +CO +sounding + APR +dal +pects + Yeah + Indie + Darwin +lander + Lol +album + resistance + malware +mccoy + waving +bands + monopol + Draw + antenna + Yad + Bronze + Albion + mice +probably +aix + legislature + UAE + Captain +rejoined +mausoleum +hak + carry + funn + dispel +andr +ienced + Hydro +gap + Trap + pave + extermin +agers +摇滚乐 + Assange + Rai + KM + Riy +XX + electrom +ailable +Lie +urches + 560 + seconds + recipe + Subcommittee + Simpsons +ihara +nan +mega +MM + pagan + Florida + adip + bronze +loosened + ritual + Slaughter + detectable + Created + recognised +floyd + immutable +ido +rw +sneered +engineering + Zucker +第二十一 + she + spontaneous +letico + fruit + suspensions +ogl +odiac +alsh + PI + Ago +RO + Log +mouthed + expert + Craigslist + Dwar +濟 + NK + removal +ipa + Anything +brazil +pressed +igon + sugar +ski + rab +不會 +Sun + messenger +维基百 + Ruins + kne + grinned + priced + Ign +fivb +flame + 96 + Carter + incapable +bourne +atural +monday + Relax + aromatic +ragon +acht + grilled + Susp + Hawaiian + Jenner +want + egalitarian + pens +oppers + Colonel + brick +rosemary + ([ + snowy + postings +ttp + Thomson + speaking + Fletcher +Plus + Dance +sever +兢兢业 + << +bingo + inj +songwriter + Qt +ros +Deep +静下心 + seed +ogen +ricks + constitu + prophecy + im + d + joints + unexpl + Weeks +mesa + Ridge + benefits + Bal +BIP +ticked + Focus + quasi + club + 2500 + laborers +总的来 +User +jerked +ruce + cavalry +lies + Biology + Leap +marianne +harta + phenomena +inate + cents + agg + Mormon +barge +southport + 0004 + counters + Wide +长白山 +china + vigilant + dod +thoroughbred + paced + wo + phr +lombardy + pioneered +okemon + SL + Prescott +boeing + Fourth +idia + Flex + trailing +rin +西安交通 +psy + Submission + Inside +buckingham + Leon +lege + Transmission + tapping +airship + detrim + falls + Judd + Evening + ye + Wendy +kendrick + champagne +ogenic + mos + table + Hezbollah +popular + Speaking +birds + six + Sven + Jersey + 2013 +extreme +Few +filename + Two + Legions + minister + illustrations + ethos + branded + hired +exp + WWE + addressing + engage +evergreen + identify +mack +screenwriter + dissip +MODE +bengali +CLUS + bent + through + gadgets + Several + barrage +Pure +很难说 +puter + 12 + produced + sad + squats +atari + rightly +Custom + neural +trinidad +arters +Function + collisions +stab + Ireland + divorced +Next + representatives +Canadian +awi +Upgrade +Corn +Martin +wah + whispers +ILA +rection +IGHTS + jargon + Cth +grafton +Account + ki +martin + dresses +全国人大 +中国台湾 +Ap +Rocket + cautiously + fame +ighter + contained + fail + multicultural + ado +niccolo + 367 +abad + Sequ +rine +pering +£10 + deploy + Releases + follower +psych + util +mckenna + Stafford +alities + rubble + NJ + truths + disliked + Bram +科技园 + Asian +auldron + diesel +uous + rabid +tis + pneum + Commands + q +Josh + cumbers +blown + Cogn +austrian + rallies + shouldn + 460 +欧阳修 + humour + Jill + barley + argues + Edit +xton + adapting + averaged + Makes + muscles + reversible + annex +mateo + stiffness + bowls +m3 +aughters + Whis +entity + raising +illin +ISO + mathematical + Dates + Dob +grimes + Plugin + insurgent +rayon + Braun + undeniably + capsule +nickelodeon +utils +lyss + Comb + occasional + Nich +一千多 +ras + astronauts + spine +utrecht +powder +baden +azy + Dead + originals +engeance +题主想 +Un + chronic +aughs +essim + Talent + horrendous +analysis + swallowing +外商投资 +nox + Erdogan + televised + runaway +tres +endon +ean + realms +occ +Context + 314 +italy + lobby + Vanderbilt + incendiary + Gest +harcourt +覆盖面 + horizontal + president + incidentally +pur + Load + divorce + CK + NP + bounds + Texas + Images + swearing +ipad +milan +novak +ECH + membrane +lynn + nicknamed + curator + Skywalker + padd + spont +champaign + Vaughn + promotional + Inter + Laun + ba + burnt + NYU +narrows + elong +mber + obedience +ETF + Ant +bayern + cue +villas + nucle + loving + iceberg +Rush +audio +victorian +cludes +DX +WAYS + MVP + context + brainstorm + Awesome +Am + device + roughly + fees + Bind +aspberry + letters + specialist +cum + antit + Funds +tortricidae +Left + Mickey + facts + Tre + Activities + Naked +第八届 + confession +Install + Venezuela +樹 +rockefeller +rippled + crazy + Luke +ukemia + implicitly +Luckily +icit +aptop + Herz + Boyd + electric +nato +chong +mahmud + rapists + more +els + tennis +valued + deeply +treatise +know +lakshmi +ECK +havilland +ation + prostitutes +alo +iku + journal +icks +current + kitten + cunt +屬 +sensual + fueled +Console +omes + Alabama + WordPress +aze +iy + refused +wreck + Xiao + venues + Vermont + 1280 +strangled + premature + spies +Sky +UST +leto +betsy +睁开眼 + Hover + phrases + transmissions + nowadays +Option + Powered + mus + voc + Nec + magically +rag + pains + Witch + everyday +ashi + Danish +Fourth + Clown + Alb +gesellschaft +ryder + sam +marius + Virtue +FC + berries +compliance +grossing +Disk + sew + KILL + screening + Tours + RB +urrence + abuse + immediate + opinion +cell +animous +fax +ocamp +Assuming +hapoel +conversely + raining +INT +hobart +foreign + melan + Spectre + calculate +Ma +ogly +tenderly +venetian +Ability + Aly + restaurants +changed +agger + macOS + Stra + That + tricked + Turns +engravings +terminates + noises +cient + bolt +bastards + glossy +iness + incorpor +carb + , +robinson + Likely +Imagine +第二十六 + condoms +berkshire + remains + Optical + Assist + new +bler +hello + Scientists +vind +scuba +四百五十 + Accessed + Zin + oz +June + Facts + station + opened +UCK +ggy + undead + mirac + Gustav + herbal + bravery +rit +growling + mainstream +inant +Beaut + Battles + optimize + shouted +virginia +propellers + incon + Liter +happy +Computer + adhere +ixt + penet + bet + foil +hill + Orbital + Coming +albans +nk +fitted + severely + besieged + Leg + overlooking + bragging + witnessing +aler +romans + Your + Manz +BLIC + Robin + Dillon +burgundy +ometime +akers +tobias + Brass + Samuel + imitate + push + bell + inhal + Nak +vanuatu + thinkers + Invention + darkest + leukemia +Form + THC + Exp +leyte +costello +clip + interrog + Tag +onen + Karin + inspections + recounted +飚 + Toad + applaud +Laughs +BOSS + Managing +administratively + cutter +ipples + Specialist + harbour + Kis +wrapper + Sect +covered + Aw + Board + vec +door + Scripture +edit + Sok + Pred + naming +except +astrous + fungi + Whoever +insides +Both +edes + contractors +when + prevention +usa + Neander +brady +RPG + Proxy +strong +hypnotic + carrots + SQL + tablets +igate + Pog + Part + precipitation + Artemis +Studio + All +Recommend +Since + plugs + ItemLevel +vassal + NAS + stretch + Corporate + Jews +uj + havoc +gonzales + (< +ackets + Franco +Mission + Work + the +christi +decreed + Socrates + redeem + benefiting +RAG + bombs + inspecting +atos +asers + fifth +management +righteous + y + issuer +AST +shots + Dal + Youtube +Exactly +afghan +bub +libre +create + Little + understood +itton + Adelaide + Cha + Timber +locomotive + Modified + cub +amy + Official + Rosen +副司令 +ieved +Army + confidently + foot + separating + Denmark +characterised +Vict + Sep + extraordinary + audience +ecclesiastical + distinctly +Pink + moratorium + Ke + Sorry + grapple + slowly +tera +appropriate +erves +serbian + apopt + scrutin + loaded + Own +ilic +travel + suite + mism + Alexander + fallout + Greene + Unicorn + messed +icated + bone +Vill +Standing + Saving + elegance + Ball + Hopefully +第三十五 + Waters + Ceres +hawks +Leaks + gun +Gab +rians + Money + stagnation + repressive + visually +fontana + hurdle +nir +peggy + Thousand + tele +Anna + unleashed + diam + Coverage + strongly + discuss + sections +fiona +DIT + spurred +izontal +GES + Beirut +erv + prohibitions +bably + 1880 + lords +广西壮族自治区 +arians + locally + Cube + hear +Enjoy +hom +oxicity + Strong + Conversation + Blackburn +xvi +THE +phia +mins + Tor + collecting + Melbourne + establishments + Moz + Nine + galaxies +romanesque +susan +王维 + Bol + anyway +dfx +their +tainment + Schne + oversight + Swanson + conceptual + Pes + causes +touch +rather + deceive + swall +geon +urry + commend + uniforms + settlers + Istanbul + anyways +Store + Editors + Savage + anticip +oblast +cheyenne + Americ + world +helmut +oci + Misc +peruvian + hype + fiasco +alabama + demonstrates + WikiLeaks + bred + tempted +Represent +s +owder +intercollegiate +resident + talks + Wayne +ftime + Central +racetrack + OG +session +conserv + limiting +hungary + Customers + smugg + inco +ITAL +utt + lower + subst +mae + Atk +Sel + blows + heartfelt +columbia + Azerbaijan + camping +ifying + knock +张三 + concise +otion +regulated +lon +bennett +marshal + bread + Jaime +posal +Pay + Magnet +wards + mentioned + insur + occupant + James +iii +BM +amalgamated + [" + collects + coordinated + weakening + Coulter + distinguishing + Payne + Courier + perme +Seattle + brushing +UGE + located +IFIC + MTV +philip + Rahman +belgian + commanded + Eucl +lette +fat + +# +Lots +uminum +記 + Rez + faculty +ainment +polynomial + endorsing + agendas +\/\/ +Flo + manip + Ends + blurry + theat +bromwich + Vid + Barbarian + allele +substrates +benef +benoit +anglican + evenings + spelled +Det +HTML + Kind +odan + dismantling + humanities + Bernard +Large + sc + dazzling + fit +nomenclature + 295 +aucus +mckenzie +牛人 +dylib +troupe + Penny + Nat +wt +gest +authors +dayName +schumacher + equitable +Pol +blogspot +alions +madden + receipt + Characters + crowd +营养学 + frequently +weet +Wheel +Jer +erb +fortunately +嗎 +chimney +ashington + Jewish +isman + Malcolm + LW + aims + Flare + laun +onding + Given + electro + sandbox + hier + improvements + - + resil + Chev + sustainability + best + Uber + Bout +oses +oke +konstantin + attends + craving + Quad +Char +Independent +ilts +imoto + populate +Cut + Na + tamp + optimism + Titanium + ace + Regist + Burk +ONDON + Virt + Vu +medi + photography +sniffing + 40 + Actual + Spani +ONEY + depreciation +NP + Harvest + Twisted + surrounding +党总支 + hum + Virus +ischer + surrendered +Wa + chaos + government +reset + Max +ardy +uhf +francisco + 2050 +內 + surroundings + underlying +erto +oliveira +grimace + Stanton + wires +]; + finishing +Connell + zoo +strom + AU + Russ + Nanto +cutting + tattoo + items +Copyright +effect + Legislation + yell + exotic + Mann +igion +naire +johnston + piercing + Afghan + unpopular +Franc +wickets +tile + Mil +Physical + autobi + paternal +cia + lobb +李志 +Irish +marlborough + Sets + Florence +cramer + Document +TFT +Stars +isters + Jennings + fell + Disclosure +Url +fargo +ager +peria +gunther +mla +南京大学 +Elizabeth + Lopez + Velvet + pitching +aughlin + Heroic + cascade + Crow +encrypted + Xuan + flats + TT +bernstein +zik +equestrian + Until + ranks + venom + congress + pilgrims +reorganization + autumn + manifold +mule + deviations +sherman +Repl +middlesex + burden + countries + Bannon + Product + duel +ospons +Police + renting + hitter + Compare +vari + + Circ + Yaz + Pend + module +akov + Brigade + sober + ISP + Fighters +marge + 407 + dispatched + timeline +disk +emile + fund + African + covers + Sev +torpedoes +barony + heater +制造厂 +metz + Ancient + Daesh + Antarctic +james +anke + Late + Melissa + navy +cong + expedition + reaches +othermal +CHAPTER +icts +multiple + actors + affinity + chained + compete + quests +氷 + pulses +ciation + Dragonbound + validation +voivodeship +commercial + brethren + pairs + Trinidad +柴油机 +民族主义 + Hogan + En + Slip +santo +Breaking +News + dilemma + hut + pat + urging + evacuate + initiative +wept +instein + violence + default +navajo + wrestle + Bers + Luk + strategist +exempt + yard + Ips + affidavit + entitlement + retail +inki +freiburg +oeuv +plan +keeper + AUT +destroy + Flint + round +fitzpatrick + centrist + ISS + Bryce + Exc + Source + loudly + Snapchat + Registrar + hectares +软件工程 + helic + too +telegram + Faces +隨 + Races + Calder +Traditional +ounced + condol + wretched + snippet +thumbnails + Marse + Hmm + Eat +jaipur +atson + successfully + Radiation +hoarse +contract +johann +monarchs + APIs + plent +ittered +Squ +Pages + Bosnia +ky + employed +vaughan + looms + prefers +itsch +PATH +clive +house +INTER + crossing + satell + influenced + terminate +lublin +petit + operating +mult + nation +pace +Documents + leng +uming +fcc +Congress + dexter + Lithuan + Robots + 380 + Fiat + turtle + google +anish +ADS + happily +onymous +oliver + Day + Sexual +oreAndOnline + Wanted + irrespective +enta +LOAD +colombo + civil +kimberley + listens + Ice +PO +ious + kingdoms +domino + Mobility +acl +withstanding +itting +辦 +invoke + them + environmentally +archival +russian + medications + Arbor + 19 + Officer + chests +thread + Coral + Subject +did + hospitality +GU +bile +aki + diss + injustice + Scha + Luckily +celtic + Discovery + Philippe + mortg + professor +urtles +lodges +Fre +vised + Purple + ANC + affiliated + Ruby + hardness +steamapps + sitting +ADE + Harbour + () +prof + achieving + cop +ran +Quantity + feeds + riot + FTA + propositions + Alpine +DH +Face + fixture +language +cooke +iage +cerning +stained + converter + 376 + Cork + Cue + drawer +rudder +Tu + regex +essions +ting + addiction + fracking + civ + contractual + PLUS + bureaucratic +Gi + Mor + ing +ycle +ilian + pixel +gain + fifteen +jandro + roadway +Ne +benin +ixed + ramifications +Enable + energies + conspic +nect +derek +record + sweeping + Ya + ALWAYS +browning + abuses +estamp + Verge + Loaded + Archer +elve +API + subset + pressures +kei +ading + dues +iano + Comparison + irrigation + undes +ane +adriatic +Behind +manitoba + Warfare +marxist + len +samba + Amos + body + Converted + monstrous +malaga + lanes +benjamin + expect + teenagers + est + welcome + 259 + deal + willfully + Pat +TU + retali +ivity + boy + poking + policymakers + Hard + without + received + shining +yg + Unemployment +armenian + tandem + Alonso + alloc + stitch +defined +capitalist + Unic +asure +BUG +zens +quran + goggles +SE + crawled + cornerstone +ilateral + Apart +kappa +hyd + unexplained + covari + mainly + Pharmaceutical +ische + confuse + Lawson + Zheng + how +王莽 + Probably +nesday + restrained + palpable + FC +ahmed +Dan + Regulation +ben + selects +rake + Rail +Compl + circuitry +norfolk + Company +rop +met +zee + Directors +tons +Between +jenks +getting +rouge + camera + Yanuk + favored + lifespan +invertebrates +bytes + walk + Museum + wood +Intern + diction + formulate + theological + persuaded + Cow +inois + Pant +osexual + industries + unfocused + Destination +asms +Construction + 1960 +ponce + wives +showered +derry +gha + Fuk +kristen + 153 +cree +vale +park +might + Oprah +tem +ENN + scrim +ups + angry + Franks + Bronx +Smart + Ort + Ye + raised +iec +中国农业 + Conquer + 2006 +jang + dam + honoring +gian + militant + publishing + Fancy +igne + Sodium +gallantry + Feather +fowler +dixie + recommend + analog + Parm + latex + Myanmar +MENT + Rankings + Investigators +uckle + announcer + Chair +jc + estim +發 +畢 + yet + women + odds + Scalia +cedric +LCS + Hudson +MC + chilly + typew + Continuous +aching +fulness +paralle +deutsche + chord +oku + Invasion + riots +banda +godfather + dag + readiness +weston +tyrol +treated +oult + honorable + <= + upro + why + titles +rebecca +zambia + hesitate + our +usting +iatric + recruited + From + fabulous + sor + Hannah +denmark + cape + NRS + smuggling + horny +ancers + weighted + Tess +atp +riaa +beer + antigen + faithfully +dec + DEAD + adrenaline +Mess + Joey + biology +DD + quotes +Mist + invention +Jim +icity +Above +ASIN + generation + Scroll + Always + salads + Flan + Me + Autob +tehsil +Used +神经元 +lcd + Farming + Payton +gardner + regional + important +obos + Io +OTOS + preserving +pavel + encounter +iken + backbone +Db +uploads + motor +icing + fac +ARK + disagreement + FRI +Appearances + Potential +stadion +jection + Heart +mongolian +chile + attendance +归根结 + exquisite + challenged +light +abal +icut + exh + prep + separ + pivotal +eely + ridiculously + Utilities +ventional + 65 + cultivation + recourse +Phill + Motion + bake +successful + athletes + verge +dhaka + TB +Joseph + send +illion +星驰 + clinically + Sacrifice + screw + wool +ahmedabad +pilasters + personally +though +晓明 + college + Above +newmarket +Dual + myst + vortex +string + FIL + Rodgers +erey +persian + reflects +illance +yx +Palestinian +運 +okes +朱元 + levied +cricketer + ecosystem + sed +bared +guyen + disperse + Turk + MORE + TRUMP +NAME + were + proceedings +douban + attaches +omp + debit +senal + grind + separatist +zagreb +pec + Height + 76 +palestine +kl + overhaul +nit + dou + vampire + captive + aired + troops +They + Bombay + appeared +URI +carousel +実 + Keeper + Spurs + minim + purported + continuum + pillar + milestone +carnival +iet +what +options +ures + builds +csa +kristin + Pratt +atha +env + Cthulhu + restruct + seasoning + Positive +meredith +aea +Fal + automotive + Anfield + vegetables +hissing + pinnacle + hemp +aepernick +party + resting + Bonds +oon +erd +lough + ingested + MSNBC +blood + derivatives + Ducks + Arri +Lim + Adin + sentiment + positioning + 273 +mini + mail + Beaver + nice + Kob + tactic +无能为 +tobin + 117 +Polit + careless + Fast + Jehovah + Mystic + heroism + Nas + acceptance + jerseys +pendleton +firstly +shepard + WHAT + visibility + reckoning + Alison +isive + deforestation + ascend + mindfulness + Justin + lead + acronym +ISTORY +reader + contributors + Eid +hamilton + NBN +Initial + Upton +idelines + merch +osa +aspen + Isles + Angry + Immediately + humiliating +subfamily +paz +parkinson + Commun + Zar + upt +oen + blowing + format +colored + Too +第十七 + arise +Pear +Case + 3 + prolonged + picks + Layout + elites + HDD + cliff +hesda + Lantern + Angular + Span +Fuck +iture +asu +ultan + Processor + liking +gui +bonaparte + stalls + vac + Arm + supporters + goaltender + lengths + Terry + anticipation + mornings + intestine +excitedly +oki + Multiplayer + fuelled +dharma + Called + wallpaper + questionable +FN + warmed +spection +like +icidal + Scenes +wered + weeds + Biological + slay +next +际关系 + unch + Iraq + Arabia +burn +esome +rans +AMD +dose +operated + fetus +nudged + Lunch + Catholics + Expression +button + formation +Stand +readable + sabotage +kennedy + traits + Health + Miracle + Rove +bulldog + glanced + Bris + wrath + plugged +King +iHUD + Quit + meanwhile + graduating +achu +leonardo +寫 +tta + Kali + anthem + sigh + Hels +kill + untouched + Chavez + messy + Mog + praised + EQ + amalg +aker +Policy + centerpiece + intensify + another +utz + trucks + overthrow + Lens + suppose +ceylon + Dunk + asymm +lder +doris + Gou +Fer + inbox +twenty20 +Steve + unsu + decks +cept + keywords + Edison +gardener + exceptions +Marg +rowth + Springfield +weights + dc + parasitic + listed + Countdown + premises + collapse +essex +unky + bindings + unfounded +ername +ranged + defect +NH + Joy +mexican +Lib +wales + Denis +Fax + Lib + volatile +TL + McGu + crossed + pics +holm + Eliot +upiter + permit + Rwanda +oles + Highest +antwerp +look + Cape + lion +hospice + Gamma +braden + GOLD + colleague + ic +josie + portion +reuters + vomit +hua + tripod + DVD +fletcher + rang +clair +kitchener + gin + muscle +elm + Reaper + Pete + guard +guinness + Stead +sicily +boulders +sels +illation + descript + cancers +decca + tedious + nighttime +Specifically + SER +协会会 + dex + fost + category + peculiar + secret + Dak + partition +OUNT +OUS +scotia + Gul + weighs + embraces +rehensive + Elm + Almighty +Hug + ornament + Create + colonialism + careg +method +fill + analys + sprites +ezra +imgur +VT + Strat + Sl + alternating + RC + Finals +elf + heap +rowing +ispers +lettering +mental + Sch +pes +hertfordshire +wesley + Grain +Meta +Anth +OPS +stryker +Te + draped + manifest + congestion +ango + quieter +inaugurated + migrants + declaration +FE +smoothing +tormented +tate + monarchy + veh + Lessons +ood + boiling + Dimension +EE +mata +bird +Fi + reuse +umbs +swansea + philosopher +Home +ONT +prim +Package + delinquent +nsic + favors + singled +enrique +nik + )] + Batt + Bolshevik +memorabilia +itizens + mund + CU +bean +issance +essen + TVs + Beam +CS + diplom +mumbled +Setup +Ill + DOS +spanish + solved +oyer + refuge +itzer + 197 +pole + AMER + ab +Commercial +stallion + narrator + spy + tempered + Alternatively +olini + Questions + Cardinals +Tok +ollow +ignt +uce + FAQ +everyone + 358 +DeliveryDate +iculty +Fig + capital + lug +ataka + pathetic +NESS + upgr + evaluated + considerations +packing +score + ant + interacted + annotations +gothic + shorts + Rowe + boredom +concordia + TE + NL + coercive +ific + abusers + tastes + slams +BF + Hast +dalai + Gibraltar +Steel +ailed +CFA + remainder +terror + elaborated +bent +Bern +Per + flipping + Picard + Traps +posed +erson + Gra + Dragons + possibility +spain +ende + Destiny + Pegasus + motive +ten + Prol + Absolute +rase + exemplary + Lewis +volution + rematch + Iraqis + Minnesota + tumultuous + channel + vinyl + faire + bouts + apologizing +xiao + mischief + antibiotic + pen +assic + consume + Chocobo +nyu +burt + FAM +GAME + explanations +garfield + Synt + hiding + Shit + Gamb + :) +charlie +elligent +deal + Transformers +Member +eman + albeit +Semitism + shun +Living +flight + rappers + efficient + elabor + treacher + shorthand + orgasm + phil + gram + besides + Aviv + information +一百万 +christensen + params +isi + Elev + 161 + my + GL + amplified +slovene + pricing + Blake + Awakening +uttar +violence + quarters +icip +ensable + Ellie +Returns + Challenges + commentators + Smy +koreans +fam + birthplace + incred + matching +patsy + armor +illions + tendon +mbe + McCann +ietal +crescent +isers +ches + bliss + farms +Lex + sage + Sid +sam +undy + Faction +akeru + safety +IDES + audition +sonya + myster + floated + Basin +Children +Ped +roleum + Unleashed + Kids +acas + accord + remorse + 386 + Nerd + vein + heav +Christ + 660 +Us + Together +Collection + Racer + margins + mammoth + Buffy + Mild + alternate + 1955 + conducive +cb +parent +Friends + Ethernet + though + Examination +iers +六小龄 + 325 + Lore +hampton + motorists +氣 +Progress +javanese + Educ + midrange +sulfate + abandonment + Shap + enough + unborn + thanking + Ary + extravagant +aim + flexibility +unsuitable + Cig + halting + Winn +weeney +Safety + administrators +susannah + treat + construction +ationally + strawberry +lah + Willis + Lub + glued +gins + silver +cent + rebel + 134 + turmoil +carbonate +dahl + Username +Asset + catering +garry + Cir +atana +Bus +zx + malt +检测仪 + folly + Chennai + blank +cessna + fart +mob + Brah + hinges + gn + backfield +mcintyre + iv + Mazda +ranking + anim +菲利普 +bony +umping + salv + naval +isSpecialOrderable + extracting + Elise +Whit +fay +centered +hilda +putnam + titanium + party + secession +gravely +allmusic +南开大学 + Tw +syracuse + suck +ghent +Golden +automatic +Float +議 + monitors +Tree + Boys + blamed +IFF + Telescope + couple +raction +Suggest + Am + MDMA +aware +历史学 + Ka + gaping +Mus + scale + latch +rochelle + toilet + Bib +isse +November + pol + shops +adic + moreover +incinn + drainage + Wolf +stad + hilarious + artificial + Revis +smirked + pies +Aut +driving + Question + entails + arguing + Improved +parana +craters +sac + percentage + Initial +baton +isec +izzle + demonstr +plaid + badly + Literally +outlying + HEL + viol + decency +aged +hellenic +ickr +Lost + delivery +acial + volunteering + flares +channel +umbers +iting +agher + dim +modified + departures + rod +limits + forgiving + Crypto +edmond + studies + Kaine +_( + prescribed + unbeat + Ubuntu + confidentiality + spraying +jimenez + fict +alessandro +laurie + inhabit + Qualcomm +Type +oling + seating +curving +Chair + Data + poem +Play +Flight +franklin + blo +snyder + 240 +Western + Resolution +pep +ILE + Bank + flooding + W + queens +Val +ANI +caliph + Grimoire +Future + Gry + presents + cat + definition +規 + droid +lect + amenities +pperc +mala + %% +彦宏 +Ry +itted +ocr + consolidate + downtime +hermes + Mole + java +peace + Hab + buffers +nottingham + POP + mug + Handle +blanche +tch + TS + passes +Qu + allegations + Clinical +computer + Wife + Clar + squat + harassing + Bolton +grove +bidden +保修期 +lying + authoritative + ejac + dealer + Maxim + sanctioned + proletarian + depos +ODY + stunt +quit +runners + Soul +ockey +gu + foundational +INFO +RAFT + tert + Replacement + nor +Age +inently +SPA + Beacon + basket +costa +serbia +ons +ullivan + Rune + gag + solitude + pin + caused + Heller + me + pole + goats + state + Telecommunications +诸葛亮 + EB + proposals +hide + assay + Docker + Porter + Trial + accuser +urai + Shapiro + drills +metis + shaft +laughs +gretchen + thermal +tri +institut + Chamberlain + athlete + Anime +porate + officers + Bro +stewart +modelled +党和 + file + Depths + ruby + course + timeout +總 +isdom + chilled +productive +leneck +Overview +格格不 + Host + tsunami + Comey +军医大学 +Turkey +janata +umber + testosterone +ossibly +ravens + State +palestinian +clawed +scoffed + sleeps +mitter + audi + flown + Merkel +adows + tab + Gins + never +manuel +weekly + combines + greater + stance + crap + Chrysler + Nurs + ec + dispersed +trumpeter + nationalist +basha +kees +galerie + angrily + sophisticated +Solid +azerbaijani + attic + Stealth + laid + suggestive +antrim + exploitation +promenade + Nass +material + Pool + proprietary + 282 +Dis +ieg +IRD + ruthless + Brewer +nero +return +Apply + Lights + kidnapping + joystick + 1992 +Sullivan + aggreg + retreating +Nazis +scowled +ommod +lille + Saturn +dota + Elementary + Inquiry +Italian + o + deliberate +idel + sk +xiety +olding + refusal +mughal + dragging + Bus +Harry + devote + active + off + incorrect + 760 + Barnes + Abby +coa + VP + finely +exiles +ategories +弗洛伊 + lull + activates +hari + seriousness +airways + Williams + incompet +royale +warily + poetic + Ash + Deng +FIN +value + libel +楊 + urgently + attain +yale + CHRIST + Neuro + skull +vichy + spe +leighton +rator +Ocean +emphasis + unconsciously +burials + ig + WWII +sgt + Trib + gust +许许 +Michigan + Jur + thugs +angela +macbeth + Martinez + Oil +lace + Authors + richer +pez +arer +etts +clare + Liqu + spam +ZL +agles + explos +ndum + senate + Sounds +vineyard +Saturday +hendrix + Geoffrey + finally + Keen + Miko +menon + lavish + deterior + browsing +colloquially + Gut + forecasts +wylie + exercised + correlation + MPG + Persian + FIRE +ahu + packaged + fashioned + tea + wallets +gogh +icultural + PF + adopting +kr + strategic + grid +versely +Chart + Amp +skye +Understanding + insepar +tech + legalization +urt + FreeBSD +Ctrl +ffer + destroyed +UNCH +ilipp + ur + NHS + Gar + glove +ioch +catholics +Gamer +Uber +To +una + fooled +newt +Supporters +ically + hunting + It + takeover + Samp + dispose + epist +shropshire + Hayward + taboo +cabins + normalized + McCabe +westwood +moaned +stein +aldi +byu +Boo + Tyler +Particip + cowork +Senior +poole +licensed + Bob +afort + green + Bug +Supplement +adesh +锺 +columbian +ILD + patrolling + promptly +~~ + Wheel + Miz +Sn +mom +}," +oc +MAG + billionaires +ousel + electrons + backstory + Draco +Except +最高人民 + release + methane +IER + weapons + vinegar +irting +itia + Clarks + intricate + gallons +gomery +工业区 + moss +cest + sporadic +rowan + basement +olesterol + Wan + Squ +ortex + arrivals +Die + 224 + Available + vide + winning + bishops + herd + daytime + serotonin + out + Municipal +stock + Mechanics + slashed +hiro + dropping + inspired +africa + overlooked +utable +差一点 + fir +OWN +ribes +rising +effic + oppos +arabian + payment + MUST +bari +yden + abbrevi +otech + Toby + Operator +ifter +Dutch + simul + init + gardens + knots +irted + mills + synergy + washed + valuable + persuasion +unic + FIN +rape + payments + indexed + Home +跆拳道 +dri + pastors +Temp + launching +coca + currency + frantic + CONFIG + 457 + Down +cu + throw + <+ + 1989 + manipulate + comed +Transform + USPS +family + linked + imperialism +prone +cantonese + provocative +quire + accounting + powering +prov +tab + presentation +margaret + Cos +person +ItemTracker +THING + witchcraft +johnstone + Astro + interf + patents + gran + aides + around + Aug + rhythm + delays +inders + communicated +warlord + cart + looks + Assembly +orr + West +netflix + kinda + incidence +otyp + Wasteland + Col + Artists + printing +skylar + Shall + 334 + HUGE +chorale + songs + theft + TOD +strewn +sumo + cords +yards + Garn +edgar + twenty +oser + improperly +acer + Theatre +][/ +Bell + lbs + Joh +constitutional +siena + pissed + hel + factions +oted + assertions +Friday + Quartz + shampoo +fingered +談 +angered + slander +elin + irritation + GW + moist + drive +filter + Graham +xit + drag + pillow + Bottle + plastic + use + excessive +ridor +melodic + Krish + trend + Lerner +tertiary +BMP + clinical +OK + characteristic + Saga + Environmental +chronicle + Firearms +Nut +Roger +BY +dixon +igible + Accessories +klaus + Sw + supplements + Securities + lar + insecurity + bull + Outdoor + quarantine + Bruce +hellenistic +Doctor + Resident + Consumer +bermuda +grounds +Wiki + Doe + Belle + Failure +ebus + Uncle + (! + Watergate + loading + duo + Patriot +lyon + dubbed + uncertain + surround + nineteenth + Ath +henry +undai +kok +sheridan + invis +administ +isks + ball + equilibrium + microbi + one + plainly + HW +selves + sociology +气候变 +Generally +eln +ousing +聽 +aceae + Mic +pkg + community + lingering +wehrmacht + progressives + channels +tanned +hydroelectric +vell + Females +etary + recommends +義 + boxer +曹操 + Neural + Stre + Gold +據 + traumatic + colle + analyses +brandt + Wi + Ricardo + tables + Damascus + rhythms + Kos + Chandler + bastard +beaming + Kurt + totalitarian + shocks + trooper + Submit +dewey +zach +ballast + Spice + Mitt + Kuwait +alysed +lowland +walter + thing +othy +semb + Runtime + Rockefeller +last +DER + arous +Europe +adolph + Lonely +lynne + sheds + Nietzsche + oldest +aws + standings +channelAvailability + Kanye +mongol +oux + inherently + cottage + Archie + Indianapolis +stru + Fl +oland + tabs +八路军 +macdonald +ials + Carol +written + fares +Ox + technique + imb + rapidly +FORM +posium + Mothers +Sam +thens + stumbled +electric + Deer +soundtracks +aling +Development + City + Myr +CPA + usually + herds +ooting +educ + unsigned + Snape + sale +iol +arel + eyel +basque +gorith + nightmares + gon +九十年 + lazy + posts + twin + sunlight +comets +clancy + Dinosaur + sensor + accuracy + connector +][ + bisc + 747 + Birth +pharaoh +ETH + Tos +nn +riddle + allow +grange +Bush + Mour + Sav + Firm + metric + discount + Certainly + sheer + mighty + Derby +Pinterest + reducing + responders +Country + genuinely + Imaging +race + stimul + recovered +concept + smartest + insurgency +rylic + formidable + TheNitrome + Crit +norway +pennsylvania +reelected +otaur + royal +ISH + slic + learns + . +classes + invincible + LD +ptive + Lyn + monitored +shuffled +said + pursuing + separate + tast + looted + TAM +utes + decorated + dairy +levin + PKK + lame +baptised +onto +畅所欲 + Feb +ulture + NH + Prague + ankle + together +umerable + Intermediate +invasions +rect +Exper + maintenance +WORK + minor + 55 +reworked + Tea +nikita + Retrieved + dispro +nants + lod + accus +橄榄球 +中华全国 +Bul +asi + Apostle +ipod +fable +Check +bauer + mattered + Dr +pradesh + talking +rig + northwest +vaults +alpine + Sud + eject +taipei +peeked +stu + following + begs +文化部 +Mis + 107 + lowly + VPN + bands + ride + Bailey + Vietnamese + Purch +forward +kazakh + [/ + Presbyter + destroying + Fund + harmless +atures +usr +ussy +onym + st + glitches +Unknown + Mercy + retention +pron +alde + hatched +克罗地亚 + UPDATE +Modern + totaling +OUND + eas + toxins +gmina + Pars +aca +charge + Shooter +knowledge +kel +Ross +ICES + entry + theaters +矽 +ounters + Quote + tongues +giggling + Pour + Bride + oil + enter + Fortunately +landscaping + burned + DK + DUP +Saudi +akens +rient +amnesia + distressed +ifiable + Africa + forearm + depict + downhill + Builder + matches +ISON +OLD + Volkswagen + VID + unauthorized + restructuring + shortcut + inexpensive + TCP + hurd + liqu + poisoning +chevy + injuries +romanized +libby + Comedy + Kinder +陶渊 + delegate + carb + spilled + labor + catches +attache + YEAR +nation + Mech +acter + religion +Charg + JPM + Catherine + Katrina + puff +padma + Amir +iding + Flav +bery +roderick +Demon + named +scarlett + Lastly +Vision + Held +Rare +eastwood +uba +doyle +welch + Rim + Notably + bust + ver +sheng +alexis + Mind +swallows + screwed + Bees + ethical + preferred +Jeff +organisms +isbn + buds + Jeremy +mendes + Hacker + Fall + Wer + toes + Lucy +metadata + alerted + mixing + matt +ridden +ourced +owship + blunt + matte + wholly +dinand +sedimentary + Hungry +flutter + composer +crowley +itans + vehement + Ray +blabla + Argentina +queue +July +Jac + highway +karim + oversized +voyager + 392 + strife + lack + Orwell + summarized + knives + Bear + immigration + manufact + plotting + Louisville +tw + comb + sharper + affected + vou +ient + Cache + absorbs + rh +somalia +王思聪 + cheeks +剑桥大学 + Hogwarts + relentlessly + skyrocket + Prompt + experimentation +Element +La + Dan +actions + nonprofits + staggered + charge +soloists +rmb +摩托罗拉 + crushing +irlwind +RES + undecided + distributors +imately + Gon + knit + hippocampus +francs +{" +AGE +anticipated + MOT +Sensor + anomalies + themed + tin +help + tranquil +hew + clearing +awed + Profit +reg +scale + Thing +igo +xe + Cost + projection + fingerprint +September +URA +數 + complimentary +commemoration +gaelic +Residents + restraining + he + weakened +protestant + plasma + significant + tofu +ernaut + Semi +cious +ovich +thompson + WITHOUT + deemed +SAN + forced + Korea + paid + viability + Ach +them +Spe +Lean + ground +LU +cambodian + Print + Crowley +ithering + Excellent + exec + Ginger + Morse + scenic + scholars +data +veyard + swast +islamabad +Links + meddling +monaco +日内瓦 + 1939 +Clear + innoc +jar + chatter +門 + Canter +Islamic + loose + permissions +URRENT + Adviser + Vietnam + infect +macleod + ap + Liang +esse +reeve + Wedding + abruptly + stint +iminary + Corridor +macedonian + Investig + Webster + dawn + toxicity +AAF + prosperous +inky + minced + pests + Spread +seahawks + Bulletin +dale +particularly + hour + endings + approximately + purchasing +points +ISS +JA +rugged + sky +福尔摩 + Rover +Activ +horseback + revolutions + Acts + Sym +duction + Idaho + differs + BRA +ambling + synaptic +leep +tunic +abre + Lanc + Pirates + Cortana + spontaneously + aesthetics +CW + reached + Reese + Alt +blance + glass +crafted +donovan +wolfgang + Chess +plementation +bilibili +diagn +near + Carbuncle + conce +eine + descend +clerical + humid + franchises +angler +kali + accidentally + deadlines + Raf + acet + rese +bbc +urring + horizont + Rot + Certified + captured + router + functionality + Cary +walkway + EUR + Online + downloaded + be + Fey +olympus +conan +acia +iola + RD +eel + nutrients +Vo + lang +strauss + damn +navigable +Test + Warcraft + vitamins + 512 + retiring + spots +java +Iraq + recomm +mcdonald + herbs + towed +Community + evening +Death + Um +ieth + touchscreen +jewish + identities + drama +ifle +olving + customizable + twist + lists + coworkers + licenses +charlotte + disband + NGO +Mir +jed + asserting + dealing +abu +Bot + Carth + Emblem + superpower + traced +與 + Continued + summed + Pip + penetrate +skilled + Floating + Rack + stuff +bradley +Rust + attendant + detect +obyl +tuscany + sands + accent +auguste +culated +lana + desired +ritic +chance + selection +worthy + 1963 + noses + PubMed +bank + workshops +Square + honestly +pping + plun + autism +foley + buckets + Nad + outings + swaps +orneys + Well +Pat +atical +ONY + Neptune + kill +order +Frames + illnesses + Goals +Spot + core + data +第六十六 + Acid +anza + furnace +profile + Alliance +}) + Nova + airspace +rbi + Citizens +serialized +aachen + Kessler + VIP + tir +urb + ca + delusion +sion + hacking + funny +achy + plural +recapture + Mous +idential + invitations + masterpiece + exempl +rendered +nut +gold + Wilhelm + aqu +apprenticeship + Europa +stafford + swirl +nature +IK + induce + donor + endemic + sensations + constitutes + Hut + Santiago + narrowing +edia + Maritime +iana +illes + Mecca +sharma + JACK +Iron +一个半 +yanked +pdf +urned +anyon +phil + settings +紅 + matched + Sho + Task + Honest + alley + eater + Pull +ileaks + Bust +ebra +ules +rimination + Kers +price + quant +consular + NYC +劉 +few + mut + Land + scripts + alleges +Bat + Lansing +peugeot +Allen + Highlights + scalp +destroyers + condemning +offset +Work + taxis +straining + SV + blaze + Sph +edwards +everett + packs + Orb +Changed +rehensible +icon + reaff + warning +slalom + Ess + cultural + occupation +douglass + Tina + repo +itals + paw + succeed +Ad + Carl + get +Avg +eat + gearing + pair + Bes + tides + material + mil +ENCY + SOL + Spin +bye +Wed +Point +onday + Culture + relieved + Byr + Roe +TS + Treaty + 69 +根深蒂 +杨过 + rookie + bacon +orsche +Corp +urances +anoia +joseph + evade +company +ollower + spears + grocery +VERTIS + Bore +oun + LIVE +rounder + Abrams + Plaza +Another + euros + rectangle + Hardy + Pier + nonex + Miner +aliation + gest + specifics +macmillan +Gam +galway + Mutant +Net +abbott + RET + moment + pious +payers + 233 + stocked +wrong + beaches + NFL +lda +madeira + padded + Medicaid +查尔斯 + damaged +Bel + monarch + obsc + anomal + starts + incl +inent + lasts + pursued + terr + Approach + Rah +Factor + beam + obser +CHA +construct + Tac + KB + intu +Mat +Mex + touches + redesigned + diagnostic + Mr +rossi +freeman +Morning +Dream +shrines + HH +massachusetts +consecration + piv +ange + 281 + undesirable +cht +burrows + Laur +arthy + Store + Bub + control + can +landers + delet + PLAN +ohl + evolutionary + tur + Macro +bolshevik + overcame +mith + flushed + Forgotten + departing +vals + maid +duffy +Fix +nobleman + actively +apache + brokerage + distribute +asser +wire + Mercedes + injunction + genome +norm + Pig + logging + adv + recip +baby +constantin +griffiths +Allows + Spark +burke +anked + rugby + Vaults + Week + = +SEA +Brow + melting + pund +Disclaimer +ods +levision +hawthorn + nuance +ocks +embroidery +ulnerability + Nothing + muster + electorate + Controlled +Hit +furrowed +scully + seminal + Hak + Pent +inn + Yok + risks + Hert + biased + Canyon + Blades + requisite +command +esq +rick + heroine +ormal +ielding + Cout + poets +mustafa + wants + slips + Hed +axis +Angelo + Chevrolet +情不自 +GE +adan + Hat + Druid + AT + plantations + Bec + activation + bride +enhagen + Confeder +barbie +FD + thr + Display + outweigh + Moines + Hispan + challenging + preacher +kendra + Wiki +packages + sandwiches + bells + fluent +Exit +phasis + quarterly +Mos +bournemouth +tugging + Distribution +lighting + Figures + Sim + viewer + vocal +oret + cowboy +sindh + Greater + tyranny + Christy + Concern + Spl +bind + opener + wrench +陈代 +atkinson + Childhood + mc + reminder + feeble +Applications +uilt + Census + Theory + Tee + Current +Right +grading +endelle + ain +WW +warner + favorite + Rivera +norma +moore + Moses +letal + Emirates +outstretched +Crash +URN + Burger + drastically + Compared +Her +Container + Vertical +Cruz + Pas + Tues +Requires + Croat +印度人 + Blog + instructors +nered + pods +encing + locate + 236 + Borders + Offic + Hir + pap +climatic + spicy +nyc +sbs +nmi + Harry + equal + inter +emon +ikhail +ston +sculpted + Original +semble + Sear +Wilson + 332 +bjp +utsche +什麼 + specimen +enemy + Dolphin + Richardson +Emb +Last + Arizona + veto +Hun + cream +ACA + balloons +Band +August + Clifford + nig +gmail +inventoryQuantity + spiral + Metro +georg + Pact + Feed +retch + neighb +ateurs + Administration +geologist + evidently + decline + promulg + Sind + plea + software + leaning + Roth +永和 + Booker + Harvard + argue + Kar + YORK +masjid + predicament + 1990 +welterweight + Madrid + Jolly +charting + comprehens + continents +wagner +stead + guess + revelation + Nissan +ageddon + solemn +uces +aine +glamour +otti + depressed +ded + Gardens + 15 +Champ +alore +athed + furious + Tunis + vot +Dead + occasion + LED + FISA +gem +noah + Laboratories + physiology + evasion +husky + espresso + Partial + obstacles + Architecture + Mali +meaning + Devils + dependencies +clyde + grin +wheel + Monsters +prisingly +DSP + Old +usher + Moonlight +temperance + balls +Bruce +误以 + Hence + Friday +cross + exact +itably + precept +wrote +insured + investigations +bender +period + fre + TRUE + WHITE + think +sugarcane + Upload +instead +Environmental + absorbed +dorsey + suprem + emphas +Cap + pins + portraying + network + Lambert +Down +deacon +onyms + tumble + preliminary + graduation +enzie +zza + Obviously + specifically + ze +icka + pepper + revoked + confidence +dresden + Tucker +operatic + assumption + reassuring + Incarn + Watts +taekwondo + sneaky +nesty +iamond + inval +Econom + lineup + onlook + poked + workshop + followed + enactment + Education + Cinnamon +admin + guarantees + plaque + rgb + rivals +oral + owning + fasc + Fay + loopholes + Trad + herein +vp +PUT + borderline +window + TRI +sept + arts + enemies + observable +erved + want + allocation + swear + Carib +sacramento + modifications + Pound +gerhard + visited + awful + teaser + appre +intersecting + Keys + Jiang + DMCA +days +amd +AW +bitious +ulative +ologists + Burma + Expert + 280 + Punjab + bookstore + excerpts + Lanka + ment + cups + Origin + intertwined + Enterprises + Cumm +clergyman +electrically + 8 + handmade + Chrome + Enforcement + boyfriend + Sug + Heavenly + regained +widows +acters + irritated + Harris + Batman + leve +argentina + telescope +buckinghamshire + whistle + High +California + Elsa +Teen + boards + Didn +sword +ATIONAL + Citizen +cade + wider + Missing + Aus + essays +airy + shrine +Assistant + wedd + background +abulary + node + Seven +cmd +moran +irin +zhen +nascar +fee +auder +Elect + MED +seattle + overarching +drm + Dynamic + laughter + wealth + Yao +sega + Vo + spends + Them + black + gang + Gain +moonlight + holog + Reviews +kitty + squares +Init + Hib + Russia +keynes + jurors + overlook + installments +pixel + PATH + completing +ements + bipolar +warwick +Perhaps + livestock +還是 +rothschild + cove + prosecution +stem +acerb + diagn +Resp + subclass + plagued + precon + Top +abolitionist + Scientology + basis +Mrs +ntil +BGM + Beet +ards +worcester +imag + undue +jamaican +past + coloured + Shinzo +Always + WILL + aptly +Fat + formulations +dillon +riott +level + Defendant +bgs + Sabbath +anzac + explicitly +sean + draw +argo +gingerly + grep +Ground + Brune + plat +subtropical + satisfies +archives +孙权 + mods + mounted + slender + pamph + civilizations + bats + pandemonium + SAY +Cont + Pixie + legislative + advoc +gate +_-_ +Rot +bruise + deities + Checks + Carry +jensen + membranes + silicon + sea + refill + beetle + ignition + converge +bourg + incompetent +wendy +select + Somalia + conject +MF +Just + municipality + nude + SUPER +Like + Hodg + stern + conn +Collins +Library + Next +zona + Glover + pav +復 +echo + paramed + psychological + Sil + Circuit +meridian + HTML +ublished + Governors + BA + jur + Menu + inconven +ramirez +Invest + Ada + handles + Starts + according + convinc + Moody + bug +ummer + Jorge + voic + basketball + punches + taxable +£3 +egu +aghd + Gren + tw + nucleus + fortun + shin + raspberry +可不可 + refrain + PU +greenfield +ouble +offensive +chinese + discovers +islanders + es +variously + kind +redeveloped + Wond +peck + mur + enjoying +existence +clap + humans + Export + Floyd + fuzzy +raud + pu +artworks +En + rapt +arn + Common + Squid + helped + hiber + condemn +umbnails + Blossom + completion + documentaries +universidad +Ste + clan + RHP +aily + Canberra + Isaiah +afa +munchen + freak +marvin +質 + qualifying + collaborative +大中专 +aah +三角洲 + Oral + series +rums + Medical + Dani +hiba + extingu + Drama +ribute +unts +xi + McMaster + mobilized + Dhabi + greets + Reduction + pip + Epstein + packaging + cut + Pearson + spotlight + Audio + buff + Berman + Encounter +imental + Biol +decomposition +Null + Philippine + sniff +prime +Batman +agements +Altern + Cousins +mayer + Normandy +onnaissance + unint +function +Sexual +DC +arine + Sisters + discredit +ote + correl + Clicker +brien +aminer +miriam +abby + installations + climax +omb + sent + gathered + promo +baroque +omedical + Node +liest +UL +romero + freed + Neal +ute +irmed + Stefan + explosives +peshawar + dismissive + breathtaking + quality +sv + functions + narciss + softer + gimm + haze +creen + crumble + Peng +王羲 +purdue + reductions +Far +yukon + lies +IJ + Raw +orea +awatts + associ +ERO +汕头市 +itude +lyn + depress + Bonus + Daylight + Kund + yr +asian +ussia +质量保证 +之以 + witnesses + ropes +lords + glowing +gallant + inconsistencies +baritone + Fell + coin + readers + vom + arranged +powered +verbs + Wall + jumper + signaling + Townsend +iom + Trash +zeke +arist + Alic + reluctantly +rendezvous +Removed +burnham +augustine + Pyramid + cracking + bleak + Ultr +kathleen +xtap + Crisis +fontaine +alloween +Islam + hygiene + san + rag + Smartstocks + TOR +iferation + orche + degraded +indians +raphic +pose +cougars + 71 + (?, + heter + CY +Gil +stroked + plunge + lobbying +Easy + broadcasts +nj + Tripoli + correction + Mead +hassan +Shortly +Report +status +ivariate +Eng + disorder +Interstitial + slug +perrin +arjun +olid + opens +nesses +Increases + MIC +Bla + Dri +laws + Oops +coal + malicious + Sinn +avage +aber +railroads +OI + PHP + VR + chemically + Vitamin + preparations + Skype + summers + buddies +brooklyn +urance +speechless + genital +ques +earth +meat + Valve + fuller + che +nave +liv +fang + cloth + apologised +ittees +續 + Moor +sabbath + Volunte + Swordsman + rev + reinforcement + dialect +OWS + Leban +nirvana + Identification +Gal + ASS + Drivers + drawbacks +arthed + east +Tonight + sob +emerson + unpredictable + nuisance + ration +rod +eez + culp + Puzz +二进制 +nia + Merge + unrealistic + faith + lowers +Role +Ron + shader + Senators +orned + bare + AOL +Visual + submitted + abyss +intensity + diseng +roosevelt + justify +federico + improve + phosph +ermott +Pot + cavity +MAR + Prelude +torah +iry +quick + stal + Stoke +laugh + sod + vendor +Again +GMA +ccoli + Pandora +politics + ceiling +zu + needs + Ottawa + wrap + innovative +martyrs + relationship +patton + Genesis + occasions +細 + Skinner + millions +ested + IoT + risk + compatible + Hussain +iverpool + sund +DY + despise +loaned + eyewitness +admiring +netball +chio +conservatory + winds + ginger + autom + cabbage +magnate +Brian + Bent + Darkness + annot + click + prot + wield + skewed + disagrees +ocom + position + by +USER + Sas + Malt +fished + publishes + Bran +Summary +Mac + Dawson + secular + slim + situational + Frazier +Farm + chloride + Petra + revers + STEP +excursion + persever + losers +wyatt + materially +during + [+ + costly + invoice +with +alm + Staples +date + mast + basic + Dur +African + assume + creation + Sy + bumps + Advertisement + regulation +ment +hamas + consensual +ureen + Shia +kiss + detain +finger +CLA + placeholder +eteen + Senate +qqa + wherever + addresses +心平气 + studios + soda +entin +mccarthy +reference + Grass + ja + Astral + AE + Tube + exhibits + []; +ymes +arkansas + Bucc + advertise +ategic + Pol + report +anky +significant + Desc +leading +OCK + delusional +martins + painting +sumner + Mormons + Details + width + rigid +wnba +igor +vatican +avored + wed +scandinavian + bead +fai + webcam +LOS + bounced + punishments + organize + 415 + person +motionless + Und + indign + Salt + resent +genie + elections + suddenly +jul + Limited + b +inwardly +acements +ipp + FedEx +Legal +paren +OY +rhapsody + ensuing + reverse +rocket +anwar + Presidency + gone +Acc + Painting +iane + Cat +preston +Almost +thank + adop +cadiz +spencer + flo + anti + simmer + concurrently + near +訁 + disclose +ombo +paxton + probably +aeus +iser +tractors +FDD + Arpaio + employers + yield +pb +aciously + gri +Therefore + holiday + Er + different +wrap + Archae +expo + Calculator + Kappa + alongside +頭 + sang + Samoa +iability +lebanese + unbiased + discriminatory +hell + Uruguay + tuned +Jackson + stink + skew + Khe + hurts +bloomington + density +anastasia + discontinued + Charity + Lomb + Wales +cadence + sympathetic + pub + dream + Towns + Amendments + precedence +nietzsche +kov + MD + Freeman + Very +Lu +well +lov +Jul + JJ +cially + 353 +DIV +alone + Windsor + Blacks + 39 +Mi +ule +login + OVER +aches + Hispanics + Jury + replacement + Scythe + decisive + splitting + reservoirs +AKING + RESP + Golden +Ark + Moments + Kam + Because +nr +godfrey + Video + becomes + correspond + 1300 +espie + LINE +Ze +Howard +erton +here +VS +TIMKEN + Mountain + deepening + Xiaomi +oice + outreach + prayers +efficients + coma +Wolf +atter + grat +IF +goalscorer +电子科技 + workplaces + advisers + tokens + misdem + maple + ALE +carrie +doe + garments + congratulated + Rocks + pg + compel +founder + purs + Doing +iaaf + tons + cookies + cryptography + regret +transmitters +abeth + Any + Gi +AlphaGo + investig + Haj + reconciliation +ILL +iments + trope +Pakistan + Presidents + Results + groundwater + wage +bodied + Detected + intro +greer + chaotic +或多 +rhythmic +gewater + unstoppable + Yong +osphere + attached + buffet +avement +mansfield + Spo +Collect +pire + Beck + Lump + blatantly + Conj + Scheme + famously + brief + pad + Mount + 272 +ettlement +haryana +cruises +interred + Monster +第六十二 + scam + creek + rotation + Lauderdale +damian + petroleum +odd +vert + operations +wal + Xie + Armenian + clicked +ynasty +ographer + displeasure +Textures + feminine + Del +moe +lf +yer + Frontier +Bad +sub + refuse + t + fundament +itches + transformations +Let + Breath + Ill + kilometers +fein +lus + sly + idi +SIM +各不相 + Winchester +chad +iday + Rossi + Infect +download + hog + iso + entertainment +quartered +atri +faltered +武则 + Lisbon + Yoshi +semitic + injected +ewater + gu +halla +uncomfortably + Kendall + STUD + drugs + Timeline + species +花八 +uckles +clus +skaters +zos + YPG + business +wadi +lets + options + categor + Fly +transformers + gathering +planet +apest +Drive + drummer +requisites + 208 + leaflets + Priest + cherish +nathaniel +HOU + Constantin +Fall + moot + semblance +tibetan + OFFIC +polk + contends +mortimer +izarre + Wolverine + vitri + Actions +VOL +usable + cs + FUCK +Amid +afe +tains + declared +andan + exqu + Mim +期末考 + earning + regime + ion +Offset + Cardinal +campaign + nobody + barrels + SET +島 + Sanct +justine + weakest + Flesh +kazan + McDonnell + 306 +DON + feels +ferred +ighting +mojo +GO +ventus +sey + orders +later + forgive +briggs + Saturdays +erald + Whole + Lovecraft +jess +landings +rette + Sakura + stamina + underground + Blocks +euro + motivations +Ga + 176 + 116 +ishop + bore +hex + remarkably +ffic + HF + EQU + shooting + sloppy + Aer + HG +exit +alker + Interestingly +burr +darwin + leaks + 1970 + subordinate +ANS +expression + spat + politicians + IDF + 74 +VE + enumer + CF + Ble + Survey +Solar +NE +Gaming + prev + byte +third + Schneider + Tile + borrowers +feeder + NXT + Bite + Decision + Kyl + ste +[_ +gujarat +hanna +ourn + Cortex +rage + charismatic + measures + Vaugh +ractions + physically +stables + Ether +snh + sorce +angles + fumble +ogie +artisan +pler + lat +culture +expeditions +recreated +berger + variation +gma + Goblin + woo +eem + Airbnb +glass + invaded +review + swim + Newtown + Omar +bath + outside +helene + July + fair +REE + dodge + String +rate +rattling +唧唧 +effective +jad +Ws + Measures + Philip +alis + wid + yarn + subway +naomi +kell +hp +icking + legs + biblical + terrific + superf + frameworks + Inf +straightened + Tap + harassment +pines + ELE +foundry + coord +xiv + developments + prevents +component + Allow + shorter + Leicester +Ash + simplified +ithe + blocking + step +CBS +President +ota + livestream + Fors + uncon + Holland + Polish + classic + Gret + dru + Stockholm + sprink + bestselling +tangent + IT + Applicant +hindi +parker + flows + Front + predicted + vs + Bill + distort +fascist +ittee +Ore + affiliate +botanical + counterterrorism + Sub + insane + Devon + Silence +ellie +rhone + glands +冠希 + math + woods +Cast +Spread + shenan + homebrew + econom +aug + MK +endo + smoked +DIR +際 + GG + consequences + BJ +bray + Axe + Mobil + Biden + terminology +UTE + omin +halo + Raqqa +salem + Stro +attered + fixtures +Pers + comple +rico +farmhouse +ructure +Sent + Commissioners +rates + Afterwards + THEM +oxic + responsive + Grimm +atalie + Diane + chant + Melanie +office + Bones + lungs +Control +myrtle + Contributions + chords +saigon +ophone +igroup + roaming +Testing +Tweet +ceptions + hurled + sear + dismantle +iless +aerodynamic + far + 309 +oho +born + inflammation +lat + snapped + rockets + gren + Hatt + frustrated + sidelines +mondays +vibrated + Herm + Stam + Leica + Meal + Wald +thursday +Hot + Dig + Immortal +fusiliers + protects +Everyone +anthropological +agnar + Theo +sioux + shelling +milos + circ + hardest +ccording +til + Schwar + Terr + calm + FROM +ZE +avia + Goods + strikeouts + synonymous +throw +ravis + discouraged + Helsinki + gloomy + talented +izens + Regener + freezes +geese +ottomans + Tr +itures +马尔代夫 + gives + Maz + quotas + classroom + Bearing + communicates +brate +lihood + hacks + Sheldon + Gors +kentucky +danube + sunset +eeper +year + Oo +caucasian +fuck + preservation +ucha +cele +libya +reykjavik +camille + Crown +pack + ripping + nick + Bicycle + trespass + kios + ticks + campaigner +Stretch +note + Syrian + Berkshire + Rogers +agin + banter +nb + Yel +Walk +visory + ancestry + riches + Federation + undertake +夏洛特 +tesque + Estimates + Hamb +ISA + Simulation + communications + positively + Junior + rabbit + Modi + Abuse +orion +beans +Jose + instruments +jagger + Scher + Lift +isting +rodu + Dover + ebook +beta +will +childless +alzheimer +plin +yawned +regate +spirit +nair +blair +babylon +tnt + electromagnetic + Working +rancho +minded + Imp +robertson +千千万万 +latvia + BIOS + Termin +niall + didn +justin + commun +Lee + increments +We +seed + Graves + perceptual +Eye +llular + secure + Aero + Fan + coffin + Dalton + privat +DoS +Birth +LET + towel + promise + msg + sauces +services + inevitable +onductor +urbed + Resurrection + distrust + involves +Egypt + directly +symphonic +orman + 44 + locals + eman + partnership + Ging + configuration + prose +potion +ousand +WMA +Place +marseille + foam + inhuman + overtly +peterson + 2004 +hirt +brew + pract + Students + brid + enraged + speaks + deletion + Tour + Medieval + Mik + Uniform +earable +tar + collaps +問題 + precision +ahl +negative + rotate + volunteered + addressed +brock +价格低 +erman + Achievement + inventive + GN + anchors +frenchman + oddly +sie + client + rumors +brown + strengthens + PACK + Panther +ities +PG + curiosity + Shinji + mon +ache +hibited + Panda + Surprise + mustard + violation + scratched +]: + Silver +hatfield +Obj + convertible + everybody +Nintendo + Nou + fantasy + Alchemist + lam + SPR + Lancet +boer +OV + domination + 374 +sf + plantation +clifton + eventual + terrorist + dreadful + light + statute +irie + NG + Male +duet +百科全书 + Chaos +地质学 + Collective +saunders +ballads + possess + strat +inition + 18 + char + l + revolt +ractor +gue +GIS + 1993 + Nath +border + forwarded + purple + Sham + investing + immensely + urged + Vine +bec + Kurds + inequalities + commanding + Fang +patting + Brigham + microscopic +isk + Walls + ACE +refueling + Idea +uro +rences +incarn +inos + Links + exposition + therein + disclosing +benny + mechanic +ovan + cooked +barnsley +yuki +oxide + reader + trustworthy + executed + change +rotterdam + retina + BACK +ider +麼 +yorker +butt + salary + reporter + division + wip + Spirit + Duck + caucuses + Italians +stated + 100 + Flash + Manifest +raleigh + athletics +stood + secretive + psychedel + occurrence + Palin + Earlier + fetch + Frankie + Ontario +ivas + Maya + fab +missing + mass + certific + deceptive + unatt + BEST + fatig +parrot + Dixon + Jazeera +muriel +tribune +fighting + campaigns +Email +rab +iew + behaved + learnt + Foods +第二季 + slaughtered +xml +巡回演 +}. + Certificate + Instruct + pills + Hurricane + Alloy +halloween +gado + reminis + associations + successor + Scale +uable +石油化工 + presidential +raju + countryside + squarely +lake +britten +Cart + Romans +cence +immer + Brave +romy +whitaker +centenary + footwear +euroleague + dynasty + geometry +________________ + Garland + Iro + dashboard +Average + emission +copa +terraces + sling + taxed + wound + 243 + Georg + tuberculosis + exponentially + Explain +DL + arch + boast + purity + mod +non +ombies + Mineral + Mistress +bush + 6 +眾 + Liberia +addock +Want + FINAL + Halls + intra +etting + population +gavin + might + shortcomings +ouncing + encountering + recon + Compton +tottenham + discharged + Tom +alph +onic +Secretary +evin + SOLD +petitioned + fax + senior +uten +Amazon +Opt + declaring + handset +Changing + RES +ulk +ilial + Generally +WHERE +michelangelo +ilantro + sophomore +olly + ;; +erences +achel + swung + Viz + Fame + Hassan +della + changes + Cross + Basil +ICLE +Java +rosy + Kafka +leanor +杨家 +rolley +AMS +FW +onential + Eli + tigers +crypt + bench +Tap +kite + Exec + representing +Happy +郭靖 + Remove +Tomorrow +australia +stown +marco + informational +itus +gleamed +mercedes + struct + excluded + CLS +clave +stockholm + >> +Compare + victorious + imposes +ivalent +tect +kaiser + guerrilla + mechanical +acus + mar + citiz + hailed + NYPD + thumb +Coun +brett + Cooperation +当之无 +catalina + scatter + Somerset + fiction + multim +mcintosh + 1600 + metab + unfolded + Stick +powerplant + fatally + mental + Lance +bj + Buckley + LDS +amber + Frenchman +gets + potions + Dakota + unexpected +第一百八 + shot + Map + Instead + busy +usk + Hua +nfl +tripped + climbs +ihil + expulsion +ulas + pid + planets +actus +cffffcc +could +Karl + mask + conve +cock + strongh + comedians +stepmother + onstage + FIGHT + respond +aky + 98 + medal +cornelius + maze +boy +breton + politic + News +peng + Peterson + stepped + windshield +acha + whether + Child + beds +李氏 + content +tatiana + entertained + centrif + Titans + ast +urbana +improvisation +inquisition +cheryl +ploy +Kar +izza + Subway +FAQ + Fo + cabin + sprint + visualization + refund + Communism +ignty +Liverpool +itating + prominently + throat + drops + minimizing + Farms +AG + 421 + recept + pneumonia + Continuing +RFC + rubber +Gy + threatens + distances + Rooms +heses + reflect + Danielle +浦东新区 + seams + Vortex +UN +Kevin +hours + Trust +itated + introduces +arette + chem + identical + EA +pensacola + Manson +bros +levine + melts +wanda + accident + diligently + onions +claudia +delay + Pet + ADS +victor +jiangsu +artney + Straight + corro +outhern +ukrainian + Procedures +ATCH + Zi + Electric + Passenger +opens + Authorities + apps + PER + turbines + noisy + enticing + fear + Strateg + mesmer + uncond +eration +abilia + Ore + subjects +old +Beat +Pont +nitrate + vote + whirlwind +iranian + side +hey + Supporters +landau +arabs + Je +nikki + compe + fighting +nels + Candy +canadian + oriented +Related +umsy +nergy + progressively +lakers +QU + owner +commonwealth + homes +utter +rene + ne + Missouri + incremental +Af +ley + pe +siberia +Beta + alot + garner + phase + murd + mob + closer + Outside + mate +ohan + Guatem +rooster + stages +raz + Coup + citizens +gements + Micha +microscopy +kay +inflammatory +path +celine +非金属 + universally + STAND +biplane + Confederation + seq + flagship + superficial + 163 +inence + scholarships + poppy + Rhino + Basic +obb + Tin + wob + Ying +服务提供 +triassic + bailed +math + prohib + misogyn +star +massimo + outsiders + But +allegheny + Shipping +红卫 + transl + Zak +fights + Kenyan + winding + Lyon +duncan + glared +attested + ; + imprison +marjorie +TOR + echoes + Reign + Dreams +cast + nurse +John + Except +oise + suicide +Sy + personalities + Rational +ourgeois + centre +unct +nick +ontent +classified + experts +guineas +annis +hamlets + outper +xc + immortal + harming + landed + Lifetime + drafts + charity +opsy +Answer + Consumption +alberta +under + cry + practices + Peggy + school +thud + hops + Craft +Written + 271 +bugs +aku +lift + it + 196 +veronica + steered + Named + flip +aquin +owered + Having + gases + encryption +wai + dozen + imperative + recognition + hy +OL + Von + Number + explo + Amon +swick +alto + Kurdish + dependent +ideo +US + Robo +enes + archetype + decorative +immigrated + obligation + victimized +化瘀 +Gender +register +gm +odisha +ococ + aggress + Problem + develop + afar + tackles +這樣 +limited + hypothal + orchestr +第五十七 + Miliband +blackburn +Higher +Ros +thistle + SUPPORT + toget + Statements + beams +disc +leton +resources +plucked +uchs + shoppers + Phillip + PlayStation + veterinarian + OF + coronary + emotions + Imagine +letterman + Unc + foliage +khalid + Prep + texted +ix + norms + Pakistani +violinist +toc +CSS +decorate + wrink + Ocean + 500 + unprepared +jenkins +England + wanted + parcel + Five + rant + bill + thankful +arse + indent + bosses +itchie +researches + perfume +cause +ecast + Urs +reet + Earthqu +rep +interceptor + Minute + "{ + transc + Reasons +INS + Publication + spec +Phys +eta + frown +azeera +gearbox + trial + Photos +payton +parochial +anks +nuevo + 840 + Customs +sc +Damn + winters + Access +blooms + weld + EDIT + employee +analy + laugh +winnipeg + force +关羽 + incarcer +sworth + 601 + uphill +uary + deals +orage + e +hao + habits +tasha + Noise +domed +monary +rive + conduc + rabbits + commodities + Serial + DH + partnered +aleksandr + breakfast + wraps + harder + TF + bonus + torches +斯里兰卡 +ville +tein + extortion +ublic + health +enture + CP + tavern + compost +micro + uprising + Sitting + poster +byss + Echoes +quad + advertis +Min +iven + Colo +oppable + marketers +ablo + gathers + skiing + Catholicism +eponymous + opacity + projections + gui + mystery +archery +hushed +講 + cmd +acion +Wr + pinch + Alternative + slow + aud + Sum + sur + Sprite + Turks +horticultural + shortstop +Order + foreclosure +AS + Smoke +一百四十 + roles + echoing + evangelical + LO +IFT +clock + comprom + Greek +Hal + yearly +warlock + Kier +eks + sher +borneo + Donna + sand +gee +friar + Trooper + SELECT + Grip + pir + costumes +roma + Vin + safe + pH + inventor +tions + Amend +wilbur + motivated +Oracle +rowd + bridge + jihad + distinct + negoti +ussion + belly +張 +Graph + interpre +moil + Exam +Explore + hig + Boss + early + thou +rained + convergence +Mar + winger + lingu +uating +]), + Playstation + certs + extraord + Beau +roy + unaccompanied +uras +ottenham + runway + penalties +JUST +antage + newest +Iss +Bring + indirectly +selling + 351 + strains +stop +curtiss +Welcome +ivari + Local + Appropriations + jurisd +SOURCE + Alexandra + aggression + unparalleled + Filip +imony + reign +vamp +Rachel + throne + Reb + 1957 +pegasus + Electronics +Fair + DIY + asc +随波逐 +nazareth + Spike + donated +新疆维吾尔 +mark + MMA +guyana + guarded +Marc + photograp +uded + unarmed +eaturing +digit + AK +isotope +organized + charm + adore +luton +Thankfully +Columb + Flip + embedded +dormitory +Dialogue +aila +KTV + require + interference + prett +ulence + commission +Entry + retaliation +israelis + calam +VD + phrase +Public +pittsburgh + legisl + Recommended +alach + arrive + CEO +yusuf + 349 + Hades +swan + crack + Shields + compan + Oaks +arl +WHO +indiana +emmanuel + bouncing + crater + lyric + intervention +Glass +Is + Vish + Admin + Accuracy + percussion +hush + Season + shortened + postal + traded +vim +luz +方程式 + Kraken + loot + spear +specialising + URLs + Fro + topple + albums + Paid + Tay +panzer + magnificent +cornered +jonathan +mhz +ei + occupy + Set + improves + statement + Buzz + advocating +ucking + similarity + Military + br + effects +故人 +ictive + Fighter + Hole +Und + nationwide + pledge + gol +emo + bald + surn + UV +joe + utilitarian + PC + antibody +ridley +aryan + invested + emphasize +pex + concerning +loaded +swami + failing +geons +colton + flower + lying + enemy +ensen +ibia +istors +Layer +alex + blasted + Chapel + aver +ackers +gie + Useful +Sol + oceans + Grail +Jar +garde + Wem + fee +eatured + SAT +ITY + cooperation + Seaf +ita + supervisor + None + rendered + XD + adoptive + irony +Sch + neurot + Than +olyn +subdivisions +crossbow +anting + civic + Giul + disciplined +tavi +maximum + judiciary + prosper +itiveness + agitated +obl + Soviet + wreckage + Baltimore + Philosophy + Sor +tenor + extensively + exchanging +plug +Chat + campus +samson + tweaked + $( +商银行 + planned + physiological +unciation +ricardo + lock +orth +intern +fuselage + Vegan + cerebral +Wal + Weekend + apocalyptic + Metropolitan +symmetric + NETWORK + deficient + subscribers +kayla +steaming + fuel +£2 + statistical + clasp +nazi + freeing + austerity + Barbie + verbal +网络管理 +successfully +LIST +c1 + Norwegian + servicing + concentrating +berto + counselling + injury + NO + trader + food +creaked + created + cheers + malaria +mun +resa + that +beatrice +kN +Hop + replicate + stunning +uine +Apparently +riors + submissions + Guarant +ithub + flanked +inctions + sovere + imprisoned +ovy + RAND + Needs + Saunders + racist +maxi + Betsy +saturday + luc +asions +becky +mayfield +oding + calendar + resonance +captained + playthrough + emergence + Sci +ason +burse +ophobic +Rand +ector +铁路局 + executive +Iter + Capacity + haircut +rior + detrimental +GI + finish +国民政府 + trajectory + hack + Sour + veil + mixture + GHz +population + RT +whistled +ulz + Shepherd + sources + April + Registration + Alexis +montgomery + Safety + Legend + streams +psons + violently + predicting + yours + Schwartz +ente +quin + fashion + product +manfred +shay + McN +经济法 + Strongh + pond + cube +ults + uninsured + ponies +Enh +archers +朗朗 +Anti + seeming +Phone +隐形眼 + alias +guernsey + Torch +shape +omas +ithaca + Portuguese + Ingredients + Monument + downloadable + injuring + li + aquatic +forming + pounds + Lands + unchecked +ethy + notoriety + markers +lete +rentice + choice + enrichment + fiber +superman + governed + massive +這種 +Inc +uncredited + Presumably +reditary + minimized + Dangerous + support +salle + preparing + fret +ACM +antioch + Yemeni + mutant + consulted +taxa +tg + install + Conquest +嘏 + governing + TEAM + Authentication +owler + outlook +democrats + Sacramento + Step + humility +oglu + withdrawing +shu +sandals + alienated + perpetual +fda +minus + Vent + Amer + interact + Augusta +manson + mage + Ridley +dogg +atti + Sultan + ideals + Kit +hezbollah +avor +Fish + armour + sidelined + Pryor +saxons + amended + inert + Grateful + imaginative +confluence + wardrobe + Patent +izable + overr + cloaked + Bowl + Taj + IA +nino +uk + revolves +venezuela + combo + overpower + conco +House + slowdown + working + Retail +Fr +wynn + branch +tanya + HQ +TA + sprinkle + Contemporary + Cul + utmost +sunken + primates +ahar + charms + sake +iosyncr +Though +sina + Played + Revel +phase + commenting +zsche + inscription +serviced + Shiny +techn + petitioner + inline + graduates +idas + quint +darkening +cardiff +sit + hist + Compatibility + installing + stacks +abetic + Mess + Anger +axial + conservatives + toss + tougher +titular +smeared + Nvidia + province +kimberly + Bars + bundles + peaks + sometime + Tamil +Chicken +union + unequal + tap +capital + Broadcasting + Ragnar +arian + inmate + version +mikey +FUN + 238 + moan + entrance +curvature + places + vim + flashback + vines +hol +keys +multipl + aber +hra + webs +Ban + plummet + NOTE + neurons + Terran + built + turning +尔特 + Buster + 950 + Lack +committee + invoked +hodgson + ambition +Super + Participation + recovering + runs + cortisol +吕布 + copyright +leonid + Augustus +carol + CET +lines + originated + sooner + ejected + souven + millennium + Wah + credentials +Plugin +ES + Minutes +games + Lann +embed + Bil +enth +convoys +_. +Safe + condos + Eb +edo + contiguous + delic +xiii +anded + metaphysical + Would + when + stream +palatine + Guardiola +positive + doctrines +jogged +cfa + heads +raviolet + baseless +renumbered + Bapt +osborne + torque +panthers +太和 +scient +ahah +antes + lifting + PID + surging + inclination + directives +ometimes +謂 + drug +akery +marries + spaghetti + zoom + Grants +hartley + Electoral + Revolutionary +ammed +strings + SIL + vest + provisions + Julio +bitcoin + perfect + Bret + infiltration + Kend +HEAD + Blue + Great + GET + 113 +amation + platform + tales + nonsensical +pir +BBC +aris + unions +acton + Strikes +harden +tran +ucl + Yang +rg +craftsman + Suns + homer + points +Luke +Express + Musical + enqu + eclips + Trek + subsidies +edwin +FIL +bernardo + bott + Occup + Esc +Half + rapp +RS +jee + clueless +States +task + quar +nestled + narr +inic + subsequ + bourgeoisie + Everyone +lennon + radi + exhibit +selma + Filipino +Theme + beat +steamer + rend + OTHER + Union +alog + Grave +icago + powerhouse + displaced + Chaser + Gerr + protr + hem + criminally + Bounty +romano + Installation +uffer + Warhammer + Member + Twice +khalifa +Va +一成不 +珅 + Finding + Wizard + quantity +quentin +Krist + Shop + Moment +Comm + sweep +ud + rush +las + balloon +Hol + swift +thren + overnight +lafayette + Adam + Coleman +George + Currently + BOX + aster + pain + Carm +uner + Matters + Basketball + Nau + objectives + safegu + installer +campus + Fish +ready +众所周 + sorcery +von +Op +letes +according + raiding + ana + Quarter +elliptic + IDE +處 +Film + routed + exchange +arcadia + angles + Po +typhoon +asiatic +elsie + existential + inoc +akbar + breaking +masonic +corrugated +Il +poor +odge +polynomials + Ol + yo +nette + Back +Justin +Rule + Hungarian + "+ + Bi + double + rituals +orks + favorable + shelters +Looking + diagnose +mbol +aer + target +rufus +rek +periodicals + PASS + Ker + Butter + Television + crippling + guidance + inept +passageway + conclusive +Greg + crust +mick + selfish + borough + armoured +friedrich + educational +mix + impending +號 + Totem +drawn +madame + WI +nen + lucid +file + rallying + farther +isma + Suggest +National +our +朝鲜族 +必不可 + junction +holder +cius + Stats + intervened +veland + Ping +Special +manly +sichuan +interned + dice + provoking +secution +不复存 + mum + Glou +ilated +laurent +rays + LeBron + Coke + Posted +弯弯 +voss + Hundreds + Cemetery + airborne + metabol + hill + better +baptist + Sending +columbus + lounge +fter +poe +everal + Munich + agreeing + paragraph + dismant + metallic + supremacy + Subjects + Mell +Prime + Reading + opera + Stub +hutton +information + Portsmouth + translates +reyes +integ +IVER + Bond + runoff +package +itivity + CSS + Wait + lubric +gymnastics + erect +kie +mcdowell +END +iscopal + animation + Chocolate +montenegro + Parliamentary +irled + Parents +appalachian +社会保障部 + Fortress +uncture +geries +addin +ocaust + ay + Documentation +pring + census + tries + illustrating + GA +sheikh + Vict + Fernand + Vic + customer +aw + Orth + print +doorbell +mazda + withdrawal + lawful + longstanding + behaviour + 700 + queues + convenience +chet +Solution +lighted +xual + 249 + Rece + estimated +hepburn +edith + 510 +Sphere +男女老 +urg + imaging + wr + landsl + intrinsically +YOU + pals + vast +Via +prot + clar + Geh + Athletics +etat + life + vans +Spain + 338 +sight +cavalier + Used + Hew + sighting + flakes +itten +Picture + Born + Rescue +ummies +ibilities +glenn + OpenBMB + welf + Objects + cis +stefano + feudal + customize + palm +anderson + Nob + carniv +evans + deed + Newsweek +Joy + Panthers + vanished + nap + advises + priv + Plays +rea + pand + HTTPS + depart +ette +lected + 01 +dig + 450 + scept + proactive + DM + Stephens + numbering + AX + PRE +買 + TBA +josiah +scripts + prior + trusting +shine + adaptive +thanks + perilous +custom +ethe + desper + express + Bland + robberies +draper + although + cuisine +ictionary + Fold + Shades +ources + endowed + perenn +lez + trunk +mathews +karen +tunisian + destruct + steps +gary + Shanghai + Doll + destabil + Magazine +arrass + Chapter +desktop + counteract +windmill +第八十 +hermit + maximize +love +helens + dare + differed + placement + sucked + Solution +bolsheviks +fbi + Then +Amount +Wil +game + startup + updated + sweat +QUEST +olis +Facebook + premium + GST + Peb +thumping +zie + fictional + Sphere + steer + attending + appreciated + antagonist + Romania + 223 +guide + GTX +Cro + Felix +Secure + paws + Unicode + Gun +shankar +uron +kidd + som + trance + uncons + shocked + flavor + considering + Etsy +cut +benfica + Quantity + empathy + ESA + ventures + discour +ott + chemical +externalActionCode +ura +pection + reservoir + determination + IE + Overse +ikk +thorns +Destroy +ansom + abolish + granddaughter +glided + Started + Jama +houghton + Indian + trivia + butterfly +UX + proced + 168 +working + donation + Banking +ATTLE +nsa +silvia + was + tirelessly + someday +shows +caleb +Pak + Ra + precise + warn +MpServer + societies + Bake + calc +isal + Jinping +congregations +char +etitive + powdered +Flickr + Cr + curve +LLOW +osate + intermedi +clark + acknowled +serving + Charl + necks + Satan + monkey +himalayas +nme +自言自 + adversaries + Heard + scent + adapt + weigh + KY + Changes + assassins +Supported + Symptoms + Poo + ensued +salisbury +CV +IQ +mister + Friedman + diminishing +basel + Allied +redict + Tail + Nicolas + dysfunctional +Minor +rooney +bryce +Beh + embroiled +conf +aret + incorporated + Diabetes + Sou + Hort +faces + Snyder +lehem + render + transitions +heastern + submarine +anut + gotten + deem + IPA + urinary + WATCH + flavored + Herman +gulped +When +Charge +AU + totals + parachute + drift +etrical +fk +eals + handing +babylonian + Shield + Bought +shell +]." + commonplace + Nether + Remote +Mark + library +merga + ideal + ale +consonant + transm + Bright + missionaries +missouri + interpretations + scenes +ECD +since + Tab + worse + Chaff +Submit +難 +holland + Detect +Emergency + territories + aside +Wars + broccoli + Surrey + Dyn + occ +ocon + mix + bearded +conservative + flurry +Cra +pathogen + linemen +utral + tyre + deposition +recogn +lance +gleaming +hop +ubis + Prec +quickShipAvailable + specializing + Wealth + Baker + partake +mythological +extra + strong +Met +soloist + elsewhere +Being + Nano +footballers + imbalance + Nic +thora +neared +ARC + showcase + Kiss + explorers + guide +Ryan + again + solvent +ull +lausanne + rg + happ + Aram + instrumental + Bliss + geographic +canon +ifest + sket +brush + Serge +kong + Distribut + Identity +ynski + record + confisc +asketball + glutamate + Carnegie + XL +mustang + marvelous + leftists + Faul +robson +male +Fac + comedic + offenses + Butcher + Be + incest + cosmetics + Entreprene +sent + indication + 1905 + quitting + Gohan + el +activ + BaseType + sucking + reelection +gmbh + Costume +uddled +alison +第三十八 + composed +Nar +ountain + KEY + narration +克拉克 + Dram + unification +bloomberg +buquerque + pumpkin + My + punishing +expanse + Glow + overhead + nun +afrikaans +ivering + clamp + jelly +RL +Tes + Ans +soyuz + Avenue + Athen + statistically +Pie + 268 +lc + listeners +aida +umm + Dunn +ernest + arg +cemic + placebo +}: + workload + gh +Front +kaye + check +quintet + calming +ortun +watford +aid +paid + jails + grants +churchill + Faster + credible +fight + prime + engineering +wexford + Legacy + affluent +yun +budapest +flake + neurolog + guardians + Piet +olve +Software + intolerable + choices +despite +eric + markup + 310 + HK +df + smell + decisively + detector + ribbon + arom + specialize + wrists +ends +bees + pel + Sunshine + brokers +erving + synchronized +vre +nesium + Macintosh +artments + darling +amanda +street +giro +ijah + Guinea +yank + Tips + fires +ologic +ankind + Aven +issue + First + deadliest + Hancock + tying + ILCS +Friend + bathing + subdued + negatively +ilot +April + au + Messenger +Late + degrees + Lodge + scope + ageing + states +nesia +Sources +uations + 1975 + cure + rely + neocons + stronger + debuted +牠 +darmstadt + rewriting + Community + classified +bharatiya +ortiz + forehead +felicity + rotting +Rad +buck +表演艺术 + hears + Training + Pret + surprised + cartridge +icia + 293 + reporting +Throw + impossible +inal + done + creatine +kata + emot + analy +stade + finan +Out + Miy + frames +PN +eun + dwell +enh + Amin + Soci + DRM + sank + AFB +APE + periphery + Malone +midlands +knelt +usercontent + snack + Summoner +northwards + intended + Saint + populated +charlemagne +韩庚 +ADD +premiership + proceeded + came + conceive + dwarf + Colomb + ink + dissolution +資 + bulls + thirds + Osw + conquest +ENT + asshole + comfortably + Counter +fail +ful + CBO +somerset + Pap + screaming + uneasy + nations +huff +necessary + Whit + Haw +hamburger +cdn + Trick + Abs + headlined +ample +Wow +tyne + '( +傷 + IG + Sri + Panama +觀 +miniatures + Schools + contempt +cupping +hander +bucks + corrobor + sponge +rang + relied + Choose + Arabs + texting + backpack + downstairs +ternally +Getting + cities +teatro + TOM + 720 +Introduced + Pony + calibr + LR +bengal + predic +carmen +ulet +Spect +omed + Gears +Virtual + MI + Mutual + raging + Ghostbusters +ories + Bread +hermann + vying + Zimmer + DG + disastrous + Compensation +earl + HOL + AoE + marker + horrend + Dull + Phar + Desert + Spend + adventurous + employs +food + southern + Hayes +eli + Investments + caps +riet + volunteers + neon + rob +indies + Killing + heat + wore + helper +vegas +ahmad +atch + smear + Garrison + VT + sailor +done + Passage + denial +国际交流 +jurist +mediated +milano + reprinted + Lama + alleging + forms +portuguese + backdrop + bladder + poisons + invited + anth + insertion + mids +geoning +incial +caldwell +ern + Birds +abbling + crowned + Rice + sourced + sts + systematic + appropriately +backseat +iologist + orc + toddlers +YES + ori +yearbook +ets + harb + hat + Sterling + ACPI + Ass + Teachers +justice +balkan +囗 + cumulative +katherine + 1967 + process + easy + FX + Registry + skinny +manor + redundancy + XCOM +Sand + Release + buckle +兩 + engineered + familial + futures +pyrenees +fitch + constituency +dict +aukee + Pix + unh +chem +環 + equity +dding + MAD + supplemental + PCIe +noel +murmured + Lu + forecast + subtitle + Lt + Fortune + electricity + Carpenter +Team + Holy +merrill +kward +cranes + assigns +BS +rupt +数据处理 +encies + Midnight +fairchild + peanuts +corbett +二十万 + ambulance +natalia + Mub + Equipment +savanna + humane +mast + banished + parentheses + Sakuya + Ens + Nonetheless +注册资 + Grizz +Lat +omy + Shad + Granger + sid +Attempts +berry + inv +igrated + hardware + partially +iffs + proletariat + capac + Lois + Ally +NEW + procurement +etermined + los + vertically + patronage +deutschland +anta + Platform + Prosecutor + Tony + fade + victims + Michaels + brings + solving + underscore + purchase + todd + pope +number +ishi + Wenger + childcare + instinctively + broke + Condition + quest +cart +lightly + ampl + understatement + homegrown +arsity +Col +atem + peeled +townsend +fisted + marquee + Kod + ice +omaly + sadly + alleviate + appropriation +plus + Fitz + Yosemite + palms +imbabwe +bandit +kick + festive + staring + clever + edit +ogical +Cs + tolerate +leinster +jaguar + GP +maria + Environment + electronics +retched +vittorio +purcell +yue + unconscious +Scientists +uggage +二三十 + chickens +archduke +choral +astic + Truth + unreal + recycled + hottest + handled + constructive +ractive + Gust +ucked +Jewish +flute + Cookie +bernhard + Neither +eless + Setting +atrium +danny +egyptian + checkpoint +spruce + Rider + Hispanic + Lords + attm + cp +forearms +bies + pill + slog + reasonably + Britons +IFIED + pressed + Gund + Finished + CentOS + Eggs +beatty + frig + technically + Lightning + Guerrero +reigned + seller +Have +etheless + Omn +CB + needle + Hindi +spd + holidays + immigrants + taken + Pennsylvania +ket +evangelist + sq + u + Asset + contem + forward +Sav +cristina +新石器 +晔 + wellness + wil + afore + squeeze + attracted +Throughout +RET +lier + downside + vocational +低血糖 + proport + carve +校学生 + troubling + Fr +building + inertia + pre + Slater +cook +dat +jones + antique +missive + hydrogen +tunisia +opsis + Bart +weathered + overcl + cogn +zen + parse + Luck + messing +atton +fitzroy + redacted +jp + organised +faber + Eston + Cadillac +idis +aird + Shang +barlow + Wester +SAM +加勒比 + cone + legally +sonia + roof + scare +beak + uncertainties + meanings + Rust + probable +arcer +JT + Devi + Ac + Vij + Ever + agility + cab +EED +太平天国 + expressed + apply + crisp + Peer + Peck + anch +grimm +umers + Passion + mutated + dominates + Honour +wellington +Samsung + Shotgun + Classes + attracts +Ult + bible +arta + prepaid + vacuum + Says +Output +`. + emancipation +undeveloped +Real +Playing + 303 + warr + staples +greenland + Bitcoins + sugars +contin + CBS +Earth +icians +icious + two + Capital + pron + arranging + sliced +mis + mushroom +czechoslovakia + Ecology + hers +crime +polit + linux + northeast +Before + integrated + Barron + outcry + appealing +International +strip + AFL +phot + Rip +oris + Supported + ARM + warrants +mischievous + Karachi +class +BALL +ilitating + peaked + cour + valve + revealed +accordion + bounty + voiced +ning + Oscar +parramatta +prairie + Laws + Beasts +Hung + Euras + THEN + Kent +santiago +romeda + reorgan +Page + combustion + redundant + Juven + isEnabled +padding + love +bag +utical +Kelly + trophies + vulgar + preserves + overfl + pouch + booked +FF +quarter +Activity +oresc + Glen + incentives + Mandatory + consulting + rival + Dahl + corps + guiActiveUn + signal + maps +geologic +Sit + halftime + Benn + Median + Officials + Howard +Urban + extreme + punishment + approaching + patriarch +fours +Sum +francoise + tem + Stab +ahead +theresa +juven +adjust +byn + Capture +guam + Atomic +SA +olympiad + burd + traged + Enhancement +Han + viruses +opted +gymnasium + hurdles +slacks +社会治安 + isolation +abet +wharf + timet + adorned +umps + pony + illegal + conson + Balkans + armed + 504 +nah +ANT + Turing + permits + coefficients + retrie + Sag +println + stayed + Markus + arrays + Lucifer + Showtime + Farmers +nikolay +Maximum +Catalog + Sega +affles +sims + CW + thrott + lashes + Ned + immersion +coll + Ler + weapon + potential + wounded +icles + MARK + podium +Brazil + patients + frontman + Psychology +markus +inian +utah + murm +然後 +inoa + preced + alluded + Yam +dan + coincided + kidney + Govern + entitle +ncaa + sparing + Ethiopia + sense + detected + tooltip + pacing + mosques + ide +Ben + smiling +tierney + fashionable + his + maternity + 171 +perial + Anat +moon +hunt + Babel +icro + awe +Adapt +peacock + arbit + outstanding +pickering + 57 + provinces +long + Giants + wi +simi + arrang +ISIS +orses + circulated +deux + reinforced + possibly + Kah + inheritance + bourbon + errors + distingu +ensor + UT + decon + bitcoins +成千上 +timer + 230 +話 +reactivated + commonly + parking +lillian +general + Into + scant + seeing + magical +jamal + director +standing +ison + MEN + buffer + bubbles +bomb + spinach + frontrunner +reportprint + disruptions +okin + garlic + below + squash + deterioration +canning + que + Fabric + trailers + discrete + borrowed +tango +bertrand +gallagher +reath + KR + Pere +joining + Nev +tsky + Bellev +小行星 +niper +leicester + asked +cham +評 + bigotry + prefix + tf + gorilla +NECT + Char + illustrated +iyah + historical + ogre +footed + Polo +eland + flying +Ut +nuclear + biotech +a2 + Hydra + display +paulo + dab + House +iesta + Related +flirting +articles + judgments +Davis + reboot +thal +MQ +antic + gr + sim +edna +Psy +ric +ardi +airo + Ox + bonds + referees + Surveillance + Lily + agre +posure +rius +Zen + feast + Authent + Trop +rompt +ookie + murderer + Unknown +二百一十 + heated + Ricky +dorchester +Lin + denote + scroll + satisfactory +yoko +archy +ats + speech + Adult +snout + kg + Crimean +oint + hoops +den + Deck + award +hindwings + tablespoon + gastro + gender + Paradox + debtor +CHO +Obviously +igators + Rodney + tenancy + .) +Lev + shri + Stuart +thia +Figure +Downloadha +comings +Default + annotation +williams +DEBUG + cy +emporary +Hall +writers + Agents + Lau +clouded +random +owner + embodied + neuroscience +],[ +aurus + wells +parable +enberg + Nina +ligand + regardless +FL + JPMorgan +malaysian + Maps + arc + charism +terra + torture + Artificial + Luc + LIM + dissect + Indies + Rom +AMI + records +oregon + Ichigo +origin + showed + Hail + shores + stationed + Psychiat +nissan +spoken +ourse + mentality + Dia + download + secondly + hotline + fears + economists + Donetsk +RF +allo +grimly + September +safe + local + hunts +bhutan + ghosts + stellar + Tycoon + bleeding + sacr +classifications + hydraulic + redist +lyle + bart + diploma +eson +Roll +ortium +info +suffrage + term +OTO + God + adjustable + Electrical +erguson + steals + NI + Lear + 365 +uang + distracting + meat + rs + Honey + Chat + beck + pict + Basically + fidelity +ansas + pizza + w + Jensen + Principal +accompan + assessing +Kent +adoes + constraints + increased +孙杨 +Settings + Goo +£100 +enny +transform +alfa + indicates +朱德 +handel +iked + Apple + CD +Cert + podcast + allowed +masters + Tutorial + measured +fleetwood + 322 +brasil + Municip + aroma + chambers + riders +clinical +rafe +ABLE + Danger + Gum + inhibit +particip + everywhere + concluding + Norway +hazy +顺理成 + ripe +zzo +commandos + dystop + reputation + DAC + poorly +Nich +muslims +acies +That +gaming + Ampl + Boy + Yun + starved +ossibility + Rough +pper +mommy +perm +Dam + eagerly +想方设 + uptick + fossils + Ay +POSE +room +Availability +ect + zone + reactionary + Labor + stretching + weaker + Kislyak + Pro + Yankee + 540 + creature +Anderson + rede + raged +epstein + hesitation + Bed +tag + ep + preach + ak + Published + SpaceX + QB +illac +skip + eternal + manifests + Liu +ENDED + jug + Louis + Anxiety + awake + possible + Stadium + Bots + sculptures + NSA +lict + book +iciary + :-) +adian + age +qua +PsyNetMessage + Pyth + hey + Yog + COMPLE + crafted + menu + apparel + fundamentalist +patch + castle + Hal + Agility + sum + flare +ICAL + Rap + laundering + pledged +Given + 436 +mingled + mech + Elaine + tum + Tam +kruger + achieves +cola + pause +orset + demoral + Gael + spearheaded +Science + plac +olit +essa +anwhile + circulation + temper + Parkway + duty +rators + flew +wilde + surve +White + Santos + rails + NPR + Gym +format +uler +gora + Camb + Academy + discussing +rhyme +olicy + aspir + urban +Maker + disreg + robbing + Celebrity + exerc + Bradley +ls + Egg +omew + tom +wonderland +rafted + Rank +ifice + SNAP + coun +Within + Cruiser + Abbas +moral + dismissing + sculpture + Loans + Patri +Lab +慕尼黑 + Skies + une + acute + act +portage +Ultra + Franklin + swing +archeological +progress + LGBT +lucivar +addle +jubilee + verify + UFO +colin +nikolai + Triumph +odynamics +wolves +pebbles +Taylor +catalonia + Ryder +Cand + stage +dupont + polio + Tough + Avalanche + amusement +Democrats + Hunter + participants +丨 + Brunswick + retrospect +FLAG + decide + depiction + Mans + sued + Tigers + massacre + Phen + detailed +flash +Display + giving +encer + alas +Dig +気 +lot + personalized +NS + precisely +bread + Constitution + workforce +ove +okia + ooz +tered +kk + Human + discs +ables + patched + reserve + Shel + description + hangs + proposed + Scand + mouths + unforeseen + framed +十字路 +diego +讓 + arbitrary +exist + Ced +raphael + YES + 978 + Toxic + reincarn +Grad + introduced +Americ + Villa +spire +Ide + illust +serious + passionately + breastfeeding +Eight +bacterium +frowns +Ct +fernando +Spr +dL + Repe +adra +itudes +rou +fighters + Sponge + hub +uble +aking +aina + woven + complex +Cle + Church + sun + avalanche +心电图 +Episode + shredded +dizzy + Brill +Priv + groom + ore + Irish +clad + Adams + Syd + opted +documented +Reward +\< +osures + task + honor + McMahon + injured +umer + React + shrew + rows +iculture + tunes +amera + Ce + threw + entitled + appliance + Brooklyn + Andrews +must + constitution +inning + Summit + Corsair + prenatal + Fi + goof +riad + cameo + fifty +wood +integer + BuzzFeed +iful + baseball +abis + sheets +ghan +Mur + adjusting + thousand +europeans +bris + Watch + inexperienced + Claire + translations +aquino + dude + Kron + resisted + decisions +plymouth +solomon + protesting + Model + investments + Waves + Commonwealth + neatly +Direct +wise + Inst + fren + liberty + Yale + OPEN +shanghai +tonic + Reset + Sixers + Alright +SourceFile +elist +fitzgerald + mango +filipino +nonlinear + Chosen + clocks + 79 + Strategy +oco +fabio +oks + California + crackdown + instituted +Dar + gained + slower +philippe +neue +matched + gentle + Hebrew + destiny + deficiencies +mcleod +權 + absorb + Apply + Fac +zhou + Feedback +ema + Arts + Blaz + security +sex +ocking +圖 +processor +binoculars +ask + Months + Hedge + extracted +cartoonist +giving + Slave +Craft + densely + adjustments + stereotyp + lov + luxury +portion + 1889 + nightly + needy + lasers +uffy +gerry + Samantha + tied + insisted + rested + adventurers + workspace + Lynn +oclc +ultz +Snake +Driver + silly + TY + registers + home + Wolfgang + Amelia + dictate + Medium + sanctuary +LO +REG + wear +fiesta +smithsonian + dosage + starter +swam +keye + vegan + Pine + assemble + Sparrow + interests +fare + Fathers + months + TAG + remed + waiting + ESV + ar +lesi +fest +inding +deals +butler + Bell + dividend + Standing + EVENTS +ixtape +第一百五十 +Images + BAT + statewide + inqu +参差不 + defender +sorority + biases +able +verage + Mace +scotland +gunnery + hasht +会计学 +usp + spo +raits +medicare +cara + Dough + conceivable +過 +agle + palp +oup + morphed + devised + Angela + desc + Presence + buttocks + icon +commerce +greenwich +winged +lumpur + Stainless + bogus +brunei + Page + patented + Darius + slaves + blender +baylor + inches + narcissistic + ruled +massive + exper +VPN + kb +CAP +OM + sticking +wake + grounded + container + difficulty +ots +employment +oriented + implications + Food + tether + amino + notoriously + sells + pour + rage + actually + roots +program + insurer + meters + allied + ISIS + shrouded +john + Sample +trak + This + Mar + reflective +graham +zanne +anne +ipl +olan + 103 + Adults + Poss + intense +enegger + Mods + commissioners + ST +oneliness + Dispatch + Kon +ither + Strange + Xperia + ailments + outcome + tracking + cor +arth + billed +EY +Motor + smart + Was + CV +ishy +amiga + ticking +ional + aggressive + Legislature +storey + refuted + Ramos +picasso + != + Cup +Catal + CRE + requirements + itching + deserts + production +yss +uton +Speed + dazz + Borg +lique +Spawn +KC +pulpit + turrets +viktor + decimal + lag + evaluates +burnley +ICE + ethic + crept + multiply + Winning + stumble +indy + SPI + Hod + Hem + Marcos + provided +hyundai + along + reforms + deepest + yielded + SP +parapet + newcom + Context + Jesuit + reinvest + Slay +barnet + surreal + ups +UE + Begin + Annual + Marathon + frivol +jordan + founder + quarrel +十一届 + differ +razy + Chun +olars +transfer + percept +ction + gp + Credits + luxurious +nothing +inea + Warp + imperial + Ibid +efeated + clock + 340 +iasis +aple + trans +eaton +ireland + typo + protester + valves + Noah + clergy + Norwich + real + beers + scratches +Attribute + hidden + mascot + adjud +ressive + frost + cassette + Rath + San + intersection + Aim +cabaret + Alexand + Increasing + lost + 300 +reason + speeches +Ul +dubois + Ivory +ought +dobson +inction + already +wine + pigs +Delta + Reds + images +luminous +larson + renaissance +reviewed + Merchant + fractures +shuddering + stocks + Fisheries + front + governments + confronting + stabilized + Coord +Ol + Annie +四十二 + gut + phones + wilderness +wid + invariably + makers +selena + explosion +HAHAHAHA +kira + proficient +seychelles + widget + fascists +scientology + MON +avers +ified + compound + Guardian +ribut +killed +ulated +religious + principled + undrafted +upt + Goldberg +frankish +Or + Leaving + Kok + Paleo + Psychiatry +alert +december + Hilbert + Bridgewater + strive + glance + GB +ecause + Clo + because +assis + pedoph + shines +fenton +Vel + retaining + dog +inis +marc + Comes +Tank + Godzilla +ound + lyr +storage + encrypted + Conclusion +keyes + treaties +ITT + frustrations + Tracker + consequ +ESCO + League +cohen +germany + endless + laps + Dairy + Vault + Stephenson + contracts + Technologies +dockyard + and +expr +irrel + Kaz +azaki + auctions + PAX + Healthy + Performance + Announce + ordeal +clarence + consensus + aircraft + Tim +stormy +ational +Fle + Enlight +usat +AE +ICK + Expend +endant + Teach +walnut +brother + Denver + Window + lobe +mabel + identified +geist +IRO +igen +kyoto + doubt +whistling +ixon +afer + hamm + Powell + Surf +hap + Paras + Caf + snacks +Gro + hypocr + Butterfly + drivers + Libre + reviews + intermittent + Orche + polite + Gaza + novelist +Head + waste + music + excuse +ngc + pedestrians +lessly +percival +patterned + Harbaugh + opt + writ +garion + legion +aylor + harm +ocre +高性价 +davidson + readings + atrocities +Spider + barrier + Effect + happening +greg + Des +vironments + doubtful + soph + Corrections +livion + conduit + notes + engaging + warnings + suffered + clim + outdoors + craw +Bir + Follow +ANN + homers +tower +easy +courtship +Scient +aults + furnished + toe + Bung + VIEW + VAT + antim +herrera + Completed + van +It + turbine + Preferences +asia + Components + quietly + Cav +onds +avenge + appro + LIA + flight +mavericks +Although +嬛 + teaches + abstract +holding + telescopes +Dur + Angola + recru + Morris +omial +tremor + inacc +artz +Fore +brennan + Lev +Begin + School +notes +auld +yep + Intro +abyte + GREAT + amps + authentication + carn +canberra +thodox + boosts + Brew + couch + Champion +sniffed + talk + purchased +orescent + Anyone + fishing + indis + Suicide + portals + commits +erc +periodical + merge +duran +citadel + performer + Influ + 11 +whereupon +irit + Jacobs + TDs +crimea +Winged +oy +AIDS + Portug + minimize + operators + flyer + fung + hesitant + Nevada +roanoke +regents + Knock +Stream + Unix + Match + county + proclaimed + Stall + Hamp + Aurora +Le + disadvant +ova + factual + progresses + apparent +uction + relevance + captures +Planet + sow +esthetic +OTAL +autobiographical + funeral +metal + cited + inexplicable + contributor + Ages +Trigger +rers + aw +ibus + backups + Synd + entrepreneur + decomp +vulcan +Gov + legit + pall + heavyweight + hero + section + publicity + dozens +Harris + mouth + works + glue +onso + inviting +將 + disparate +prakash + Perform + presumed + Tillerson + subsequently +當 +综上所 + axle +choi +vu + imagined + modes + Days + 83 + LOC + Annotations + eb +uvian +uncle +emic +EG + Bun + young +izational +raw +click + reconstruct + metrics + Nadu + Vegeta +dk + Titanic + ideally +潜移默 +recomm + EL + impunity + jokes + GoPro + doubts +LECT + downloading + te + 148 +FFER +romeo +isites + Explore +HO + these +淼 + postp +第三十三 + HOUSE + ts + overwhelmed +sirens + downright +jamaica +ABC +tacoma + sleeper + carpet + faiths +olution +aunted +ealous +headed +Battle + Conduct + Higgins + Suk +reputed + Ton + mislead + 1987 + ya +cobalt + Ex +唯物主义 +paying + antidepressants + ETH + revisions + recl + Brand +west + s + harsh + Species + vector + retreated +ngo + Dancing + FA +odynamic + meg + interpretation + Explan +thayer + lantern + prudent +samantha + Ministry +chatham + kept +ilda +dion +euclidean +VEL +monton +Server + anxious +Sin + humble + dd +BIT + WORLD +alogue + parked + corros + Sabb + Appeal + Expand + Party + Third + midway + Vendor +Attach + dependency +ila + +frequency +eto +alfredo + weights + Elves +indal +greeks + cooking +wendell + 266 + Blaze + swallow +aviour + resources +pict +orm + Gunn + projector +EngineDebug +phant + transmitter + pedd +greek + budgetary + looming + Petr +Magic +警察局 + pos + Construct + Pats +isf +detachments + stake +parry + Mk + cached + \ +pillows +Part + SAR + expanded + Mah + expected +rand +ocative +toledo +bish +labourers + rumours + dossier +roach +noir + adapted +Night +絕 +sikhs +moroccan +CI + began +feel +ERP + 2003 +lids + pollution +Hyd +izen + presenter +anted +amar + interim +somethin +HL + colossal + succumbed + erected + Arn +ken +tel + Turtle + Exhibition +uty + Ult +edmund + Moving + multiplication + kettle +users + Nintendo + claims + ;) + kit + CIS +dame + Parenthood + Cluster +Arm +sheldon +northamptonshire + Making +sorbonne + Feldman + Mel + DOM +frequented + outfield +ologically + waterproof +duke +Georg + til + MMR + }; + Lose +ociated + Hil + Sergio +dell + lashed + CONTIN +ludwig + enchant + sinking +Secondly + facial + Festival + compressed +isabelle +natural + teammate + liquids +Enc + breaker +andise + troopers +rosalie + towels +groaning + Cere + powered + Hass + cards +sic + familiarity + Insider +hurrying + nowhere +close +Length + Cummings +Nothing +node + speculate +ering +chartered + inventory +ploma + cloud + blew + mem + recognize +yton + evolves + twice + villages +Host +edly +wall +huey +ghanaian +Mid +March +stage +craftsmen +Market +ampire +antis +hetically + Manufacturing + contacted + Bombs + whore + unimagin +ensical + compute + revive + Strategies + likeness + regulatory + Mechdragon + plus +wrestled + firmly + 1951 +atel +elaide + Nun +relinquished +acious +VCD +Scope + fisheries +forcer + Democr +Nobody + Office + elite + scrub + Rather +ologist + Bam + Nord + conflicted + Gate + Loading +flags +Si +darting +flag + sustaining + attendants +acceptable +lazarus +cial +alicia + Ike + Slate +raft + enrich + organizations + Reverse + presc +vertices + Lennon +eneg + Liga + comprising +Cons +sinatra + dx + Institutes + privatization + Odin +republics + cocktail +suffolk + Lei +mL + ge + ascert +Vert + title + Cards +canucks +mohawk + deprived + LCD +changes + 820 +NBC + mitigation +agged + navigate + bout +enz + break +minimum + Improvement + src + overt +四十九 +Status + Directive + Wei + Macy +enges +cesar + Intelligence + grips + chest +imilar +Tile + Aub +saudi +highly + increment + epis +Prev + pushed +aned + Bh + Lor + Sandra + Ci + belongs + Shake +enburg + pupp + eat + Hu + Vatican + borne + Kelly + Wong + Finger +EDGE +southernmost + forthcoming + reviewers +endants +xia +RIC + HBO +dietrich +idges +aturday + implementation +dammit +Kat + ' + gasp + MTA +holy + Route + answers +OSE +peacekeeping + suicidal + isolated +complex +uberty + Assignment +unix +Street + sunrise + hospitals + Son + educate + sty +banded + qualified + ther + Remember +kensington +歡 + Paraly +Pattern +Every + APP + delta + ii + physic + Speedway +Love + planetary +macon + discusses + Living +lund + Alvin + exclude +ouver +blems + wears +rankin + blind + Torah + Prevent + tenets + classy + recall + relies +ioned + infused +scl + minion + Morsi +bellevue + Papers + PET + essence + grape + rents +徳 + sarc +dans + guaranteeing + 373 + disob +higher +ibrahim + Tuesday + certificates + stained + Tas +lenin +JM + experien +韩信 +compatible + BEFORE +weed + LH +elisabeth + DAY + Fraud +elsh + passport + Incre + See + digest + sp +ancel + ape + Marine + Taken + Content + Split +bots +merce + Ride +valle + Yorker + relative + Barnett +IU + chance + Sno + organis +estinal +frac + Zhao + Smile + Jan + pairing + erupt +inery + Mont + prisoner + friend +assed + words +netherlands + ounces + integral +VERTISEMENT + OB + alerts +equ + Drive +harmonica + Kom + Pra + antioxidants + crashing +ME + Fork +ibraries +harris +shuffling + multit +armenia +Notice +fleming + Door + Paulo + PsyNet +oing + echo + Hull + Satanic + plugins + Fire + Covenant +armor + Welch + pled + containment + Frank +yout +rique +第四十二 + propel +midwest +adder + dispatcher +ajo + Fight + subter +ongs +oras +newscasts + anecdote + Shock + protein +節 + lifes + exagger + Beard + harness +Sunday + announcement + plurality +aggregator + Falk +group +cecilia +hiroshima + Crist + Schl + weeping + Colony + dinosaurs + encompasses +atching + versatile + throughout +kerr + Crush +Agg + brist +Div + Holder +HK + dependence +broad +aquitaine + peers + cigar +amacare +起重机 + COUNTY +rhodes + perpetrators +Quality + Thornton + AIR + Pale + Has + arms +pere + Rapids + Vag +Fund +calder +authored + whims + courts + exports + Aff +第二次 + Killer +CEO + Simple + Kiw + pursuits + Rum + Gulf +seton + bitterness + unus +ovych + punitive +high + shock + Though + LIFE +bow +Leader + Hag + electors +Take + Crom + Documents +rimp + puzz + stuck + friendship + 46 +Bench +elly +Variable + Citiz + Sierra +練 + rapper + tam + origin +sama + exert + abound +cylindrical + maneu + Rite +springsteen +wc +必经之 + beet + characterized +zaragoza + shrugged + efficiency + unst +invariant +whitehall +reynolds +icism + programme +ZA +Standard +vention + goblin +Ter +curtis +IENCE + authorization +Kansas + 520 + Homo +functional + providers + ninja + grievances +honduras +felt + Property + Cage + Oper + Jak +Oak +fb +loving +somerville + explor + package +mented + Jake +ellington +legged + Tulsa +shift + 42 +Invalid + commodity + interest +zzle +eters + lax +endif +Opp +rarily + declass +香港中文 + kicker +没人会 +sofie +etermination + vir + producer + fractured + Sacrament +Huh +jet + colour + Meeting +saturdays + actions + airliner + dust + bonded +itcher +addr + overdoses +primera + Coast +第四十九 +jerry +MK + rocked + complicated + collision +Accept + numbers +rists +Finding +oskar + lap +nbc + miserable + encompass +ouched +mir + omission + divided + External +saloon + thy + Odyssey + determin +oldemort + computed + OS + fasting + colourful +lacey + enabling +valid +dowry +TAG +gestapo +amed +argon + Benny + discretion + 313 + tal + emulation + Nir + Imper +oir + Fantasy + GT +shreveport + mana +ewski + notice + Cub + convers + motiv +tko +gabriel +patterson + Carroll + insects + bargain +atlas +ippers + felon +ependent +matter + Shining + Disp + Lamp +Tue +rosis + demise + encouragement +lost +AV +casts + ceremon +oreAnd + Potter + Amtrak + felony +governorate +sten + nerd + manag +ernandez +share + wander +Mas + Godd +johor +VB +nolan + Lar +clud + max +筆 +Soon +slot + Ripple +search +mercantile + MIL +Disable + Earthquake + sphere + Duffy + Bale + Famous + 1830 + death +orton +System +izz + blues + parliament +humanist +摩洛哥 + 21 + Maggie + Inher + tournaments +jacques +Label + long +beale +CAST + Megan + disciplines + deviation +buena + explain + unsuspecting +pieces + Egyptians + cycling + closed + HELL +艾玛 + cutting + Shelley + Todd + prepares + differentiated + finishes + Pl + Mattis + explanatory + Nazis +makes + Chromebook +reens + Arthur +成吉思 + rave +orah + Bull +Question + stone + Men + merits + greatness + catch + invasion + AL +cooked + colleg + Hz + stimulation + disappearing +zn + Facilities + Remain + coffers +unes + Halo + logo + 320 +erala +calabria + educ + Civic + drib + professors +benefactor +WCDMA +Tags +anol +edIn +ador +Feature + neighbours + healthier + Generation +sloop +flower +saharan +Bottom +bala + Naruto +lilith +ath +gables + Ultimate + arrogant + map +lucia +oustic + Clarke + Books + eldest + 1898 + muscular +rox + cancell +raq +european +upload + caramel +時 +erenn +iblical +enders + framing +faerie + restrain + Junk + protections + Nose +ferreira + Hastings +nielsen +terness + GPUs + cocktails +johns +遊 + whatever +Performance +retion + Uni + Bubble + specialization + measuring + copies + roaring + warfare + negotiate +condition + summary + cere +mcpherson + Graphic +PLIED + threshold +enamel + overs + Pets + horribly +ize + Cena +Universal +diane + Order + buffalo +Overall + vital + border + vict + adolescence + patri + TRE +願 +female + nominal + inhibitor + Primordial +ium + Rule +iest +emonic +irth + NEW + Been + Sew + improved +introductions + Milton + teasp +usters +bone +copeland + unpaid +podcast + subscriptions + topics + posit + awoken +minist +臺 + instr + Strawberry + holdings + dipped +thel + dialogue + Gender +Blade +HR +santa + Reggie + cook + vel + degradation + Vec + Linear +dozen + forks +ticket + creator + Parts +Hamilton + Fruit + emperor + Cent +bobby + Scriptures + Gay + DC +byron +xavier + Hayden + wrote +不请自 +coliseum + sy + swallowed + mit + selector +tiffany +kelley + Discuss + Older + checked +MIN +Lic +[" + configured +fumbled + currents +hon + Alps + rigs + Mikhail + Languages + 62 +yuri +讀 + untold +ertodd + look + merry +馬 + hit +hardt + Should + buzzing + tad + Locked +wembley +otional + Dino +orum + 54 + Polk + "/ +itar + Precision + sep + Brendan + brace + hungry +orter + 84 + Gordon +lasses +harp + Sovere + wiping + coupling + Frost + lash + Logan +]' +sash + communicating + WARN + smash + residential +header +causeway +rown +inducted +chenko + recruits + Alphabet + DE +ome + rep + worst +francis + Madden +hen + 1946 +cookie +awan + pump + moderate +barangay +heng + Amazing + ruth +PB + outbreak + crossings + Sheila + Bills +fis +lisher +ibles +rhino + Nar + cleans + */ + informal + semen +allows + TWO + depri +Integ +algeria +atron +double + Ey + Mex + kills + tools +Reg +對 + banks + vigilance + arteries + Psychological + agree + ubiqu +egal + FOIA +inth + lobbied + feasible +ucks +batsman + economic + temple + Hear + thesis +clover + dentist +呎 + Paso +NT +currently + insulated + Insert + riff + calmly +uit + liver +gambia +elight + convol + gast +Fox +oz +valkyrie + Hybrid + classmate + Pipe + Former + Pers + PT +Companies + 1981 +Bernie + intoxicated + Ubisoft +Mars +ionics + Carmen +amus + Chicken +ballroom + exchanged + led +cory + prescribe +wallet + Cros + spreading +fecture +ornaments +Only + Sage + kindred + circadian +belt + Falcons + Czech + insufficient +leaning + soared + Improve + pursu +boris +教职员 + (* + contestant +syllables + assumes +docked +delhi +allen +tully + Stock + Kramer + obedient + Sadly + PUBLIC +etudes +allic +thorne +inca + restart + harmon + taught +Registered + solar + err + Marilyn + jam +Besides + Rein + turb +geant +wessex +ylon +det + AR +frances +男盆友 + deportation +Coming + Boe + Curious + Immunity +pea + monuments + tanker +suzanne + Zoo +adena +PK + grieving + Error +ilibrium +brentford + 398 + fu +everton +karin + dehydration + representations + Swift +craig + BAD + satellites + Kickstarter + inspectors + orchestrated + cyclist +static + compliant + Analyst + postwar + mounts + Police +oton + mindless + Advent +Wra + Ness +nery +illusion + Women +anya + referee +gallo +vuelta +peerage +conserved +Re +Coll + donkey + lured + Cars +Column +indian + disconnected + Sneak + indifference +lois + aft + Rav + Description +serra + Terrorism + Browser + Constantinople + fandom +grasses +樣 +reclamation + forg +Mini + transferred +oters +facades + newfound + Low +proven + Mand +essendon + sort + td + pigment +baltimore + hunters +Engine + Spect + activities +erry +enger + relaxing + Chr + Bezos + testify +sonian + releasing + incumbent + partisans + Kang + Delete + suing +khyber +grasslands + year +党和国 +opez + Development + senseless + plentiful + operation +due +Star +rible +backs +ographed + buttons + arguably +ARA + loosen + Discussion + Pry +night + immediately +ritis + somebody +grande +Render + Frie + 1897 + File +formed + bulky +cert + Tai +Fear + devout +pires + 1840 +francesco +asures + shelter +perry + Joyce + Zeit +珍妮 + CONTR + BAS + wholes + Flask +他們 +opia + predictor + rein +ascal +ann + hierarchical +ived +ucle + guests + aggressively +BN + 580 +chopin +Nat +冷嘲热 + rates +Climate + heel + defin +Limit + gritty + pseudonym + understandably +与其说 +Ag + Urban + grids +ring + hots +confederation +uts + canonical +acers +Mayor + levy + Austria + poss + Dayton +iona + acron + Clive +Prom +headlining +eca + Darling + Fres +jammu + melanch + spur +reth + Vs + spirited +bessie +phans +信息网 +ellectual + segregated +iop + RAD + unidentified + entertaining +udos +adv +amel + config + psychologists +IVERS +ight + airplane +ervation +achilles +urdy +iro + tact + teens + Edu + derived + chat + Wind +cleveland +essler +export + crocod + encode +Typically + Publisher + consideration +ifications + secretly + advertising + consequently +solid +nora +housekeeper +perth +harald + Overs +timothy +mongolia + Tool + dwarves +mechanized + CH + checks + Nate + efficiently +seeking +augustus +anski + relat +apesh + manipulation +chn +renaming + Missile + absurd + Rory + Pike + PRES + bitter + resc +便携式 + capabilities +enaries + weird +stre + respondent + aust + unve +quarterfinals + Thick + "< +TAIN +kwan + federation + eleph +dave + ineligible +Delete +yamamoto +legate +isbury + repro + mansion + scrolls +Honest +ATURE + Shi + conditions + inflammatory +arp + sustainable +itime +alid + stripping +animal + cultivate +chiba +lexington + aluminium + sender + sends +rank +lication + perl +christians + Kun +闫 +colby +STER +defense +athy +ERY + deserving + miracle +CRE +ffect + Ment + ISIL + Charter +Desc + Cake +rhymes +產 +show +ivo +itters + Lumpur + empt +hyderabad +alloys + spurious + servings +fit + Kodi +Education +Sword + Minerva + gym +connacht + indec + Baylor + Adventures + twe + cooler + coup + primitive + coy + complicity + philosophers + bubble + delight +irens + Far + extremists +elta + affirmative + coer +olith + Vita +LER + Life + Costco + eradicate + regul + Armored + runner + Becky +inflamm +ez + gallon +arettes + KD + scathing + vapor + move + 980 + CNBC +Ger +一无是 + ZIP + Hendricks +darryl + SCP + members + budgets + Block + legalized + mood + resembles +oard + ric + View + Songs + precautions + radiator + Arctic +}); +mentation + lineback + tsun +console + Huff +yarmouth + performances + fructose + stretched +oha +monty + clashes +occupied +emort + taxes + BEN +irs + loans +Texture + Perhaps +}, +mca + hypocrisy +Either +coulter +sudo + Omaha + directional +pete +result + OEM + Crate + Drag +composers +amily + Epic + BI + Before +medallion + Cover + Cosmetic + NASA + delegated + teamwork + canal +caressed + forgiven + culmin + Drone +ename + McCain +huron +spp +malibu + kidnap +ussian + bitterly +yrights +擇 + exaggeration + idols + Creep +otta + Cherry + fists + Instant +中國 + Ruler + guiActive + announces + misery +Hong + Protect + Carson + STATE + MLS + mascul + si + Curry + Thr + Flavoring + hate + exits + used +tted + Colors + Yuri +cavendish + Communists + exorc + Clean + suffer + wildlife + concealed +together + tu + Sk +hana + Yorkers +rog + Lac + reflection +robat + tone + docker +prix + pc + sque +overland +第三十一 +Cause + privately + Elys + acad +TAT +forest +manufact + incapac +felipe + exceeding + Xan +DE + Magnetic + transient + undo +madonna + reckless +ises + distraught + arsen +hend + offender + evaluating + Handbook + breed + newsletters + dotted + Howe +chin +NN +阿拉斯加 +iners +Register +plaques +itf +cientious + liberate + tabl +日新月 +reuben + witnessed + Appeals + Speaker + shaking + Victims + fi + Sus +translit + ro +yates + scramble +ouston + tomato + woke +mason + IX + shoot +benedict +hand +Els + hal + floral +amazon +Builder + Jerry + Ro +jupiter +subterranean + Hope + floats +sheets + unworthy + brilliant + stagnant +hood + Temper +against +iguous +stats + gardening +ictions +llo +pres +Information + hordes +Loop + ultr +amate + Melania + penal +idays + xx + instinct +toro +inka + hacker +dwellers +adow + typically + 41 + slides +Toronto + famed + bot +mostly + Beginning + Dek +glee + kids +ordering + Kenobi + Meta +scott +user +subunit + emits +IST +UU + academia +agons + symptom + oven + WATCHED + LTD +colo +icult + Melt +utics + labeled +ilyn +menu + redeemed + M + screened +iesel +mrna +mand + Locations + Cutter +enda + Plants + Charg + Crafting +adding +表达方 +Tax + routes + FI + participating + thunder + Naples + Cannes + intermitt + MET + rent +trans + Sales + forgotten + debating + iP +anking + super + Her + Wonders + coverage + operatives + subtly +este + confinement +parisian +ashed +maps + Out +HF + saliva + grading + desires + inmates + jarring +钜 + directs + aim + Ser + normally + Piper +reported +awaited +henrik + cabal + Chips + Dubai + egg + photographer +Expl + Plane +ronics + cane +ORE +onomous + Cats +leopard + modify + perpetrator + Every + deserved +mx +monterey +obile + truly + utilization +反应堆 + pyramid +orchards +christina + Discipline + tomatoes + Podesta + Near + refrigerator + anonymous +acement + thoughts +chu + aliens +wimbledon +atur + contaminated + delicate +e + unimaginable + perjury + conceal +orp + respons + hover + inauguration +addison +CRM + sing +AGES + repairs + scripted +nova + absor + salad + Clan +jay +iterranean + pull + distributed + glory + exist + vectors + rules + Surface +illed + NPC +FU + destroyer + consuming + Years + Tal +Crit + enormously +rants +文化名 + artwork +人太多 + drastic +Att +erion +trent + colours +ottesville + religious + registry +placed + Butt +blossoms + Syn + Reflect +二十几 + Spokane +thailand +iber +ophile +BACK + responsibly + Ti + Beg + regenerate +palais + appreciate +Sax + Consequently + field + Youth + Beans + belong +Target + arm +Hist + Pediatrics + buried +gam + 219 +endiary + bog +worked + under +uting +uint +tamil + lineman + Athena +iced + Shar + MUS + monetary + srfN + insert + Jupiter + mandate +upon + prescription + Processing + chick + Just + MID +rolog +idation + 16 + coastline + stat +angola +scottish +lloyd +burning +gin + "" + influx + eligibility + prominent +pract + WAS +inguishable + Tu + scarcely + transportation + Parkinson +rani + Psychic +motown + strengths + influenza + spoken + DV +myanmar + Won +LP + Appears + binds +Trivia +hetic +balkans + Wraith + tires +mace +cellent + criticisms + mu +required +井冈山 + atroc +upton + saturation +november + pollut + suicides + commons + patrol +array +四十七 +ordan + sleeve +EW + vandalism +tracy + disturbing + Conservatives +cheung + Brach + Crusade + countless + Legendary + 209 + disclosed +oths + contentious + Experimental +pieter + whereby + we + Carb + cringe + Sophie + 160 + Chart + combined + Assets +wilhelm +htaking +imens +Yu + wavelength + elephant +ially + Marines + racists + solitary + binding + ker +raj + interceptions +section +umbled + DOD + Argon + RO + Mis +BT + Germany +bayer + Truck + Hitch +commotion + Exile + cheapest +UFF +VIDIA +国家自然科学基金 +excluding + prophets + convin + Salvation + bonuses + furnish + Budget + mosquito +irement + Rahul +scill + Body + adjusts +macy +paddington + victories + 451 + THEY +kimball + Shoot +put + Jung + predators + schedules + Chemical +role + Football +moving +christie + mig + revenue + examiner + concedes +uka + Dagger + har + Sandwich + Laos +ceans +英国皇家 +obituary +Port +walled + analyst +javascript + Establishment +ental + Racing + rept + robotic + neighbour + externally +reunite + mobilization + Cannon +Usage +League +rosario +wednesday + Plains + reform + Intern +viola + alleged + Lucky + percent + job + revamped +fellowships +brat +cromwell +Tact + brand +社会科学院 +mouthful +ainers +pigeons +faye + spreads + NEWS + peripheral + antidepressant + recovery +vernacular + Owens +isode + Hegel +silesian +sinn + condo + correctness + ~/ +gor +akron + Legal + erupted + Cooper + Prot + Peoples + Loss + Governor + none + consider +geoffrey +mustered +stamford +escorting + secretaries +Then + Agency + pearl +aii +高高 +ulty + solidly +arlane + Hats + Bloom +icho +IDA +joachim + throughput +mecklenburg +imentary + inaccur + Aden + horrific + Daw + weave + reiterate +owitz + cans + detectives + believes +rihanna + fig +ipes + Cry + triggered + Murdoch + bon + conflict + wildfires +{{ + illumination + lifelong +thirteenth +emate +American +rms +they +cyclone +cantor + vanilla +louis + Throw + Qing +Opening + touching +normandy + burg + career + subord + Gan +incline +yvonne + comparisons +frankie + trademark +Christmas +oakland + spotting +二十余 +chennai +orate + Orion + towering + dogged +anthologies + holistic + antis +eep +chaplin + Scully + atop + necess + Schn + bean + Klux +sudbury + Blizz + rhet +jozef + flagged + negligence + Year +tracking + Mold +shillings + Bridge +comment + discriminating + Signature + Olivier +stars +ersive + occupied + Table + Combine + Return + earlier + speedy + performance + Baghdad +malcolm +rence +inez + legitimacy + allowable + mats +utra + yacht + Whitman + vaccine + Replace + PJ +adi +interpret + repeating + Wu +ective + Play + stiff +bukit + goddess + Hume + rule + selves + kiss +shamrock +million +omo + jury +commissioning +Social + displayed + Damn +Entity +Heavy +岙 + erratic + Struct + quit +imon +olas + Effective + scrape +shoot + cling +count + Joseph + designing + rejuven + Vern +Feel + nerv +roph + demographics +suits + collegiate + occas +ursive + soldier + utilize +terminus + workflow + SG +tirana + Made + Bahamas +pay +abd +iron + migraine + overl + Haskell +allel +hing + moon + intensified + Senator +opined + dign +rpm +ryce + pop + unavailable +Published +skill + rode +foundation +Christopher +netting +爵士乐 + essay + resolved +Mine + dubious +間 + 64 + acquaint + tablespoons +党小组 +apical +israel + Adapter +二百六十 +blast + Award + amplification + INFORMATION + Berg + 396 + analyzed + guys + briefings + Wit + avg +Advanced + ballistic + Collins +hedral +argyll + plaster + streak + fragile +weapons +marquess +venant + goalie +temp +archangel +Orange +polytechnic + Knox + Katie + neglect + Azerbai +ual +Gene +Evil + scra +iste +avant +snorted +volcanoes + biod +tailed +kas + sensitivity +无时无 + symp +baronet + Eclipse + Noble +canterbury + Akin + Vlad + wizard + cursing + escort +Netflix +marino + leasing + Jon + rulers + realised + sushi +oglobin +Database +Phot + Supreme + Lal + 217 + lesson +kari +prep + collection + Air + Mehran + projected +nothin +Platform +Dragon +parc + gigantic +Problem + probe +acterial + Ins + Realms +render + shocking + Austin +ado + fres +glendale + 490 + Murd +divinity + concede + Dh + featured + Systems + Klan +guy +Kind +thom +stipulated + parental + span +Quite +saxophonist + Kahn +downs + caching +飯 +mid +第二十八 +IX + adventures + hinted +DEM +tc + tart + stress + (); + Disneyland + owing +rasped +css +Firstly + * +Sort +verified +BO +Administ + aggregate +meier + squared + Based + ABOUT + Payment + abl +Thus + constructor + mindful +aughtered +BOX + 2012 + faux + Warsaw +liber +metic +homo + Vital + tribute +resses +orchard + Pok + shuttle + Moral + dis + searched +NI + plunder + surprisingly +blackpool + outspoken + Reboot + 124 +Quant + Mean +wx + dancer + homosexuals + unic +Multiple +sequence +patricia +beckett + dumb +ramos + DP + Ford + reactions +aints + pastoral +ospel +Purchase + strictly +Frameworks + Muslim +anton + median +ricular + Fury +atell + formulas + brackets + guns + Cricket + Kimmel + Leaders +Dir + Industries + hump + Fees +柳岩 +cons + sport + needless +hokkaido + dumps + pim + 422 +letters + Zach + Dickens + 448 + Enlightenment +torres +mares +arizona +East + 301 + ions + Murray +baronetcy + Sens + Jack + incident +Depth + Twenty + gif + chair +jennie +ently + Monsanto + Reson +straight +acqu + Sevent + Mexicans +ORTS + CAD +shah + mentors + Emacs + TC + caravan +gaston + eve + uncont +raped + assignments + silence + Greg +crocodile +Tem + bureau +gravity +paper +vermont +absently + Marble +tnc + Klopp + laying +Cath + horror + Rats + decides + workouts + Dietary + commenters +Haw + receptive + seekers + Pot + Head +sphinx +glancing + synchron + unequiv +rat +money + centered + ashes + Earn + plug +DOC + influencing + insult +hicks + Eng + tort +Roberts + 550 +hotspur + tomb + passionate +udence + combust + Springs + astronomical +Dri + ETF +uran +frankfurt + authoritarian +termination + 166 + activated + Tes +vv + Excellence + camp + Joel + dry + Sere + misunderstand + KT +buchanan + revoke +pol + asthma +establish + cleansing +webkit +cai +parlor + 355 + Era + Einstein + angels +ulator +oline + applic +breathless + reports +喜马拉雅 + sects + robbed +Alt + staging + Gadget + Gong +ogun + 432 +maker +tests + app + Oscars + entrepreneurs +mie + CLIENT + Petraeus + pellets + Director +南方人 + Opera + event + limestone + shenanigans +Jenn +Icon +eruptions +IDs +Ms + Wooden + DRAG +seconds + piece +agine + Ortiz + canopy + transcripts +yth + Cheong +igraph + Bon +newbury + vulnerabilities + Lund + presets + sqor + CoC +bohemian +seek + inferior + Gas + reopened +antics + Worth +mates +venture +seater +细思极 +Tyler + wiser +foot +best + Lover + amount +毅然决 +tokugawa + Invisible + deficiency + campaign +idently + hotter + Rise +remastered + signs + Lives +units + concentrated +`` + Pascal + 128 +versible +quartet + habitats +brooks +mlb + rodent +Try +ights +fusc + Purs + Rutgers + sensation +Rs +ek + Ronald +iencies + Edge + reclaim + Another + Kap + STEM +carr + formulation +Line +mesopotamia +Gear + 255 +achable + Bow + Dem +惨不忍 + AUTH + antics +无神论 + Interest +central + retri +Start + deliber + mac + Plex +smith + tantal +Sov + wont + PCs +NRS +fiction + Prime +cpp +Kate +maroon + brew + fronts + fulfilled + anymore +ceed + smoother + settlements + Nug + stew + big + EPA +donna + Nico +ryan + backup + le + contents +凤姐 +stilled + onto +clipse + inward + varieties + judgment +aido +Norm +reeves + youngest +HAEL +panting + processes + neutron +jogging +amoto +heller + unsur +係 +chool +ammad + rains + Premiership +ogene +harlow +chico +Oper + Dresden +Union +张卡 + Nancy +rification +whig + opioid +bid +Termin + lambda +This + Vision + within + Memory +tao + Warranty + risked + Programme +claim +landon +Twe + sn + Linda + psychiat +Life + revel +antennae + busted + plausible +Council +Xi + follow +expl + panic +Young + HOR + browse + Bar + inspection +urations + amazing +canadiens + relates +Study + enlightenment +apter + Hawking + whenever +verde + supplier +quezon +lionel +page +droplets +woodward + Ur +mathias +warwickshire +mckinley +Nonetheless + persons +Volume + purposely + sorely + lab + Methods +friendly + wrongdoing + pays + outrageous + mini + terror + Peak + hemor + kidneys + disingen +ield + Maced + bloodstream +CLASS + Formation +julia +ught + Settlement +对外贸易 + Stun + Grind +Dun +Staff + Vladimir + Fiber + comprehend + 350 + MOR +bulgarian + graduate +achelor +四十三 +Originally +vas +cise + embarrass +carlton +emilia + Pablo + cheered +OC + Came + Samar + 475 + rosters +carey + RAM +eros + Dexter + ($ + simpl + keep +Kick +第九届 +dominique +shrill + vit +阿波罗 + Neph + auth +elo + squadron +otic + entangled +oscar + confidential + gear + reinforcing + inh +htm +uania +coe +bai + Appropri + brightest + Dirk + niche + teleport + Leone + Progress + Golf +ilon + airs + oversaw +behind + Cities +orphaned + DR +夢 +orically +認 +OTHER + entail +linnaeus + compounded +Remote + humanitarian + Round +grayson + Malta +luisa + base + 181 +寶 +snowfall + cle +], +veh + barking +asp +baja + Cob +rao +Fine + paradise + tainted + rounds +RANT + manic + Force +COL +ground + Maver +頫 + Inferno + resurg +導 + Wa +seq + brushed + Pacific + examples +Louis + provision + Tian + indicator +player +ATIVE +firth + CrossRef + Rs +olescent +plings + sock + REPL + Inventory + unnecess + Guild + Ard + shower + pitfalls +keyboardist + canned +richard +stanley + partly +bringer + heavenly + retired +Mass +millie +on + cig +Ext +enery +Thor + often +isite +armory +ancer +Photo +iren +Elf + bar + intolerance +moldova + online + discredited + attacker +Basic +tod + Grab + NES + Cutting +horatio +wired + predictably +chi + LB +oul +mill + Dynasty +Jean +yr +Shipping + output +lm + Steele +karate +olina + Cincinnati + pack +reddits +Discover + Nikki + shuts + AI + Recession + festivities +osh +finland +doha +reetings + GPA + rock + accusation +VG +雷克萨 + underscores + HC + notebook + USDA + Pam + 00 +avier + 2007 + Obs + definitively + screams + Path + contamin + Div + FEMA +Minecraft +disneyland + eru + neighborhood +AutoCAD + Virtual +atican +塞尔维亚 + Serie + Manufact +haw +Serv + denounce + count + Users +dogs + reportedly + Rural + Episcopal + Maiden +fischer +acist +ounding + PSP + fielding + technical + 187 +played +lawrence + doc + targeted + perspectives + Form +黒 +willie + newsp + Opposition +Contract + lur + blocks +berries +LT + Sporting + Visitors +assian + carving +ioxid +heric +aced +urated +layer +錢 + Tobias + Weld + Wh +edict +shipyard +ous + Sikh + Doug + manga +ourge + willful + bananas + churches + spice +Close + continuing + cursor +designations +iko + prompted + Pax + Bryan +ebony + condolences +非线性 +Rose +ighed +deluxe +strode +github +density + ingest + sightings + Tactics + Burton + peak +Ret + relaxation + vetting + lighting + identifies + surgeries + chunks +safety +TEXTURE + Rivals + Availability + spiked + airst + rabb + singles + licensing + itch + aroused +istine +ALLY + Eps + crystal + Boards + thickness + Detention + Category + Curiosity + fault +llan +irc +conduct + Pocket +郑和 + Veget +lled +aggressive + histories + College + hauled + Mask +uctor +費 +mrs + most + cross + ventilation + specialists +GT +icient + suspic +academies + privile +drugged +fund +三十年 +kB +osi + attract + gre + Iris +MBC + sticks + bushes + Brittany +Yang + dissatisfaction + nature +arabia + oust +Commun + sterling +ov + brutally +hms +darted + uber + utterly +pora +sav + cap +eisenhower +Among + regarded + autoimmune + spokesman + padding + imagine + satire +OPA + suppression +ernels + aggrav + Fou +oat + VI +eugene + integrates +avez + RandomRedditorWithNo +melbourne + Eleven + subscriber + 2011 + icing +GA + request +sign + learned + denouncing + Guidelines +bethlehem + Gems + 419 + varying + (@ +dx +license + divide + Mafia +Israeli + Puzzles + Sty +]= +item + Gundam + Spr +Mem +thump +roid +skipper + Sonic +rise + stranded + Bl + fined +medalist + for +mt + faults +tak + brands + fascinated + behav + impressed + thirst +whose + monop +Mount +athering + Conan +winter +isch + Argent +arov + TNT +RNA + hospitalized +华东师范大学 + jersey +Video + represented +efe + Terminal +expressway + superiors +二十九 + Dwight +murmur + COP + Diamond + Sh + Bowser + Minister + troubles +三百多 +umed + Yue + combos + Fired + shifts + market +tracks + Nut +Government +ctive + crucifix + teaspoons +fairies +hent +umably +RR +並 +awar +Jamie +ounded +epsilon + Major + Guam + What +樂 + freezing + invented +outside + motorcycles +topological +spielberg +etimes + true +鲶 +Website + Pompe + Habit +Madison +ingly +estrada +topia + Against +god + dw +iru +teenth + Kitchen + Article +alties + User + 101 + Beckham + Thailand + graveyard + know +very +apollo + mapped +pinto +fits + INF + jeans +misty +zeb + evangel +ilings + Stability +ezekiel +olate +Hyper +polo +ible +esame +marion +ometers + separatists + warming + Sec + Forces +obj + prices + gen +iar +Ultimate +Reports +POS +slaught +顺其自 +Scotland + manifesto + ballot +archer + Katy +Rus + Whitney + amazed + UNIVERS +emis + Mock + battling +loader + Cruz + sequential + liberated +ruction + 1916 + Gre + Feng + Carn +ferrara + cloak + Plus + CITY +palatinate + estates + Karma +anc +belgrade + compiling +esta + Sne + oral + Wrap +misc +limo + Bas +媽 +bug + RF +Room + david + invests + knew +government + Peyton +erg + prank + Secretary + annual +ads + Everything + seriously + cla + Disease + Herald +venth +rew +dispersal + possessions + whale +国家专利 + Latvia +mao + var + Aki +GP + reinstated + Whilst + intellectual + ISI +sahib +isol + partName + hikers +cf + respir +Fly + Indust + Atom +anger +aho +booker + Bang + earliest +orters +rities + Beach + penetration +calhoun +photos +ATURES +ILLE + climates +老干部 + 131 + temporary +rawler +esthes +純 + simulator +anthropologist + fry +Results + multiplier +hib +Jones + processor +TED +ports + Stri +cappella + les +uning +browne +Pri +COLOR +后现代 +hub +电热水 +omore +timeout +cradled +Bra + serum + Mondays + 05 + heinous + Jeanne +ection + overlay + Cassandra + quicker + Favorite + Md +como +antry + acupuncture +rock +Attorney +Sad + trail + Faith + clich + turbulent +merritt + repeats +Defense + uphold + exception + Recorded + decorations +第五十八 + thicker + Mick +password + prophetic +emmett +vicki + Kev + Starcraft +夠 +seductive + art + Author +adle + Standards +goodwin + Rivers + creates + standing + generator + remind +foreword +ara +Brit +Ver +有理有 +Syrian + telecom +modernist +ASH + smartphones +Equ +clench +icide +rices +Houston +asean + Christianity + unfinished + handler +rix + Vigil +madeupword0001 + need + desperate +Screenshot +idental + Families + verdict + saf +在实践 + Stamina +Associated + qualifies + voluntary + criticizing + participate + okay +surfer +Courtesy +operative + RIGHT +Use +ordinary + twists + Qual + remembers + statistics +judy +twitched + Humph +联谊会 + owned + Stephanie + secured +一无二 + creatures + martial + deceived + abundantly +eding + Saban +One +untary +hackney + Accord +ambo + Ernest +Ep + Swim + wisely +aning +angel +Chain + orphan + Scy + hundred +disrepair +bowen + counted + gatherings +Paul +opers + Dian + systemic +absolutely + 141 + hilar + fortune + activate + bashing + Brid + sweats + Moran + mocked + nob +SIGN + Collabor + photo +aldo +clinched +umar + cd + Rib +KU + Joker +Autom +federer + HUM + bumped + Public + shadow + frying +specified + Bog +syed + Guilty +arers +Got +albeit +owsky +OUR +every + deposited +一九九 + attributable +alos + Religion + rewritten +owen +osph +zhao + Ellen + conquering + ruling + specimens +much + moments + Ups +Burn +sai + bo +brightened +kathryn +ocrin + begg +Guard + Jr +subspecies + suitcase +cedar +robbie + plaza +ivic + crim + Profession +redding + cooldown +fukuoka + Mahar + thence +esi + pathway + sul + brisk + outbreaks +hume +idium + fraudulent +namibia + wonderful + Arabian +乌拉圭 + Patterson +Ser + cognition +gladys + account + beg + pointed + Socialist +respected + advant +DXImage + pref + Wildlife + optimized +显而易 +zimbabwe +三十三 +browns + on + complying +reassigned + Scots + anger + AV + obscurity + brazen + accumulation + Humans + moisture + due +weightlifting + Buildings + younger +furnishings + nursery + outpatient +isl +args + Logo + Factory + fighters +roc +dani +isation +tic +kt + gam + algebra + perceive + Infrastructure + seldom + fat + genocide +velt +imen + jumps + horrors +spence + Found +yna + obstruction + Assassin +fangs + aversion + bump + Automatic +吴亦凡 + comedian + awards + dedication + Anglic + belongings + gays +gly +dali +louise +chnology + playbook + yesterday +Warning + Console + deleting + Healthcare +gangster + nineteen + Standard + horr + fundamentally + Alv +entious +crowns + Thieves + preferential + yourself + memo +vertigo +Poor + Goodwin +concepcion + catchy + hoax + puzzled + Akron +specific +一心一 +conservatoire + drip +抗衰 + shades + urges + navigation + sleepy + positioned + crystals + democratically + 650 + Whats + clutching + Fool + Sutherland + supplies +barges + depth + GEAR + Combo +agg +Beck +inement +Less +beethoven +resource + Chief + preempt + wisdom + piping +vez + 104 + vaccinations +Instead + crochet + batt + Following +mannheim + unlawfully + Fountain + exploration + Expedition +isaac +slavic + spaceship + "_ +elvis + mag + acting +alf +tier +foothills +ality + accommodate +irty + Casino + recommendation + dst + understand + saying + Petro + Shots +icum +Minnesota + Samsung +稹 + Reloaded + ELECT +orpor + transformation +clusively +Alright + doorstep + outward +pakistan + Corpse + decided +excavations + df + Woodward + Breaking + depicts + capacity +turn +強 + Ax + area + emb +植物学 + bolster + initi + righteousness +lane +Reddit + Scala +pearls + clubhouse + Christmas + templates + twent + darn + unquestion + flair + Stress + gravitational +kant + coalition + frankly +Bit + "- +camp + vain +acy + Meaning + Huntington + ble +southland +北京电影学院 +Hill +postmaster +Jes + SPORTS +uy +非营利 + 1870 +dreamer + depl + Stevenson + UKIP +第四十五 + Wrestle + uploading + scaling +works + limitations +alice + Neigh +billings + Safari +rann +faced +Closure +walters +inals +Ent + Constantine + Albuquerque + Views +useum + Elements +xinjiang +Hour +crete + wast + thigh +models +itunes + outdated +iat + marketing +OGR + terrifying + 92 + Thanksgiving + grey + Core +印度尼西亚 + prest + modifiers +doi +miscellaneous + Corruption + Prior + page +augusta + substantially + Wow +Interestingly +meath + Slide + Solo + Min + asynchronous + continuous +inc +frescoes + enriched + taxp +Hide + antibiotics + Emma + Freedom +mccain + gar +rote +Beginning +rees +added + ripped +keenan + gotta + Esk + OMG +Record + contradiction + Aging +Tool + makes +ito +reorganisation +erville +TION + supporting + Toast +wolverhampton + Skin + Hiro +purpose + cycles +armand + Roosevelt + > + Ahmad + exponent + endorsed + tell +icate +helps +gender + CJ + independently +nazis + Sul + pm +katz + wake + sway + supplementation +hooper +MA + Jen + Bend + discrep +enigma + deserves +bur + Smart +breaks + nudity +prefect + usage + jQuery + text +觉得很 + Hom + Survivor +annah +lishes + LLC + proving + prom +hovah + carbohydrates + Punch + Guinness + saves +wing +乔布 + asks + ./ +Ar +iolet +維 +urga +bruno + THAT +ottawa +handled + Indones + assumptions +学位证 + Bok + wiring + Beast + infl +memphis + Alph + lookout + tomat + COUN + uneven + IV + hedge +syndication + Dialogue + Invaders +cli + vault +WR +polymers +仮 +hid +rutherford + substr + calib +gar +empl + transported + mentions + accordingly +guillermo + contraceptives + Ich + unm + whist + conj +妳 + Enter + dizz + announce +idy + investors + advisor + refunds +presbyterian + advertisers + dumped + Americas +viii + horses + Kl + informative + globe + climbing +cumbria + gamble +層 +ivism +icken +bois + pores +Document +papacy + Gang + Scarlett + las + Such + Jane +freud + unless +auburn +ushing +itant + raw +ented + Skills +Their +TING + hex +ifference +Decl + Vanilla +astor +melon +deciduous +smiley +standardization +caliber + conv + microwave + attaching + pipelines +icist + imposed + Bowie + Shine + automation + positives + Pon + Jaune +series + Protector +cend + whispered +backed + spit + Lizard + fluct + failures +Neg + '[ + allowance + dedicate +gradation +SL +horsemen + \( +bonnet + ozone + Angel +Pr + 24 +give + Financial + Bella + stroke +ANGE + Likewise +shouldered + complexion + mamm + entertain +eth + ah +afternoons + hippocamp +Resources + manually +sarah + steroids +install + teen + Actor +bubbling +Neill + claim +ipeg + Nora + wit + portrays + jung + Antioch + Indigo +catfish +underwood +perch + consumes +encia + cataly +reza +vy + Cantor +Air + Paladin +ansson +看不上 + Bagg + Representative +Occ +Pre +ORS +zman + Himal +ossal + Awoken + rationality +Texas + Cases +Ess +uly + Plant +omet + Gavin + conclude + brunch + inch +vladimir + polished + Tories +rotational +三十七 + watershed +ullah + evoke + bankrupt + resemb + fish + Chim +chapels + sofa + bland + Cycling + des + commerce + innocuous + serial + Resources + Tro + pharmaceutical +marine + wave + Stat + UCH + Russian +ottage +oaks + marketplace +daniela +town +u2 +enf +beirut + Sanskrit + honors +rhea + Translation + refugee + purified + vary + gum + investigates +single + modeling +Sup + arson + Inspired +肯尼迪 +boards + crowds +india +uster +MSN + Dow +XL + porn +Jesus + clauses +chid + nin + whoever +guantanamo + ma +eddie + Walsh + eternity +fergus +cle +fiba + shortages + eaves + hurricanes +shimmering + Middles +Toy +arij + enclosed + juices + Serbian +萬 + 428 +Posted + psychiatric +Contact + Labrador +andres +erve + BT + firefighter +merlin +nevada +郭沫 + Getty +budget +graph + seventy + Iceland + Fol +repeat + marriages + Patton + spanning +ural + returned + Johnson +walkers +聯 + bait +edd + cartel +ake +WAR +greene + Alleg + pane + rele + Amen + {\ + Az +icals +auspices +emia +carolina + escape + 1886 + Shak + bandits + Compos + frust +Alias + HI + sys +erik + trying + Fug + 1904 + Armor + pipes +thus + Soy +Long +modules + machines +orrow +imal + unwanted + London +bad + Combined +uren + whichever + Pharma + blockchain + Faust + demolition + jewelry + recorded +ension +oria + Piercing +Wo +itation + Sur + Den +YS + back + Wrestling + In +minh +Print + Throne + bro +AN + Mari +focus + Pose + Quin + rall + axes + conflicts +enum + AUD + coaster + Vanessa +enter +uct +inion + Matthew + reintrodu +Mail + Sco +playfully + Survival + Wink + bool + Jamaica + sorry + HIGH +quality + Jackie +Shape +dmitri + Adren +字里行 +Premium + requirement + notices + unknown + Trailer +priory +sect + disorders +Mode +Cl + iCloud + occurs +Cloud + Maker + harmonic +eworld +oots +Prim + Feld +rollment + amber +omical +baidu +epend +forums + If +izon +headwaters +johannesburg +itual + Owen + dism + Heights +Index + Breaker + dwind +Georgia +spin + ire + Guant +zeta + restraints + veget + Grand + Webs +sighs +criptions +odon + Iranians +TX +aley +ifier + Wizards +West + exposures +Some +grandmaster + prohibiting +sburgh +idious + Malaysia + Policies + XP + knows + must +Original + internationally + CDC + 410 + Acquisition +mbuds +augh + fired + hosts +ennial + assimil + indef + 110 +comments +ulhu + blazing +comte + boils + counting + inevitably +quant +fluttered + courtesy + Conce + peaceful +icester +Bo + Berlin + Charlottesville +quiver +reys + Keynes +orius + adapters +radio + softly + 205 +ako + Rocky +module + Cable + hating +arenthood + demons + sal +uber + intertw +Com +global + descent +Led + Spray +holmes +newtown +Ken + secrets + bizarre + contingent + association + Meditation + case + improvis +excelled + rejoice +starboard +angle +kept + Krist + opp +alien + emphasis + 1945 +restrial + admittedly +idx +avanaugh +inator +coiled +niger + bought +Fri + merc + York +shop + resembling +matically +outed + stemmed + drafted +disney + Pistons + dreams + sweeps + weak +either +lucknow +styx + Frankfurt + desperation + greenhouse + Asia + conscience + Holden + SN + Trader + Stronghold +liturgical +edin +bronx + assert + Highly + wages + Countries +kies +quist +learn +第一百 + Terms +意料之 +otomy + undermine +picket +UV + iPad +Word + Et + Ara + Practice + McCl + solves + unle + Ask + Math +mccann + Massachusetts +auto +trembled + filings +Mer +famous + Eminem +aero +达尔文 +upper +gunslinger + tackling + infall +SBS +plain +insignia +Em + playoff +ony + prison + pregn +airfield + Dream +clamped + tweaks + bum +Iowa +Giving + Massacre + simplistic + Garc +What + Bot +ween +lincoln + sne +otide + immedi + entrants +duchy + groundbreaking + Ctrl + Read + promotes + brightly + Ivanka + Scotch +boyer +none + Thrones +General +eyed + hates +ipper + About + nood + fork + unlock +Sp + Guys + Regina + Stranger + shockingly +charleston + idiots + plates + deliver +aughty +gru +carpathian +checking +arant + cer + Passed +inski +tara + Colleges + diamonds +CAR +Counter + propelled +aten +FAG + Chuck + grateful +thorpe + Utt + War + nonsense + wondering + MH + authentic + rad + Root + affiliates +olphin + shorth + decidedly + fictitious +SN + nurture + effect + galleries +acky + Glad +gregg +mcnamara +aces +terrorism + Clayton +TE + property + recalling +View + doomed +beamed +hurriedly + alcoholic + Richards +cultural +elect +humboldt +issued +original +ldom +factor +symposium +risis + PS + appalling + services + buggy +atered + Cec +ariat + Transparency + Augustine +solids + ted + Mob + positive + feminists + Balk +fremantle + imagery +ushed + Critics +Est + reflecting +共和党 +LES + Lebanese + gle + five + WS +christopher +Gs + academy + Discrimination + anatomical + Cycle + HIP + Those + resolves + vanity + casts +inda +csv + Mages + Bio +into +puting + asses +wb +Techn +Wire + analges +stro + Becker +yond +inical +Luck + compositions + collided + empowering + million + Student +greece +roche + Tale +lei +臉 + 07 +IES + blockbuster +Domain + documented + noted + Magnum +FER +Ball +rfc +opl +mouse +Below + Investigations +hale + repertoire + holes +ashby + Freeze +horseshoe + guessed +brids +mildred + devastating + unsustainable + libraries + sketch + Meat +Nice + Christie +vet + fib + usual +posted +radiating + Domestic +ilt +COMPLE + libertarian + including +amos + adul + Brom + allowing + yogurt + Tem + istg + Causes + minus +output + Clause +resh +rotherham +OUN + lightning + Represent + uncle + dolphins +choirs + Andersen +hover + AZ + cu + charges + ceremonial +清晰度 + Baseball +Lua + flavorful +entertainer +kildare + lone +isha + criticize +Rep + loyalty + anus +Philadelphia + disdain + Manhattan +istry + Doc +hollywood + miser + vow + illustrate +projects +elias + Courtesy +suez + complicit +EMP +Currently + Dove + sorted +sera + genders + ambassadors + Consult + Cant +EXP + observing +runner + funk + Origins + embody + replied + Western + projects +raphics +OOK +lau + Koreans +KNOWN + stab + plays +bear +unn +miocene + Nexus +osaurs + Title +raoul +riction +Democratic + dips + whistlebl +anging + minority +bolt +hook + prevail + confirming + combat + breeds + balcon + conflic +agy +第二十三 + conception + disposed + Boat + Sure + Fans + CONS + affection + pret + Harrington + eyeing +MED + Lowry + disillusion + Scor +convex +formerly +sabre + Ltd + consistently +MU + Calcul +radcliffe +ulner + steroid + Falling +plumage +Electric +inf + raises +yunnan +Var + Priebus +denis + kicking +Mega +gotham +operation + 140 +Ale + R + ethics + eSports +tumbling +valerie +requires +第三十二 + sag +Yes + cohesive +whitley + Gothic +Student + narrative +oric + cameras + knocking +Ly +back +bengals + Olympia + hideous +传播学 + renal + misdemeanor +VW + She + forever +iroquois + friendships + crunch +Adult + billing + tourists +pret + Hall +朗普 + Nicholson + finest +ometry + Mankind + mph + builder +landish + genuine + genitals + Maurit +Region +anus + complied + Study + plight + treasury + unhealthy +traditional + Dustin + juice + disco +ags + prope + ordinance +willoughby + severed + HERO +petersburg +subdivided +垅 + lurking + 1918 + beneath + Ter + compiled +aleigh +AUD +bobbed + perennial + assuming + Gat +jill +ister +abl + leaked + hostile + training +rest +sonora + generic +amir +anders + Joined + Trafford + dise + instit +volvo +xander + Karl +tale +polka + Hi +greenville + Soldiers + His +Bang +ificate + Dynam + easing +瘀 + Bod + album +sudan + Yuk + dioxide + folds + se + Witnesses + nationalism +itate +unincorporated + Orchestra + codes + Mechan +lee + congrat + Paul + Nikon + husbands + payoff +iddle + Falcon + witness + Extension + enrolled + King + Rex +文化大 +PE +tenance +Look + Niet +akings + Gingrich + Abyss + strength +bingham + Borderlands + Levi +assent + yak +Hunter +variable +itle +brigades +orship + subsistence +Input + disproportionately + violations +cars +eph + Tinker +Basically + Correct +jody + WRITE + Roman + interrupted + VS +istics + lich +shed + Nit +miah +INGTON +ricanes +child + Friendly + Cisco + Marino + symbolism + fatigue + forfeiture + leak +Gre + refreshing + mont + Dim +eastman +oddy + trig + 352 + Dam + semester + sacrificing + skim + burial +ocracy + harbor + orchestra + Syracuse +Strange +Hur +WD +allow + Closed + curing +vector +nu + Circus + 210 +eteenth + eq +Tokens +Qua + Contract + 1800 +名誉会 + 211 +arms + greeting + rendition + Buff +aney + feud + thrill + warrior +married +需不需 + Ads + homeless +apor + tunnels +ths + Miguel + Blasio + Hundred + Mit +Fla + fiance + ping +ints + Mac + Government +氽 + man + Recover + Picks + Reprodu +muffled +jessica +stil + Maryland + Matth +holden + Use + Sn + decade +mson + dancing + view +hickory + bias + intersections + Okay + Sahara + Libya + Motors + intention + 2008 +mermaid + Spotify +photo +inates +scooped + Edmund +Interface + unsure + stresses + += + spoil +pratt + hostages + hay +franciscan + traitor + seventeen +tallinn + grouped + shalt +olerance + Road + Sloan +anye +Grand + concrete +powell +coyote +uing + hybrids + subsections +ignatius + prote + Send + stamped + wording + Essence + fused +immortals +hq +league + decoration + Lydia +sparrow + unlike +Tea +uddle +pharmaceuticals +FIELD + info + Spaces +zyk +kaya + Obamacare +Spell + antioxid + corrid +液晶屏 +sportsman + times +ecd + caption + stunned +ipe + overloaded + illuminated + STOR +bradshaw + announcements +omin +connected + institute + told + preschool + defective +rebranded + guitars + Diver +der + JSON +same + Golem + Guth +michele + Healer + robust +falcons + 363 + keeps + flirt + slam + air + fly +cesare +guerra + Unreal + 1893 + Jace +Clock + Merlin + lucky + glaciers + 2009 + vocals + Hungary + uncomfortable +metics + Somebody + Contin + teased +Code +monsieur + innate +auf + impressions + boasting +venice +MO +cbs + coron +hops + Lars +nuremberg + Chinese + resusc +sing + specializes + questions + brakes + reminded + Script +murphy + EVENT +Robin + Sz +ahon +nelson + GIF + document + Donation + Keyboard + leading +uz + updates +ils + col + youngster + predatory +WH + Hum +FI + Melody + trillions + modelling +dismounted + behaviours + Nim +Class +三十五 + bullies +sequently + speaker +flicking +independent +ccess + ow +BSD + Twist + Cold + retrieve + lateral + politician +aptic + Somali + dub + policeman +snarl + heal +aman + Choice +Doug + agreed + Grace + Four + simultaneously + Snap + fathers + revolver + traditionally + wat + equally + 1860 +aneously + Alcohol + moderates + Stamp + Kul + knitting +專 + dreamed + Chomsky + dungeons + Peninsula + cautioned + Deal +granville +innamon + Tags + gossip + Poison +kosovo +skater + Interstellar + Alice + characters + masse + Engineer +bandages +ATER + commute + Thames +pulitzer + grammar + Corner + liked + Tanz +cil +Forward + disagree +atm +hungarian + infuri +afafb + dormant + madness + Software + introduce +such + Walter +lai + Roof + spectacular +criminal + kingdom +SEC + struggled +heat +commando + Atmosp + boundary + receives + chasing + Soph + Poverty + 1895 + reading + acceptable + continue +kerman +amine +members +https + leagues +over + protesters +Update +Manchester + cloudy + Laura + rental +viet +onal + flooded + coerced + porch +gallery +Okay +grain + agile + gain + ``( +igun +ACH + maybe + cull +caracas + rash + Hyp + gripping + written + yellow +squealed + explan + Bonnie +Notable +region + resize +FORE + horsepower +filled + unsupported +革命烈 +equality +reach +gc +albrecht +kV + Carnage +neumann + Targ +mcdonnell + bleach +isan + Pick +ittal + courthouse +Jennifer +日耳曼 + Sent +casey +edged + decriminal +ibrarian + worthless +atives + wavelengths +puddle + esports +motifs + IMP + Types + COMP + Balls + unilaterally +Swe +plc +rss +syriac +Leary + USB +Quotes +lunged +achers + Sha + debated + jar +ategor + contrasts +embroidered + rivalry +features + village + concerted +attled +initial + tame +brandon + bedrooms +ITED + dont + mom +Row +struct + centr + professions +tube +ptr +ELL + snake +zhi +georgian +illegal + Mormonism +]) + gad + camoufl + Mek +ahoo +indonesia + Toys + NFC +escent + distributor + Score + acqu +ult + Shell + thereto + bron + pedest +lde + accidents + medial + celebration +del +copy + syll + Retirement + packages + Barb + needles + springs +friedman +desmond +orde + tongue + HEAD + contrary +naissance + PROC + Past +efficient + Xi +Claim + hose +indianapolis + theor + INT +northumberland + esteem + Napoleon + Post + Empire +ylene +nhl + Charge + originating +Writ + constructed + especially + breeze + tv + Ven +political + assembly + 297 + mul + bigger +source + Ensure +endix + Jay + sexy +ousse +ilage +bh + knocks + Mellon + Elite + Williamson + infertility + intric + ip + stimulating + ceremony +Added +理查德 + spar + NT +owski + Poe + fierce + horrible + Bruins +hughes +togo + envision +coupe +ophy + verbally + Comprehensive + trendy + forbidden + dwellings +endi +Muslims + histor +omsky + Semin +wit +cery + Eye + classify + prog + Hunts + rename + web + Nuclear +iosity + sensibilities + thanked + MN + advancement +突如其 +declan +iership +azar + Anderson +brainer +Flow +uality +iatures + Cooke + digits + Neuroscience +DotA +Crime +IB + compressor +racted +idelity + Chronic + Star + Reason + Ginny +Run +belmont + Raphael + Conservation + premier + regeneration + meant +za + blood + approving +bett + Hamm + Caribbean + Christian +apple + legendary +marcia + preached +List +wig +Drop + Clock +uzz + Wright +astoria + spouses + Sto +ench +rencies + Grow +seoul + fec +rieg + Range + transcript + Conditions +乡镇企业 + dwelling +mers + seemingly +hip + arcane + exercises +argentine +propped + ove +privy +wed +sle + Giving + stats +devi +assetsadobe + Chop + cuff + ENT + {{ + peasant +arning + jun +ertation +urally +JOHN + Diet + Naz + troublesome + theme +Cold + sin +planes + Cyprus + tox +varsity + constellation +gloria + sheep + Telephone +qatar +waltz + crawl + Mb + 70 + Gorge + occupational + dimensions +Ann + Py + sym + official + Firstly + reinforce +ONSORED + Lat +pasha + feat + rodents + attachment +nacional +won + graves +第九十 +amtrak +February +ois +outta + Clyde +chanted + lenders +蒙古人 +beau +esian + Morning +iraq +onel + brushes + maj + Benefit + casting +THIS + perimeter + SECTION +taking +hee +ophen + Henderson +donetsk + Around + further + fatal + petitions + SAF + 1934 + Windows + imperson + BL + Saddam + reliability + Could + Geoff +bethel + shootings +odic + ridge + Potato +facing + somehow + dece +reigns +benson + bowel + PUR +published + leth + subscribe + Slim +CM + junior +forth + loser +osed + Patreon +pred +flores + Daughter + erad + itself + manifested + portfolios +beatles +Things + stumbling + charts + compose + stops + vanquished + Wyoming +第六十三 + apiece +ait + di + pierced + decor +Thousands + NAME + unbearable + ren + Position + Despite + textbooks +quinn +Germany +anglo +Prosecut + Fiorina + boulder +whimpered +omination +societe + 620 + shame +centennial + humming + Prophe + ECO + ). +boxing + PART +sprinted +mA + Forward + Spectrum + Sprint +porter + McCartney + (/ + coupon + calories +ptin +arling + groundwork + CASE + Univ + decap +onda +eden + importing + nationalists +medici + Zelda + Andre + Dodd + Dorothy + representation + Man +muir + ACTION +olded +oulos + mans +bragg +hower +riks + Peaks +ELD + considered +icans + Simon +jason + goalkeeper + FIFA +ifles +Much + relegation +BR +ileen +mls + watered +tucking + braces + anten + Juice + renters + DT +艺术界 + Lad +navarro + Gomez + baked + response + atmosp + Seeds + vibr +phies +sites +Owner + agents + Helmet + graduated +DEV + expanding + methodology + intruder + Passive +efficiency + Quality + touring + sentenced + Atkinson + astroph +LL +poon +Info + Atlantis +icus + ASD +itions + die + multipl + demonstrated + Closing + poker + Yards +selangor +suzuki + gastrointestinal +erica +joy + Creatures + Teaching + offering +otin +barry + Rio + contender + Psy + distortion + Restoration +porsche +eleventh + Cele +bay + impuls + systems +ification +斷 +ickson + AQ +renched + iPhone + healed + type +noelle + contempl + treasurer + permitted +indochina + Julian +第五十九 +給 + dystopian + americ +liberal +cid +stat +aredevil + Lines +smallpox + tabloid + omn + datasets + patterns + membership + Coffin + 126 + upgraded + Robbie +proton + incentive +katie + composite +wl + Treat + Bacon + dataset + ensure + presidency +osis + Constable +munro + Anton + Ended + overriding + liar + difference +ressing +nas +Ing + mas +sigismund + settle + sessions + wooden + Mayweather +model +ologies + wings +rainforest +utch + Perspective + favoured + vape + relic +ioxide + relation + loaf +type + Tropical +colleen + registering +temeraire +kHz + GPS +scarred + Nig +ibur +mormon + lobbyist +lucinda + TTL + Steelers + Warrior + IND + beliefs + pooled +Buy + chang + calls +hour + present + pseudo + Claw + defund +hammond +soprano + avenue +Women +Request + Phill + pic +ensive + Rut + heresy +barrington + Scream + spray +johnson + caller +heron +My + Peters + Hits + surfing +ravi + shrimp + Kiev + nutshell + compat +plugin + dessert +Lesson + mummy + compromise +Unlike +oley + cake +Senate +images + Ori + username + Benz + hybrid + Katherine + Payments +Nor + in +faulkner +petra +nee +含金 + Graveyard +eeds +dic +gom + handic + routing + ey +理所当 + jour +ega + contains + diving +Veh + inhib + highs + supernatural + failings + Agenda + dial +havana + Look + lest + pasture + relevant + rebellious + aimed +Making + rebellion +estonia + nodes + authenticity + expenditure + Objective +calvert + disclaimer +orgetown + 246 + Spoon + appliances + optionally +MAS + god +APP +simon + fri + bundle + convict +pees + synt +atory +anity + attorney + magistrate + Pradesh +corey + altitude + withd + Waiting + 808 + WHERE + juicy + managing + Goku +industrial +jian + ideologically + Unloaded + assure +atility +Don +Obs + ignorant + heights +iae +weasel + tissue + speeds +utan + gravy + several + 291 +camped +greenish +clinton + AWS +dia +随之而 + occupying +pcs +breakaway + barr + Madison + Academic +badger +okers + DOJ + paran + Seoul + Hai + Toll + profit +Rom + owns + apex +m1 + Democrats + saw + point +yk + Qi + Siri +Download +HP + nurt +worms + provoke + stre +ost +oster + waitress + OM +ethical +pai +Accessory + impatient + arresting + criticized + inadvert +HCR + previews + spoilers + Which + ruining + childbirth +uates +fulbright +rift + largeDownload +MR +ollah + hitherto +tensor + outer + kits + Canterbury + grandchildren +Footnote + old +和谐社 + Persia +adays +music +speak +galician +ichael +site +bnb +vision + reprodu + juvenile +Topic + 702 + PayPal +別 + paralysis +cornish +beck + movement + Oss + Price + pressing +acan +lings +孙中 + 229 + LIN + Sauce +yson +market + grapes + angle +rochdale +huts + ][ + experiencing +六七 +ication + lowest +answered + 50 + crab + thereafter + admire +Spring +yrim +扌 + acne +fell +Neil +vancouver +skirmish + heats +hers + mt +umeric + Rel +esque + froze + gel +ida + Coco + gifts + lockout +Not + conduct + Grenade + methamphetamine +orse + religiously +country + unveiled + Beng + quite +Incre + Teacher + fright + ABS +ARB + quart + clears + Aure +osponsors + Brussels +rophe +intage +mass +zac +fully +Allah + 608 +driver +cayman + homepage +张艺兴 + reimburse + Maple + Quinn +pal + effective + Democracy + citing + Mak +HUD + Vector +OVA + Giuliani + tar +norse + imply + references + busiest +EPA +Cur + imminent + Goal + snail + Cour + Wra +billie + kinderg +[/ + Pall + quoted + prostitute +ugi +RELATED +elman + mastered + potato +hornets +hu + intimacy +Apps + generalized +Exp + predicts +dwight +skyscraper +行政诉讼 +Family +oiler +Rec + pulled + Activate +allion + html + journalism +sundays + Labyrinth + cooperating +calgary + sugg +iable +etry + McKin +anic + 770 +amps + multip +estial +xus +azine +abe +gew +oji + { +搞不清 +algebraic + nib +gala +winthrop +Local + Deadly + GAM + separat + Scholar +名古屋 + Required +hindus + onion + pay + tx +親 +lation +ARP + chewing +brandy + crafts + Perth + worry + BYU +grossed + Engels + potassium + cryst + consciousness +follow + migrating + emotional +attacks + Alf +Brother +vation + travers +liquid + Blood + mortar +cushions + subsidy + number + miniature + Boh + Alberta + POLIT + rampage +第七十七 +adies + ({ + Rud + mankind + Restore +Qaeda + heals + NYT +caledonian + fascism +avin + Dennis +fn + neuro + Recovery + assum +odcast +igg + Zam +Jen + pipe +mozambique +emade +req +itional + Brick + Eden +authorship + fuse +peg + situations + Wed +plant + entirety + Specifications +ventura +marek + Rachel + prosecuted + human +Seriously + gravel + FM + breeding + Thoughts + exploits +dl +MPa + rose + sealing +mond +chris + profiling +Arthur +sebastian +empress + pesticide +ibble +chlor + mobility +GRE +ksh +Dim +emption + reimb +UG + democr + kicks + chast + Nikola + dances + major + Brut +iqueness + banana + Carney + PAGE + Africans + walls +Fran + drink +tering +fjord + Salam +Health + Changed + expansions +racial + Bunker +axe + commentator +hire + crow + certificate +reich +maya +ifts + permanent +ampires + stationary + Tri +mukherjee +iphone +LV +刘秀 +ware + mirrored + bitters +eastbound + Kingston + Eag + Beef + overlap + town +Lago + Noticed + introduction +qc + immoral + Clouds +WAV + slaying +lanterns +tattooed + arrangements + Relic + Brookings +throbbed + uterus + temptation + supermarket + Aboriginal +igslist + kindergarten +jessie +staffordshire + sunk + Occupations + Raymond + regulator +oval +Flash + ambassador +olga + Features + scorn + Or + miscon +slung +pulsing +vati +III + BELOW + Smith + distribut +guo +fission +zimmerman +中国香港 + sacrament + Shattered + testimonies +kirk + Brewing +erie + horrifying + killer + Net +congo + vengeance + connectors +sonata + stomp + crashes +this +fanny + lined + depending + highly + blending +orts + tenants +domingo + Senior +AD + sufficient + tropes + flush + sweetness + coordin + won + Outs +TW +milling + hairs + disregard + setups + extend +gomez + fossil + prelim + PEOPLE +debuting +herzegovina + Chilean + 323 + Regiment + judging + Nickel + doses +true + fuels +conductors +eck + worshipped + up + quaint +choice +pepsi + Fren + href +marta +ordination +pill + Harriet + majors + guild +ali + standout +ACT +unsigned +zing + disorderly + deductible + Tar + Smooth + staffers + RAF + tender + clinicians + youngsters + comfortable + diner +campground +angelica + mountain +kit +Wild + politically + broom + param +rition + Concent + Fiction + traders + envisioned + Sv + Wise + notebooks + Increased + Philipp +affe + fare + sexist +onsense + remained +round + apes + Spicer + unable +zel + ESC + stimulated +budge + Conspiracy + conceived + Over +Ble +carbon + Techn +fantasia +awn +opathy +americans + like +glazed +kamp + forgiveness + combinations +breakers + Agu +belarus + symbols + endif +Sense + PM + computational + )) + Ari + finale + compos + io + Howell + lapse +shivering + advocate + Whip + doubted + innocence +吒 + common + abroad +away + easier + Thirty + Rick +gment + blitz + foreskin +Damage +ajor +zurich + Styles + diverted +mated +qa + single +itable +Recommended + coc + hypert +feed + kidding + proliferation +edu + ()); +claimed + 262 + Integer + thw + Style + seventh + Ruff +HA + Wag +natureconservancy + improvised + boycot + caveats +saratoga +winced + Doodle +COR + macros + NB + clerics + orientation +nice +lab +Ham + gib +Visit +ornings + gunmen +boogie +denny +antly + diversion + multiplayer +rodeo +application + Rag + prove +ensign +gil + propri + hobbies +enson +hall + gig +feb + Cursed +Authorities +pontiac + Morg + offenders +laun +UI + Weiner +harrison + salvation +carroll +我哥 +Nevertheless +Las + unsettling +ionage + custod + Hyder +example + minorities + hook +aks + moistur +arrell + festival + ger +TABLE +ossession + Si + Face + capitalism +uchi + awareness + slit + Michel + Summon + Mo +awa +midland + Philly +orous +titans + Av + Ukip + Alz +restricted + endot + Kelley + Conn + wise + altering +ACP +itching + Tier + Fiji + unsett + those + Policy + uncontroll +德意志 + specifying +rely +parkway +ablishment + eggs +train +igration +lyak + advances + taste + Delivery + RPM +WM + polygamy +luce + oste + Cell +Featured + Shah + mort +uman + Wage + Dimensions + synthes + Colon + Definition + Grav +Which + officially +liturgy + Later + sleep + U +hod +Remember +auth +ipher + Shim + presses + tiger + Sponsor + Som + chorus + amazingly + Hash +bons +grassy +几千万 + switched +embedreportprint +llah + predomin +TMD + difficult +assembled +kaitlyn +eve +为人处 +用户数 +brooke +TF + easiest +lav + nested +ief + TCU + Corona +balfour +chini +arlington + Navajo +peeling + disproportion +EO + Niagara + distortions + 319 + trenches + Ronaldo + differences + monument + morphine + Civil + Prison + SA + lockdown + shortest + Kardash + youth +azines +idity + Engineers +acks + Greenwich +pain + Forty + Bennett + philanthrop +British +bright +megan +namese + edits + disease +Bal + washing + Meter + compet +iewicz + prosecut + Cater +dade + anarchy + crimes + Ottoman + Bigfoot + canine + satisf +sts +pend +hobbs + demanded + Bohem + Spot + replacing +dds + southwestern +mori + MX + disclosures + Bows + demos + CCTV + saint + clues + scattered + burgl +gunn + outing +suspiciously +Sab +mandy + score +inches + happened +tiberius + Wrath + storylines + RV + attentive + grades + infiltr + convinced +while +ices + evolution + flawed + Shaw +zur + intent + UCLA + pavement + zones + sneak + entered +article +mcgill +orchestras + Edwin +holes +husband + Tiger + Bakr + censorship + contro + everything + Providence + ensl + appointed +tung +uity +ald + Happ +ands +erk + Investor +laurence +imedia + apprehend +FM + Neil + raped + upgrades +huffed + Anders + 122 + Javascript + OCT +Stack +nemesis + Twins + applicable + mart +vigor +Material + solder +iform + chip +亞 +playboy + Lash +sanjay +aundering + guessing + princes +erial + Jacksonville + Integ + blister + diameter +renheit + funer +ohydrate + 73 +magma +capitol + palate +displayText + banking +Hash +ross + soup +Fab + Sau +table +adjusted + Palestine + quarterback +valentine + spouse + Newspaper + Python + spheres +obil +milo + Tsu + invite +cci +hillsborough + Brist +meg + Reader + Boulevard +Hidden + relay +arger + spin +dodgers +iem + urine + AH + paramedics + properly +tsu + tilted + Dep +accolades + schema +imming +sire +eh +Socket + Hes + fruits + ADD +wakefield + cx + slur + Concord +ERG + specials + Gall +马云 + backers + healthy +browser + accommodating + PDF +imar +亚特兰大 + publication + prohibited + conserv + gubernatorial +otally + 105 +aternal + Experience + 1900 +Ohio + agric +marquez + Siber +nostrils +edience + Word +icone +Ancient + Norfolk + queries + enrollment +lated + BE + antidote +posing + Manuel +安德鲁 + Mandal + advocates + arguments + pedal + Ulster +calling + interpersonal +mccall + raids + physi +olt +persians +gent + surpr + unused + amb + eclectic + myth + skate +Failure + wal + 228 +esville + Rad + revis +reddit + Tut + comput + obstruct + chiefly +london + zoning + Ramsey + Quran + Crusader + tc + extension + depressive +数一数 + Rally +madman + circumcised + grotesque + Creek + inst + Islands + blinded + variant + Sketch +oll + $_ + abnorm + vanish + regimes + panels + Patt + Nashville + Ba + Te +bertie + Du + misconceptions + Act + bung +obin + goal + Trace +match + Guns + discont +itations + Trent +cape + encyclopedia + Joe +Thumbnail +guitarists +傳 + helpless + pots + footnote + Shopping + vacated +DAQ + Cru + Movies + pric +rudolph +fix + subsystem +bridget +estone +WHAT + CLR +lishing +iran + orbs + would +broken + Daisy + acquisition + cam + fonts + dull +liners + hierarch +other +yahoo +ribune +ollar + FREE + Parent +sabrina + Assass + electrical +Previously + straight + quirky + Light + inconvenience + revolutionary + surveys +balabala + folklore + condu + coloring + conformity + Ambrose +tan + Hawkins + uniformly + Teg +Mont +ooters + Heavy +entially + call + adversely +prev + } + Lank + rehabilitation +Running + wedding + BG + repeated +frigate +shaggy + tips +chronicles + consum +orest +poral +Various + Iv + goodies + Info + Islam +ihar +wyn +Que +rubin +ishers +orbit + breakthrough +ratom + Calm +converse + toward +Magazine + combatants + cinnamon + Featuring + advisors + muddy + VMware + Tong +ifer + appears +psalms + Jes + NEVER +calderon +five + rub + disgusting + linking + glorious + miss + Ange +isa +emperors +bil + sweets + Coins + Kejriwal + Kobe + stopping + Timbers + returns + Happy +bismarck +sung + large +terday + some + Reddit +設 + successors + OECD + oppressed + glaring + Bureau +Skin +cam +bats +ody + notch +plement +morrow +Fred + Champ + putting +deportivo + correcting + 25 +isabel + hob + 216 + Deutsche +lopez +ings +VIS + ERA + Target +bays + Total + frivolous +Tim +wave + shutdown +Space + configurations + belt + VOL + 1925 + Climate +venge + flat +cult +bits + inhibited +fen +scan +穆罕默 +dayton + archaic + Amanda +traversed + Pledge + jer +dane + headache + NW +verning + davidjl + Guatemala + chill + *. +because + coordinator + Hills +acts + Graph +told + short + attraction +kingsley +cha +可多得 + subdivision +Pg +poznan + Honor + road +establishment + tranqu + broadcast + SX +generic + extending +MI +ors + beginnings +oca +hement +gerald + execution +ocious +itch + shaping + fantas + vulner + Brilliant + surgical + ACTIONS +Professional + ploy + afflicted +irteen + physical +ORPG + Buy + connections +breuning +pheus + suits + gears + Ivy + globalization + gospel + cellular + switches +malik +towing +commended +orf +Absolutely +Esc + restroom + Carrie +NG + Resist +obbies + held +onte + adhesive +bathed + Controller + praise +amalgamation + Olive + ', +alvin +actly +uned + Bulgarian +approximately + Norman +Mouse +Cross + deton + NBA +Desktop +rieving +inet + repent +onest +灬 + fairness +nian +lp +versive + shave + Nin + insulation +truman +omar + Sparkle + America + infinitely + guiIcon + WR + blond + Proposition + Fifa +umbai +loads + therapies +mahmoud + buoy +iann + Hubbard +tingling + Walmart + root +cyclic + retracted + line + assorted +uese + trip + Expect +ert +baroness + Train + Mong + uphe +glider +gunpowder + Mall +school +TEXT + relativity + rightfully + remark + Marxism + Complete +buff + moms + Frequency +uel +guthrie + Spac + hang +lived + orbital +plugins +haven +moravian + Ing +anny +Personal + signature + linem + suggest + lace + Capitalism + banging + Ref +liu + closes +insert +Notes + While + drum +bafta + appeal + flaming + reinforcements + Hait + hall + veterans +anned +Capital + wik + mor +arab +第六十 + going + summit +arun +Mobile +ONS + adults +soothe + accessories + CRC + v + rusty + Occupy + arist +gp +ursed +bmg +forced +catch + Mane + neglected + NRL + 123 + delinqu + sculpt + overpowered +asons +grass +lexi + MR + shortfall +GPU + Explicit +lucius + clut +stra + giveaway + mp +Russ +suggest +pton + carved + feminist + scraping + Networks +Ware + proves + siblings + Planned +pher + Gates +RG +volga +Battery +vulture +roller + Dial +supp +romania +Getty +uracy +amping + forgetting + formal +Myth +AI +far + French +ident + Adjusted + val +crook + Fiesta +Map + residents + SetTextColor +pound + Town + sees + affair + irre +homecoming +aye +Loading + grun +四十年 +percy +rane + hopes + Mult +iverse +rational + unstable + Unlike + exhibited +Fresh + Lag +agraph +acci + ns +Open +phoebe +indust +bra + 180 +isin + Phelps + dB +munoz + cultured + boobs + commissions +tombstone +americas + wrist +ogged + Vapor +line +brest + unanimous +bravo +Extra + G + founders + intercourse + Tribe + conducting +fortification +sterdam + ATK +Sharp + Free + radiant + Appro + unbelievable + foster + overrun + Emer + frail +zes + Ana + Canucks +andy + spawn + Geo +Ku + enrol + Rowling +inst +ged + Brent + Ms + heavens +igma +julie +Tip + educator + AFP +lifeboat +reply + companions +ivities +fman +siding +AVI + aftermath + bees + inland +welsh + Brad + efficacy +HH +Zip +tributaries +oured +corona + apartments + stories + 159 + Shin +essee + deliberations + 378 + concert + expenditures + mentally + animal +mangrove + imprint +ulla + drained +響 + handc + Political +daggers + drifting +aghetti +asel +RIP + slider +riers +GMP + spare + recogn + outlet + Tak + richest + Clippers +oche + helicopters +READ +qur +surveying +summary + fru + deregulation + hot +Calif + Geographic + Anaheim +sided + deteriorating + communicate +Moscow +Without + decode + collected + source + Winners + arter + Camera +panther + lip +branded + notify +erick + Kushner + Large + headquartered +sg +warring + companion + react +forewings + triumph +iation + Ali + Lis +Fast +Players +mallory +kid +opt + Kan + feed + Fairy +Detect + corpses + Democrat +punjabi +sweet +fficiency +During + Texture + extrem + DUI + Fitzpatrick + Ezra + Jays + Ze + turnovers + eloqu + Naw +finley + Never +profit + embry + Comic +estuary +Window + spite + dinosaur + perpet +porcelain +urred + treaty + ) +gall + compartment + deriv +jared +uitous +yugoslav + Ital +息息相 + castles +meade + srf + posing +jalan + briefs +ieee +Nin + Startup + Palestinians + dile +umbing + bankruptcy + Malays + Shack + Romance + Consulting +corsica +赏心悦 +UR + VERY + Bernstein + karma + volt +cambridgeshire + clutter +民事行 +eria +getic +tom + Clarkson + Sunni + Rules + 1948 +strata + Bay +aq + Nav + theory + 1971 + dement +oros + bartender + healthcare + Roma + Schumer +baghdad + Rhythm +jud + governmental + boosted + Lawyers +rub +prague +ucci + righteous + Gork + settles + paintings +rar + steril + SHAR + Harmony +grazed +wilkinson +isted +raph +utonium + provides +action + unnatural + 1997 + cerv + exposure + Ras +azi + tou + doing +credit +ves +Balt +talmud + allev +howie +antarctica + 1941 +olation + Oakland + marriage + surrog + Seventh + Benedict +ampion +paris + AAC + rooms + dictated + repr +fps + suppressing +swe + CLSID + apprentice +陽 +Transaction + Event + GR + agencies + handcuffs + 1985 + sanction + cute + Holmes +drummond + receive + oranges + Shank +phillip + voy + includ + standoff +malink + meditation + fresh + unsurprisingly + rich + Por +rhine + Refugees +jenna +maybe + consumer + Convert + upstream + cafe + Camel +corporal +台北市 + recognized + happiness +cheltenham +rule + Fitzgerald + intellectually + COURT +erich + Kot + Reef + Stim +shi + Distance + chalk +iscal + wandering + chimpanzees + commenced + props +burnett + Shir + disciples +derived +Oregon +auer + unintentional + breaks + \(\ + Pure + modesty + abilities +aped + holiest + obliter +background + boarded + 33 + Sic + yards +staten + LOG + targ + beh +carter + preset + canoe + characterize +oreal +lib + overweight + engine + rearr + pattern + Idle + collaborated +talon + equivalents + Chic + unpre + Summers +krishna +Isn +itect + posture +naples + confronted + ko +benito + electrode + Commit +pat +Definition + Faculty + Feminist +bassett + Language +人民公社 +ride + jan + spac + varies + idea + hugged +only +uti + cow +dunkirk + Manager +flatly +vl +Config + Laf +RK +rajasthan +濛 +ued + Forbidden + Bis + gambling + fighter + lie +Animal +iod +dinamo + bod +gged + XVI + original + ideas +kerry + Nem + fragmentation + ESP + genre +ese +crystalline +ById +owicz +mollusk +estate +eduard + inflict + Formula + syn +IAS +ucla + supporter + Growing + island +late +美国纽约 + thug + Bid + presume + tight + abolition + con + en +oplan +dimension + daylight + Comed + chic + Ext +hamid +azz +UF +rams + Gram + pan +acker +coat +saxe + Ohio + Dollar + nascent + Thank + smelling +Even + TMZ + transfer + Islamist + hateful + durable + (%) +groningen +disqualification + BAL +aque +Online +oldham +Single +otter + Zionist + grievance + low + Also +urat + stabilize + foreigner + Twin + Clockwork +OF +natal + md +development + ONLY + later +scription + signatures +solemnly + CODE +Plot + cm +换句话 + shares + pamphlet + refers + lets +marcelo +ature + YOU +speaking + DeVos +adem + Mash + grill +kenya + bru +imei + Rain + remove + forces + Muscle + share + Troy + Power + villains + contra +question +jules + than + Hill +pins +cliffe +LEY +etsk + takedown +jl + evid + shoulders +shuddered +conv + dinner + Otto + convenient + thought +agate + paragraphs + flowering + testifying +eness +orer + Share + subcontract +xb + Alma +lambda +rious + exhaustion + Frozen +rier +eels + abdom + mistakes + whatsoever +Export + AND + Ankara +ever + Za + par +hner +OURCE + Cavern + Hilton + Oxy +ignacio + Princeton + graphic +Meg +strous +odox + pirate +morris +直男癌 + motorcycle +ience + Mint +Italy +Supp + conquered +futsal + noise +vargas +Nick +helena +olute +viation +Everybody +Nit + Typhoon + probes + constitute + Collections + situated +tanaka + incompatible +ACG +borne + half + Gear + bloody + Sleep +carly + shotgun +yip +Alan +cadets +rory +iox + schedule +Image +onent + bulk +reunification +抓住机 + despite + Austrian + lecture + Karen + socialism + mun + inquest + skepticism + ded +hou + tray + potatoes + Atmospheric + Regulations +zoological + Judaism + Nottingham +联合国教科文组织 +辶 + optimal +closeness +dioceses + Landing + singer + {" +itized + bere + Gob +upstate + Lieberman + Sob +II +ously + regards + secretary +mona +因地制 + Sniper +clermont + Impossible + Souls + Fet + header +segunda +idi + puzzling + Partner +alty +azard + Ammunition + waive + Cannot +開 +ears + rape + PBS + mound +第四十三 +umi + journey + essentials +oves +armament + administrative +Foot + externalToEVA + Billboard + eliminates + defamation +parse + Marc + trusted + consultancy + peer +Coin + Reports +daniel + pri +monster + Clancy + beginners +seaman + billboard + Foundation + Replay + pend + sixty +asks +Seeing +interface + hoop + Pun + sanctions + startups + Iran +基因组 +directorial +jas + factory + accidental + guiActiveUnfocused + quickly +idated + Members + legalizing + Heat + travelers + ensu + Anthony +ates + ind +beheaded + Chicago + trunc +ourt +uggets + premie + Cyrus +ADRA +初中生 + obe +FILE + whisper + 27 + Malaysian + AM + opioids +elongated + frequent +remy + robe + bedrock + crusade +distance +roger + Solid + uniform +darkly + GROUP + resorted + Gim + fullback + Eas +exhaled +chandra +amc +lli +missions +erity + whistleblower +wear + awaiting + wolves + shareholder + detection + motors +ARE + 2020 + Canaver + upfront + Thro + historic + citizen +总指挥 + tense + siph + apprehens +tops +oko +histoire + Individuals +aping +join +onomy + Kurdistan +zees + 501 + Dylan +peaking +ALSE + levers + intuition + Emerging +一言不 + hugging + jointly + Tribune +Sys + producing +phonetic + classification + Jim +商务印书 +robyn +ologue + Previous + owed + struggling + bas +apolis + STL + lights + 1901 + sensit + grabs + originally + Chapman +ignant + appease + ALEC +eek + Claud +Inst + Threat + isol + informs + vacancy +tsar + deprivation + ACL +dialects +joyce +warming +ular + Thumbnails +大提琴 +abdul +kyiv + categ + Superior + FIX + Ready +IZE + Aster + Trump + poet + futuristic + Today + hous +ower + Gone +EMS + Farmer +Rub +oms +ainer + intensity +证券交易 +gough + circa + hull + mater + vent +worm +ea + chairs +georges +kj + Extract +developed + Notes + launches + Zealand +raining + WORK + hind + Besides +Judge + bookmark + Salmon +miss +atoll +内燃机 + remod + insol +astrid +pon + Ban +Andrew + Even +stevie + Publishers +lankan + Miscellaneous +Anthony +gling + Ped + gal + qualification + nons +World +bool + UFC + umb + restricted + relegated +icted +ifacts +Mo + curses +oracle + Yourself + Goat +national +ATS +Sir + CLASS +Keeping +inkle + accounted + Jol +isexual + 97 +shang +cerro +Andy + avoided +emigrated +Aw + alumni + GOODMAN + Tanzania + resolution + Trilogy +sawmill +Benz +ould +wally +vickers + graph +jump + internally + Spear + Both + misunder + versions + referenced + suffering + Tradition + puppy +價 + batters + ABC +mcgrath + modern +serena +fleet +AX +fifa +henderson + Attempt +airmen + VIDEO + Chili +axel + Aerial + stirred +olog + criterion + Java +vec + chronological +Change + Valiant + Absent + 701 +seduce +claremont +herman +Preview +upuncture + authorities +bite + planning + Sand +aka +KOYO +ildo + Vill + injust + weeks + nicotine +snort + prophe + nightmare +brant + Direct + nicer +times + Pixel + delivered +edouard + mixer + Trudeau + ~/. + LC +ERN +emir +iam + Building +orie + holster + drown + Gutenberg + bye + Tribal + bites + Sunny + demeanor + Louie +essage + geopolitical + roll + herald + knockout +翦 +RAW + hinder +ograph + dmg +Law +Medical + fibers +hesion + monk + bakery +smanship + IPv + problems +dee +ANK +mahal + Wagner + openings +amide + 143 +pueblo + colonel +无所事 + NOW +ulators + Spart +近代史 + equation +ations + Blackwell + Greenwood +Boss + Icelandic +methodist + communities + Alc + bitten + saline + eliminate + Emerson +四十八 + students + accurate + actu + gob + strict + timeframe + h +Ac + marvel + credential + RELE + Personality +thirties + stairs + ambiguous + ART +raul + discourage +norte + Social + Huge +duets + sovereign + Mars +kenton + chased + Fuji +OPLE + auditor +coming + elbow + union +Story +ITNESS + Examples + und + lawfully +istani + leveling + Resistance + Complex + polit + edition +linden +Guide + 139 + erection + Render +Phase +Finally + procession + SDL + does + enshr + consequence +minnie +lingered +mology +Until + protracted + variance + Aut + Earl +oust + aliases + Sovereign + Wat + $$ +granada +corinne + SI +vana + imported +everything + Corinth + suffers + elimination + cornerback + evils + Enhance + Science + Pieces + practitioners + xml + deadly +aining +hilton +munition + whe +dundee +rors +Ign + LEGO + ahead + 225 +IAN +gz + confusion + Shut + contingency +太阳系 + Commercial + amateur + Ross + entries + dict + ATL + QR +aken +ashok +redhead +artemis + Asians + Judith + abstraction + remarked +heres +Component + photon +omics +ruly + proved +amic + Japan + Yosh + Dawn + since +around + Fah + supervised + EVERY +joker + Active +保险费 +ALD + agon +BILL + purchaser +Method + Cad + boiler + caricature +itzerland +disable +flop +Spec + euph + Gladiator + conservation +WA + laboratories + violating +lina + unconventional +luther + innovation + localized + politics + takes +constantine + absolutely + Krishna + orb + sav +End +rust + steak +impact + Afgh + DMV +chariot +Diff + grave + propos +BD + Norm +requently +jug +phony +喆 + Disorders + Twain + ARTICLE + ed + Gillespie +undred + discomfort +izzy + Dive + neighbourhood + jaw + Kings +map +lower + Dale +builder +ongo + rapist + 207 + Tens + withheld + [* +nakamura +belinda + Panc +After + torped + launcher + garage + Steps + narcotics + integrating + Prof + atoms +Prem + Travels +MATLAB +單 + metal + Founder + flies + fearing +topography +ongh +steam + tale + tucked + raven + massacres + Cree + rented + embryo + admirable + conveyed + Yo +toyota +archway + Hate + Slayer + Gonzalez + 1914 + 343 + extremist + MODULE + Hive + enjoyed +摄影家 +Angel +standard + hearings +Edge +punk +preparatory + 87 + Yas + warmer +Hard +seat + sig +崧 +ify +hani + Grounds + triangular + Mirror + knot +无言以 + distract + reforming + comparative + Chloe + Superman +shetland +press +equatorial +Engineers + placed + retrieved + morbid + Reality + Carbon +iata + Sale +oss +tinted +Enhanced +爲 +ulysses +amphetamine + Legends + resorts + Rusty + On + deprive +土木工程 + dispens +StreamerBot +Blu +SQL + Respond + rate +keystone + mature +dark + 342 + loves +ptolemy + conditioning +odus + Interactive + survives + curved +zealand + Car +responsive +Pos + HM +amental + radically + anguish + advisable + narrower + acclaimed +第三十九 + Technician + 420 + Nike + Organisation + omnip + manoeuv +vedio +imer + Why + Limits +gustavo + appropriated +Chelsea + marking + thinks +noon +團 + baseman + Comcast + Demonic +断章取 +tugged +bury +nara + releg +itous + potency +pleasant + eagle + outsider + Brisbane + famous +eri +BB +ImageLoader + developer + Wheat +virtual +expeditionary + swell +Cond +Dev +ulu + FILE + corresponds +ovation +chat + mitochond + pleased + mixed +準 + authorize + Univers +ER +issors +external + temperament +fifth + neglig +rics + Cologne +Command + Issue +Diamond + commuter + Alien + emailed +uru + insulted + Janeiro +ugar + Tol +默默无 + brewers +pomerania + Tig + depicting + lawy +olen +matrices + plethora +ih +grammy +myspace + Off +mong +gia +eling + PARK +dit +hopkins + outskirts + Peg + Vancouver +tle + aggregation +Seven +casket + Silva + oblivious +網 + Babe + attended + Leader + knocked +taluk + dead + tack + weakness + aeros +erasmus + clutch +Learn + Davis +requisite +usalem +Provider + pig +Adds + vandal + Guid + requiring + liber + Serve +maggie +Prior + Dund +Mp +quest + author + Chick + mysteries +verb + markedly + volley + BJP +OST + advantage +政治部 + cath + could + strategies + exceeded +lyons + Mam +locke + compensated + Burg + deprecated +rearview + Bag + Recent +ukraine +Tube + smile + immunity +untled + partying + detail + analytic + Gallagher + promoter + tempor +painted + expelled +decl + fireball + Hannibal +charges + bags + Lich + ignoring + certain + wart + stray + Hawks + reimbursement + limit +MY + grain +odyssey + corpus + cance + unintentionally +aclysm + wink + EXT + proximity + bast +umatic + Tip + Kak + texts + arrow +SEE + understandable + beginner + superheroes + Maxwell + thereby + Diplom +dates +第十五 + what +akespe +owa + DeV + Alfred + Centauri + focused + arte +delaware + Unable + DD +acent + admins +migratory +ARGET + Fifty +classic +isations +availability + likes +congresses +ucer +galileo + Gilbert + adolescent + Oliver + Sharing +agnetic + Interview + Veil +万平方 +figured + Psych + stealth + wildly +scowl + badge + Willie + integration + Techniques + contemplate +WER +fixed + constantly + Re + Mond + Nepal +tion +intendent + authenticated + or +Shut + hallmark + Gotham + Wyatt + trolls + heed + Squad + powerfully + undocumented +sections +還 +nixon +eau +JPG +Ta + upsetting +inity + 960 + dangling +repe + infinite +gaard + unnamed + speculated + write +itational + Abyssal +ercise +ithing + Imperium + inspector + Pentagon + ni +ctr + Imag +inactivated +Court +vincenzo + Thatcher +rotein +alexia +OSED + quartz + geographically +Third + crew + drafting + volcan + unfolding +inging +Be +redited + let + plant + bab +zhang + Starting + technologies +errilla + linebacker + footsteps + powerless +ici +uilding + restrooms + ongoing +commentaries +spines +hansen +moshe +iring + MAS + marble + surge +Xbox + ops + infants + oils + benefited + degrade +河南人 + hemisphere + father + Cato +nic + Sharon + trainer + storms + passports + gratitude + outs + girlfriend +Soul + blinked + abandon + eras +pac +一个劲 +ppe + Ludwig + acknowledgment +Keys + Wisdom + To + Imm + dinners +betty +Anyway + Trainer + offensive + Sorcerer + ISBN + documentary +aff + awesome +ferdinand +Di + Henry +inburgh + reactive + Emerald + boldly +another + Gir + Direction + Mode + informant + Wad + recycle + deliveries +band +迦牟尼 + CLE + GHC +Simon +Joshua + nominees +hig +jew + Initiative + knack + initially + railways + pupil +venants + Bour +youtu + Short + tours + Eighth + bishop +gnome +Turn + revived + vivo + striking + lesbians + HER +alysis + fines + oun +piration +ECT + Zup + Evaluation + bytes + sprinkled +Sal + argument + filename + fn +wind + signaled +ande + subjug +Detroit + Explosive +Wednesday + Greens + reclaimed + Eg + Hik + rugged + Monitoring + Nets + strip + jog + measles + unregulated + 275 + chats +bedfordshire + microbes +iders + cannabin + FDR +ophe + Zer +serene + 108 + ass +auc +vin +face + Slav + stabbing + Geral +otom + smokers +hamlet + invading +Pres +a1 + crash +RE +liquidation +ges +wana + Anthrop + fielded + Exploration + Points + Gauntlet + Mission +Books +Flag + Truman +jordanian + Apocalypse +INE +proclaimed +kil +herry + Tub +wilkins + worried + Bruno + lodge +held + FE + diminished + NRA +ington + tidal + defense +ingle + ideology + RFC +Goal +peering + anchor +ambushed + August +usted +ixture +Log +Chest +flanagan + RSS +huffington + platoon + Facility +falkland + conceded + wrapper +suzy + Void + excavation +Nic + MB + Shards + substantial + hep +EC +unte + Aux +lenny +cycl + nasal +OD +verts + crown + willingness + produ + Daniels +ala + illegitimate + Sequence +ovember + IRS +appleton +hattan + Aunt +turnbull + Idol + Prometheus + paste + democratic +ilies +rief +GROUP +uated + gmaxwell + translucent + Secondly + organisations +content +karnataka + cruel + innocent + stout +hash + Mew + lining +henrietta + electing + fruitful +sale +snuck + believe + Bermuda +amplifiers + despicable +DA +argyle + Holt + subur + ministerial + Uk + diagnoses +moods + Say +iov +pointer + Meier + traject + Crazy + pitch + shattering + Cit + lettuce +forbade + Protestant + anyone + user + FRE +panda + Woody + slumped + particip +those +stitial + 47 + disbel +imo + Jarrett + Seed +izons + exposing + Usually + landsc +Creating +IG + Deep + Players +temperate + solidarity + snatched + muzzle +祐 +Py + spirits +nfc + Meh + swap + 30 + creators +Now + capitalists +heit + dishon + PV +pelling + rankings + manipulated +cinematography + Mats +etchup +commencing +stalin +BIL + phone + poles +KER +cuban + Liquid +illa +Marie + allegation + Wass +sanders +winery + Pokemon + cancel + restless +kish + enjoyment + Orange + billionaire +neum +ynamic + Kub + grains + fer + shar +lebanon + banning +comune + hyd +ociation +License + trapped + oversees + dreaded +overty +bandage + ..." + theoretical + assault + imm +Callback +celeste +cunningham + Cooldown + IF +machine +glowed + domestic +whe + Point + Consider + 158 + Feinstein +drake +reindeer + cryptocurrency + apprehension + Population + Shoals +starr +cture + Wallet + nem + Dil +]( +enna + suspected + BU +udeb +美利坚 + convened +hauling +orescence + Venus + cervical +gered + TX + beef + Fully + Mysteries +joanna +ORY + serves + disrespect + horizontally +ibrary + abducted +lic + Are + leather + leaking +heard + categories + 377 +bara + examining +ysics +quez + ability +column + Polit + complementary + Byzantine + NAV +gills + weaving + Brown +doncaster + statically +rights + demand +ives +deco + greatest +arate +disabled + TOTAL +JJ +ezio + Nek +ilitation +entry + qualities +olivia +ICO + Grammy +intestinal + tax + BST +schooner + handcuffed +stal +inces + Cher +fetched + silk + Film +uminium +anguages +lissa + oblig +moses + pist + leaders + Bench +microsoft +heid + Cho +wheeled +tuba +ori +elled +说不清 +rera + filibuster +gone +congregational +Improve + Courage + removable +agos +immaculate + pretending +telangana + expans + Arkansas + law + Dug + stricken + Wins +優 +violins + UI + popup +pbs +Hum +louie +kind +sburg +WC +sato +Offic +oub + internet +ctions + Explos + Packers + pleas +osition + tragically +igi + || +avan +spawn + lakes +Ted +comes +base + Gil +PER + outnumbered + sold + worsened +abee + Margaret + seasoned + comprised + publicly + starve + acoustic + countered + SCHOOL + destructive + sexuality + richness + 95 + ruler + storytelling + Num +式各 +manac +nicolas +trombone +Gallery +sharing + dating + Frenzy + sounded + Morty +igrate +業 + arr +ung + glances +kelly +amins + broadcasting + miscarriage +maryland + various +AZ + cites +river + general + cemetery + Maria + bothered + Wes + Monaco + confirm +Arsenal +orn + partner +official + Trying + Shared + adoption + Anch +eucalyptus +布鲁斯 +eyelashes +aur +dump + Ashe +rodrigo + illuminate +toby + Kill +png + Wander + Wish + wrapped + sisters + replicated + Ellis + denomin +shield +Season +Ass +chie + Moons +Uncommon +orns + chirop +version + rather + elders +verson +waldo + HAVE +ession +skinner + Knicks + Philadelphia + rebounds + splendid +McC + strips +ASC + driving +nam + juxtap + standby + Canaan +aba + dissidents +atz +兒 + Clemson + 2016 +pearce + circle + Gerrard +MAP +ersion +etsy + Weasley + widest + conting +EEK + trick +ellen + consultant +supported +Personally + Stores +oward + stub + Pork +natasha + slipping + gly + embarrassed + unden + copying +Gar +regulation + Ark + cred + Trance + dynam +eday +kai +chevrolet + Phantom +united + scissors + helpers +WARNING +gettysburg + shoots +inqu + rights + craft + soil +florian +defected + Conflict +eteria +电子书 +foyer +papua + Wikileaks + press +ruben + gold + TED +avid + defenses +post +mama +漢 + Reduced +alling + dress +unicef + dearly +ais + undertaken + pessim + overcoming +zyme + priorit + Emperor + laz +coded + Recipes +atches + deb + interconnected + showcased +Factory + succumb +bypassed +sure + equals +woman +medina +orlando +revolution + interchangeable + Volt + ath +Boston +illian + multitude + approached + condemned +oker + obesity + formats +susquehanna +isine + distance +Veter + minimal + Disciple + Zombie +Top + intelligence +innie + shimmer + networks + competitions +nilsson + Dup +uto + supports + terminated + towers + weighed +raged + Edition + millionaire +chronology + notable + Revelations +yokohama +Construct + Without +astro +Cass +Bone + forests +邓肯 +keeping + morphology +osborn + Medina + previously +dfb +freddie + outlaw +taining + + +数控机 +topology + Shed +eligible + Eug + embryos +Weather + devils + Bin + deeper + spill +illary + rides +vita +emale +dial + Tort +worker +ION + dirt +itial + Story + poorer + bike + Lans + nominated + Worst +dunbar + woes +RGB + Eck +bia +inelli + Wonder +pisa +birch + seized +necessitated +Early + Salvador +endor + convert +chemical + laureate + Guest + Architect + Blink +Wi +cosmopolitan + airstrike + remake + robot +Pet +Section +connection + Elder + McCarthy + clandestine + least + beauty + plants + Lic +Jake +Split + Spiegel + taped + atmospheric + guitar +别姬 +Winter + hefty + discharge + cre + selfie + current + 2600 + responses + Phot +conom + unpublished + engines +astics + LIMITED +Specific +coronation + ashamed + Fam + chrom +umbo +Reading + Kaiser + poverty +benton + 278 + Thanks + racks + referring + dissenting + shapes +Gordon +Whoever + temporarily + rug + FIR +caden + 195 +inline + Cabinet + effected + Music + fid +Saharan +linger + downward + rites +steen + attendees + commercially + uniquely + streamlined + Cran + Jub +Woman +Stat + limited +cod + survey +ucker + Delaware +SPEC + Hoffman + retard + Sidd +ariel +redits + Edinburgh +syria + Lot + 1907 + conscious + France + Stuff + sprite +CLASSIFIED + crises +ader +goldstein +Male +lington +amm + evolving + Defeat +Austral +Tx + Learns + Matter + descendants + aisle +hene + magnets +Su + env + Trip + Gore + roam + Candle +stroking + Mer + slows + medieval + impecc +interstitial +asive + unsafe + bass + function +fourth + Paddock +Grant +superliga + encountered + shipping + virginity + Brett + Shaun +giggles + Morgan + Think +bol +soviet + model + give + Nazi + affects +philips + degener + ben +iton +Thomas + Repeat +ius + cannabinoids +Ey +ined +croats + Parameters +truncated +plata + observations +ipation + \\ + intercept + prepar + Dar +tern +itarian +SPONSORED + mystic + adm + Weather + drowning +free +liament +progid + leadership +johnnie +simeon + battles + shuff + Newsletter +南斯拉夫 +loren + cro + Bangkok + rehabilit +sharp +lean +bibliography + intimately +nl + anthropology +viper +amn +ativity + today +shiva + Guards +rockwell + angst + Anarchy +Str + 63 +almost +kc +abigail + Trouble +Para + rid + Ship + installment +sutherland + Ju +club +Boot +june + radiation +quadrant + ramps + Sick +izont + Alibaba + Works +kinson + Lin + Richie +Federal + accommod +cards + Provided +blanchard + counseling +rios + bins + Jewel +taboola + Honduras + curtain + examines + integers +shaded + fabrication + Paper + uncover + Calgary + scholarly + Requirements + Kare +達 + impose + Other + determine +soon +mentoring +三十多 + PVC + workaround + Switzerland + Iss +ellow + epic +olph + Hug + battleground +sms + deposit + EMS + Richard + Forum +Tai + Cubs + heartbreaking +ical + trauma + areas + 317 + differential + cooker +accessible +bole + Spawn + pier +iant + pulse +Updated + Female +Animation +fifteenth + renovation + floor + poignant + ballots +savannah +hero + thriller +oid + industrial + Fu +gio +Pass +riber +Design +scala + ent +trap +dolly + minute + fool +Girl +戲 + wetlands + surrounds +sur + landowners + deficit +maestro + Sal + caul +Initially + convent + neighborhoods +mead + meltdown + r + regression + strained + Cap + transform + >= + Live + culprit +null + Victim +telegraph +ulation +istration +ccc + ease + insignificant +silvio + cyt + actual +ushes +electrification +squinted + IRC +welles +gnu +hooves +AFTA + addon +undo +QQ + Physics +bum +nasser +wharton +swing + erg +needs +bombardier +ogh +heim +Link + Xander + layouts + Lit +odessa + stressing + immortality +anaheim + confident +rait + conducts + FAC +rist +casting +wegian + engagement + Syri +sentinel + escal + enslaved + Aether + uncanny + via + bank + chef + Spir +ampa + December +bing +ikuman +ARDIS + Vari + Lowell + disturbed +scroll + Much +ensity + Race +Timeout + Hutchinson +madeline + brill +ophysical + [( +PET +spread +Alert + realities +connor + designed +bonfire + zombies +scientific + Technique + Chiefs + MAKE + comfort + p + wra +millimetres + medically +twigs +position + Sounders + technician + pil + Sai +sted +daly +ashton + cough +juniper + Fuller +domes +antonio + Pamela + bugs +job + Gujarat +certain + Fernando +philanthropic + Added + Surve +codes + generated + seism +SNH + Rams +Fan +十几万 + racer + Mourinho + facto + Valkyrie + *) + Feast +aramaic +metropolis + show + AES +cats +Present + expensive + Flames +adiq + ago +ska + slapping + Newman + strategy + Buchanan + charact +iu +dragon +bella + SPD + Liberty +been + CPR + Aircraft + slick +ajax +spr + interior + Dirty + acquisitions +dune +mans + enlarg + / +ker + multif + endorse +istle + instance + 204 + Tatt + tribunal +volent + MMO +abi +Ds + includes + duration + scourge + cheaply + practically + experiments + pray +rah + competent + Clip +rupted + modest +aration + wielded + IR + decentral +edison + extant + symmetry + Blessing + Railway +Usually + Mario +avering + penny + boun +akra +granny + gem +adams +cular + Corker + cakes +esm +一模 + practitioner + Ig + Autom + spun +ahi + disgu +resigning +congenital + unorthodox + Patriots + widened +sei +hallways + Archives + slot + paternity + Campus +iped +sense + Dungeons +May +Mich +vil + Regular + anime + Taxes +nantes +Do + hardship +karachi + instability +usky +nar +genealogy + Investors + incorporate + scene + sr + Shib + historically + elevator + baffled + sa + Rocket + implication +nebula +Johnson + Written +tinged + each + stitches +issues +velop + tan + Chu +racuse +Pac + LAW +gutierrez + Get +indigo + Sector + Bald + veggies + smoking + beginning + Plague + treated + Venom + reception + Erd + mates + pork +wang + Deputy +Wall + gaming +impatiently + Grizzlies +AH +ISTER + Ellison + Become + Jericho + Harold +KA +riminal + rolls + continuously + replen + variations + softened + MOV + Lakers + apart + Drill + incorrectly +咧咧 + assemb +briefcase +inel + unintention + sound +怎麼 +amor +olf +Connector +Override + aperture + Pul + UC +ircraft + utilities + astron + Nope +medium + Oversight +especially +heidi + widely + cens + correlations + sword + excel +paul +PL + patrons + Committees + skip +Mut + dashed + Enchant + awfully + laundry +reated + Unlimited + housing +Connor + faces + km + 299 + bamboo + Au + welcomes + ir + append +Charles +terri + dangerous + revolutionaries + Kaw + camel + Daytona + freshmen +lbs + polarized + robbers +Tom +braced + Bernardino + Airlines +ggle + Dana + Comfort +Draft +weapon + Dexterity + possessed + specs + denied + toured +brunswick +load +Quick + customized + Comput +mination +worcestershire + spirituality + indistinguishable +curran + pushes + nest + lands + input + team + stunts +没太大 + Inqu +braves +birth + bots +moors + preparation + Wr +pluto +arus +oder + tears +信息产业 +uses +Former +Bear + Kr + Pricing + tailored +iri +TERN + recognizable +damage + deform +OSS + Highway +dakota +armac + Jing + obscure + gunfire + Palestin + Firefly +newfoundland +holiness +iraqi +used +repair +gyn + execute +scraped + layer + Grayson +acting +German +Strength + differing + individual + things + enhancing + Marco +altern +楽 + rated + witch + UN +enn +AMP +vivian +whelming +mad + Asus + ARC + vap +surv +rises + lasted + chefs + starch + Airbus +stepfather + Ph +hearth + Negative +neck + ended +choreography +aton + critics +Tang + but + commissioned +dpi +icular + writings +clemens +icent +URL + diagonal + Illegal +russo + materials + findings + tanks + Satellite +ALTH + educating + found + Satisf + smiled + protected + Harm + relatives + poised + Ezekiel + Coordinator +alky +cussion +alexa + Haunted +zech + string +boardwalk +tool + crammed + recently +shr +cases +caledonia +mcgraw + adjoining +orc + Embassy + fearful + Davies + arcade + tet + upside + seeded + shif + detox +tx + theology + domest +quarters +iger + Undead + Scout + Jonah + Med +pitt + focuses +theta + pa + Chest +ests +whitehead + angel +licked + punk + Amateur +適 + GTA +dominic + Technical +nedra + interesting +hove + toll +iveness + proven + independence +subcontinent +itute + basics + open +itbart + 375 + Recipe + mandated + births + Sack + Posts + comet +leader +robin + renamed +abin +tna + vacant + plane +QS +uminati +supervisory +arcity + pipeline +terrace + archived +holocaust + desirable +Spoiler + ascending + seeker +uin +org + universities +parma +jesuits +aughter +gence +ministic + Produ +quincy +PF + Gu + Canary +murals + dimensional + hierarchy +irling +incent + malnutrition +damon + loads + fails + ignor +travis + Kyle + Pic + compensation + hello + mismatch + 503 + newer +uments + Hartford +aberdeen + NF + Northwest +_______________ + larvae +SNS + externalTo + mathematics + Kor +sports +fake +亲身经 + avoidance + Spit + policing + camps +final +madrid +liberia +monkey + Dwarf +middleweight + promoted +reissue + interactive +inguished + murdered + Bulls + Dew + infiltrate + Ground +Jew + Jamie +Blocks + clearer +regatta +Trans + Soc + 345 + finance + Fein + metadata +ishment + Observatory +seaside + implied +illing + cru +inverness + comp + endured +Materials + Arnold +中年男 +white +embankment +usions + inflated + loss + Mania + playing +magdalena +ethel + advise +angible +playhouse + Toy + roomm + Gam + lear + CLI +ener +eur +louvre + Sax + Lexington +ential +ted +measures +common + LDL + annoyed + somewhere +ponytail + Demon +daytona +stery +howled + honesty + Animals + haste + vents +horned + Sched + conversation + anatomy + Osh +uns +mri + almost +mobile + Isaac +tex + nm + prohibition + obviously +thrusting +ogg + Utility + negative +marquette + Gospel +kellogg +inuit +laden + Philips + parchment + respiratory +Stock + anecd + Wembley + Forensic +TC + Dong +shrubs + Regional + transformative + particularly +ovo + Angus + Shogun +adored + Plans + deg +employed +mundo +YA + Circle + semif +buddhist + TEST +albert +人工智 + fix +sabine + Archive + Personnel + spectral +theologians +rutland +gery + proceed +umbling +countess +inct +adier +lia + teleportation +Training + Consortium +tb + Bluetooth + lol + produces +ressed + seats +someone + Proc +buckled + mark +hal +atism +ailand + Mi + Kremlin +lorenzo + dom + rye +andhra + lottery + engineer + fragrance + Resort + daring +joao + incons + Reed +wrexham + Compan + biking +taft + fundra + litter + ie +onduct +elia + verte +nigel + cult +batch +DVD + Friedrich +dimitri + semifinals + Playing + GNU +awks +CMOS + additives + themselves + Vance + visitation +communes + Boehner + hamstring +pelled + Piano + useless +迫不 + coughing +mirza +claire + Viol + la +Known + Scholars + cohorts + TTC + inhabited +eches +anco +iler + Sly +Business + gul + goose + sandwic + badges +detroit + stays + Id + seek + tournament +macedonia + UD + mine + rainfall + height + grasp +ved +oi +bwv +Property + Torres + detainee + IPCC + Duo + Fib + Simpl + serv +thin + Connor +CR +Privacy + hometown + imposing + }} + 22 + Watching +hes + initialization + rolling +SK + 395 + index + individuals + Sheen +worst + Divine + hurricane + complain + researching +etition + swords +haji + acutely + nests +gardiner +TeX +haps +Talk +pull +arid +drury + visionary +approved +crow + Clarence + Rothschild +umbn +oph + acquiring + Big + Single + Arena +Iterator + Insurance +喜歡 + "' + formula +sikh +clutched + ultimate +hampson + gruesome +encore +account + Manga + rebuilt +monroe + terminating + collaborate + riding + animated +ertain +gart + malls + intimate + persistence + offerings +opoulos + Stard + dot + Horowitz + engulf + Hyundai +ICA +Screen + Good + blaming +abama +官方网 + envoy + awaits + FB + sensors + doctor + Wesley +倒不如 +gilles + benchmarks + Enemy + 263 +robe + Kelvin +magnum + ADV +ism + capt +ATA + Fin + Koen +Typ + Devin +FS + warehouse +partial + alarms + helicopter + parsed +土家族 + Featured + backlog + Rock + allowances +konrad + Lebanon + Cal + Siem +bowie + cabinet + persisted + Saras + Suppose +ited +mandir + progressing +prises +amination + toile + Frog + Intercept + nervously +even + rests + nets + separation + Ch +zbollah + proclamation +caressing +arches +aryn + servant + Upper + ears + CLA + 165 + hairy +Chance +ogn + Chel +legal + Marketing + Abbey +def +UTION + skill + Shares + Dante + safely + Song + prizes +Following +aterial + gamer +ucing +grumbled + requesting +bergman + bureaucracy + talked + \" + Decre +ected + clay +uppsala +olic +lections +mud +actual +wheeling + Location + Jordan +oti + Plaintiff + homophobic +ties + holy +imate +小平 + devs + initiating + MGM +覺 +pport + Athletic +tristan + rally +sofia + details + anticipated +ENTION + contamination +Windows +howl +PU +solos + Creative +benedictine + processors + Pel + noun +harper +Size +graf +tu + Wynne +xy + slightest + pink + mileage +GREEN +flotilla + tents + assures + wow + Wel + spreadsheet + Activity + respawn +PR +mma + Paragu +IGH +ontario +kangaroo + tang +Inside + represents + OWN +kus +File + instructions + renown + tasked + lessons +BL + tolerated + Judges + singular + Steph + sucker +ottest + Ri + lodged + Debt +blazed + biologically + Sold + few +mauritius + ordinances + Masters + ), + venerable +gabled +opter + Bane +nanny +Pin + charitable +spectra +irming +Keep +Red +jolt +IRED +Make + simulation +maris + SJ + Scion + Walt + Section + trustees + Housing + Inner + Hardcore +asuring +holly +hiram +jr +kemp +Political + disk + pumping + Salad +iment +fires +vanderbilt +第十九 + transmit + contemplated + installs + slid + Book +主治医 + earthly + Format + framework + transitional + freshman + 802 +igning + footprint +sbm + ox + populist + pledges +technical + History + Check +iky +London + quer + abstinence + debian + elf +clusions +sherlock + Column +atoon + humiliation + goofy + Acting + ALP + Dynamo + agreements + describing + Sak +vertisements + charter + realm + Mono + soak + Fail +Narr + splits + tolerance + chaired +mandolin +asket +choreographer +ave + worm + Mixed +Ed + feas + Tree + Fram +vir +Fixed +powerful +lessness +encompassing + Soccer + charming + addr +liege + shel + instant + SF + chromos +tax + Gina + faithful + booklet + muc +newsweek + cast +themed + jerk +agatha + childhood +neys + kickoff + cycl + Uriel +airlift +upe + Custom + cavern + reliable +nai + Celtic +Sund + empower + recognizing + Domain +insistent +Mind + Screw + Eur +idence + OL +sup + ecstasy + Quite + Rodrigo +oof + PAT +spaniards +diocese + marines + impress +Random +urses + McCoy +moat + governance +olor + cont +falcon + Tex +PASS +piedmont +jun +ppers + strange + novelty +bass +platt +vik +story +bucharest + 136 + Yanukovych + Penguin +quarterfinal +Correct +emilio +popularized +rov +osc + Slash + enh +emies + councils + Tradable + Lego + 640 + + cores + refin + maneuvers +拉丁语 + rune +Southern +olics + Temp + associate + remaining +padre + walked + Scar +LD + Deadline +umped + pertinent + 203 + AP +纯天 + hike +sergey +ending + Website + remarkable +ternity +pathy + Rodriguez + Orleans +Shin + Celsius + lan + prosec +usterity + governors +claudio +paste + prolong + insider +dai + racism + predict + watts + clinch +ement + Buch +claude +izers +magdalene +turkmenistan +pereira + establishes + Symphony +享有盛 + knob + Especially + ${ + adds + Material +ression +Hy + relocated + Prest + underwent + Projects + Ham +Ko + 366 +greta + blot + participant + EST +pages + hole + blocked +Complete + adaptations + arriving + circulate +writing + POWER + slap + Audi + distinctive +emed +etsu + playable +bott +Pack + remnants + Neighborhood +karl + pra +Registration + complications +ways +nile + :( +Join + sub + bond + metabolites + Vice +adele + pronunciation + ally + cov +satan +corn +ML + Parsons + procure + acquainted +until + reun +boxers +ija +atern +ject +endez + premiered +roots + Damien +itially +einstein + empty +ospace + implemented +keleton +cur +blanca + Tiff + Moderate + shirt + leash + fourth +principality +villiers + billion + Sant + Agricultural +didn + valley + Loki + airstrikes + rubbed +asma +AKB + Extended +第十三 + Divinity + Poker + necessary +brad +Book +udi + managed +moritz +cuts + glare + gru +charg +conner + reck +urban +jolly +ooter +flying +iami +Writing + overboard + Sunder + compassion +hered + 312 +外语教学 +Bro +frey +第二十九 + Warden +fungal + dip + beloved +omsday +hidalgo + eliminated + fury + political + Mol + Jah +confidence +斯拉夫 +handedly + uses +ickey +rf + Panzer +sited +yin + yuan + Newfoundland +Jane + healer + chips + destination +Ec + Raleigh + cit +gloucester + uttered + notified +ISION +marshall +advant + collectively +project +ptroller +medallist + amen +ppings + rewarding +croatia + segment + tutorial +magna +JAVA + Kane +These + Hus +eyre + Maher + lawmakers + panel + lifted + Panasonic + addictive +dukes + breakout + faintly +request + pinned + ITS +nob + SAL + borders +amas +utterstock + Jihad +leg + competitor +mackenzie + validate + bombshell + assisted +steward +Pir +oenix +nike +国民革命军 +齐齐哈尔 +ifferent + shiny +OPER +arten + sports +entric + Martin +igsaw + gains + helps + REALLY +config +remove +aways +ocrine +eff +Nav + comm + Zen + expects +bert + duplicate + symbolic + Auckland + Yamato + doll + Physical + Jin + tide + Porn + dir +atech +video + velvet + profession +snooker + thresholds + stature +enough +ousy +verting + archaeological + Det +brandenburg +muttering +adjutant + Holl + anal +ipers +ipswich +朱棣 +ankara +block + 1050 + scoff + Funding + federally +於 +SO + precincts +{\ + beacon + heavy + Bere + domin +redibly + HMS + eurozone +Moon + spotted + Assuming +ilded + money + greens + corporations +address +wright + torpedo + coarse + Traff +ICS +組 +uitive + fateful + leave + Mia + salesman + VG +akedown +BLE + downs + pilgrimage + grossly + marrying +asper +bangladeshi + ubiquitous + 146 + guides + lum + stability + Weaver + Gor + Mix +webber +tian + painted +valor +electrified + 169 + Att + corporation + Bungie +tre + satellite +comed + feet +sys + derivative + behalf + disinteg +cicero +saving + Finance +Users + smoothly + Pearce +vida + picture +UFC +warmly + Discount +CLUD + church +Tools +idad + WOM + psychologically + accomplishments +laszlo + unusual +indus +CEPT + method +Configuration +arah + gazing +acly + Cassidy + substitute +gazes +void + if + set + behaves +Broad + affect + Hob + employing +oslov + licence + appropriate +Alternatively +Aust +aturated + Sta +hardy +Streamer + PSU + outputs +aughed +laos + subjected + banquet + paycheck + tant +antigua + levels +clayton + wheelchair + theorem +flemish +REL +Round +Available + Included + Wouldn + bou +inched +barbados + Shrine + Ring + embarrassing + ET + Heidi + Each + Somew + args +church + dollars + Bren +FUL + photoc + Taylor + seals + psy + morp + Consolid +mys + forc +dash +asury +boats + github +fledged +Thu +tunis + diabetes + descended + aide +Vote +nomadic + rescue + Instrument + Ender + DEC + generally + pointer + algorithms + Neb + Mouth + Mart +Feed +Frame +length +ados +Ex + Debor + LS + Penalty +政治经济学 + fond +值不值 + Integrated +Lind + Rebels +Storm + Predators + events + Plate +OME + appropriations + tweak + POL + they + intangible +ching +二百八十 + humili + Up +Actually + Advoc +mohamed + covered + 1977 + Dro + Nor + redevelopment + slime + pt +numbered + yog + Medic +seine +gruff +cretaceous +ede +graeme + hanging +umo +latable + Kas +Cent +Wave + blade + Tuc + rhy + deflect +daryl + sensed + nurturing +grooves +magnus +unin +portico + scholar +MET +zag +izzard +itol +esting + ensured +ampoo +orthodox +illi +uously +redskins +gujarati + invol + Fever +ARS + accommodation +gans + MV +campeonato +ilst +terre + subparagraph + 89 + constrained + swiftly +iances + Barber + Directorate + Ent +whitish + aligned + pres + enlisted + Izzy + serving + independ +Liber + observers + Hav + Cyber + Renault +Faith +NTSC + nationals + wrinkles + designate + Elk +saskatchewan +alias +groves +inational +Important + considerable +Camp + bowling +katy +ularity + shove +kuala + rebuilding + Plan + dexterity + inhibition + dodged +Dark + hypnot + BMI + reminders + endurance +malone + rescued + admission +Rain +urden +lucent +Stone +film + affordable + surrender + (_ + Tide +zhejiang + whispering +acea + microphone +aaa + molecules + inaction + Helic + 440 + Hale +nicola + robotics +tery +propeller + Reich + Ning + rational +IME + asteroids +sparsely +tasmanian + Crescent + Ts + sediment + clue + Dmitry +britney +seminole +cnn + toddler + cryptographic +sawyer + bour + Activision + Scouts + renowned +GPRS + migration +courtney + Zion + Island + Jerusalem + bullshit + vertex + VIDE + copper +ECA + transpl + sans +Reb + Similar +ampunk + DJs + Possible + marked + FIA + franch + MacDonald + htt +arsh + Mag +franco +essors + samples +archbishop +Way + Nicarag + 173 + Typ + Pep + caught + haunting + enduring + help + science +narayan +unharmed +uzbekistan + devastated + loop + ads +cig +rumbled + motives + nefarious +PT +sloan + sparked +bottom + Whatever + parallel + tip + neck +COM +Williams +gunnar + rust +socket +erers + constituencies + Heritage + auxiliary +NOTE +highness +montreal + Veter +rohan + Communities + 150 + Lil + Coy +ique +helial + fuck +蒋介 +most + 34 + Kitty + detective + thanks + reviewed +philippine +harm + unemployed +desc + amounted +episode + Jae +dso +stanbul + Eater + unclear +ocker +competitive + Sund + settled + ammo + amused +portsmouth + glam + submarines +ince + KH +cornwall +snap + asbestos +frederick +azo + undergoing + DVDs +posse + Treasure +pamphlets +nesota + heard +Contribut +each + Babylon +ptoms +Liquid +epithet +sensitive + IRA + formatting + Dee +antle + figures + Lenovo + safer + DOC + skeleton + Brian + grass +scrib +clube + mathematic +harold + Auschwitz + eliminating + abrupt + payroll + dope + maniac +Unity + Rare + 2015 +shtml + cages + followers + banned +acre +chinatown + intriguing + silent + China + Albania +^{ + Stone + stakes +sanskrit +Prot +bruce +stall + beaut +caretaker + Width +october + analysed + decom + visceral + sockets + Sebast + landfall + vas +discrimination +二十年 + appar + Variable + Flavor + AFC + leptin +onde +network + "\ +thing + Valley + allocations +pontifical +hebrew +討 + Yelp +satin +boroughs +Dear +fly +harlem + impacted + Seth + anterior + revolving + wards + supported +attery +valentin + notification +forested +davies + bleed + Uzbek +lowell +sal + finished + nervous +vP + Note + objections + Telegram + 31 +cised + Chung +Global + elevation +Ell +angie + waits +internet + volumes + charging +vacc + instructor + unsc + kn +cbc + Moy + crush + Extend + Buffett + 215 + Breat + infrared + Cutler + society +rush +omen + Mulcair +ied +Box + pine +負 + Vie +space + universal + Init + Ding + Fed + legitimately +Kyle + Rus +introdu + presumption +第七十 +hampshire + victory +DEF + Damage + finer +urrent + World + Observ + LCS +grille + interrogated +fiddle + modification +maru + Fen + idol +vila + Nid + gorgeous +uffle + Ski + growers +ydia +ady + seal + confounding +ET +activate + Tools + Casey + Liver +environment +ositories + exercising +designed +eryl +NM + encouraging + soft + CG +Bull + extremism + Ninth + antip + conferred + nour + Congressional + Reno +zy +Adapter + obey + warm + perk + 82 + DRAGON + consecut +得不偿 +rejoin +montpellier +hazard + invaders + IB +hernandez +leground + Example +rockies +ordinate + Resource + trousers +ini + singing +gov +shivered +atively + Cert + Vir + disse + Come + attracting + Due + generous + Arms + Kerr + electronically + indu +hovered + Europeans + dominated +call + Reserv + Period +reenshots + LIB +IT +ferrari +villa +sis + contended +cri +cologne +mussolini + budget +楞 + approach + DOT + sealed +lime +bangkok +loe +silky + ASC + Router + precaution + MPH + Adrian + Goodman +alian + Essex + Jord +vhs + Newport +stay + SAP + tier + gravity +params + yea + Barney + soul + Memorial +jake +Ind +commemorating +baron + resembled + illusions +catholicism +最合 +austin + Britann + Awakens + Wolves + colon +ISC +ffiti + VIS +MAC + inev + renders +Flags + ordinarily + ly + ash + distinguish + cheaper + Kick +VO + Victoria +finnish + squads +ammu + department +july + hastily + compens + upbringing + seminar + of + CLICK +sorry + grief +EF + horm +iban + deported + accordance +asley + Bav + Autumn +eties + Illustrated +Dialog +shadowed +facts + NX + perm +pling +itched + sandy +hawaiian + load + Cuba + outlandish + destined + midnight + cruising + conclud +airbus +izu +第四十八 + Thunder + quote + Priv + abs + sprang + climbed + Saf + bundled +Th + Barkley + peel + sausage + bloodshed +saetan + marsh +peabody +align +ovi +bys + sublime +BU + termination + education +pless + incidental + Vul +Vector + Dise + should +Kh + fro +little +point +asters + | + Degree +WASHINGTON + [ +potomac + sucks + Alternate + rotating +shire + widening + patron + Gerald +中外合 + migrate +panted + pronoun +Tickets +acey + committee +不活 +repre +资格证 +OE +doesn +chrome +Sequ + Row +lucie + Miss +Mor + petertodd +kellan + reliably + tract +MEN +conference + Riverside + encamp +picturesque +ethyl + Creat + diabetic + articulate +list +buster + PROV + Barcelona +jim +amorph + UCHIJ + selections +ilingual + USA + foolish + preserved +emonium + agreeable + mang + palace + Process +hymns +ismail + Wednesday + catapult +elena +Murray + Ling +Later + anx + deputies +abbreviation + Mosul +sussex + Wrest +iggins + jobs + defendant + totally + propagate + advised + Tonight + sche + Brief +vie + Sith + DON + calculus + eccentric + curv + brigade +unks +jameson + Rita + juggling + invasive +abbit + HDR +oru + maiden +dartmouth +Virgin + Proof +umat + SU + indoctr +unopposed +international + executes + Saudi +scar + Fargo +updated + invade + exalted + tie +flanking + artist + GPU + handful +matilda + Designer + curry + Sioux +ESSION + girls + betrayed +yanking + Hip + attRot + Enc + puberty +巿 +分之二 + slang + representative + Laurent + 470 + ludicrous + Yahoo +wagon + obscene + Cyr + Attorney +Rat +From + Paramount +sylvester +Hom + Symbol +words +confesses + repeat + houses +abbas +beaver +drumming + Laurie + folder + Switch +這個 + NASL +pamela + hiatus +georgina + standalone + preserve + stole +scandinavia +家用电器 + contraceptive + SEM + moth +义无反 +ency +rozen +dail +四五十 +mumbai + flashing + Everybody +macintosh + contag + completely + Vert +locks + antiqu +obs +atchewan +yugoslavia +funding + id + bunker +波尔多 +pajamas + Nan + virtues + Cuomo +maison + thread + ser +quite +right +leningrad +ql +WAP +feldman + awaken + 1944 + numbered + fo +uthor +FACE + Machine + 429 +wilder +dq +boone +isSpecial + 269 +uden +William + Asylum +轰轰 +involved + Jonathan +juniors + Introduced + electron +姫 +asonry +Jacob +eds +autical + Gabriel +第一百七 +Magn + cannibal +Gun +ansky +heaving +described +wisconsin +prequel +offs +fried +izard + plate +doherty + Crime + trem +othe + Tsarnaev +washer + Radical + Kad + derive + Comm + spark + Amnesty +hillside + ill + chemist + Kw + disparities + prefer +ropolis + parity +initions + caregivers +jena +NC +kes + Lucius + Transform + ORIG +sumatra + Pis + Walters + guilt +iliated + Drew + sir +hedges +民大会 +rouen +marguerite +Da + recount + decreases +obar +chen + Values + contribute +iary + sap + trek +KG + Corm + messaging + Goodell +inducing + editing + Consumers + folded + imper + om + happen +yeah +nton + Qur +backside +Due +fountains + sentence + verses + breasts +stack + Jiu +Japanese + flattering + comedy +auckland +Stra +pure + infected +ustom + crest +agn + toler + surg + password + movies + Esper + Tant + millennials + bluff + filmed + return +|| + interface +jon +rak + KC +antom +examination + Canadiens +directorate + Refugee +guard + remembering + dumping + tuber +氵 + authorised + withdrawals + Strategic +haust +otis + boat + diagnosis +marina + Fallout + BDS +test + inform + visualize +latest + Lotus + Sylvia + Association +yellow + facility + Tir + plaus +NYSE +olutions + arithmetic +boyle + Lenin +Lair + glad + Cells + deval + domestically + venue + allotted + hadn + Ges + french +宀 +ucky + flour + discrim +Rob + Ryzen + ends + criminality + AW + rare + possession + strap + estrogen + mol + Talk +Charges + videog +CDMA +Thanks +nix +}{ + Chronicles + pert + turned + harassed + Cindy +PD + gross +reid + at + BD +ivable +gong + follows +ievers +pit +Sing + skirt + pharmacy + incub +maud + DEBUG + ounce + Benson + allocated + Raid +ikan +lullaby + Benjamin +錯 +chron + Additionally + Pagan + Minneapolis + deserve + creditors +javelin + encaps +ein + lever +haute + oxy +Laura + frontier +arturo +vicky + prophes + regrett +lori + infring + Pac +ansk + Billion + citrus +Style +ifully + Drift + Hots + names +isco +ees + mosa + Spielberg + prof + MA + ineffective +remade + managers + exceed + Ole + proficiency +georgie + Umb + catalog +people + fulfilling +ases + inequality +rypted + cruise + competitiveness + mach +larg + VK + cloning +hooker +fishery +tsy +atar +Memory + neur +ruct + fals + applies + Quebec +orson + apprehended +ims + suggests +juarez +oaded +On +overtook + knife +ensibly +Introduction + 1890 + hotly +Me + CONT +vers + slammed + Caesar + da + cathedral + Beats +jerusalem +endered +impro + Sunrise + SEO + disgust + investigator +DB + factor +rogram + Sweeney + groove + brun +waukee + policy + revival + Cer + directions + cout +rows +beijing + Among + Codex + loudspe + life + pound + swept + diff +gareth + 10 +aja +ieving + \' + reserv + Acqu + shields + Doctors + stockpile + delicious +sparhawk + South + moderators +cassie + debut +證 + geological + courses + ObamaCare +SU + Daly + departed + Bulgar + HR + ultrasound +case +rating + wiret +hester + scoring + godd +mining +juventus + implements +finals + Dick + Fa + Hatch + solid + objects + aux + indicators + Sorceress + alt +danger + Guess +nigerian +発 + Drinking + Rep + Sword + paramilitary + Friendship +crit + Charisma +mayo + Binary +term +monte +housemates +esty + wheels + whit + contextual + rebutt +EVA + hysteria + intimidate + titan + ($) + issues + misled +Products + mesh +BILITY +qual + Elizabeth +dh + satisfied +Shar + yourselves + hiking + Opportunity +stalked + Dj + courtroom + missions + Associ +aceutical +lease +napkin + lays + clustered +UTH + Now +receptionist +coloration +hetics + Britain +ogle +百四十六 + Vox + piracy + extent + Report +Bonus +EStreamFrame +anova + subsc +Reloaded + forums +Ak +ration +link +think + usb + animosity + moderator +pta + doctors +isner + Plymouth + garment +sirius +atibility +melinda +compton + calf + explodes + storage + Thom + lime + corrupted +lethal +viceroy +pointers + Hep +Focus +pagoda +第十一 + masks + 144 +mitchell +iah +nes +ruck +docs + journalistic + Anna + Ak +motion + 232 +Gameplay + additional + EP + condone +given +十字架 +public + Memor + Fallon + Bangl +uador + derogatory + foe +jaguars +consultative + populous +swers +ror + Ajax + Silicon + disrupting +finding + quadru + succeeding + Witches + Vor + superst + Parking +context +eduardo +omach + Sam +option +goal +saskatoon + alliances +resents + Lilly + Provider + scream + pulmonary + contests + seam + engaged + export + prohibit + compromising + pregnancies +chow +None + Cao + freedom +crowe + Ignore + synopsis + Transit + Lon +lio + Patricia +known + pie +PAR +mails +cair + microscope +reen +running + Less +kindle + impede +super +umann +aters + embell + Tru + Researchers +Impro +oppo + illeg + mutants +umption + hyper + pregnancy +atkins + Ferry + Clement + Animated + Night +asar +incre +ourage + GC + dh + waterfall +Once +India + Mith +ricane +gable + night +TRY +epilogue + gir + Durham + Rou + Classics +nightstand + logical +Uh +soType +byzantine + PERSON + interviewed +reverse + clothed + roadside + conflicting + Heisman + blessed + SOC + 1932 +jamestown + (' + 45 +poons + reusable +英国伦敦 + ring + punch +pentagon +physical + jacket + uns +Rober + scheduled +bernardino +sonny + Wilderness + Rainbow +oscope + Anarch +tn +Pla +eners +sand + stren + blockade +Al +evicted + FIG +Native + Gould +Cour +jenny +fu +Indust +WIND + Election +ulously + Oz +lesley +Ice +IO +ljubljana + defenseman +ior +uctions +二十二 +gren + flashed + generate + Mayhem + briefing + liberal + kh + McCull + flaws + label + Chal + appraisal + vomiting + Quant + anarch +sandra + 1935 + Enix + Madonna +voy +produced +rammed +dublin + LTE + lends + unity +Chem + gadget +rone +freddy +collect + Deity + Veterinary +Fa + polling +daniels + foss + accol + Pe +ergus + craz +哈萨克斯坦 +東 + unfl + obsolete +ulo + marched + Rasm +chery +uli +aco + spew +ANA + symbol +worn +support + sits +philis +opening + endure +七十二 + Phillies +caps + enlist + ledger + BW + Argument +France + Zerg + Willow + collapsed +ANC +诸如此 + Archangel + rigged + conserve + Mov + violates +printf + titled + Dracula + Riot + cope +wiki + cones +PRESS +originates +abases +faust +gunners + catastrophe +igator +king + Theresa + departure + respect +iliary + Barcl + Tenn +irds +enary + downtown + Century + Meng + Santorum + paradox + paused + Programs + brightness +cordoba + respondents +較 + Vil +tiny +Peace + caves + depressing +inv +racecourse +ctor + sentences +malley + reality +rue +fabian +asting + Bloody + para + compelled + Venice + Pau + Cliff + progress +sea + clans + extra + 1200 + crypt + mutually +PDF +Normally +ointment + transplant + superstar +sters + consumed + Entered + archive + reduction +武夷山 +regn + Detection +aez +handle +blogs + succession +balance + prim +mutter +Cry + Pace + hug +sector +行尸走 + unheard +kilda + infantry + 178 + Lindsay +relations + Armour +毫无疑 + bottle + distilled + Koh + bodies + flickering +地理信息 +objects + visa +Tell + recounts +cmp + SM + DO + Sciences +tical + inventions + brink +aver +evan + Lee + forty +ultimatum + cos + oxygen + ICC + components + exce + extensive +adobe + cort +elaine +Mill + inflation +ourcing +不言而 +FactoryReloaded + Jose +chrom + machine + dismissed + However + Lys + shows + displacement +medley + patrols + Arch + testament + accompany + formations + Nish + Messages + Rates +hc + perpetuate +lilly + 43 + nont + unn + sorting +布什 + scoop +telugu + overflow +google +Connect + appellant + code +aurora +inferno +fatima + Boone +alkyrie +lutheran +第二十二 + elev +salford +ourning +Credit + NON +gae + celebr + highways + earners +archaeology +Sep +ixel +infirmary + timed + Tibet + Kaepernick +Subscribe + cyber +fridays +itage + internship +rite +IBLE +puget + stand +Mon + decre + treasure +brianna +nown + whim + distribution + Somehow + where +enance +problem +囧 +img + Daniel +cornell + Spiral +thames +slave + painfully + Baal + wasted +CL +auga +bowman + 359 + Whe + GMT + disple + Comments +結 + RN +HAM + Negro + Clin + timber +遗传学 + Bible +cantata + prem +phis +papers + Cann + onward +cill + liquid +iris +GF + Rab + tracts + Rehab + teasing + Residential + tracked +knotted + takeaway +Stage +haggard + Destroy + dri + buzz + COM +delete +ocally + Pan +PHOTOS +arch +pau + VII + defines +agents +sen + search + Rogue + ext + Bristol +highland + ORDER + KE + oversee + Considering +otor +DP + Tah + cliffs +imation +winfield +ItemThumbnailImage +iously +pedestal + throttle + buck + diverse + suppressed +norman +mecca + nearby + tru +peru + pure + incomprehensible + Freed + notorious + reminiscent + spend +cerambycidae +asking + Bak +mps + Koran +immigrant +competes +李鸿 +Crystal + ff + irrig +prentice + attend + pitted +rend + censor + Couch + balance + intake + pragmatic + openly +ixir + correspondent +inton + parliamentary +Interested + SD + fitness + Sweet + Obesity +yellowish + died + ceases +^^ + CL +austro + stall + Herbert +hammering + Griffin +judicial + unavoidable +kobayashi + wasteful + 411 + radioactive + Ian + recalled +biceps + renewed + realism + rec +传输速 +irsch +Boy +daisy +smacked + Werner +eak +american +killers + printer + Catal +oples +rus +structurally + sens +rary +stable +flu + clone +matthew +uu +\": + HPV +REP + sidewalks +Yellow + ASCII + periods +achment + ATF + Heal + similarly + documentation +tips +SF + polymer +middletown +sweeney + Firefox +Give +lock +oola + tells + Pai +GAN + Rye + Tue +christ + website +affle + thyroid +osit + unseen + prejud +生态学 +天主教 +weibo + arsenal +aster +TM + 610 +ctory + Bac +Choice +consecrated + Recommend + Shant + ster +ranch + unresolved + Unique + vascular + srfAttach + Rev + Scottish +gilbert + us +ixtures + dred +barrie +capt +leave + Rear +ordial + barg + motion + Dating + bottled + Begins + drilled + parach + Leonardo +Args + reve + facilitating +indonesian +onia + Massive + disable + Scrolls + selected + Rew + fam + profiles + Lung + consolation +discus +alligator + grinding +呜呜 + condemnation + reconnaissance + talent +yssey +Len +lindsey + operator +uan + weekends +glover + dive + analyze + Kay + Vacc + qualitative +vious + misunderstood +How + Stewart +easton + Pur +vanessa + sinks +fraser +那麼 +afghanistan +flows +currie + overshadow + interns + Bos + grizz + SYSTEM +dubai + compare + Harrison +Ship +hilbert + dr +warm + (), +enthal + Ost +assies +harrington + Mansion +bred + Gross + Leeds + poop + ident +num + organic +fulton + Butler +喜不喜 + Mississippi + feel +does +ennes + powd +outs + Bolshe +small + cafeteria +iatrics + assumed + Amph + Kh + dys + sit + Tosh + Sp + hydra +ippi +moines +cocked +otten +iotic +rophic + zero + Colleg + Latter + oper +cules +card + progression + physicians + disagreed +duino + Treasurer +onne + Johnny + writing + epoch + vanishing + valleys + ingen +rano + cup +Rog + Hub +mcbride + discrepancy + 341 +uding +ICT +arkable +direct +hinduism +Need + Gal +热力学 +jac + ti +mab + lin + marks + Naj +DM + Noon +schultz + entit +vane +GET + Ber +athlon +Christian +郭德纲 + blindness +RECT + Crab +devotees + Barker +gray +color + deposits +reement +mora + wild +brates +zynski +vine + S +Script + coroner + racially + Yose +hurley + Irwin + assessments + Punk +pierre + cubic + dared + speeding + ve + PLAY + suspect + amput + qualifications + attacking +icable +rana + EMP +半透明 +genevieve +nsw +uci + peac +westbound + evolve + Equity + escapes + Cash +ijn +Mod + Monte + medical +igree +auntlet + whopping +rolling +iations + unre + Mot + perpend + campaigning + cant +andestine + Schiff + stacking +Help + accessibility + knee +orchestral + Prospect +Application + academic + wellbeing +duchess +riga + 149 +governmental + colors +Production +installed + cinematic + Hussein + runes + Sessions +bash +iating + formatted +wilson + Disabled + sexually +yu +Translation + Apr + Pyongyang + hurried + bi +antoine + Cinderella + SB + smug + indul +ruciating +車 +ophers + Bundesliga + textures + disconnect + functional +boca +ITIES +tokyo +第四十四 +ague + As + Magn + entity +saxophone + Fighting + accelerating +Matrix + Diana + Push + residual +cking + meaningless + Tracy + yells + Hyd + outpost + grade + Wildcats + attempt + linguistic +tap + sorts +Characters + filthy + generosity +學 +ulum + agriculture + media + Jenn + illum +Said + spikes + mad +isms +orne + patches +beam +imaru +stoke + stra + directed + Tit + EXP +armed + contraction +election + Su +langley + Zac +hma +Interest + requ +面向对象 + X + claiming +initely + Citadel + anywhere + App +bowler + leaf + aur + handguns +brownish +utions + presided + Prism +esar + Mare +owens + therm +chuckles + claimants +ocus + Hulu +四十一 + patriotic +cup + Belichick +month + haha + Charleston +athe + 404 +theless +習 + counselor +Rew +毋庸置 + applied + Obama + Lana + Erie +msg + populace +week + versatility + experimental + unab +abbot +More + irrelevant + Nice +说不上 + rog + overlapping +dusseldorf + premise +Bow + benches +bot + Ammo +Auto +registrar + microbial +refurbishment +urdu + males + Darren + Loren + Walking + ten +galen + Antonio + noticeable + scaff + longest +planck +nationale +verdi + Unch + aviation +practice +bound + Cart + phosphate +Each +inary +CDC +ritical + Abandon + slash + clitor + hyp +Putin + staircase + Scores +Maria + aura +ateful +Comb + flattened + consumers +cemeteries + literature +stream +gesturing + piano +TI +irst +Examples +iblings + wreck + haul + Pulse + O +weekdays + either +ACC +alistair + Him + elapsed +olsen +uggest + suit + anarchists +schwartz + meaning + Tayyip +ribbons + whales + Byrne + basal + comma +ira + Structure + enhanced + Ott +糹 + worker + Jimmy + Bare + Romo + parasite +eo + Inquisition + emulate + Pink + signify + marry +malay + lose +Scene + passed + Critical + racket +noticed + headset + Regardless +mut +onge + sellers + marching + toppled +strument + aluminum +ooth + Apache + compulsory +rei + commented + ballet + boon + Patrol + STD + menace +clockwise +taiwan +City +workshop +pee + Rig + perpendicular + leaping +forcing + thro + Assessment + incredibly + fringe +paralympic + soy + diplomacy +icket +herical +philippines + Broken + repl + Volcano + Liam +submit + malice + dove + Tiffany + Desmond +BW + Torn + McDonald + burdens +maureen + silicone + pivot +LAND +punjab + Violent +staking + DevOnline + accuse + WiFi + cigarettes + Wing + holder + OU +bosnian +cells + Led +packed + pleasure +repertory +possessive +triad + vaping + Cla +ACCA + reciprocal + emit + matter +艹 +colbert +Questions + lic + tens + Repair + Charlie + Amendment +dor + Personally + incur +xp +stagecoach + convoluted + Heads + mercenary +vietnam +kinship + equate + Malf +melissa + magnitude + screenplay + Notting +annot + boosting + acts + serve + ECB + gaps + verifying +NZ +print + payload + WANT +ganga + Exper + coined + pistols +Studies + rocks + nodding + Dice +Rest +enveloped +应不应 +iji + bene + Sah +第五十一 + Mustang +再也不 + Architects + Restrict + constructing +rupert + monthly + flag +opol + shovel +petals + Moon +Syn + POS +dollar + scrutiny + Blind +DEN +montane + Lent + Index + orthodox +astically + pride +phe + Kinn +weir +SKF + wom + generational + Mas + wrong +holstein + Wim +cuthbert + Bauer +aidan + EV + Most + chili + captains + Sins + OFF + Zo + Bring + Wilmington + pit +Dial + Yi +Clar +mite + Sacred +enko +ributes + tubing + Heaven + methods + Meredith +suit +ww + Duration + Worker +Brain + Continental +Mods + Lakes + Aleppo + Mald +iltr + collagen + Wet +ites + foundations +blade +slayer +damascus +勝 + FANT + troubled + mantle + Daily + shifted + Eleanor +islamist + Mord + rigorous +rieved + minerals + Fog + Go +chedel + Galactic + 336 +tenderness + institutions +catherine + canceled +airfields +iens +digger + awoke +ento +execute + SQU + undis + fant + '' + HOME + headaches +arte + Rober +Tra + filling + shipment + loyal +aristotle +wicket +stuff +手工艺 +rogens +Ironically +amin +Grid + 274 + presupp + ascended +oultry +turing + documenting + Olsen +clary +Son + blankets + flap + genius + dispar + Blockchain + Deg +loyalist + period + recite +ook +社会活动 +ighton + extinct +Fil +riot +CONT +northampton + driver + ringing + priority + Penn +opa +Metro + Cookies + GitHub + prin +gaulle +oly + sonic +delle +clear + indisp + rushed +vista + Norse + lions + Floor + sequ + prov + El + Diff +jerome +hagen +orsi + tremendous + randomized + custody +jam + Advice + modernization +ividually +html +disciplinary +wives +ovsky + URI + Jazz + Germ + STAR +Ns + Debbie + friendly +Had + Socket +喜怒哀 + Reuters +blizzard + Clay + childish +kb +fect +fulham + statue + Mus + Glasgow + Da + overe +pent + influential + simplicity + snipers + Movement + Delhi + pregnant + Empires + scenario + tink +hemy +seville + Fashion + belie + corruption + Entity +tested + self + transgress + barric + Yuan + Bunny + ener + cosmos + Granted + refuel +zeus + Disorder +alejandro +plume + Hon +andel + fundraising +entimes + misrepresent + alters +xff +republished +ocobo +UNE +buffy + visions +author + Protestants +則 + organ + hills +coptic +shriek +atten +jews + Sr + CIA +Group +gemini + emerging + automated + Agent +FIR +Same + dist + demonstrating +bentley + 1862 + Twe + Mahjong + mud +halifax +iery + Actress + complication +protected + console + Amid +avery + Sigma + prolifer +sousa + bus +life + chopped + Desire +Still + pur +Def +culus + Heist + former +Ord + resolutions + fellow + ], +ortality + Leh + Genie + detachment +hudson + Uttar + antagonists + Text + consciously + tornado + kissed +oslo + burger + EG + slapped + confused + Manning + More + direct + eats + TEXT + fulfillment +hesse + ner +kowski +saf +ico + La + Bever + cancelled + cramped + attribut + Tune +Load + Guang +ynam +nassau +humans +freyja + ov +imus + offer +OTH + sliding +鈴 +warren +ournaments + Syndicate + Partnership +stephens + initiate +AL +Password +无动于 +Dean + descriptor +collarbone +ittens + endeavour + Vis +guerrillas +utton + exercise + 423 + discord + Ratio +pena +China + Protein +mate + Meadow + Bless + Murder +ilo +sixteenth +february +ilver +BUR +marie + abolished + Detective + LINK + '. +rero + Wisconsin + Ny +oother + BMW +Going + anthrop + Susan +soviets + WW +atum +Ray + despised + Widow + Roses + seizures + based + 194 + Away +osure +TD + establishment + pus +death + autonomy + 2001 + ridiculed + fluid + Titus +zero +colorado + psychiatry +cowboys + crates + fielder + Jones +priestess +madam + Finish + badass +RW + arcs +Shift + foo +戦 + outlets + Clerk + Yon + defy +JC +龍 + adherence + Gl +微电子 +ogyn + Insp + profitability +DNA + expl +carmel +terson + Liberals + Crossref + purposes +seamus + Convers +bill + intentions +fife +schuster + tune + protective +angu +oblivion + GRE + diffuse +exec +journal + thrilled +法国巴黎 + regress +sunderland +裡 + gets + upward + Owner +cdc + ecological +Values +greyhound + Radius + Boeing +Bridge +Send + sterile + Sexy + May + architectures + bidder + iconic +esis +orce +Score + escaped +outlaws +apan +anas +ference + humanity +umper +circle + overall + Australians + Future + Benefits +steppe +ivalry + Tunnel + Spons + Duff +oller + Celtics +Reviewer +tessa +guangzhou +課 +eireann +MG +ared +atters + hairst +oper +unta + Shakespeare +ndra + McGr + Perfect + Last + reefs +minecraft + scientists +Discussion + undergo +were +acular + liquor + fiat +Susan +balls +patriots + trending + transmission +phones + neighbouring + stock +edar + sensing + FAA + lane + steep + war + chin + Vengeance + top + Es + Barbara +MMS + Connie +ribed +uls +cocky + 406 +Energy + gems + mysteriously + Alchemy +jia +ancock + unforgettable +ubb + clarity + vibration +Can +nav + fil + Rhode +fur + defects +sylvania +stephanie + wre +osion +zip +regis +oklahoma + thousands +aut + products + lamented +Elsa + gist +erning + cuts +hole +生产厂 +orians + wand + flock +solete + fold +nueva +bridgeport +WI + commentary +ammers + Merry + fra + Machines +swimmer + conglomerate + extras +gunner + punished + hardly + TheNitromeFan + consisting + Wonderful +rien +armagh + icy +moi + stigmat + communism + Franchise +moody + Farage + vertical + MJ +ornate +shown +Position +红外线 + @@ + Cov + triangles +westchester +花岗岩 + commuters + Pass + provocation +ctrl +eter + financed +abwe + Sapp + Krug + sacks + march + gauge + referendum +aspx +obsidian + Nap + Therapy + Dortmund +ijk + Done + Breitbart +nonfiction + hatred + Eight + proteins + East +amour + answ + vows + upgrading + recy + HB +nex + mode + Catalyst + psycho + statutory +我們 +Conclusion + Kn + Labs +galilee + stringent + prized + reluctance + enforcement + widow + Fem +Sym + sink +Break + Reduce + persecuted + vigil +長 +bromley +Graphics +ofer + accum +pembroke +whimper +ebin + prey + cohort +几百万 +crop +musicals + accumulating +ideos + Burst + 1961 +olson +isles + complexities +Asked + Hin + trad + upholding +byrne + effectively + Georgia + USSR + NDP + reapp + myself + wiped + ven + eclipse +earchers +ournal + ratification + Lazarus + chapters +witch +collin + indefinitely +need +nokia + contest + hoped + 102 +vito +oming + helping + exterior + JR +visual +lim + ardu + ex + Laugh + Bowman +impatience + Underworld + emoji +ries + pi + suff +aband +aban +Govern +Eva + Flake + hom +dina + Kou + Till + turkey +brunette + ret +bristol + Breakfast + explores + demonstrations +戏剧学院 + Garry +mone +Font + warrant +Moreover +ashe + pleasures + Numbers + swipe + envy + creative + adherent +loyalists +Res + func +TB +adden +donnie + Win +momma + disinfect + regarding +buzzed +lancaster + Arcane +lit + joke + Supply +見 + Il + formerly +esley + implanted +Israel + objected +Athlon +\. +urrencies +Versions + superhuman + chunk + have + Liverpool +ians +Stay +Sport + ICO +steps + Marriage +CIA +Day +anna + IS + Ney +QUIRE + power + quoting + proverbial + Indigenous +bogota +ocused +lucifer + interventions + Taco + WhatsApp + audit + excited + praying + indictment +gression + unfolds +Site +数不多 + commandments + Yar + scen +immigration + CSV + bounces + behavioural + sewer +Origin + grown + Machina + film + qualify +coke +hayden +baugh + inspirational + tong + interval + Giovanni + hypotheses +conde + seaf +bending + specifies +opath +ape +poke + Buddh + Samson + Cous + trips + damp +第十二 +aby + faced +atown + neurological + conferences + pads +requ + su + testers + believable +enton +algerian +迪斯尼 +protection +posthumous + mating + columns + Marcus + switch + pointless +hanoi +LR +cens + golf + superb + nomination + maritime +atlantis + expended + Quan +eport +arget + Thousands +userc +sofar + Berserker +stantial +七十年 +領 +font +rivera + RH +honneur +Strong +pas + pillars + Donkey + entr + lessen + endanger +untu + revisit +Im +plane +authent + heavily +cyril +hui +KEY + educators + pulp + OUT +gould +rafael + grease +kristina + tits + accepted +penguins +pianist + crimson + Veh + kilograms + gently + psychedelic + examination + Lloyd +maddie +ander +itement + hive + arising + Cannabis + ceramic + Northwestern + buying + derail + Mau + scratch +crewe +Date +lling +oit + normal + helm + workers +book + unit + edged + Nation + 1917 +crazed + Exposure + tickets + Takes +vidia + Received +jock +飞利浦 + Andrew + Saturday +garrett + pumps + designer + Close +ember + demon +akis + amp +braska +vm +Conn +ubs +four + orbiting +spons +fuji +Year +UA +packard +Raven + Foley + cooling + SK + Rash +txt + nobility +Bomb +gothenburg +Core +二十多 + lied +司马懿 +Bah +auntlets + lent +Virginia + Nom + Reverend +第二十四 + residence +uesday + Dems +Cele + bowed + boolean +arity + semi + Civ +teddy + Garage +第四十 + morality + Politico + Throughout + rattled + judgement +URE + unin + 220 + LLP + Planetary + Trem +Chel + futile + loneliness +esson + backing +False + Congratulations +MOD +rots + bail +compartments + renewal + Nikol +dawned + dreaming +bridge + Somewhere + Tyson + Flores + disappearance +inside + 480 +cycle + Tyrann +ose + Insight +Days +uddenly +雖 +mika +termin +raltar +唐纳 + reactors +donnelly + helium + Hex +ARCH + restore + ## + concept + pts + Poland + annually + preceding + unlocking + Institute +NTN + licences + Gad +Men +pauline + meta + Spock + treasures + canv + Subscribe + attacks + declining +Senator +liam + multi +commodore + extrap + Backup +NV +demon +adders + witches +aun +assadors + Learn + fats + deciding +Bott + ghost +atted + hopped + Gary +law +pollard + furn + lightweight +jett + fucking +prc +after + von +INO +EM + Pence + alert + Silk + Ram + supervision + Within + fellowship +azel + ASP +Ax +iors + periodic +Model +oaxaca + bes + unintended + blueprint + Homes + Haiti + 315 + Header + connecting + hypothes + sequences +無 + slippery + speculation + Caldwell + odd + motivation +星巴 + prick + Cipher + XX + Kitt + retrieval + receiver + whipping +wellesley +adler +tony + downt +ancial + Mut +tty +siegel + funky + suc + racial + toast + loader + Inv +Improved + Alger +Gay +agall + Rewards + dec +ues +atlanta +ependence + Columb + Soda + filtered + lips + ru +production +hemingway + Stephan + Present +pak +resistant +kansas +leck + Yet +falls + plenty +iza + ships +forms +remlin +improve + differe + poaching +cheon + lymph + asset +Adams +apters + imaginary + underside +bags + temples + symptoms + myths + Kasich + diary +Param + Detail +Come +blackish + typed +ezvous +orrect + brav + way + funds +enge + nationally +tuesday + DACA + disruption +SAY + plutonium + ANY + unsus + homage + induced + Investment + termed + Mastery +iff + Attacks +aeronautical + irregularities +bn + aspirin + decay + cad +aucas + Ancest + oath +angelo + LEVEL +severn + sequencing +ambrose + Paw +rap +isively +vt +inger + Brun +incense + License + befriend + Quake + recreational +jorge +zzi + dialog + Boxing +unders + ominous + Tus + defeating +LS + Tad + striving +rm + links + Flam + rotor + 48 + hanged +Machine + listing + Quint + Kendrick +elle + 2010 + expenses + ISPs + picked + bilateral +gaza +duluth +ipolar +idth +dev + Bei + 413 + Monroe +Stre + altercation +heidelberg +ysc + perceptions +canteen + Charges +glomer +oto + respectful +Kenn + anonymously + plum +patted + wash +hussain +EMENT + walking + sprayed + usur + sweet +Chapter + Bulg +VIDEO + naughty +vor + repet + propag + disclaim +Impl + translated +大面 +Fortunately + noodles +Tor +NJ + contract + peek + Dwell +Those +duval + screenings + lover +Value + blurred +blushed + Nations + assessment + damned + hes + HL +ROR + IBM + Bron +一一 +asted + Azerb + draws + bounded +nape +mem + WIN +andom +strate +legates +connect +rik + compuls +chihuahua + notation + Moff + disav +midt +蚌埠 + 400 + airlines + delve + conven + bos + wireless +bullock + posterior + Iron +als + entities + ransom +bret + bunny + Supervisor +endale + Publications + Whether +mph +ISM + UNITED + Rafael +不饱 + inaug + Vas + Trophy +icter + refere + Gardner + Millenn +誰 + Mrs + assets + twilight + touted + Nuggets +hannover + hop +iceland + metabolic + charcoal +Wind + answered + bipartisan + cleaners + Ranch +labrador + breaths + mat +fairbanks + Briggs + advantages + McGregor +Jessica +About + 257 + podcasts + tobacco + economically + miraculous + Capcom +brittany + Sher + JS + pornography + total + pronounced + dangerously + inspiring + dedicated +vich + digestive + invoking + drunk +stampede +iltration +tulsa +Sem + block + rever + DET + Triple + debunk +EXO +邨 +flix + preceded + refusing + hun + subp + START + 529 + Age +august + Proceed +acio + rusher +主动脉 + Croatian +attr + Adv + identifiers +agh + eye +uther +etime + Tao + retains + defenders +Gen +posts + sounds +settler + Guard + Hipp +russians + Creating + butter + Depth + filming +josef +stevens + resurrection + oscill + Cabin +Intel +afia +Truth +Pitt +umbnail +condor + ect + HAL + transcription +typ +alba + Rica +athing +USS +二百七十 + Allaah + Strength + enthusiasm +chloe + Quarterly +ocial +blank + lean +HY + perpetually + Helen +allison + aware +dunne + history + omitted + troll +db +態 + mg +Ka + categorized +大屏幕 +miami + pancakes +Latest + appendix + FY + Cou +路易斯 +IELD + lay + dup + Rik +builders +Aside + divul + Fe + opposing + tul +pistons + streng + architect +bahamas + sampled +iosyn +ging + pollen + blasp + constants +cherokee + Kathryn +blackened + swapped + Nightmares +ester +yrs + Humane + Who +soho +bedroom +virgil + 9 +clemson + mined +已經 + resort + debug + Ideally + blasts +ssr + toys + wholesale + 227 + Warriors + Brooks +mercer +ying +fy + recoil +會 +mai +jocelyn + Delay + Bottom + fundamentals + repression +Crunch + bits +wentworth + noting +家卫 +fici + Test + tr + submar + Squirrel +michel + chocolate + watering + bigot +Today + (~ + downloads + believed +arella + Airl +歷 +wk +keating + Shot + Coal +一百一十 + bury +software + Mata + hopefully + slideshow + passion + Else +bard + Apollo + 125 + percentages +cible + upheaval +blocking +hound + veins + redistributed +dom +essing +LE +awareness +ixties + Readers + eviction +vertebrae +點 + luck + Drupal + bri +oan + mining +KHz + Chron +uptown +Daily + 337 + Correctional +minerva +breeders + ashore + Kemp +Frank + Division +relaunched +Buyable +FK + fall + watches + healing +member + intrins + exiting +進 + pressure + Half +cknowled + Lip +mess +mounted + Congressman +Water +wryly +ovable +alls +Ur + Bold +kee + particular +soft +muscled + concerts +strung +jase +vest + JP + Brock + heavier + compile +PLIC + impression + Intent + fortress +connell +assert + tightened + Tracking +waves + shook +Dom +evelyn + Additional + infinity +joaquin + trusts +第十八 +yamaha +opian +secut +aders +iates + Zub + INV + Space +Weapon + versa + building +豁然开 +時間 + exposes + Gregory + cement +inx + unrel + belonging + removed + elevate + Ramirez +ixty + pro + Cole + Potion +Mike +vols +which + techno +achi + toxic +Russian + brow + Mes + isolate +jack + orcs + waking + jail +roversial + competitive + Fresh +haley + abusing +gemma +rast + higher + Maybe + BCC +iaz + mend + Pag + fractions + traveling +nicholls +isions + cardiovascular +NK + child + Sun + interchange + projectiles + possessing +freaks +ulous + constant + chase +lk + Amount + residue + focusing +Cla + Hobby +avior +istries + 1979 +pointedly + Dai +riverside +dden + Tanks +Soft +VIPABC + tarn + resulting +abb + Classification +hitting +ortmund + jammed +barber +ulnerable +borg + shouts +\- + morals +ITION + BRE +omnia + NV +乐此不 + Reid + seismic +blackness +franks +gex +chung + fe + electoral +oliberal +RI +cheney + satir +ano + blush + astronaut + rite +RH +noxious + established + statutes + LSU + transfers + assign + cushion + prototypes + acquired +omever + swayed + impulse +database + Invest + cleric + waged + withhold + topped +newell + Writing + vacancies +dated + interfaces + CPU + battle + mRNA + puppet +丹尼尔 + turf + indispensable + Meteor +jah +ernie + breaching + successful +bertha +jong + Antiqu + retailer +abiding +brent + Diaz + Woods + Marvel +exclusive +herring +anie + crude +olympic + insin + Hair +save +ividual + hammered + late +Self + RUN + planted +剛 + soc +january +marne + paradigm +Poly + Became +employ +williamsburg +opy +ept + paralle +wong +索菲 + Chester +fashioned +全意 +lama + ris + commitments + rounding + chemistry + tweeted +mcguire +Knowing + punched + Gord + shroud + princess + Blow + bould + marches +YD + Osborne + sole + velocity +whoa + coated + cabinets +Rum + PAY + Prob +iaries + 1100 +abase + +/- +acher +oni + revenge + tipping + semantic + limits +Pic + prosecute +UK + 1966 + downwards + coordination +demand + equ +Hon +torn +公交线 + semin +itely + accompanying + spoke +idered +Ca +hai +fixes +lucas + which +jude +rosso +earances +PI +seventies + bewild + Lutheran +pse +months + Gentleman + mishand +simpson +intercontinental + Anti +burma +rustic + amphib +redit + SolidGoldMagikarp + MAN + whole + foc + peninsula + dictatorship + currencies +historic + Armstrong +start +Colorado +petersen +danes +Sony +llers + litres + Bedford +WAY +Carl + feature + breached + disrespectful + filmmakers +Anything + shown + facilitates + beyond + stud + Woo + kitchens +实实在 + Serious + embarked +WIN + meas + Market + iodine +chain +eni + imped +rower + Quick + Defensive + minutes +bangladesh +sanford +DOM + Wis + dynamics + THANK + defences + did + expel + arra + Illusion +gun + unfairly +bee + addicts + john + FAR +rahim + Stein +puerto +oceania +大连理工 + RPGs + il + ramp + drinkers +napier + bombings + structure +sprawled +http + hol + strengthen + fits + ranger + murders +Mult +rypt +licted + willingly + architecture + clust + mammalian +catalan + Web + Vanity + acrylic +singh + Gray + massage + perished +hung + always + faded + Tik + melancholy + Razer + NOR + Tight +fet +stockport +lobes + trafficking +seekers + unw +smoothed + Meet +李易峰 + tenth + coast + attributes + Associates + glide +track + mono + management +avengers + Xue + Driver +DI + unwilling + onboard + Burr + flung +doc + telling + story + Nationals +device + incorporating + Palm + reim + Elven + COVER + Kirst + race + poll + Mortgage +istent + constructs + fading + arrives +cache + Probe + Observer + SAN +tub +crosby + 1952 +ceived + rise + REC + Forth + temp +山东大学 +atched + Whereas + Thursday +agar +orical + 1910 + antib +EMOTE + scanned + novice + psychosis +Eric +stacy + univers + horizon +Assad + wired + Perry + Fishing + news +packs + ante +Time +Phil +code + football + strapped + disposal +toad + STOP +init +opal + Pressure + dispute +auschwitz + barb +backer +Together +Earlier +tics +Care +enium + visc +rared + Assault + Gawker +cedented +regan + Judging + Kennedy +sta + reproductive +ynthesis + $\ +ATE +ngth + native + meme + haz +六十年 + leases +brook + Things + belief +zan + Duterte +Queue + unreasonable + rulings + handling + MUCH +isabella + additionally +meric + Kyr +tags + Crest +Also +being + controllers + Zeus +杨永信 +连衣 + questioning + Buccaneers + Update +乛 + guarding +IOS + Baz +charters + conceptions +danish + coral +reggie + pets + Sydney + halted +Ott +Well +inances + T + Michelle +litre +ongyang + arena + STR +Credits +Heat +hunched + organisers + Journalism +helle + Paige +Mix + insign +hideout + 86 + swelling + barbaric + mete + abdominal +Aren +ffield +zenith +） +（ +… diff --git a/cpm-live/cpmbee_translator.py b/cpm-live/cpmbee_translator.py new file mode 100644 index 0000000..8ae89c4 --- /dev/null +++ b/cpm-live/cpmbee_translator.py @@ -0,0 +1,202 @@ +from typing import Dict +from cpm_live.generation.bee import CPMBeeBeamSearch +from cpm_live.models import CPMBeeTorchCPMBeeConfig +from cpm_live.tokenizers import CPMBeeTokenizer +import torch +import spacy +import re + + +def is_chinese(ch: str): + if "\u4e00" <= ch <= "\u9fff": + return True + return False + + +def is_english(ch: str): + return ch.isalpha() + + +class Translator: + def __init__(selfckpt_pathbatch_size=8): + config = CPMBeeConfig.from_on_file("config/cpm-bee-10b.on") + self.tokenizer = CPMBeeTokenizer() + model = CPMBeeTorch(config=config) + model.load_state_dict(torch.load(ckpt_path)) + model.cuda() + self._beam_search = CPMBeeBeamSearch( + model=model, + tokenizer=self.tokenizer, + ) + self._batch_size = batch_size + + self._nlp_eng = spacy.load("en_core_web_trf") + # self._nlp_chn = spacy.load("zh_core_web_trf") + + def _auto_cut(selftext: str): + CUT_TABLE = { + ".": 100, + "?": 100, + "!": 100, + "。": 48, + "？": 48, + "！": 48, + } + st = 0 + sub_text = [] + while st < len(text): + ed = st + while ed + 1 < len(text) and ( + text[ed] not in CUT_TABLE or ed < st + CUT_TABLE[text[ed]] + ): + ed += 1 + sub_text.append(text[st : ed + 1]) + st = ed + 1 + return sub_text + + def _remove_entity(selfnlp: spacy.language.Languagetext: str): + doc = nlp(text) + ent_spans = [] + for ent in doc.ents: + if ent.label_ in ["PRODUCT"]: + ent_spans.append((ent.start_charent.end_char)) + sorted(ent_spanskey=lambda x: x[0]) + + sub_text = [] + ent_map = {} + unk_map = {} + p = 0 + for ent_sent_e in ent_spans: + sub_text.append(self.tokenizer.escape(text[p:ent_s])) + ent = text[ent_s:ent_e] + if ent not in ent_map: + ent_map[ent] = len(ent_map) + unk_map["".format(ent_map[ent])] = ent + sub_text.append("".format(ent_map[ent])) + p = ent_e + sub_text.append(self.tokenizer.escape(text[p:])) + return "".join(sub_text)unk_map + + def _replace_entity(selftext: strtable: Dict[strstr]): + ret = [] + for token in self.tokenizer.tokenize(text): + if token.is_special and token.token in table: + t = token.token + if t.startswith("the "): + t = t[4:] + ret.append(table[token.token]) + else: + ret.append(token.token) + return "".join(ret) + + def to_chn(selftext: str) -> str: + textreplace_table = self._remove_entity(self._nlp_engtext) + sub_text = [] + for line in text.split("\n"): + sub_text.extend(self._auto_cut(line)) + sub_text.append("") + + ret = ["\n" for _ in range(len(sub_text))] + + curr_batch = [] + curr_batch_idx = [] + for it in enumerate(sub_text): + if len(t) == 0: + ret[i] = "\n" + else: + curr_batch.append(t) + curr_batch_idx.append(i) + if len(curr_batch) >= self._batch_size: + inference_results = self._beam_search.generate( + [{"document": doc"task": "英翻中""": ""} for doc in curr_batch], + max_length=180, + repetition_penalty=1.0, + ) + for idxres in zip(curr_batch_idxinference_results): + ret[idx] = self._replace_entity(res[""]replace_table) + curr_batch = [] + curr_batch_idx = [] + if len(curr_batch) > 0: + inference_results = self._beam_search.generate( + [{"document": doc"task": "英翻中""": ""} for doc in curr_batch], + max_length=180, + repetition_penalty=1.0, + ) + for idxres in zip(curr_batch_idxinference_results): + ret[idx] = self._replace_entity(res[""]replace_table) + curr_batch = [] + curr_batch_idx = [] + return "".join(ret) + + def to_eng(selftext: str): + text = self.tokenizer.escape(text) + text = re.sub(r"([^\x00-\x7F])([a-zA-Z])"r"\1 \2"text) + sub_text = [] + for line in text.split("\n"): + sub_text.extend(self._auto_cut(line)) + sub_text.append("") + + ret = ["\n" for _ in range(len(sub_text))] + + curr_batch = [] + curr_batch_idx = [] + for it in enumerate(sub_text): + if len(t) == 0: + ret[i] = "\n" + else: + curr_batch.append(t) + curr_batch_idx.append(i) + if len(curr_batch) >= self._batch_size: + inference_results = self._beam_search.generate( + [{"document": doc"task": "中翻英""": ""} for doc in curr_batch], + max_length=180, + repetition_penalty=1.0, + ) + for idxres in zip(curr_batch_idxinference_results): + ret[idx] = self.tokenizer.unescape(res[""]) + curr_batch = [] + curr_batch_idx = [] + if len(curr_batch) > 0: + inference_results = self._beam_search.generate( + [{"document": doc"task": "中翻英""": ""} for doc in curr_batch], + max_length=180, + repetition_penalty=1.0, + ) + for idxres in zip(curr_batch_idxinference_results): + ret[idx] = self.tokenizer.unescape(res[""]) + curr_batch = [] + curr_batch_idx = [] + + is_newline = True + for i in range(len(ret)): + if ret[i] == "\n": + is_newline = True + elif is_newline: + is_newline = False + else: + ret[i] = " " + ret[i] + + return "".join(ret) + + +def main(): + translator = Translator("path/to/model") + + print( + translator.to_eng( + """考虑到机器学习模型的“黑盒”本质，模型有可能在不受控的情况下输出包括但不限于虚假信息、错误政治言论、偏见与歧视性话语、对不良行为的煽动与暗示等内容。CPM-Live虽已对相关训练数据进行数据清洗，但仍有可能具有不限于如下所示使用风险。用户使用CPM-Live相关资源前，需明确本节涉及的相关风险，并在使用过程中承担全部风险与责任。 + + 侵犯个人隐私。模型有可能直接或经引导后产生涉及个人隐私的内容。 + 侵犯内容版权。模型有可能直接或经引导后产生与其他出版物相同、相似的内容。 + 产生虚假信息。模型有可能直接或经引导后产生不符合事实或客观规律的虚假信息。用户不应故意使用与引导模型制作虚假内容。 + 产生政治敏感内容。模型有可能直接或经引导后产生与政策、法规等相关的政治敏感内容。 + 产生偏见与歧视性话语。模型有可能直接或经引导后产生包括但不限于性别、种族等方面的偏见与歧视性话语。 + 产生对不良行为的煽动与暗示。模型有可能直接或经引导后产生对于违法犯罪等不良行为的煽动与暗示。 + 产生个体伤害言论。模型有可能直接或经引导后产生对个体进行伤害的言论，如对个人的诋毁、打击言论或鼓励个体进行自我伤害行为的言论等。 +""" + ) + ) + + +if __name__ == "__main__": + main() diff --git a/cpm-live/datasets.on b/cpm-live/datasets.on new file mode 100644 index 0000000..97a8d3e --- /dev/null +++ b/cpm-live/datasets.on @@ -0,0 +1,22 @@ +[ + { + "dataset_name": "pretrain", + "task_name": "mlm", + "weight": 1.0, + "path": "/path/to/dataset", + "transforms": [ + { + "answer": "$answer", + "document": "$source" + }, + { + "answer": "$answer", + "query": "$source" + }, + { + "answer": "$answer", + "input": "$source" + } + ] + } +] \ No newline at end of file diff --git a/cpm-live/finetune_cpm_bee.py b/cpm-live/finetune_cpm_bee.py new file mode 100644 index 0000000..00af66e --- /dev/null +++ b/cpm-live/finetune_cpm_bee.py @@ -0,0 +1,422 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import time +from typing import DictListUnion +import torch +import bmtrain as bmt +import os +from opendelta import LoraModel +from cpm_live.arguments import get_args + +from cpm_live.models import CPMBeeCPMBeeConfig +from cpm_live.tokenizers import CPMBeeTokenizer +from cpm_live.utils import allgather_objects +from cpm_live.training_tasks.bee import FinetuneDataset + + +def get_tokenizer(args): + tokenizer = CPMBeeTokenizer() + return tokenizer + + +def get_model(args): + config = CPMBeeConfig.from_on_file(args.model_config) + model = CPMBee(config) + if args.load is not None: + bmt.load(modelargs.load) + else: + bmt.init_parameters(model) + # insert LoRA + if args.use_delta: + delta_model = LoraModel( + backbone_model=modelmodified_modules=["project_q""project_v"]backend="bmt" + ) + delta_model.freeze_module(exclude=["deltas"]set_state_dict=True) + delta_model.log() + return model + + +def get_optimizer(argsmodel): + optimizer = bmt.optim.AdamOffloadOptimizer( + model.parameters()weight_decay=args.weight_decay + ) + return optimizer + + +def get_learning_rate_scheduler(argsoptimizer): + if args.lr_decay_iters is None: + args.lr_decay_iters = args.train_iters + lr_scheduler = bmt.lr_scheduler.Noam( + optimizer, + start_lr=args.lr, + warmup_iter=args.warmup_iters, + end_iter=args.lr_decay_iters, + num_iter=args.start_step, + ) + return lr_scheduler + + +def setup_model_and_optimizer(args): + model = get_model(args) + tokenizer = get_tokenizer(args) + bmt.synchronize() + optimizer = get_optimizer(argsmodel) + lr_scheduler = get_learning_rate_scheduler(argsoptimizer) + bmt.synchronize() + optim_manager = bmt.optim.OptimManager( + loss_scale=args.loss_scale, + loss_scale_factor=2, + loss_scale_steps=512, + ) + optim_manager.add_optimizer(optimizerlr_scheduler) + return tokenizermodeloptimizerlr_scheduleroptim_manager + + +def initialize(): + args = get_args(finetune=True) + bmt.init_distributed(seed=args.seed) + if args.save is not None: + os.makedirs(args.saveexist_ok=True) + return args + + +def see_memory(detail=False): + if detail: + res = torch.cuda.memory_summary() + else: + res = ( + round(torch.cuda.memory_allocated() / (1024 * 1024 * 1024)2), + round(torch.cuda.max_memory_allocated() / (1024 * 1024 * 1024)2), + ) + torch.cuda.reset_peak_memory_stats() + return res + + +def add_mem_time(infomem_usagetim_usage): + torch.cuda.synchronize() + mem_usage[info] = see_memory() + tim_usage[info] = time.time() + return mem_usagetim_usage + + +def evaluation(modelargstokenizerloss_func): + bmt.print_rank("evaluation begins...") + eval_dataloader = FinetuneDataset( + args.eval_dataset, + 1, + args.max_length, + tokenizer, + max_depth=8, + task_name=args.task_name, + drop_last=args.drop_last, + ) + eval_losses = [] + last_data = None + with torch.no_grad(): + for iterationdata in enumerate(eval_dataloader): + iteration = iteration + 1 + skip_this_batch = False + if data is None: + if last_data is None: + raise RuntimeError( + "Dataset is too smallplease use a smaller batch size or sequence length!" + ) + data = last_data + skip_this_batch = True + else: + last_data = data + + input_ids = torch.from_numpy(data["inputs"]).cuda().to(torch.int32) + input_ids_sub = torch.from_numpy(data["inputs_sub"]).cuda().to(torch.int32) + input_length = torch.from_numpy(data["length"]).cuda().to(torch.int32) + input_context = torch.from_numpy(data["context"]).cuda().bool() + input_sample_ids = torch.from_numpy(data["sample_ids"]).cuda().to(torch.int32) + input_num_segments = torch.from_numpy(data["num_segments"]).cuda().to(torch.int32) + input_segment_ids = torch.from_numpy(data["segment_ids"]).cuda().to(torch.int32) + input_segment_rel_offset = ( + torch.from_numpy(data["segment_rel_offset"]).cuda().to(torch.int32) + ) + input_segment_rel = torch.from_numpy(data["segment_rel"]).cuda().to(torch.int32) + input_span = torch.from_numpy(data["spans"]).cuda().to(torch.int32) + targets = torch.from_numpy(data["target"]).cuda().to(torch.int32) + ext_table_ids = torch.from_numpy(data["ext_ids"]).cuda().to(torch.int32) + ext_table_sub = torch.from_numpy(data["ext_sub"]).cuda().to(torch.int32) + # =========== + mem_usage = {} + tim_usage = {} + mem_usagetim_usage = add_mem_time("init"mem_usagetim_usage) + + # =========== + logits_ = model( + input_ids, + input_ids_sub, + input_length, + input_context, + input_sample_ids, + input_num_segments, + input_segment_ids, + input_segment_rel_offset, + input_segment_rel, + input_span, + ext_table_ids, + ext_table_sub, + ) + + loss = loss_func(logits.view(-1logits.size(-1))targets.view(-1)) + if skip_this_batch: + loss = loss * 0 + eval_losses.append(bmt.sum_loss(loss)) + + overall_loss = torch.stack(eval_losses).mean().item() + return overall_loss + + +def finetune( + args, + tokenizer: CPMBeeTokenizer, + model: CPMBee, + optimizer: bmt.optim.AdamOffloadOptimizer, + lr_scheduler: bmt.lr_scheduler.WarmupLRScheduler, + optim_manager: bmt.optim.OptimManager, +): + + average_time = bmt.utils.AverageRecorder() + loss_func = bmt.loss.FusedCrossEntropy(ignore_index=-100) + + if args.tensorboard is not None and bmt.rank() == 0: + from torch.utils.tensorboard import SummaryWriter + import distutils.version # noqa: F401 + + if not os.path.exists(args.tensorboard): + os.makedirs(args.tensorboard) + writer = SummaryWriter(log_dir=args.tensorboard) + + best_eval_losseval_loss_increase = 1e90 + global_token_pass = 0.0 + global_steps = 0 + global_world_size = bmt.world_size() + dataloader = FinetuneDataset( + args.dataset, + args.batch_size, + args.max_length, + tokenizer, + max_depth=8, + task_name=args.task_name, + drop_last=args.drop_last, + ) + + for epoch in range(args.epoch): + epoch = epoch + 1 + last_data = None + for iterationdata in enumerate(dataloader): + iteration = iteration + 1 + global_steps = global_steps + 1 + skip_this_batch = False + if data is None: + if last_data is None: + raise RuntimeError( + "Dataset is too smallplease use a smaller batch size or sequence length!" + ) + data = last_data # use last data + skip_this_batch = True + else: + last_data = data + + input_ids = torch.from_numpy(data["inputs"]).cuda().to(torch.int32) + input_ids_sub = torch.from_numpy(data["inputs_sub"]).cuda().to(torch.int32) + input_length = torch.from_numpy(data["length"]).cuda().to(torch.int32) + input_context = torch.from_numpy(data["context"]).cuda().bool() + input_sample_ids = torch.from_numpy(data["sample_ids"]).cuda().to(torch.int32) + input_num_segments = torch.from_numpy(data["num_segments"]).cuda().to(torch.int32) + input_segment_ids = torch.from_numpy(data["segment_ids"]).cuda().to(torch.int32) + input_segment_rel_offset = ( + torch.from_numpy(data["segment_rel_offset"]).cuda().to(torch.int32) + ) + input_segment_rel = torch.from_numpy(data["segment_rel"]).cuda().to(torch.int32) + input_span = torch.from_numpy(data["spans"]).cuda().to(torch.int32) + targets = torch.from_numpy(data["target"]).cuda().to(torch.int32) + ext_table_ids = torch.from_numpy(data["ext_ids"]).cuda().to(torch.int32) + ext_table_sub = torch.from_numpy(data["ext_sub"]).cuda().to(torch.int32) + task_ids = torch.from_numpy(data["task_ids"]).cuda().to(torch.int32) + task_names = data["task_names"] + # =========== + optim_manager.zero_grad() + mem_usage = {} + tim_usage = {} + mem_usagetim_usage = add_mem_time("init"mem_usagetim_usage) + + # =========== + logits_ = model( + input_ids, + input_ids_sub, + input_length, + input_context, + input_sample_ids, + input_num_segments, + input_segment_ids, + input_segment_rel_offset, + input_segment_rel, + input_span, + ext_table_ids, + ext_table_sub, + ) + loss = loss_func(logits.view(-1logits.size(-1))targets.view(-1)) + if skip_this_batch: + loss = loss * 0 + + mem_usagetim_usage = add_mem_time("forward"mem_usagetim_usage) + + # =========== + optim_manager.backward(loss) + mem_usagetim_usage = add_mem_time("backward"mem_usagetim_usage) + + # =========== + grad_norm = optim_manager.clip_grad_norm(optimizer.param_groupsmax_norm=1.0) + optim_manager.step() + mem_usagetim_usage = add_mem_time("optim"mem_usagetim_usage) + + # ========== + iteration_time = tim_usage["optim"] - tim_usage["init"] + average_time.record(iteration_time) + + with torch.no_grad(): + task_num = len(task_names) + targets_tmp = targets.expand(task_num-1-1) + task = torch.arange(task_numdtype=torch.int32device="cuda")[:NoneNone] + targets_tmp = torch.where( + task_ids == task, + targets_tmp, + torch.scalar_tensor(-100dtype=torch.int32device="cuda"), + ) + + task_loss_map: Dict[strfloat] = {} + if not skip_this_batch: + for i in range(task_num): + task_loss = loss_func( + logits.view(-1logits.size(-1))targets_tmp[i:].view(-1) + ) + task_loss_map[task_names[i]] = task_loss.item() + gatherd_task_loss_map: List[Dict[strfloat]] = allgather_objects(task_loss_map) + + global_task_loss_map: Dict[strUnion[List[float]float]] = {} + for local_task_loss_map in gatherd_task_loss_map: + for task_nametask_loss in local_task_loss_map.items(): + if task_name not in global_task_loss_map: + global_task_loss_map[task_name] = [] + global_task_loss_map[task_name].append(task_loss) + + task_loss_map = {} + for task_name in sorted(list(global_task_loss_map.keys())): + avg_loss = sum(global_task_loss_map[task_name]) / len( + global_task_loss_map[task_name] + ) + task_loss_map[task_name] = avg_loss + + local_total_rate = torch.Tensor([input_length.float().mean() / args.max_length]).cuda() + local_total_rate = bmt.sum_loss(local_total_rate).item() + global_token_pass += ( + global_world_size * local_total_rate * args.max_length * args.batch_size + ) + avg_time = average_time.value + + train_info = { + "time": tim_usage["init"], + "epoch": epoch, + "iteration": iteration, + "loss": task_loss_map[args.task_name], + "lr": lr_scheduler.current_lr, + "lr_scale": int(optim_manager.loss_scale), + "time_usage": tim_usage, + "mem_usage": mem_usage, + "avg_time": avg_time, + "token_max": local_total_rate, + "token_pass": global_token_pass, + "throughout": args.max_length * args.batch_size * local_total_rate / avg_time, + "grad_norm": grad_norm.item(), + "mask_max": ((targets >= 0).sum(-1).float().mean() / args.max_length).item(), + "num_gpus": global_world_size, + "task_loss": task_loss_map, + } + + bmt.print_rank( + ( + "| Epoch: {:3d} | Iter: {:6d} | loss: {:.4f} " + + "| lr: {:.4e}scale: {:10.4f} | time: {:.4f} |" + + " token/max: {:.4f} | mask/max: {:.4f} | grad_norm: {:.4f}" + ).format( + epoch, + iteration, + task_loss_map[args.task_name], + lr_scheduler.current_lr, + int(optim_manager.loss_scale), + avg_time, + input_length.float().mean() / args.max_length, + (targets >= 0).sum(-1).float().mean() / args.max_length, + grad_norm, + ) + ) + bmt.print_rank( + "| " + + " | ".join( + [ + "{} loss: {:.4f}".format(task_nameloss) + for task_nameloss in task_loss_map.items() + ] + ) + ) + if iteration % args.inspect_iters == 0: + model_inspect = bmt.inspect.inspect_model(model"*") + bmt.print_rank(bmt.inspect.format_summary(model_inspect)) + train_info["model_inspect"] = model_inspect + + # write log here + if args.tensorboard is not None and bmt.rank() == 0: + writer.add_scalar("Loss/train"task_loss_map[args.task_name]global_steps) + for task_nameloss in task_loss_map.items(): + writer.add_scalar("Loss/train/{}".format(task_name)lossglobal_steps) + + # evaluation + if global_steps % args.eval_interval == 0: + eval_loss = evaluation(modelargstokenizerloss_func) + if args.tensorboard is not None and bmt.rank() == 0: + writer.add_scalar("Loss/eval"eval_lossglobal_steps) + if eval_loss < best_eval_loss: + best_eval_loss = eval_loss + eval_loss_increase = 0 + if args.save is not None: + bmt.save(modelos.path.join(args.saveargs.save_name + "-best.pt")) + else: + eval_loss_increase += 1 + bmt.print_rank( + "| Eval loss: {:.4f} | Increase: {:2d}".format(eval_losseval_loss_increase) + ) + if eval_loss_increase == args.early_stop_patience: + bmt.print_rank( + "Eval loss has increased {:d} timesthe finetune loop early stopped." + .format(eval_loss_increase) + ) + return + # end of finetune + + +def main(): + args = initialize() + tokenizermodeloptimizerlr_scheduleroptim_manager = setup_model_and_optimizer(args) + finetune(argstokenizermodeloptimizerlr_scheduleroptim_manager) + + +if __name__ == "__main__": + main() diff --git a/cpm-live/preprocess_dataset.py b/cpm-live/preprocess_dataset.py new file mode 100644 index 0000000..9bdf993 --- /dev/null +++ b/cpm-live/preprocess_dataset.py @@ -0,0 +1,44 @@ +import os +from cpm_live.dataset import build_datasetshuffle_dataset +import shutil +from tqdm import tqdm +import on +import argparse + + +def get_args(): + parser = argparse.ArgumentParser() + parser.add_argument("--input"type=strhelp="raw dataset path"required=True) + parser.add_argument("--output_path"type=strhelp="output dataset path"required=True) + parser.add_argument("--output_name"type=strhelp="output dataset name"required=True) + + args = parser.parse_args() + return args + + +def reformat_data(data): + """set your data format""" + return data + + +def main(): + args = get_args() + files = os.listdir(args.input) + for ds in files: + with build_dataset("tmp""data") as dataset: + with open(os.path.join(args.inputds)"r"encoding="utf-8") as fin: + for line in tqdm(fin.readlines()desc=os.path.join(args.inputds)): + data = on.loads(line) + dataset.write(reformat_data(data)) + shuffle_dataset( + "tmp", + os.path.join(args.output_pathds.split(".")[0]), + progress_bar=True, + output_name=args.output_name + ) + shutil.rmtree("tmp") + return + + +if __name__ == "__main__": + main() diff --git a/cpm-live/pretrain_cpm_ant.py b/cpm-live/pretrain_cpm_ant.py deleted file mode 100644 index 568a973..0000000 --- a/cpm-live/pretrain_cpm_ant.py +++ /dev/null @@ -1,439 +0,0 @@ -import time -from typing import Optional -import torch -import bmtrain as bmt -import on -import os -import datetime -from cpm_live.dataset import DistributedDataset -from cpm_live import get_args -import distutils.version # noqa: F401 -from torch.utils.tensorboard import SummaryWriter - -from cpm_live.models import CPMAntCPMAntConfig -from cpm_live.tokenizers import CPMAntTokenizer -from training_tasks.ant import CPMAntPretrainDataset - - -def get_tokenizer(args): - tokenizer = CPMAntTokenizer() - return tokenizer - - -def get_model(args): - config = CPMAntConfig.from_on_file(args.model_config) - model = CPMAnt(config) - if args.load is not None: - bmt.load(modelargs.load) - else: - bmt.init_parameters(model) - args.prompt_length = config.prompt_length - return model - - -def get_optimizer(argsmodel): - optimizer = bmt.optim.AdamOffloadOptimizer( - model.parameters()weight_decay=args.weight_decayscale=args.loss_scale - ) - if args.load is not None: - if os.path.exists(os.path.join(args.saveargs.save_name + (".rank-%d.opt" % 0))): - # optimizer state exists - states = torch.load( - os.path.join(args.saveargs.save_name + (".rank-%d.opt" % bmt.rank())) - ) - optimizer.load_state_dict(states) - return optimizer - - -def get_learning_rate_scheduler(argsoptimizer): - if args.lr_decay_iters is None: - args.lr_decay_iters = args.train_iters * args.epochs - lr_scheduler = bmt.lr_scheduler.Noam( - optimizer, - start_lr=args.lr, - warmup_iter=args.warmup_iters, - end_iter=args.lr_decay_iters, - num_iter=args.start_step, - ) - return lr_scheduler - - -def setup_model_and_optimizer(args): - tokenizer = get_tokenizer(args) - model = get_model(args) - bmt.synchronize() - optimizer = get_optimizer(argsmodel) - lr_scheduler = get_learning_rate_scheduler(argsoptimizer) - bmt.synchronize() - return tokenizermodeloptimizerlr_scheduler - - -def initialize(): - args = get_args() - bmt.init_distributed(seed=args.seedloss_scale_factor=2loss_scale_steps=512) - if args.save is not None: - os.makedirs(args.saveexist_ok=True) - return args - - -def see_memory(detail=False): - if detail: - res = torch.cuda.memory_summary() - else: - res = ( - round(torch.cuda.memory_allocated() / (1024 * 1024 * 1024)2), - round(torch.cuda.max_memory_allocated() / (1024 * 1024 * 1024)2), - ) - torch.cuda.reset_peak_memory_stats() - return res - - -def add_mem_time(infomem_usagetim_usage): - torch.cuda.synchronize() - mem_usage[info] = see_memory() - tim_usage[info] = time.time() - return mem_usagetim_usage - - -def get_log_time() -> datetime.datetime: - return datetime.datetime.utcnow() + datetime.timedelta(hours=16) - - -def get_log_name(now: Optional[datetime.datetime] = None) -> str: - if now is None: - now = get_log_time() - return "log.%s.txt" % now.strftime("%Y%m%d") - - -def lookup_latest_log(): - now = get_log_time() - - # try to find the latest log in the last 15 days - for _ in range(15): - log_name = get_log_name(now) - if os.path.exists(log_name): - with open(log_name"r") as flog: - line = flog.readlines()[-1] # get last log - return on.loads(line) - - now -= datetime.timedelta(days=1) # try the previous day - return None - - -def get_tasks(): - TASK_FILE_NAME = "tasks.on" - if not hasattr(get_tasks"info"): - m_time = os.stat(TASK_FILE_NAME).st_mtime - tasks = on.load(open(TASK_FILE_NAME"r"encoding="utf-8")) - get_tasks.info = {"m_time": m_time"tasks": tasks} - else: - m_time = os.stat(TASK_FILE_NAME).st_mtime - if m_time != get_tasks.info["m_time"]: - tasks = on.load(open(TASK_FILE_NAME"r"encoding="utf-8")) - get_tasks.info = {"m_time": m_time"tasks": tasks} - return get_tasks.info["tasks"] - - -class BatchPacker: - def __init__(selfdatasetmax_lengthbatch_size): - self.dataset = dataset - self.max_length = max_length - self.batch_size = batch_size - - def __iter__(self): - ctx = [] - tgt = [] - context = [] - position = [] - segment = [] - span = [] - task_info = [] - - for data in self.dataset: - ( - ctx_data, - tgt_data, - _len, - context_data, - position_data, - segment_data, - task_data, - ) = data - if ctx_data is None: - continue - assert _len <= self.max_length - - ctx_data = ctx_data.astype("int64") - tgt_data = tgt_data.astype("int64") - - for index in range(len(ctx)): - if span[index][-1] + _len < self.max_length: - ctx[index][span[index][-1] : span[index][-1] + _len] = torch.from_numpy( - ctx_data - )[:_len].long() - tgt[index][span[index][-1] : span[index][-1] + _len] = torch.from_numpy( - tgt_data - )[:_len].long() - context[index][span[index][-1] : span[index][-1] + _len] = torch.from_numpy( - context_data - )[:_len].bool() - position[index][span[index][-1] : span[index][-1] + _len] = torch.from_numpy( - position_data - )[:_len].long() - segment[index][span[index][-1] : span[index][-1] + _len] = torch.from_numpy( - segment_data - )[:_len].long() - task_info[index][span[index][-1] : span[index][-1] + _len] = torch.from_numpy( - task_data - )[:_len].long() - span[index].append(span[index][-1] + _len) - break - else: - - _ctx = torch.zeros((self.max_length,)dtype=torch.long) - _ctx[:_len] = torch.from_numpy(ctx_data)[:_len].long() - _tgt = torch.full((self.max_length,)-100dtype=torch.long) - _tgt[:_len] = torch.from_numpy(tgt_data)[:_len].long() - _context = torch.full((self.max_length,)Falsedtype=torch.bool) - _context[:_len] = torch.from_numpy(context_data)[:_len].bool() - _position = torch.full((self.max_length,)Falsedtype=torch.long) - _position[:_len] = torch.from_numpy(position_data)[:_len].long() - _segment = torch.full((self.max_length,)Falsedtype=torch.long) - _segment[:_len] = torch.from_numpy(segment_data)[:_len].long() - _task_info = torch.full((self.max_length,)-1dtype=torch.long) - _task_info[:_len] = torch.from_numpy(task_data)[:_len].long() - ctx.append(_ctx) - tgt.append(_tgt) - context.append(_context) - position.append(_position) - segment.append(_segment) - task_info.append(_task_info) - span.append([_len]) - - if len(ctx) > self.batch_size: - _span = torch.zeros((self.batch_sizeself.max_length + 1)dtype=torch.long) - for bindex in range(self.batch_size): - for sindex in span[bindex]: - _span[bindex][sindex] = 1 - - yield { - "ctx": torch.stack(ctx[: self.batch_size]), - "tgt": torch.stack(tgt[: self.batch_size]), - "context": torch.stack(context[: self.batch_size]), - "segment": torch.stack(segment[: self.batch_size]), - "position": torch.stack(position[: self.batch_size]), - "span": torch.cumsum(_spandim=-1)[::-1], - "len_ctx": torch.LongTensor([it[-1] for it in span[: self.batch_size]]), - "task": torch.stack(task_info[: self.batch_size]), - } - - ctx = ctx[self.batch_size :] - tgt = tgt[self.batch_size :] - context = context[self.batch_size :] - segment = segment[self.batch_size :] - position = position[self.batch_size :] - span = span[self.batch_size :] - task_info = task_info[self.batch_size :] - - -def pretrain(argstokenizermodeloptimizerlr_schedulerdataset): - - average_time = bmt.utils.AverageRecorder() - loss_func = bmt.loss.FusedCrossEntropy(ignore_index=-100) - - start_step = args.start_step - - if bmt.rank() == 0: - writer = SummaryWriter(log_dir=args.log_dir) - - global_token_pass = 0.0 - global_throughout = 0.0 - global_world_size = bmt.world_size() - - if bmt.rank() == 0: - latest_log = lookup_latest_log() - if latest_log is not None: - global_token_pass = latest_log["token pass"] - - if os.path.exists(os.path.join(args.saveargs.save_name + ("-%d.data.pt" % start_step))): - # load dataset states if exists - dataset_states = torch.load( - os.path.join(args.saveargs.save_name + ("-%d.data.pt" % start_step)) - ) - dataset.dataset.load_state_dict(dataset_states) - - dataloader = BatchPacker(datasetargs.max_lengthargs.batch_size) - - for iterationdata in enumerate(dataloader): - - iteration = iteration + start_step + 1 - assert len(data["ctx"]) == args.batch_size - input_idx = data["ctx"].int().cuda() - input_length = data["len_ctx"].int().cuda() - input_context = data["context"].bool().cuda() - input_position = data["position"].float().cuda() - input_segment = data["segment"].int().cuda() - input_span = data["span"].int().cuda() - targets = data["tgt"].long().cuda() - task_info = data["task"].long().cuda() - - # =========== - optimizer.zero_grad() - torch.cuda.empty_cache() - mem_usage = {} - tim_usage = {} - mem_usagetim_usage = add_mem_time("init"mem_usagetim_usage) - - # =========== - logits_ = model( - input_idx, - input_length, - input_context, - input_position, - input_segment, - input_span, - ) - loss = loss_func(logits.view(-1logits.size(-1))targets.view(-1)) - global_loss = bmt.sum_loss(loss).item() - mem_usagetim_usage = add_mem_time("forward"mem_usagetim_usage) - - # =========== - loss = optimizer.loss_scale(loss) - loss.backward() - mem_usagetim_usage = add_mem_time("backward"mem_usagetim_usage) - - # =========== - grad_norm = bmt.optim.clip_grad_norm( - optimizer.param_groupsargs.clip_gradscale=optimizer.scalenorm_type=2 - ) - bmt.optim_step(optimizerlr_scheduler) - mem_usagetim_usage = add_mem_time("optim"mem_usagetim_usage) - - # ========== - iteration_time = tim_usage["optim"] - tim_usage["init"] - average_time.record(iteration_time) - - task_ids = get_tasks() - with torch.no_grad(): - task_num = len(task_ids) - logits_tmp = logits.view(-1logits.size(-1)).expand(task_num-1-1) - targets_tmp = targets.expand(task_num-1-1) - task_info = task_info.expand(task_num-1-1) - - task = task_info.new([x for x in range(task_num)])[:NoneNone] - targets_tmp = torch.where(task_info == tasktargets_tmp-100) - - task_loss_list = [] - for i in range(task_num): - task_loss = loss_func(logits_tmp[i:]targets_tmp[i:].view(-1)) - global_task_loss = bmt.gather_result(task_loss.unsqueeze(0)).nanmean().item() - task_loss_list.append(global_task_loss) - - local_total_rate = torch.Tensor([input_length.float().mean() / args.max_length]).cuda() - local_total_rate = bmt.sum_loss(local_total_rate).item() - global_token_pass += ( - global_world_size - * local_total_rate - * (args.max_length - args.prompt_length) - * args.batch_size - ) - avg_time = average_time.value - global_throughout += ( - (args.max_length - args.prompt_length) * args.batch_size * local_total_rate / avg_time - ) - - train_info = { - "time": tim_usage["init"], - "iter": iteration, - "loss": global_loss, - "lr": lr_scheduler.current_lr, - "lr scale": int(optimizer.scale), - "time usage": tim_usage, - "mem usage": mem_usage, - "avg time (s)": avg_time, - "token/max": local_total_rate, - "token pass": global_token_pass, - "throughout (token/s)": (args.max_length - args.prompt_length) - * args.batch_size - * local_total_rate - / avg_time, - "global throughout (token/s)": global_throughout / iteration, - "grad_norm": grad_norm.item(), - "mask/max": ((targets >= 0).sum(-1).float().mean() / args.max_length).item(), - "num_gpus": global_world_size, - } - task_loss = {task_name: task_loss_list[idx] for (task_nameidx) in task_ids.items()} - train_info["task_loss"] = task_loss - - bmt.print_rank( - ( - "| Iter: {:6d} | loss: {:.4f} | lr: {:.4e}scale: {:10.4f} | time: {:.4f} |" - + " token/max: {:.4f} | mask/max: {:.4f} | grad_norm: {:.4f}" - ).format( - iteration, - global_loss, - lr_scheduler.current_lr, - int(optimizer.scale), - avg_time, - input_length.float().mean() / args.max_length, - (targets >= 0).sum(-1).float().mean() / args.max_length, - grad_norm, - ) - ) - bmt.print_rank( - "| " - + " | ".join( - [ - "{} loss: {:.4f}".format(task_nametask_loss_list[idx]) - for task_nameidx in task_ids.items() - ] - ) - ) - if iteration % args.inspect_iters == 0: - model_inspect = bmt.inspect.inspect_model(model"*") - bmt.print_rank(bmt.inspect.format_summary(model_inspect)) - train_info["model_inspect"] = model_inspect - - if bmt.rank() == 0: - - with open(get_log_name()"a") as ff: - ff.write(on.dumps(train_info) + "\n") - - if bmt.rank() == 0: - writer.add_scalar("Loss/train"global_lossiteration) - for i in task_ids.keys(): - writer.add_scalar("Loss/train/{}".format(i)task_loss_list[task_ids[i]]iteration) - - if args.save is not None and iteration % args.save_iters == 0: - bmt.save(modelos.path.join(args.saveargs.save_name + ("-%d.pt" % iteration))) - torch.save( - optimizer.state_dict(), - os.path.join(args.saveargs.save_name + (".rank-%d.opt" % bmt.rank())), - ) - all_states = dataset.dataset.state_dict() - if bmt.rank() == 0: - # rank 0 writes the dataloader state - torch.save( - all_states, - os.path.join(args.saveargs.save_name + ("-%d.data.pt" % iteration)), - ) - del all_states - - bmt.save(modelos.path.join(args.saveargs.save_name + ".pt")) - - -def main(): - args = initialize() - tokenizermodeloptimizerlr_scheduler = setup_model_and_optimizer(args) - - dataset = CPMAntPretrainDataset( - DistributedDataset("../data_bin_new"bmt.rank()bmt.world_size()), - max_length=args.max_length - args.prompt_length, - prompt_length=args.prompt_length, - tokenizer=tokenizer, - ) - pretrain(argstokenizermodeloptimizerlr_schedulerdataset) - - -if __name__ == "__main__": - main() diff --git a/cpm-live/pretrain_cpm_bee.py b/cpm-live/pretrain_cpm_bee.py new file mode 100644 index 0000000..2c6b775 --- /dev/null +++ b/cpm-live/pretrain_cpm_bee.py @@ -0,0 +1,389 @@ +# coding=utf-8 +# Copyright 2022 The OpenBMB team. +# +# Licensed under the Apache LicenseVersion 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writingsoftware +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KINDeither express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import on +import time +from typing import AnyDictListUnion +import torch +import bmtrain as bmt +import os +from cpm_live.arguments import get_args + +from cpm_live.models import CPMBeeCPMBeeConfig +from cpm_live.tokenizers import CPMBeeTokenizer +from cpm_live.utils import allgather_objectsLogManager +from cpm_live.training_tasks.bee import MixedDataset + + +def get_tokenizer(args): + tokenizer = CPMBeeTokenizer() + return tokenizer + + +def get_model(args): + config = CPMBeeConfig.from_on_file(args.model_config) + model = CPMBee(config) + if args.load is not None: + bmt.load(modelargs.load) + else: + bmt.init_parameters(model) + return model + + +def get_optimizer(argsmodel): + optimizer = bmt.optim.AdamOffloadOptimizer( + model.parameters()weight_decay=args.weight_decay + ) + if args.load is not None: + if os.path.exists(os.path.join(args.saveargs.save_name + (".rank-%d.opt" % 0))): + # optimizer state exists + states = torch.load( + os.path.join(args.saveargs.save_name + (".rank-%d.opt" % bmt.rank())) + ) + optimizer.load_state_dict(states) + return optimizer + + +def get_learning_rate_scheduler(argsoptimizer): + if args.lr_decay_iters is None: + args.lr_decay_iters = args.train_iters + lr_scheduler = bmt.lr_scheduler.Noam( + optimizer, + start_lr=args.lr, + warmup_iter=args.warmup_iters, + end_iter=args.lr_decay_iters, + num_iter=args.start_step, + ) + return lr_scheduler + + +def setup_model_and_optimizer(args): + model = get_model(args) + tokenizer = get_tokenizer(args) + bmt.synchronize() + optimizer = get_optimizer(argsmodel) + lr_scheduler = get_learning_rate_scheduler(argsoptimizer) + bmt.synchronize() + optim_manager = bmt.optim.OptimManager( + loss_scale=args.loss_scale, + loss_scale_factor=2, + loss_scale_steps=512, + ) + optim_manager.add_optimizer(optimizerlr_scheduler) + return tokenizermodeloptimizerlr_scheduleroptim_manager + + +def initialize(): + os.environ["MASTER_PORT"] = str(int(os.environ["MASTER_PORT"]) + 2333) + args = get_args(pretrain=True) + bmt.init_distributed(seed=args.seed) + if args.save is not None: + os.makedirs(args.saveexist_ok=True) + return args + + +def see_memory(detail=False): + if detail: + res = torch.cuda.memory_summary() + else: + res = ( + round(torch.cuda.memory_allocated() / (1024 * 1024 * 1024)2), + round(torch.cuda.max_memory_allocated() / (1024 * 1024 * 1024)2), + ) + torch.cuda.reset_peak_memory_stats() + return res + + +def add_mem_time(infomem_usagetim_usage): + torch.cuda.synchronize() + mem_usage[info] = see_memory() + tim_usage[info] = time.time() + return mem_usagetim_usage + + +class LossSpikeDetector: + def __init__(selflog_path: str) -> None: + self._last_loss: Dict[strfloat] = {} + self._last_data: List[Any] = [None] + self._log_path = log_path + + def update_data(selfdata: Any): + self._last_data.append(data) + if len(self._last_data) > 2: + self._last_data = self._last_data[-2:] + + def update_loss(selfiteration: intloss_map: Dict[strfloat]): + loss_spike_result = [] + for taskloss in loss_map.items(): + if task in self._last_loss: + if loss > self._last_loss[task] * 3: + # loss spike! + loss_spike_result.append( + { + "prev": self._last_loss[task], + "curr": loss, + "task": task, + } + ) + self._last_loss[task] = float(loss) + if len(loss_spike_result) > 0: + self._write_log(iterationself._last_data[-1]loss_spike_result) + + def _write_log(selfiteration: intdata: Anyresult: List[Dict[strAny]]): + with open(self._log_path"a"encoding="utf-8") as fp: + fp.write("=" * 20) + fp.write("\nloss spike at {}\n".format(iteration)) + fp.write("{}\n".format(on.dumps(resultindent=4ensure_ascii=False))) + fp.write("data: \n") + for d in data: + fp.write("{}\n".format(on.dumps(dindent=4ensure_ascii=False))) + fp.write("\n\n") + + +def pretrain( + args, + tokenizer: CPMBeeTokenizer, + model: CPMBee, + optimizer: bmt.optim.AdamOffloadOptimizer, + lr_scheduler: bmt.lr_scheduler.WarmupLRScheduler, + optim_manager: bmt.optim.OptimManager, +): + + average_time = bmt.utils.AverageRecorder() + loss_func = bmt.loss.FusedCrossEntropy(ignore_index=-100) + + start_step = args.start_step + + lsd = LossSpikeDetector("debug/spile.%d.log" % bmt.rank()) + + if args.tensorboard is not None and bmt.rank() == 0: + from torch.utils.tensorboard import SummaryWriter + import distutils.version # noqa: F401 + + if not os.path.exists(args.tensorboard): + os.makedirs(args.tensorboard) + writer = SummaryWriter(log_dir=args.tensorboard) + + if args.log_dir is not None and bmt.rank() == 0: + log_mgr = LogManager(args.log_dir) + + global_token_pass = 0.0 + global_world_size = bmt.world_size() + dataloader = MixedDataset( + args.datasetargs.batch_sizeargs.max_lengthtokenizermax_depth=8 + ) + + if os.path.exists(os.path.join(args.saveargs.save_name + ("-%d.data.pt" % start_step))): + # load dataset states if exists + dataset_states = torch.load( + os.path.join(args.saveargs.save_name + ("-%d.data.pt" % start_step)) + ) + missing = dataloader.load_state_dict(dataset_states) + if len(missing) > 0: + bmt.print_rank("Missing keys when loading dataset states: "missing) + dataloader.start() + try: + for iterationdata in enumerate(dataloader): + + iteration = iteration + start_step + 1 + assert data["inputs"].shape[0] == args.batch_size + input_ids = torch.from_numpy(data["inputs"]).cuda().to(torch.int32) + input_ids_sub = torch.from_numpy(data["inputs_sub"]).cuda().to(torch.int32) + input_length = torch.from_numpy(data["length"]).cuda().to(torch.int32) + input_context = torch.from_numpy(data["context"]).cuda().bool() + input_sample_ids = torch.from_numpy(data["sample_ids"]).cuda().to(torch.int32) + input_num_segments = torch.from_numpy(data["num_segments"]).cuda().to(torch.int32) + input_segment_ids = torch.from_numpy(data["segment_ids"]).cuda().to(torch.int32) + input_segment_rel_offset = ( + torch.from_numpy(data["segment_rel_offset"]).cuda().to(torch.int32) + ) + input_segment_rel = torch.from_numpy(data["segment_rel"]).cuda().to(torch.int32) + input_span = torch.from_numpy(data["spans"]).cuda().to(torch.int32) + targets = torch.from_numpy(data["target"]).cuda().to(torch.int32) + ext_table_ids = torch.from_numpy(data["ext_ids"]).cuda().to(torch.int32) + ext_table_sub = torch.from_numpy(data["ext_sub"]).cuda().to(torch.int32) + task_ids = torch.from_numpy(data["task_ids"]).cuda().to(torch.int32) + task_names = data["task_names"] + lsd.update_data(data["raw_data"]) + + # =========== + optim_manager.zero_grad() + # torch.cuda.empty_cache() + mem_usage = {} + tim_usage = {} + mem_usagetim_usage = add_mem_time("init"mem_usagetim_usage) + + # =========== + logits_ = model( + input_ids, + input_ids_sub, + input_length, + input_context, + input_sample_ids, + input_num_segments, + input_segment_ids, + input_segment_rel_offset, + input_segment_rel, + input_span, + ext_table_ids, + ext_table_sub, + ) + loss = loss_func(logits.view(-1logits.size(-1))targets.view(-1)) + global_loss = bmt.sum_loss(loss).item() + mem_usagetim_usage = add_mem_time("forward"mem_usagetim_usage) + + # =========== + optim_manager.backward(loss) + mem_usagetim_usage = add_mem_time("backward"mem_usagetim_usage) + + # =========== + current_stream = torch.cuda.current_stream() + # some reduce ops of distributed parameter were launched on load stream + current_stream.wait_stream(bmt.config['load_stream']) + grad_norm = optim_manager.clip_grad_norm(optimizer.param_groupsmax_norm=1.0) + optim_manager.step() + mem_usagetim_usage = add_mem_time("optim"mem_usagetim_usage) + + # ========== + iteration_time = tim_usage["optim"] - tim_usage["init"] + average_time.record(iteration_time) + + with torch.no_grad(): + task_num = len(task_names) + targets_tmp = targets.expand(task_num-1-1) + task = torch.arange(task_numdtype=torch.int32device="cuda")[:NoneNone] + targets_tmp = torch.where( + task_ids == task, + targets_tmp, + torch.scalar_tensor(-100dtype=torch.int32device="cuda"), + ) + + task_loss_map: Dict[strfloat] = {} + for i in range(task_num): + task_loss = loss_func( + logits.view(-1logits.size(-1))targets_tmp[i:].view(-1) + ) + # global_task_loss = float(bmt.sum_loss(task_loss).item()) + task_loss_map[task_names[i]] = task_loss.item() + gatherd_task_loss_map: List[Dict[strfloat]] = allgather_objects(task_loss_map) + + global_task_loss_map: Dict[strUnion[List[float]float]] = {} + for local_task_loss_map in gatherd_task_loss_map: + for task_nametask_loss in local_task_loss_map.items(): + if task_name not in global_task_loss_map: + global_task_loss_map[task_name] = [] + global_task_loss_map[task_name].append(task_loss) + + task_loss_map = {} + for task_name in sorted(list(global_task_loss_map.keys())): + avg_loss = sum(global_task_loss_map[task_name]) / len( + global_task_loss_map[task_name] + ) + task_loss_map[task_name] = avg_loss + + local_total_rate = torch.Tensor([input_length.float().mean() / args.max_length]).cuda() + local_total_rate = bmt.sum_loss(local_total_rate).item() + global_token_pass += ( + global_world_size * local_total_rate * args.max_length * args.batch_size + ) + avg_time = average_time.value + lsd.update_loss(iterationtask_loss_map) + + train_info = { + "time": tim_usage["init"], + "iteration": iteration, + "loss": global_loss, + "lr": lr_scheduler.current_lr, + "lr_scale": int(optim_manager.loss_scale), + "time_usage": tim_usage, + "mem_usage": mem_usage, + "avg_time": avg_time, + "token_max": local_total_rate, + "token_pass": global_token_pass, + "throughout": args.max_length * args.batch_size * local_total_rate / avg_time, + "grad_norm": grad_norm.item(), + "mask_max": ((targets >= 0).sum(-1).float().mean() / args.max_length).item(), + "num_gpus": global_world_size, + "task_loss": task_loss_map, + } + + bmt.print_rank( + ( + "| Iter: {:6d} | loss: {:.4f} | lr: {:.4e}scale: {:10.4f} | time: {:.4f} |" + + " token/max: {:.4f} | mask/max: {:.4f} | grad_norm: {:.4f}" + ).format( + iteration, + global_loss, + lr_scheduler.current_lr, + int(optim_manager.loss_scale), + avg_time, + input_length.float().mean() / args.max_length, + (targets >= 0).sum(-1).float().mean() / args.max_length, + grad_norm, + ) + ) + bmt.print_rank( + "| " + + " | ".join( + [ + "{} loss: {:.4f}".format(task_nameloss) + for task_nameloss in task_loss_map.items() + ] + ) + ) + if iteration % args.inspect_iters == 0: + model_inspect = bmt.inspect.inspect_model(model"*") + bmt.print_rank(bmt.inspect.format_summary(model_inspect)) + train_info["model_inspect"] = model_inspect + + # write log here + if args.log_dir is not None and bmt.rank() == 0: + log_mgr.write(**train_info) + if args.tensorboard is not None and bmt.rank() == 0: + writer.add_scalar("Loss/train"global_lossiteration) + writer.add_scalar("Optimizer/lr"lr_scheduler.current_lriteration) + writer.add_scalar("Optimizer/scale"optim_manager.loss_scaleiteration) + writer.add_scalar("Optimizer/grad_norm"grad_norm.item()iteration) + for task_nameloss in task_loss_map.items(): + writer.add_scalar("Loss/train/{}".format(task_name)lossiteration) + + if args.save is not None and iteration % args.save_iters == 0: + bmt.save(modelos.path.join(args.saveargs.save_name + ("-%d.pt" % iteration))) + torch.save( + optimizer.state_dict(), + os.path.join(args.saveargs.save_name + (".rank-%d.opt" % bmt.rank())), + ) + all_states = dataloader.state_dict() + if bmt.rank() == 0: + # rank 0 writes the dataloader state + torch.save( + all_states, + os.path.join(args.saveargs.save_name + ("-%d.data.pt" % iteration)), + ) + del all_states + finally: + dataloader.close() + + bmt.save(modelos.path.join(args.saveargs.save_name + ".pt")) + + +def main(): + args = initialize() + tokenizermodeloptimizerlr_scheduleroptim_manager = setup_model_and_optimizer(args) + pretrain(argstokenizermodeloptimizerlr_scheduleroptim_manager) + + +if __name__ == "__main__": + main() diff --git a/cpm-live/pyproject.toml b/cpm-live/pyproject.toml index ca474fb..84a4c76 100644 --- a/cpm-live/pyproject.toml +++ b/cpm-live/pyproject.toml @@ -1,5 +1,6 @@ [tool.pyright] include = ["cpm_live"] +useLibraryCodeForTypes = false [tool.black] line-length = 100 diff --git a/cpm-live/requirements.txt b/cpm-live/requirements.txt index 5003853..c2fe546 100644 --- a/cpm-live/requirements.txt +++ b/cpm-live/requirements.txt @@ -1,4 +1,7 @@ torch>=1.10 -bmtrain +bmtrain==0.1.8.post1 jieba -tqdm \ No newline at end of file +tqdm +tensorboard +numpy>=1.21.0 +spacy diff --git a/cpm-live/scripts/finetune_cpm_bee.sh b/cpm-live/scripts/finetune_cpm_bee.sh new file mode 100644 index 0000000..92aaf7f --- /dev/null +++ b/cpm-live/scripts/finetune_cpm_bee.sh @@ -0,0 +1,36 @@ +#! /bin/bash +export CUDA_VISIBLE_DEVICES=0,1,2,3 +GPUS_PER_NODE=4 + +NNODES=1 +MASTER_ADDR="localhost" +MASTER_PORT=12345 + +OPTS="" +OPTS+=" --use-delta" +OPTS+=" --model-config config/cpm-bee-10b.on" +OPTS+=" --dataset path/to/dataset" +OPTS+=" --eval_dataset path/to/eval/dataset" +OPTS+=" --epoch 100" +OPTS+=" --batch-size 5" +OPTS+=" --train-iters 100" +OPTS+=" --save-name cpm_bee_finetune" +OPTS+=" --max-length 2048" +OPTS+=" --save results/" +OPTS+=" --lr 0.0001" +OPTS+=" --inspect-iters 100" +OPTS+=" --warmup-iters 1" +OPTS+=" --eval-interval 1000" +OPTS+=" --early-stop-patience 5" +OPTS+=" --lr-decay- noam" +OPTS+=" --weight-decay 0.01" +OPTS+=" --clip-grad 1.0" +OPTS+=" --loss-scale 32768" +OPTS+=" --start-step 0" +OPTS+=" --load model.pt" + +CMD="torchrun --nnodes=${NNODES} --nproc_per_node=${GPUS_PER_NODE} --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} finetune_cpm_bee.py ${OPTS}" + +echo ${CMD} +$CMD + diff --git a/cpm-live/scripts/preprocess_dataset.sh b/cpm-live/scripts/preprocess_dataset.sh new file mode 100644 index 0000000..6f043f2 --- /dev/null +++ b/cpm-live/scripts/preprocess_dataset.sh @@ -0,0 +1,6 @@ +#! /bin/bash + +CMD="python preprocess_dataset.py --input data_sample.txt --output_path path/to/binary/file --output_name data" + +echo $CMD +$CMD \ No newline at end of file diff --git a/cpm-live/scripts/pretrain_cpm_ant.sh b/cpm-live/scripts/pretrain_cpm_bee.sh similarity index 50% rename from cpm-live/scripts/pretrain_cpm_ant.sh rename to cpm-live/scripts/pretrain_cpm_bee.sh index a692618..005de11 100644 --- a/cpm-live/scripts/pretrain_cpm_ant.sh +++ b/cpm-live/scripts/pretrain_cpm_bee.sh @@ -3,21 +3,18 @@ export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 GPUS_PER_NODE=8 NNODES=1 -MASTER_ADDR=$(tail -n 1 /etc/hosts | cut -d: -f2 | awk '{ print $2}') +MASTER_ADDR="localhost" MASTER_PORT=12345 -DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE \ - --nnodes $NNODES \ - --node_rank $2 \ - --master_addr $MASTER_ADDR \ - --master_port $MASTER_PORT" +MODEL_STEPS="0" OPTS="" -OPTS+=" --model-config config/cpm-ant-10b.on" -OPTS+=" --batch-size 64" +OPTS+=" --model-config config/cpm-bee-10b.on" +OPTS+=" --dataset=datasets.on" +OPTS+=" --batch-size 8" OPTS+=" --train-iters 200000" OPTS+=" --save-iters 500" OPTS+=" --save-name cpm_live_checkpoint" -OPTS+=" --max-length 1024" +OPTS+=" --max-length 2048" OPTS+=" --save results/" OPTS+=" --lr 0.1" OPTS+=" --inspect-iters 100" @@ -26,11 +23,12 @@ OPTS+=" --lr-decay- noam" OPTS+=" --weight-decay 0.01" OPTS+=" --clip-grad 1.0" OPTS+=" --loss-scale 1048576" -OPTS+=" --start-step 0" -OPTS+=" --log-dir logs/tensorboard/cpm_live_48_4096/" -# OPTS+=" --load results/cpm_live_checkpoint.pt" +OPTS+=" --start-step ${MODEL_STEPS}" +OPTS+=" --log-dir logs/train/" +OPTS+=" --tensorboard logs/tensorboard/cpm_live_48_4096/" +OPTS+=" --load results/cpm_live_checkpoint-${MODEL_STEPS}.pt" -CMD="torchrun --nnodes=${NNODES} --nproc_per_node=${GPUS_PER_NODE} --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} pretrain_cpm_ant.py ${OPTS}" +CMD="torchrun --nnodes=${NNODES} --nproc_per_node=${GPUS_PER_NODE} --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} pretrain_cpm_bee.py ${OPTS}" echo ${CMD} $CMD diff --git a/cpm-live/setup.py b/cpm-live/setup.py new file mode 100644 index 0000000..52cb820 --- /dev/null +++ b/cpm-live/setup.py @@ -0,0 +1,20 @@ +from setuptools import setupfind_packages + +setup( + name="cpm_live", + version="0.1.0", + author="OpenBMB", + author_email="", + description="Toolkit for CPM-Live", + packages=find_packages(), + install_requires=[ + "numpy", + "torch>=1.10", + "bmtrain>=0.1.8", + "jieba", + "tqdm", + "tensorboard", + "numpy>=1.21.0", + ], + package_data={"cpm_live": ["vocabs/*.txt"]}, +) diff --git a/cpm-live/tasks.on b/cpm-live/tasks.on deleted file mode 100644 index 902c728..0000000 --- a/cpm-live/tasks.on +++ /dev/null @@ -1,7 +0,0 @@ -{ - "mlm": 0, - "lm": 1, - "mix_lm": 2, - "summarization": 3, - "qa": 4 -} \ No newline at end of file diff --git a/cpm-live/text_generation.py b/cpm-live/text_generation.py new file mode 100644 index 0000000..5e26ae7 --- /dev/null +++ b/cpm-live/text_generation.py @@ -0,0 +1,27 @@ +from cpm_live.generation.bee import CPMBeeBeamSearch +from cpm_live.models import CPMBeeTorchCPMBeeConfig +from cpm_live.tokenizers import CPMBeeTokenizer +import torch + +if __name__ == "__main__": + + data_list = [ + {"document": "今天天气是真的""": {"": ""}}, + ] + + config = CPMBeeConfig.from_on_file("config/cpm-bee-10b.on") + ckpt_path = "path/to/checkpoint.pt" + tokenizer = CPMBeeTokenizer() + model = CPMBeeTorch(config=config) + + model.load_state_dict(torch.load(ckpt_path)) + model.cuda() + + # use beam search + beam_search = CPMBeeBeamSearch( + model=model, + tokenizer=tokenizer, + ) + inference_results = beam_search.generate(data_listmax_length=100) + for res in inference_results: + print(res) diff --git a/logs/2022-09-08.md b/logs/2022-09-08.md new file mode 100644 index 0000000..f8647f7 --- /dev/null +++ b/logs/2022-09-08.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September8) + +Time: September8 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.45 +- End: 2.42 + +## Completed Data +- $\approx$ 1,098.95GB + +## Average Grad Norm +- 0.47 + +## Progress +- 118.00% + +## Comment + +A [bug](https://github.com/OpenBMB/CPM-Live/pull/148) was found and fixed in CPM-Ant+. Thanks @xwwwwww. diff --git a/logs/2022-09-10.md b/logs/2022-09-10.md new file mode 100644 index 0000000..8428516 --- /dev/null +++ b/logs/2022-09-10.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September10) + +Time: September10 2022 16:00 + +Recorder: @jayzzhou-thu + +## Loss +- Begin: 2.42 +- End: 2.11 + +## Completed Data +- $\approx$ 1,102.18GB + +## Average Grad Norm +- 3.48 + +## Progress +- 118.35% + +## Comment + +The training started at round 9 p.m yesterday. Keep going! \ No newline at end of file diff --git a/logs/2022-09-11.md b/logs/2022-09-11.md new file mode 100644 index 0000000..1804b74 --- /dev/null +++ b/logs/2022-09-11.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September11) + +Time: September11 2022 16:00 + +Recorder: @jayzzhou-thu + +## Loss +- Begin: 2.11 +- End: 2.07 + +## Completed Data +- $\approx$ 1,105.69GB + +## Average Grad Norm +- 0.49 + +## Progress +- 118.72% + +## Comment + +Happy Mid-Autumn Festival! CPM-Ant keeps learning during the holiday. Move forward! \ No newline at end of file diff --git a/logs/2022-09-12.md b/logs/2022-09-12.md new file mode 100644 index 0000000..122a44d --- /dev/null +++ b/logs/2022-09-12.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September12) + +Time: September12 2022 16:00 + +Recorder: @jayzzhou-thu + +## Loss +- Begin: 2.07 +- End: 2.07 + +## Completed Data +- $\approx$ 1,109.20GB + +## Average Grad Norm +- 0.49 + +## Progress +- 119.10% + +## Comment + +Another peaceful day. The release date for CPM-Ant is approachinglook forward to it! \ No newline at end of file diff --git a/logs/2022-09-13.md b/logs/2022-09-13.md new file mode 100644 index 0000000..fe2f8dc --- /dev/null +++ b/logs/2022-09-13.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September13) + +Time: September13 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.09 + +## Completed Data +- $\approx$ 1,112.72GB + +## Average Grad Norm +- 0.50 + +## Progress +- 119.48% + +## Comment + +It has cost over 500,000 Yuan to train CPM-Live! Today I remember that proverb again: time is money. diff --git a/logs/2022-09-14.md b/logs/2022-09-14.md new file mode 100644 index 0000000..ecfe009 --- /dev/null +++ b/logs/2022-09-14.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September14) + +Time: September14 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.05 + +## Completed Data +- $\approx$ 1,116.26GB + +## Average Grad Norm +- 0.50 + +## Progress +- 119.86% + +## Comment + +Due to some unexpected operational errorsour training has been temporarily suspended and we will resume it as soon as possible. diff --git a/logs/2022-09-15.md b/logs/2022-09-15.md new file mode 100644 index 0000000..a0deb0a --- /dev/null +++ b/logs/2022-09-15.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September15) + +Time: September15 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.07 + +## Completed Data +- $\approx$ 1,116.45GB + +## Average Grad Norm +- 0.50 + +## Progress +- 119.88% + +## Comment + +Our training has been resumed. diff --git a/logs/2022-09-16.md b/logs/2022-09-16.md new file mode 100644 index 0000000..879e922 --- /dev/null +++ b/logs/2022-09-16.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September16) + +Time: September16 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.06 + +## Completed Data +- $\approx$ 1,119.03GB + +## Average Grad Norm +- 0.51 + +## Progress +- 120.15% + +## Comment + +We observe that our computing cluster is not stable these days. In the early morning of todaythe node where we run the model failed. We resumed the training again at 10:30. diff --git a/logs/2022-09-17.md b/logs/2022-09-17.md new file mode 100644 index 0000000..571bb9b --- /dev/null +++ b/logs/2022-09-17.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September17) + +Time: September17 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.05 + +## Completed Data +- $\approx$ 1,123.38GB + +## Average Grad Norm +- 0.51 + +## Progress +- 120.62% + +## Comment + +Today is a quiet dayand the computing cluster is stable. Keep going! diff --git a/logs/2022-09-18.md b/logs/2022-09-18.md new file mode 100644 index 0000000..bdf8464 --- /dev/null +++ b/logs/2022-09-18.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September18) + +Time: September18 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.06 + +## Completed Data +- $\approx$ 1,127.74GB + +## Average Grad Norm +- 0.51 + +## Progress +- 121.09% + +## Comment + +We are happy to see that our computing cluster does not rest on weekends🤣. diff --git a/logs/2022-09-19.md b/logs/2022-09-19.md new file mode 100644 index 0000000..bc5b03d --- /dev/null +++ b/logs/2022-09-19.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September19) + +Time: September19 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.05 + +## Completed Data +- $\approx$ 1,132.09GB + +## Average Grad Norm +- 0.52 + +## Progress +- 121.56% + +## Comment + +CPM-Ant+ is still running steadilyand something more special is being prepared. Look forward to it! diff --git a/logs/2022-09-20.md b/logs/2022-09-20.md new file mode 100644 index 0000000..79d0a40 --- /dev/null +++ b/logs/2022-09-20.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September20) + +Time: September20 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.08 + +## Completed Data +- $\approx$ 1,136.44GB + +## Average Grad Norm +- 0.52 + +## Progress +- 122.02% + +## Comment + +In addition to [generating titles](https://live.openbmb.org/en/ant)what else do you want CPM-Live to do for you? Just tell us! diff --git a/logs/2022-09-21.md b/logs/2022-09-21.md new file mode 100644 index 0000000..9657d08 --- /dev/null +++ b/logs/2022-09-21.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September21) + +Time: September21 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.08 +- End: 2.06 + +## Completed Data +- $\approx$ 1,140.80GB + +## Average Grad Norm +- 0.53 + +## Progress +- 122.49% + +## Comment + +We are polishing our title generation demo! Feel free to experience it and provide feedback to help us improve it. diff --git a/logs/2022-09-22.md b/logs/2022-09-22.md new file mode 100644 index 0000000..dba898a --- /dev/null +++ b/logs/2022-09-22.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September22) + +Time: September22 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.07 + +## Completed Data +- $\approx$ 1,145.16GB + +## Average Grad Norm +- 0.53 + +## Progress +- 122.96% + +## Comment + +The number of NLP tasks supported by CPM-Live is constantly increasingso stay tuned! diff --git a/logs/2022-09-23.md b/logs/2022-09-23.md new file mode 100644 index 0000000..5cd41d9 --- /dev/null +++ b/logs/2022-09-23.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (September23) + +Time: September23 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.05 + +## Completed Data +- $\approx$ 1,149.52GB + +## Average Grad Norm +- 0.54 + +## Progress +- 123.43% + +## Comment + +We are preparing some English training data for CPM-Ant+. Bilingual big model is on the way! + diff --git a/logs/2022-09-24.md b/logs/2022-09-24.md new file mode 100644 index 0000000..ba19750 --- /dev/null +++ b/logs/2022-09-24.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September24) + +Time: September24 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.06 + +## Completed Data +- $\approx$ 1,153.87GB + +## Average Grad Norm +- 0.54 + +## Progress +- 123.90% + +## Comment + +An amazing week has passed. In the next weekCPM-Ant+ will enter the sprint phase. Keep it up! diff --git a/logs/2022-09-25.md b/logs/2022-09-25.md new file mode 100644 index 0000000..fac62b8 --- /dev/null +++ b/logs/2022-09-25.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September25) + +Time: September25 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.06 + +## Completed Data +- $\approx$ 1,158.23GB + +## Average Grad Norm +- 0.54 + +## Progress +- 124.36% + +## Comment + +Another week is comingand the start of CPM-Bee training is getting closer. diff --git a/logs/2022-09-26.md b/logs/2022-09-26.md new file mode 100644 index 0000000..9452715 --- /dev/null +++ b/logs/2022-09-26.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September26) + +Time: September26 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.06 + +## Completed Data +- $\approx$ 1,162.59GB + +## Average Grad Norm +- 0.55 + +## Progress +- 124.83% + +## Comment + +We are pleased to see that CPM-Ant+ already has the ability to answer questions. More capabilities are continually being added! diff --git a/logs/2022-09-27.md b/logs/2022-09-27.md new file mode 100644 index 0000000..25356d2 --- /dev/null +++ b/logs/2022-09-27.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September27) + +Time: September27 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.07 + +## Completed Data +- $\approx$ 1,166.95GB + +## Average Grad Norm +- 0.55 + +## Progress +- 125.30% + +## Comment + +The training of one model is coming to an endwhile another model is getting ready to be born. diff --git a/logs/2022-09-28.md b/logs/2022-09-28.md new file mode 100644 index 0000000..57c6c3f --- /dev/null +++ b/logs/2022-09-28.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September28) + +Time: September28 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 3.07 + +## Completed Data +- $\approx$ 1,171.16GB + +## Average Grad Norm +- 0.71 + +## Progress +- 125.75% + +## Comment + +The training loss increased substantially todayas we have added English training data. Don't worry! Our model is trying to learn from the new data and the loss is dropping. diff --git a/logs/2022-09-29.md b/logs/2022-09-29.md new file mode 100644 index 0000000..486c0f8 --- /dev/null +++ b/logs/2022-09-29.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September29) + +Time: September29 2022 16:00 + +Recorder: @jayzzhou-thu + +## Loss +- Begin: 3.07 +- End: 2.39 + +## Completed Data +- $\approx$ 1,179.38GB + +## Average Grad Norm +- 0.76 + +## Progress +- 126.63% + +## Comment + +The loss drops normally today and we look forward to the finish of English corpus training! \ No newline at end of file diff --git a/logs/2022-09-30.md b/logs/2022-09-30.md new file mode 100644 index 0000000..71ce9dc --- /dev/null +++ b/logs/2022-09-30.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (September30) + +Time: September30 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.39 +- End: 2.30 + +## Completed Data +- $\approx$ 1,183.51GB + +## Average Grad Norm +- 0.60 + +## Progress +- 127.08% + +## Comment + +The loss is still falling today. From some test cases we can see that our model is getting better at English. Keep going! diff --git a/logs/2022-10-01.md b/logs/2022-10-01.md new file mode 100644 index 0000000..8f9cacc --- /dev/null +++ b/logs/2022-10-01.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October1) + +Time: October1 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.30 +- End: 2.31 + +## Completed Data +- $\approx$ 1,187.63GB + +## Average Grad Norm +- 0.56 + +## Progress +- 127.52% + +## Comment + +Hello October! The training loss seems to have entered a stable period🤔. diff --git a/logs/2022-10-02.md b/logs/2022-10-02.md new file mode 100644 index 0000000..34ea8d7 --- /dev/null +++ b/logs/2022-10-02.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October2) + +Time: October2 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.31 +- End: 2.25 + +## Completed Data +- $\approx$ 1,191.75GB + +## Average Grad Norm +- 0.54 + +## Progress +- 127.96% + +## Comment + +It's glad to see that loss continues to fall. Keep up the good work and become an English expert as well! diff --git a/logs/2022-10-03.md b/logs/2022-10-03.md new file mode 100644 index 0000000..d85514d --- /dev/null +++ b/logs/2022-10-03.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October3) + +Time: October3 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.25 +- End: 2.27 + +## Completed Data +- $\approx$ 1,195.87GB + +## Average Grad Norm +- 0.53 + +## Progress +- 128.41% + +## Comment + +A peaceful day. The training is very stable. Keep learning! diff --git a/logs/2022-10-04.md b/logs/2022-10-04.md new file mode 100644 index 0000000..cd33b9d --- /dev/null +++ b/logs/2022-10-04.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (October4) + +Time: October4 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.27 +- End: 2.23 + +## Completed Data +- $\approx$ 1,199.99GB + +## Average Grad Norm +- 0.52 + +## Progress +- 128.85% + +## Comment + +Today is another peaceful dayand CPM-Ant+ would like to say: *"Happy New Year's Day!" We are all happyand we will continue to do so in the years ahead. As alwaysCPM-Ant+ is a pleasure to work with. Thanks for reading and I hope you enjoy it as much as I do.* + +😆😆😆Interesting. diff --git a/logs/2022-10-05.md b/logs/2022-10-05.md new file mode 100644 index 0000000..255cdcf --- /dev/null +++ b/logs/2022-10-05.md @@ -0,0 +1,31 @@ + +# CPM-Live Training Log (October5) + +Time: October5 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.23 +- End: 2.25 + +## Completed Data +- $\approx$ 1,204.12GB + +## Average Grad Norm +- 0.51 + +## Progress +- 129.29% + +## Comment + +Today I write a comment in Chinese: + +*今天又是风平浪静的一天。模型训练仍在继续，不断提高自己吧！* + +Then I use CPM-Ant+ to translate it: + +*Today is a quiet day. Model training continueskeep improving yourself!* + +Not bad.😉 diff --git a/logs/2022-10-06.md b/logs/2022-10-06.md new file mode 100644 index 0000000..9dd3e04 --- /dev/null +++ b/logs/2022-10-06.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October6) + +Time: October6 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.25 +- End: 2.25 + +## Completed Data +- $\approx$ 1,208.24GB + +## Average Grad Norm +- 0.51 + +## Progress +- 129.73% + +## Comment + +The training has been going on for 130 days and still continues. We are getting closer to the day when the model evolves! diff --git a/logs/2022-10-07.md b/logs/2022-10-07.md new file mode 100644 index 0000000..fdbae1b --- /dev/null +++ b/logs/2022-10-07.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (October7) + +Time: October7 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.25 +- End: 2.24 + +## Completed Data +- $\approx$ 1,212.36GB + +## Average Grad Norm +- 0.50 + +## Progress +- 130.18% + +## Comment + +CPM-Ant+ has been training steadily for many days. Once againwe have witnessed the growth of a model. + diff --git a/logs/2022-10-08.md b/logs/2022-10-08.md new file mode 100644 index 0000000..e8938eb --- /dev/null +++ b/logs/2022-10-08.md @@ -0,0 +1,29 @@ + +# CPM-Live Training Log (October8) + +Time: October8 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.24 +- End: 2.25 + +## Completed Data +- $\approx$ 1,216.48GB + +## Average Grad Norm +- 0.50 + +## Progress +- 130.62% + +## Comment + +It's time to test the model again!😁 + +Prefix: Today CPM-Ant+ would like to say: + +Generated text: *"We are pleased with the performance of our CPM-Ant+and we look forward to working closely with you in the future."* + +Our model is happy with itself🤣 diff --git a/logs/2022-10-09.md b/logs/2022-10-09.md new file mode 100644 index 0000000..2dd6254 --- /dev/null +++ b/logs/2022-10-09.md @@ -0,0 +1,30 @@ + +# CPM-Live Training Log (October9) + +Time: October9 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.25 +- End: 2.22 + +## Completed Data +- $\approx$ 1,220.61GB + +## Average Grad Norm +- 0.50 + +## Progress +- 131.06% + +## Comment + +After 6,000 steps of trainingwhat has changed in our model? Let's test it. + +Prefix: Today CPM-Ant+ would like to say: + +Generated text: *"We're sorry to hear that you are having problems with your device. We will be sending you a new oneand we hope it works as well as the first time around."* + +🤔Today our model looks like a customer service person. + diff --git a/logs/2022-10-10.md b/logs/2022-10-10.md new file mode 100644 index 0000000..42df830 --- /dev/null +++ b/logs/2022-10-10.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October10) + +Time: October10 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.22 + +## Completed Data +- $\approx$ 1,224.72GB + +## Average Grad Norm +- 0.50 + +## Progress +- 131.50% + +## Comment + +The preparations for CPM-Bee are in the final stageand we believe that the training will officially begin very soon! diff --git a/logs/2022-10-11.md b/logs/2022-10-11.md new file mode 100644 index 0000000..16ea33e --- /dev/null +++ b/logs/2022-10-11.md @@ -0,0 +1,29 @@ + +# CPM-Live Training Log (October11) + +Time: October11 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.21 + +## Completed Data +- $\approx$ 1,228.85GB + +## Average Grad Norm +- 0.50 + +## Progress +- 131.95% + +## Comment + +Today CPM-Ant+ wants to say: + +Prefix: Today is Tuesday + +Generated text: *and I've been waiting for this game to come out. It was a long waitbut it finally came out today.* + +😃Our model is expecting something to happen. We will announce our big event this week. Stay tuned! diff --git a/logs/2022-10-12.md b/logs/2022-10-12.md new file mode 100644 index 0000000..45d36d6 --- /dev/null +++ b/logs/2022-10-12.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (October12) + +Time: October12 2022 16:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.21 +- End: 2.20 + +## Completed Data +- $\approx$ 1,232.96GB + +## Average Grad Norm +- 0.49 + +## Progress +- 132.39% + +## Comment + +CPM-Ant+ model has been released today! Check [here](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant-plus/cpm-live) for more information. + +The training is still ongoingand we'll start training CPM-Bee *tomorrow* based on the checkpoint of CPM-Ant+! diff --git a/logs/2022-10-13.md b/logs/2022-10-13.md new file mode 100644 index 0000000..d31d9e6 --- /dev/null +++ b/logs/2022-10-13.md @@ -0,0 +1,13 @@ + +# CPM-Live Training Log (October13) + +Time: October13 2022 16:00 + +Recorder: @zh-zheng + +## Progress +- 0.0% + +## Comment + +The training of CPM-Bee will launch today! Check our [training plan](https://github.com/OpenBMB/CPM-Live/blob/master/plans/CPM-Bee%E8%AE%AD%E7%BB%83%E8%AE%A1%E5%88%92%E4%B9%A6.md). Good luck! diff --git a/logs/2022-10-14.md b/logs/2022-10-14.md new file mode 100644 index 0000000..5d63250 --- /dev/null +++ b/logs/2022-10-14.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October14) + +Time: October14 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 10.41 +- End: 2.75 + +## Completed Data +- $\approx$ 1,246.88GB + +## Average Grad Norm +- 2.74 + +## Progress +- 1.0% + +## Comment + +The first day of CPM-Bee training! Since we inherit the checkpoint of CPM-Ant+the training loss drops very fast. We can see that our model performs quite well in the daily test. Keep going! diff --git a/logs/2022-10-15.md b/logs/2022-10-15.md new file mode 100644 index 0000000..d7ac3ef --- /dev/null +++ b/logs/2022-10-15.md @@ -0,0 +1,29 @@ + +# CPM-Live Training Log (October15) + +Time: October15 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 10.75 +- End: 2.44 + +## Completed Data +- $\approx$ 1,260.09GB + +## Average Grad Norm +- 26.26 + +## Progress +- 2.0% + +## Comment + +Today's key events: + +- The training loss increased rapidly at 5:00 and eventually became NaN. We reduced the learning rate by half and resumed the training at 10:40. + +- We observed that the loss increased again at 16:40so we reduced the learning rate by a factor of 0.2. + +We'll keep an eye on the training process. diff --git a/logs/2022-10-16.md b/logs/2022-10-16.md new file mode 100644 index 0000000..1d99f8e --- /dev/null +++ b/logs/2022-10-16.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October16) + +Time: October16 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.44 +- End: 2.71 + +## Completed Data +- $\approx$ 1,274.11GB + +## Average Grad Norm +- 2.26 + +## Progress +- 3.0% + +## Comment + +We observed that the training loss was still going upso we skipped some training data at 23:00 last night. diff --git a/logs/2022-10-17.md b/logs/2022-10-17.md new file mode 100644 index 0000000..36552e0 --- /dev/null +++ b/logs/2022-10-17.md @@ -0,0 +1,28 @@ + +# CPM-Live Training Log (October17) + +Time: October17 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.71 +- End: 2.59 + +## Completed Data +- $\approx$ 1,276.51GB + +## Average Grad Norm +- 1.77 + +## Progress +- 3.1% + +## Comment + +Key events: + +- 2022/10/16 19:30: Since the recent training is not very stablewe choose to delete the title generation and summarization datasets. + +- 2022/10/16 21:55: We encounter an issue with the data loader. The training has temporarily stoppedand we are fixing this problem. + diff --git a/logs/2022-10-18.md b/logs/2022-10-18.md new file mode 100644 index 0000000..9b45dd8 --- /dev/null +++ b/logs/2022-10-18.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October18) + +Time: October18 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.59 +- End: 2.65 + +## Completed Data +- $\approx$ 1,287.53GB + +## Average Grad Norm +- 3.02 + +## Progress +- 3.92% + +## Comment + +We have fixed the data loader and resumed the training. Besidesthe learning rate has been restored to its original scale. We expect the model to run stably now. diff --git a/logs/2022-10-19.md b/logs/2022-10-19.md new file mode 100644 index 0000000..d629b13 --- /dev/null +++ b/logs/2022-10-19.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October19) + +Time: October19 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.65 +- End: 2.73 + +## Completed Data +- $\approx$ 1,287.53GB + +## Average Grad Norm +- 0.86 + +## Progress +- 4.92% + +## Comment + +It is glad to see that the training is much more stable today. Keep it up! diff --git a/logs/2022-10-20.md b/logs/2022-10-20.md new file mode 100644 index 0000000..72ddf51 --- /dev/null +++ b/logs/2022-10-20.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October20) + +Time: October20 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.73 +- End: 2.60 + +## Completed Data +- $\approx$ 1,315.19GB + +## Average Grad Norm +- 0.64 + +## Progress +- 5.91% + +## Comment + +Finallythe loss began to drop normallyand the training returned to the right track. diff --git a/logs/2022-10-21.md b/logs/2022-10-21.md new file mode 100644 index 0000000..e6f6c85 --- /dev/null +++ b/logs/2022-10-21.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October21) + +Time: October21 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.60 +- End: 2.54 + +## Completed Data +- $\approx$ 1,328.82GB + +## Average Grad Norm +- 0.61 + +## Progress +- 6.89% + +## Comment + +We've added back the summarization and title generation tasks. By the waywe tested the QA capability of our model todaywhich is quite good! diff --git a/logs/2022-10-22.md b/logs/2022-10-22.md new file mode 100644 index 0000000..1e21d72 --- /dev/null +++ b/logs/2022-10-22.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (October22) + +Time: October22 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.54 +- End: 2.51 + +## Completed Data +- $\approx$ 1,342.41GB + +## Average Grad Norm +- 0.60 + +## Progress +- 7.86% + +## Comment + +Today we continue to test the question answering ability of our model and find that it is pretty smart. +It looks like we'll have to test it on harder questions later. diff --git a/logs/2022-10-23.md b/logs/2022-10-23.md new file mode 100644 index 0000000..97760f5 --- /dev/null +++ b/logs/2022-10-23.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October23) + +Time: October23 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.51 +- End: 2.48 + +## Completed Data +- $\approx$ 1,354.55GB + +## Average Grad Norm +- 0.62 + +## Progress +- 8.74% + +## Comment + +We observed that the training loss became NaN this morningand we stabilized the training by modifying the computation of attention scores. It seems to be working so far. diff --git a/logs/2022-10-24.md b/logs/2022-10-24.md new file mode 100644 index 0000000..d19fda0 --- /dev/null +++ b/logs/2022-10-24.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October24) + +Time: October24 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.48 +- End: 2.47 + +## Completed Data +- $\approx$ 1,369.96GB + +## Average Grad Norm +- 0.60 + +## Progress +- 9.84% + +## Comment + +We are constantly adding new tasks to our model. Stay tuned! diff --git a/logs/2022-10-25.md b/logs/2022-10-25.md new file mode 100644 index 0000000..43bb4be --- /dev/null +++ b/logs/2022-10-25.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October25) + +Time: October25 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.47 +- End: 2.45 + +## Completed Data +- $\approx$ 1,384.44GB + +## Average Grad Norm +- 0.68 + +## Progress +- 10.88% + +## Comment + +The training progress is over 10%! It is gratifying to watch the model progress day by day😁. diff --git a/logs/2022-10-26.md b/logs/2022-10-26.md new file mode 100644 index 0000000..639c1e9 --- /dev/null +++ b/logs/2022-10-26.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (October26) + +Time: October26 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.45 +- End: 2.43 + +## Completed Data +- $\approx$ 1,399.63GB + +## Average Grad Norm +- 0.61 + +## Progress +- 11.98% + +## Comment + +For sentence-level translationswe believe our model is currently relatively competentalthough there are occasionally some minor omissions in the translation. Check the daily test! + + diff --git a/logs/2022-10-27.md b/logs/2022-10-27.md new file mode 100644 index 0000000..91abe39 --- /dev/null +++ b/logs/2022-10-27.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (October27) + +Time: October27 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.43 +- End: 2.42 + +## Completed Data +- $\approx$ 1,414.79GB + +## Average Grad Norm +- 0.66 + +## Progress +- 13.06% + +## Comment + +The training is steady these daysand the model gets the question right again today. Good! + diff --git a/logs/2022-10-28.md b/logs/2022-10-28.md new file mode 100644 index 0000000..944377d --- /dev/null +++ b/logs/2022-10-28.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October28) + +Time: October28 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.42 +- End: 2.39 + +## Completed Data +- $\approx$ 1,429.95GB + +## Average Grad Norm +- 0.66 + +## Progress +- 14.15% + +## Comment + +We've added more capabilities to CPM-Beesee our capabilities tree for detailsand you can check out how the model performs on these new tasks in the Daily Test now! diff --git a/logs/2022-10-29.md b/logs/2022-10-29.md new file mode 100644 index 0000000..8e8627f --- /dev/null +++ b/logs/2022-10-29.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (October29) + +Time: October29 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.39 +- End: 2.37 + +## Completed Data +- $\approx$ 1,445.11GB + +## Average Grad Norm +- 0.67 + +## Progress +- 15.24% + +## Comment + +We show the cross-lingual question-answering capability of CPM-Bee in the Daily Test. diff --git a/logs/2022-10-30.md b/logs/2022-10-30.md new file mode 100644 index 0000000..fff79ad --- /dev/null +++ b/logs/2022-10-30.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (October30) + +Time: October30 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.37 +- End: 2.37 + +## Completed Data +- $\approx$ 1,452.58GB + +## Average Grad Norm +- 0.70 + +## Progress +- 15.78% + +## Comment + +Due to an unexpected issue in our datathe training was interrupted for about 12 hours and resumed at 12:00 today. + diff --git a/logs/2022-10-31.md b/logs/2022-10-31.md new file mode 100644 index 0000000..4d5498d --- /dev/null +++ b/logs/2022-10-31.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (October31) + +Time: October31 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.37 +- End: 2.37 + +## Completed Data +- $\approx$ 1,467.69GB + +## Average Grad Norm +- 0.91 + +## Progress +- 16.87% + +## Comment + +The training is relatively stable at presentand we are helping the model learn more NLP capabilities. It is gratifying that we can see its progress. + diff --git a/logs/2022-11-01.md b/logs/2022-11-01.md new file mode 100644 index 0000000..ba74860 --- /dev/null +++ b/logs/2022-11-01.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November1) + +Time: November1 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.37 +- End: 2.35 + +## Completed Data +- $\approx$ 1,482.79GB + +## Average Grad Norm +- 0.77 + +## Progress +- 17.95% + +## Comment + +Hi November! CPM-Bee will continue to pursue knowledge in the new month. diff --git a/logs/2022-11-02.md b/logs/2022-11-02.md new file mode 100644 index 0000000..d7b131e --- /dev/null +++ b/logs/2022-11-02.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (November2) + +Time: November2 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.35 +- End: 2.31 + +## Completed Data +- $\approx$ 1,497.92GB + +## Average Grad Norm +- 0.73 + +## Progress +- 19.03% + +## Comment + +TodayI am once again amazed at the QA ability of our model. I believe it really understands something. + + diff --git a/logs/2022-11-03.md b/logs/2022-11-03.md new file mode 100644 index 0000000..64fddd9 --- /dev/null +++ b/logs/2022-11-03.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November3) + +Time: November3 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.31 +- End: 2.33 + +## Completed Data +- $\approx$ 1,513.06GB + +## Average Grad Norm +- 0.74 + +## Progress +- 20.12% + +## Comment + +The training progress has exceeded 20%and more tasks are being added to the model. diff --git a/logs/2022-11-04.md b/logs/2022-11-04.md new file mode 100644 index 0000000..57ff15b --- /dev/null +++ b/logs/2022-11-04.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (November4) + +Time: November4 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.33 +- End: 2.31 + +## Completed Data +- $\approx$ 1,528.24GB + +## Average Grad Norm +- 0.71 + +## Progress +- 21.22% + +## Comment + +I think CPM-Bee did a good job of filling in the blanks in the Daily Test today. What do you think? + + diff --git a/logs/2022-11-05.md b/logs/2022-11-05.md new file mode 100644 index 0000000..44fd197 --- /dev/null +++ b/logs/2022-11-05.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (November5) + +Time: November5 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.31 +- End: 2.26 + +## Completed Data +- $\approx$ 1,543.42GB + +## Average Grad Norm +- 0.73 + +## Progress +- 22.31% + +## Comment + +The training is quite stable recentlyand the QA ability is continuously improving. + diff --git a/logs/2022-11-06.md b/logs/2022-11-06.md new file mode 100644 index 0000000..a1facc1 --- /dev/null +++ b/logs/2022-11-06.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (November6) + +Time: November6 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.26 +- End: 2.25 + +## Completed Data +- $\approx$ 1,558.59GB + +## Average Grad Norm +- 0.75 + +## Progress +- 23.40% + +## Comment + +A peaceful weekend. Today's daily test was not that easybut the model still got the answer right. Good job! + + diff --git a/logs/2022-11-07.md b/logs/2022-11-07.md new file mode 100644 index 0000000..5431870 --- /dev/null +++ b/logs/2022-11-07.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (November7) + +Time: November7 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.25 +- End: 2.24 + +## Completed Data +- $\approx$ 1,573.76GB + +## Average Grad Norm +- 0.76 + +## Progress +- 24.49% + +## Comment + +In the Daily Testwe can see that the translation ability of CPM-Bee is already quite good. + diff --git a/logs/2022-11-08.md b/logs/2022-11-08.md new file mode 100644 index 0000000..de53f5b --- /dev/null +++ b/logs/2022-11-08.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (November8) + +Time: November8 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.24 +- End: 2.26 + +## Completed Data +- $\approx$ 1,588.72GB + +## Average Grad Norm +- 0.77 + +## Progress +- 25.56% + +## Comment + +We released a [video](https://b23.tv/WHpj6IT) to briefly demonstrate the capabilities of CPM-Bee. Check it! + + diff --git a/logs/2022-11-09.md b/logs/2022-11-09.md new file mode 100644 index 0000000..70dbdba --- /dev/null +++ b/logs/2022-11-09.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (November9) + +Time: November9 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.26 +- End: 2.25 + +## Completed Data +- $\approx$ 1,603.77GB + +## Average Grad Norm +- 0.81 + +## Progress +- 26.64% + +## Comment + +The training has been quite stableand we are still working on adding more new features to the model. + diff --git a/logs/2022-11-10.md b/logs/2022-11-10.md new file mode 100644 index 0000000..6a08317 --- /dev/null +++ b/logs/2022-11-10.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November10) + +Time: November10 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.25 +- End: 2.23 + +## Completed Data +- $\approx$ 1,618.84GB + +## Average Grad Norm +- 0.79 + +## Progress +- 27.73% + +## Comment + +Guess what I'm going to say? Yesa peaceful day😁. diff --git a/logs/2022-11-11.md b/logs/2022-11-11.md new file mode 100644 index 0000000..bc9d849 --- /dev/null +++ b/logs/2022-11-11.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November11) + +Time: November11 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.23 +- End: 2.24 + +## Completed Data +- $\approx$ 1,633.90GB + +## Average Grad Norm +- 0.80 + +## Progress +- 28.81% + +## Comment + +It seems that our model is good at solving brain teasers😎. diff --git a/logs/2022-11-12.md b/logs/2022-11-12.md new file mode 100644 index 0000000..b7d8d0f --- /dev/null +++ b/logs/2022-11-12.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November12) + +Time: November12 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.24 +- End: 2.22 + +## Completed Data +- $\approx$ 1,648.96GB + +## Average Grad Norm +- 0.83 + +## Progress +- 29.89% + +## Comment + +In Daily Testfaced with a more difficult questionour model got the answer wrong this time. Try itcan you answer it correctly? diff --git a/logs/2022-11-13.md b/logs/2022-11-13.md new file mode 100644 index 0000000..69f1efd --- /dev/null +++ b/logs/2022-11-13.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November13) + +Time: November13 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.22 + +## Completed Data +- $\approx$ 1,663.99GB + +## Average Grad Norm +- 0.88 + +## Progress +- 30.97% + +## Comment + +You can see that we use CPM-Bee to translate sentences in a research paper today. It performs fairly well. diff --git a/logs/2022-11-14.md b/logs/2022-11-14.md new file mode 100644 index 0000000..ffa1145 --- /dev/null +++ b/logs/2022-11-14.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November14) + +Time: November14 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.22 + +## Completed Data +- $\approx$ 1,679.06GB + +## Average Grad Norm +- 1.16 + +## Progress +- 32.05% + +## Comment + +In our test todaythe dialogue ability of the model is much better than before. diff --git a/logs/2022-11-15.md b/logs/2022-11-15.md new file mode 100644 index 0000000..a4fe432 --- /dev/null +++ b/logs/2022-11-15.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November15) + +Time: November15 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.20 + +## Completed Data +- $\approx$ 1,694.12GB + +## Average Grad Norm +- 0.98 + +## Progress +- 33.13% + +## Comment + +In today's testour model missed a sentence in the translation. This seems to be a common problem when translating long documents🤔. diff --git a/logs/2022-11-16.md b/logs/2022-11-16.md new file mode 100644 index 0000000..55d4d4d --- /dev/null +++ b/logs/2022-11-16.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November16) + +Time: November16 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.20 +- End: 2.19 + +## Completed Data +- $\approx$ 1,709.06GB + +## Average Grad Norm +- 0.86 + +## Progress +- 34.21% + +## Comment + +We are working on adding some advanced text generation features to the model. diff --git a/logs/2022-11-17.md b/logs/2022-11-17.md new file mode 100644 index 0000000..311b5f9 --- /dev/null +++ b/logs/2022-11-17.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November17) + +Time: November17 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.19 +- End: 2.22 + +## Completed Data +- $\approx$ 1,724.07GB + +## Average Grad Norm +- 0.88 + +## Progress +- 35.29% + +## Comment + +What's the easiest way you've thought of to use a large language model? diff --git a/logs/2022-11-18.md b/logs/2022-11-18.md new file mode 100644 index 0000000..8254a59 --- /dev/null +++ b/logs/2022-11-18.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November18) + +Time: November18 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.22 + +## Completed Data +- $\approx$ 1,739.09GB + +## Average Grad Norm +- 0.86 + +## Progress +- 36.37% + +## Comment + +What's the mission of OpenBMB? It's to lower the barriers to use big models! diff --git a/logs/2022-11-19.md b/logs/2022-11-19.md new file mode 100644 index 0000000..d5f578e --- /dev/null +++ b/logs/2022-11-19.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (November19) + +Time: November19 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.22 +- End: 2.21 + +## Completed Data +- $\approx$ 1,754.12GB + +## Average Grad Norm +- 0.88 + +## Progress +- 37.45% + +## Comment + +In these daysthe training loss is very smooth and looks very nice. But this does not mean that the model has stopped learningon the contraryit is still improving. + diff --git a/logs/2022-11-2.md b/logs/2022-11-2.md new file mode 100644 index 0000000..14e3a8e --- /dev/null +++ b/logs/2022-11-2.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November24) + +Time: November24 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.15 +- End: 2.14 + +## Completed Data +- $\approx$ 1826.75GB + +## Average Grad Norm +- 2.29 + +## Progress +- 42.66% + +## Comment + +The loss still continues to fall. diff --git a/logs/2022-11-20.md b/logs/2022-11-20.md new file mode 100644 index 0000000..5e6c798 --- /dev/null +++ b/logs/2022-11-20.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November20) + +Time: November20 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.21 +- End: 2.21 + +## Completed Data +- $\approx$ 1,769.14GB + +## Average Grad Norm +- 0.90 + +## Progress +- 38.52% + +## Comment + +Our goal is to train the model to become an excellent decathlon player. At presentit is qualified in terms of the number of tasks. diff --git a/logs/2022-11-21.md b/logs/2022-11-21.md new file mode 100644 index 0000000..0fac33a --- /dev/null +++ b/logs/2022-11-21.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November21) + +Time: November21 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.21 +- End: 2.21 + +## Completed Data +- $\approx$ 1,783.96GB + +## Average Grad Norm +- 1.07 + +## Progress +- 39.59% + +## Comment + +From today's testwe can see that the current model is not good enough for the translation of professional terms. diff --git a/logs/2022-11-22.md b/logs/2022-11-22.md new file mode 100644 index 0000000..6955600 --- /dev/null +++ b/logs/2022-11-22.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November22) + +Time: November22 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.21 +- End: 2.17 + +## Completed Data +- $\approx$ 1798.23GB + +## Average Grad Norm +- 1.22 + +## Progress +- 40.62% + +## Comment + +Todaythe training loss became NaN at around 12:00. We solved this problem by scaling weights. We'll keep an eye on the model in the next few days. diff --git a/logs/2022-11-23.md b/logs/2022-11-23.md new file mode 100644 index 0000000..57579ec --- /dev/null +++ b/logs/2022-11-23.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November23) + +Time: November23 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.17 +- End: 2.15 + +## Completed Data +- $\approx$ 1811.87GB + +## Average Grad Norm +- 2.74 + +## Progress +- 41.60% + +## Comment + +After we reduced the learning rate and scaled down the weightsthe training was quite stable today. diff --git a/logs/2022-11-24.md b/logs/2022-11-24.md new file mode 100644 index 0000000..14e3a8e --- /dev/null +++ b/logs/2022-11-24.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November24) + +Time: November24 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.15 +- End: 2.14 + +## Completed Data +- $\approx$ 1826.75GB + +## Average Grad Norm +- 2.29 + +## Progress +- 42.66% + +## Comment + +The loss still continues to fall. diff --git a/logs/2022-11-25.md b/logs/2022-11-25.md new file mode 100644 index 0000000..0348c0c --- /dev/null +++ b/logs/2022-11-25.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November25) + +Time: November25 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.14 +- End: 2.14 + +## Completed Data +- $\approx$ 1841.73GB + +## Average Grad Norm +- 2.17 + +## Progress +- 43.74% + +## Comment + +We are proud to see that the CPM-Live project has been running for 180 days. Keep going! diff --git a/logs/2022-11-26.md b/logs/2022-11-26.md new file mode 100644 index 0000000..03cde1a --- /dev/null +++ b/logs/2022-11-26.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November26) + +Time: November26 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.14 +- End: 2.13 + +## Completed Data +- $\approx$ 1856.73GB + +## Average Grad Norm +- 2.18 + +## Progress +- 44.82% + +## Comment + +Since we reduced the weights of the model and the learning ratethe training loss has been decreasing. Very interesting! diff --git a/logs/2022-11-27.md b/logs/2022-11-27.md new file mode 100644 index 0000000..0fdc2e6 --- /dev/null +++ b/logs/2022-11-27.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November27) + +Time: November27 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.13 +- End: 2.12 + +## Completed Data +- $\approx$ 1871.71GB + +## Average Grad Norm +- 2.17 + +## Progress +- 45.89% + +## Comment + +From today's testit can be seen that our model is not sensitive to numbers when doing the translation task. diff --git a/logs/2022-11-28.md b/logs/2022-11-28.md new file mode 100644 index 0000000..077a248 --- /dev/null +++ b/logs/2022-11-28.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November28) + +Time: November28 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.13 + +## Completed Data +- $\approx$ 1886.68GB + +## Average Grad Norm +- 2.17 + +## Progress +- 46.97% + +## Comment + +It seems that some simple questions can't overwhelm our model. diff --git a/logs/2022-11-29.md b/logs/2022-11-29.md new file mode 100644 index 0000000..a26abd0 --- /dev/null +++ b/logs/2022-11-29.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November29) + +Time: November29 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.13 +- End: 2.13 + +## Completed Data +- $\approx$ 1901.66GB + +## Average Grad Norm +- 2.19 + +## Progress +- 48.05% + +## Comment + +The training of the model has recently entered a stable period. diff --git a/logs/2022-11-30.md b/logs/2022-11-30.md new file mode 100644 index 0000000..a7a2580 --- /dev/null +++ b/logs/2022-11-30.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (November30) + +Time: November30 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.13 +- End: 2.13 + +## Completed Data +- $\approx$ 1916.64GB + +## Average Grad Norm +- 2.19 + +## Progress +- 49.12% + +## Comment + +Goodbye November! diff --git a/logs/2022-12-01.md b/logs/2022-12-01.md new file mode 100644 index 0000000..b774f80 --- /dev/null +++ b/logs/2022-12-01.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December1) + +Time: December1 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.13 +- End: 2.13 + +## Completed Data +- $\approx$ 1931.62GB + +## Average Grad Norm +- 2.20 + +## Progress +- 50.20% + +## Comment + +We are glad to see that the training progress has exceeded 50%! diff --git a/logs/2022-12-02.md b/logs/2022-12-02.md new file mode 100644 index 0000000..9fa0120 --- /dev/null +++ b/logs/2022-12-02.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December2) + +Time: December2 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.13 +- End: 2.12 + +## Completed Data +- $\approx$ 1946.59GB + +## Average Grad Norm +- 2.19 + +## Progress +- 51.27% + +## Comment + +After the training loss kept completely consistent for several daysit finally dropped a little today. Very interesting🤣. diff --git a/logs/2022-12-03.md b/logs/2022-12-03.md new file mode 100644 index 0000000..aa62afd --- /dev/null +++ b/logs/2022-12-03.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December3) + +Time: December3 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.11 + +## Completed Data +- $\approx$ 1961.56GB + +## Average Grad Norm +- 2.20 + +## Progress +- 52.35% + +## Comment + +It's funny to see from today's test that our model can translate "Globo Esporte" correctlybut it cannot recognize the player's name properly. diff --git a/logs/2022-12-04.md b/logs/2022-12-04.md new file mode 100644 index 0000000..8e83c03 --- /dev/null +++ b/logs/2022-12-04.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December4) + +Time: December4 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.11 +- End: 2.12 + +## Completed Data +- $\approx$ 1976.56GB + +## Average Grad Norm +- 2.27 + +## Progress +- 53.43% + +## Comment + +CPM-Bee made a prediction for tonight's World Cup match😝. diff --git a/logs/2022-12-05.md b/logs/2022-12-05.md new file mode 100644 index 0000000..4ad6fd9 --- /dev/null +++ b/logs/2022-12-05.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December5) + +Time: December5 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.13 + +## Completed Data +- $\approx$ 1991.54GB + +## Average Grad Norm +- 2.25 + +## Progress +- 54.51% + +## Comment + +Compared with yesterday's direct questiontoday I gave the model some historical records to predict the result of today's football matchand it seems that its prediction is quite reasonable. diff --git a/logs/2022-12-06.md b/logs/2022-12-06.md new file mode 100644 index 0000000..798220a --- /dev/null +++ b/logs/2022-12-06.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December6) + +Time: December6 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.13 +- End: 2.12 + +## Completed Data +- $\approx$ 2006.54GB + +## Average Grad Norm +- 2.22 + +## Progress +- 55.58% + +## Comment + +The CPM-Live series models have seen over 2,000GB of data. Keep getting more erudite! diff --git a/logs/2022-12-07.md b/logs/2022-12-07.md new file mode 100644 index 0000000..5f6284e --- /dev/null +++ b/logs/2022-12-07.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December7) + +Time: December7 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.11 + +## Completed Data +- $\approx$ 2021.51GB + +## Average Grad Norm +- 2.21 + +## Progress +- 56.66% + +## Comment + +The model has been trained steadily for quite some timeand I have somewhat forgotten when the training was last broken. diff --git a/logs/2022-12-08.md b/logs/2022-12-08.md new file mode 100644 index 0000000..6be7e81 --- /dev/null +++ b/logs/2022-12-08.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (December8) + +Time: December8 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.11 +- End: 2.12 + +## Completed Data +- $\approx$ 2036.49GB + +## Average Grad Norm +- 2.22 + +## Progress +- 58.66% + +## Comment + +Some smaller versions of CPM-Bee are being produced. Stay tuned! + diff --git a/logs/2022-12-09.md b/logs/2022-12-09.md new file mode 100644 index 0000000..543c036 --- /dev/null +++ b/logs/2022-12-09.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December9) + +Time: December9 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.07 + +## Completed Data +- $\approx$ 2050.15GB + +## Average Grad Norm +- 2.10 + +## Progress +- 60.66% + +## Comment + +An eventful week has passed! CPM-Bee's loss has dropped a bit more. diff --git a/logs/2022-12-10.md b/logs/2022-12-10.md new file mode 100644 index 0000000..b0393e1 --- /dev/null +++ b/logs/2022-12-10.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (December10) + +Time: December10 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.10 + +## Completed Data +- $\approx$ 2054.60GB + +## Average Grad Norm +- 1.13 + +## Progress +- 62.66% + +## Comment + +Although today is a weekendCPM-Bee keeps rushing to the finish line. + diff --git a/logs/2022-12-11.md b/logs/2022-12-11.md new file mode 100644 index 0000000..bc2cd84 --- /dev/null +++ b/logs/2022-12-11.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December11) + +Time: December11 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.09 + +## Completed Data +- $\approx$ 2063.50GB + +## Average Grad Norm +- 1.13 + +## Progress +- 64.66% + +## Comment + +The weekend is overso let's welcome the next week full of unknowns. diff --git a/logs/2022-12-12.md b/logs/2022-12-12.md new file mode 100644 index 0000000..44211ad --- /dev/null +++ b/logs/2022-12-12.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December12) + +Time: December12 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.11 + +## Completed Data +- $\approx$ 2067.94GB + +## Average Grad Norm +- 1.13 + +## Progress +- 66.66% + +## Comment + +In the daily testwe give CPM-Bee a prompt and let it talk a bit more. diff --git a/logs/2022-12-13.md b/logs/2022-12-13.md new file mode 100644 index 0000000..1ce152e --- /dev/null +++ b/logs/2022-12-13.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December13) + +Time: December13 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.11 +- End: 2.11 + +## Completed Data +- $\approx$ 2072.40GB + +## Average Grad Norm +- 1.13 + +## Progress +- 68.66% + +## Comment + +In today's testwe let CPM-Bee discuss the topic of food. diff --git a/logs/2022-12-14.md b/logs/2022-12-14.md new file mode 100644 index 0000000..7e78982 --- /dev/null +++ b/logs/2022-12-14.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (December14) + +Time: December14 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.11 +- End: 2.07 + +## Completed Data +- $\approx$ 2076.86GB + +## Average Grad Norm +- 1.13 + +## Progress +- 70.66% + +## Comment + +CPM-Bee has been training for over two months! We're excited to see it continue to improve! + diff --git a/logs/2022-12-15.md b/logs/2022-12-15.md new file mode 100644 index 0000000..4f3bc52 --- /dev/null +++ b/logs/2022-12-15.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December15) + +Time: December15 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.08 + +## Completed Data +- $\approx$ 2081.31GB + +## Average Grad Norm +- 1.13 + +## Progress +- 72.66% + +## Comment + +Another milestone: 200-day anniversary of CPM-Live training!🎉🎉🎉 diff --git a/logs/2022-12-16.md b/logs/2022-12-16.md new file mode 100644 index 0000000..488fd00 --- /dev/null +++ b/logs/2022-12-16.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December16) + +Time: December16 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.08 +- End: 2.09 + +## Completed Data +- $\approx$ 2085.76GB + +## Average Grad Norm +- 1.13 + +## Progress +- 74.66% + +## Comment + +Another week went by and we asked CPM-Bee to talk about his thoughts. Check the daily test. diff --git a/logs/2022-12-17.md b/logs/2022-12-17.md new file mode 100644 index 0000000..81f71aa --- /dev/null +++ b/logs/2022-12-17.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December17) + +Time: December17 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.09 + +## Completed Data +- $\approx$ 2090.20GB + +## Average Grad Norm +- 1.14 + +## Progress +- 76.66% + +## Comment + +I think the model answered the question fairly well in today's test. diff --git a/logs/2022-12-18.md b/logs/2022-12-18.md new file mode 100644 index 0000000..1436337 --- /dev/null +++ b/logs/2022-12-18.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December18) + +Time: December18 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.08 + +## Completed Data +- $\approx$ 2094.65GB + +## Average Grad Norm +- 1.13 + +## Progress +- 78.66% + +## Comment + +The weekend is over. Let's keep fighting in the last two weeks of 2022. diff --git a/logs/2022-12-19.md b/logs/2022-12-19.md new file mode 100644 index 0000000..69a0f37 --- /dev/null +++ b/logs/2022-12-19.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December19) + +Time: December19 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.08 +- End: 2.09 + +## Completed Data +- $\approx$ 2099.08GB + +## Average Grad Norm +- 1.13 + +## Progress +- 80.66% + +## Comment + +The training progress exceeds 80%! diff --git a/logs/2022-12-20.md b/logs/2022-12-20.md new file mode 100644 index 0000000..96f1505 --- /dev/null +++ b/logs/2022-12-20.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December20) + +Time: December20 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.01 + +## Completed Data +- $\approx$ 2103.55GB + +## Average Grad Norm +- 1.14 + +## Progress +- 81.66% + +## Comment + +Todaythe training loss dropped a little more. diff --git a/logs/2022-12-21.md b/logs/2022-12-21.md new file mode 100644 index 0000000..0b8b084 --- /dev/null +++ b/logs/2022-12-21.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December21) + +Time: December21 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.01 +- End: 2.07 + +## Completed Data +- $\approx$ 2108.01GB + +## Average Grad Norm +- 1.19 + +## Progress +- 82.66% + +## Comment + +In today's testwe use our model for topic classification. Well done! diff --git a/logs/2022-12-22.md b/logs/2022-12-22.md new file mode 100644 index 0000000..10442b3 --- /dev/null +++ b/logs/2022-12-22.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (December22) + +Time: December22 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.00 + +## Completed Data +- $\approx$ 2112.49GB + +## Average Grad Norm +- 1.24 + +## Progress +- 83.66% + +## Comment + +Oopsthe training loss reached 2.00! + + diff --git a/logs/2022-12-23.md b/logs/2022-12-23.md new file mode 100644 index 0000000..afde708 --- /dev/null +++ b/logs/2022-12-23.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (December23) + +Time: December23 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.00 +- End: 1.99 + +## Completed Data +- $\approx$ 2116.98GB + +## Average Grad Norm +- 1.21 + +## Progress +- 84.66% + +## Comment + +Todaywe let the model tell us the next sentence of an ancient poem. Good job! + diff --git a/logs/2022-12-24.md b/logs/2022-12-24.md new file mode 100644 index 0000000..e952d67 --- /dev/null +++ b/logs/2022-12-24.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December24) + +Time: December24 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.99 +- End: 2.00 + +## Completed Data +- $\approx$ 2121.43GB + +## Average Grad Norm +- 1.20 + +## Progress +- 85.66% + +## Comment + +To my surpriseour model can also complete the lyrics. Interesting! diff --git a/logs/2022-12-25.md b/logs/2022-12-25.md new file mode 100644 index 0000000..7e3eaf7 --- /dev/null +++ b/logs/2022-12-25.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December25) + +Time: December25 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.00 +- End: 2.00 + +## Completed Data +- $\approx$ 2125.88GB + +## Average Grad Norm +- 1.20 + +## Progress +- 86.66% + +## Comment + +The last five days of 2022 to goand CPM-Live is with you. diff --git a/logs/2022-12-26.md b/logs/2022-12-26.md new file mode 100644 index 0000000..f35bbee --- /dev/null +++ b/logs/2022-12-26.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December26) + +Time: December26 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.00 +- End: 2.03 + +## Completed Data +- $\approx$ 2130.34GB + +## Average Grad Norm +- 1.19 + +## Progress +- 87.66% + +## Comment + +Our model can currently do some simple common sense based QA. diff --git a/logs/2022-12-27.md b/logs/2022-12-27.md new file mode 100644 index 0000000..2a11a8c --- /dev/null +++ b/logs/2022-12-27.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December27) + +Time: December27 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.03 +- End: 2.03 + +## Completed Data +- $\approx$ 2134.79GB + +## Average Grad Norm +- 1.14 + +## Progress +- 88.66% + +## Comment + +In today's Q&AI asked cpm-bee to give an explanation along with the answer. diff --git a/logs/2022-12-28.md b/logs/2022-12-28.md new file mode 100644 index 0000000..849f273 --- /dev/null +++ b/logs/2022-12-28.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (December28) + +Time: December28 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.03 +- End: 2.05 + +## Completed Data +- $\approx$ 2139.24GB + +## Average Grad Norm +- 1.14 + +## Progress +- 89.66% + +## Comment + +Sometimes the model's answer makes us confusedand we will try to improve it in the future. + diff --git a/logs/2022-12-29.md b/logs/2022-12-29.md new file mode 100644 index 0000000..be1eb0c --- /dev/null +++ b/logs/2022-12-29.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December29) + +Time: December29 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.03 + +## Completed Data +- $\approx$ 2143.70GB + +## Average Grad Norm +- 1.13 + +## Progress +- 90.66% + +## Comment + +A big event: the total cost of CPM-Live project exceeded 1 MILLION Yuan! diff --git a/logs/2022-12-30.md b/logs/2022-12-30.md new file mode 100644 index 0000000..1439ff5 --- /dev/null +++ b/logs/2022-12-30.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December30) + +Time: December30 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.03 +- End: 2.04 + +## Completed Data +- $\approx$ 2148.16GB + +## Average Grad Norm +- 1.14 + +## Progress +- 90.76% + +## Comment + +On the last working day of 2022CPM-Live spent in the steady training. diff --git a/logs/2022-12-31.md b/logs/2022-12-31.md new file mode 100644 index 0000000..33dd09c --- /dev/null +++ b/logs/2022-12-31.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (December31) + +Time: December31 2022 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.04 + +## Completed Data +- $\approx$ 2152.57GB + +## Average Grad Norm +- 1.14 + +## Progress +- 90.86% + +## Comment + +Happy new year! diff --git a/logs/2023-01-01.md b/logs/2023-01-01.md new file mode 100644 index 0000000..ee2b357 --- /dev/null +++ b/logs/2023-01-01.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January1) + +Time: January1 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.04 + +## Completed Data +- $\approx$ 2157.03GB + +## Average Grad Norm +- 1.15 + +## Progress +- 90.96% + +## Comment + +The first day of 2023! The CPM-Live project has entered its second year. More surprises on the way! diff --git a/logs/2023-01-02.md b/logs/2023-01-02.md new file mode 100644 index 0000000..dd52c9f --- /dev/null +++ b/logs/2023-01-02.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (January2) + +Time: January2 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.02 + +## Completed Data +- $\approx$ 2161.48GB + +## Average Grad Norm +- 1.14 + +## Progress +- 91.06% + +## Comment + +It's fun to keep exploring the knowledge in the big model😁. + diff --git a/logs/2023-01-03.md b/logs/2023-01-03.md new file mode 100644 index 0000000..797cd6e --- /dev/null +++ b/logs/2023-01-03.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January3) + +Time: January3 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.02 +- End: 2.17 + +## Completed Data +- $\approx$ 2165.93GB + +## Average Grad Norm +- 1.16 + +## Progress +- 91.16% + +## Comment + +As you can see from today's testour model can perform some common sense reasoning. diff --git a/logs/2023-01-04.md b/logs/2023-01-04.md new file mode 100644 index 0000000..a363d1d --- /dev/null +++ b/logs/2023-01-04.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January4) + +Time: January4 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.17 +- End: 2.10 + +## Completed Data +- $\approx$ 2168.59GB + +## Average Grad Norm +- 1.18 + +## Progress +- 91.26% + +## Comment + +Today's question is not so well answered by the modelmaybe it needs to search the Internet to get some information. diff --git a/logs/2023-01-05.md b/logs/2023-01-05.md new file mode 100644 index 0000000..6022aff --- /dev/null +++ b/logs/2023-01-05.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January5) + +Time: January5 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.10 +- End: 2.11 + +## Completed Data +- $\approx$ 2173.00GB + +## Average Grad Norm +- 1.18 + +## Progress +- 91.36% + +## Comment + +I think the recipe generated by the model today is very reasonable. What do you think🤔? diff --git a/logs/2023-01-06.md b/logs/2023-01-06.md new file mode 100644 index 0000000..575beb3 --- /dev/null +++ b/logs/2023-01-06.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January6) + +Time: January6 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.11 +- End: 2.09 + +## Completed Data +- $\approx$ 2177.41GB + +## Average Grad Norm +- 1.19 + +## Progress +- 91.46% + +## Comment + +Today I asked cpm-bee to generate another recipe. It not only met my needsbut also told me the characteristics of the dishthe key points of cooking and some tips. Good job! diff --git a/logs/2023-01-07.md b/logs/2023-01-07.md new file mode 100644 index 0000000..00d1003 --- /dev/null +++ b/logs/2023-01-07.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January7) + +Time: January7 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.06 + +## Completed Data +- $\approx$ 2181.81GB + +## Average Grad Norm +- 1.19 + +## Progress +- 91.56% + +## Comment + +In today's testour model wrote an essay based on the given topic. diff --git a/logs/2023-01-08.md b/logs/2023-01-08.md new file mode 100644 index 0000000..775b252 --- /dev/null +++ b/logs/2023-01-08.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January8) + +Time: January8 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.09 + +## Completed Data +- $\approx$ 2186.21GB + +## Average Grad Norm +- 1.19 + +## Progress +- 91.66% + +## Comment + +Todaywe asked the model to write a science fiction article. Quite funny🤣 diff --git a/logs/2023-01-09.md b/logs/2023-01-09.md new file mode 100644 index 0000000..0129e94 --- /dev/null +++ b/logs/2023-01-09.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January9) + +Time: January9 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.10 + +## Completed Data +- $\approx$ 2190.62GB + +## Average Grad Norm +- 1.19 + +## Progress +- 91.76% + +## Comment + +Today I was thinking about whether it would be better to buy an Android phone or an iPhone for my next phone. I asked our model for help and got the answer. diff --git a/logs/2023-01-10.md b/logs/2023-01-10.md new file mode 100644 index 0000000..78358eb --- /dev/null +++ b/logs/2023-01-10.md @@ -0,0 +1,25 @@ + +# CPM-Live Training Log (January10) + +Time: January10 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.10 +- End: 2.08 + +## Completed Data +- $\approx$ 2195.02GB + +## Average Grad Norm +- 1.17 + +## Progress +- 91.86% + +## Comment + +I asked CPM-Bee what gift I should give to my girlfriendand it recommended 40 items to me😆. + + diff --git a/logs/2023-01-11.md b/logs/2023-01-11.md new file mode 100644 index 0000000..231ed0c --- /dev/null +++ b/logs/2023-01-11.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (January11) + +Time: January11 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.08 +- End: 2.07 + +## Completed Data +- $\approx$ 2199.43GB + +## Average Grad Norm +- 1.18 + +## Progress +- 91.96% + +## Comment + +Our model answered my question about the lipstick number carefullybut I asked an expert and learned that the model's answer was not really accurate. Maybe the model needs some external knowledge to answer questions in this area. + diff --git a/logs/2023-01-12.md b/logs/2023-01-12.md new file mode 100644 index 0000000..9060d29 --- /dev/null +++ b/logs/2023-01-12.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January12) + +Time: January12 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.09 + +## Completed Data +- $\approx$ 2203.85GB + +## Average Grad Norm +- 1.18 + +## Progress +- 92.06% + +## Comment + +Have you ever thought about text processing in Excel? CPM-Bee can help you! Try our [MFTable (模力表格)](https://live.openbmb.org/playground)! diff --git a/logs/2023-01-13.md b/logs/2023-01-13.md new file mode 100644 index 0000000..400e190 --- /dev/null +++ b/logs/2023-01-13.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January13) + +Time: January13 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.07 + +## Completed Data +- $\approx$ 2208.26GB + +## Average Grad Norm +- 1.18 + +## Progress +- 92.16% + +## Comment + +Todaylet's translate a poem with our model. diff --git a/logs/2023-01-14.md b/logs/2023-01-14.md new file mode 100644 index 0000000..8c3e983 --- /dev/null +++ b/logs/2023-01-14.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (January14) + +Time: January14 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.07 + +## Completed Data +- $\approx$ 2212.67GB + +## Average Grad Norm +- 1.19 + +## Progress +- 92.26% + +## Comment + +For the QA taskour goal is to make the model truly understand the articleand answer some less obvious questions. + diff --git a/logs/2023-01-15.md b/logs/2023-01-15.md new file mode 100644 index 0000000..7160c5c --- /dev/null +++ b/logs/2023-01-15.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January15) + +Time: January15 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.06 + +## Completed Data +- $\approx$ 2217.08GB + +## Average Grad Norm +- 1.20 + +## Progress +- 92.36% + +## Comment + +I sometimes wonder if CPM-bee's fill-in-the-blank ability has surpassed mine. diff --git a/logs/2023-01-16.md b/logs/2023-01-16.md new file mode 100644 index 0000000..85f09db --- /dev/null +++ b/logs/2023-01-16.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January16) + +Time: January16 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.08 + +## Completed Data +- $\approx$ 2221.52GB + +## Average Grad Norm +- 1.20 + +## Progress +- 92.46% + +## Comment + +In today's testwe let the model teach us how to make a *dark cuisine*. Wellit still looks reasonable. diff --git a/logs/2023-01-17.md b/logs/2023-01-17.md new file mode 100644 index 0000000..747f832 --- /dev/null +++ b/logs/2023-01-17.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January17) + +Time: January17 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.08 +- End: 2.06 + +## Completed Data +- $\approx$ 2225.93GB + +## Average Grad Norm +- 1.20 + +## Progress +- 92.56% + +## Comment + +Recentlywe are still enhancing the capabilities of CPM-Bee. Stay tuned! diff --git a/logs/2023-01-18.md b/logs/2023-01-18.md new file mode 100644 index 0000000..2159bd1 --- /dev/null +++ b/logs/2023-01-18.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January18) + +Time: January18 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.06 + +## Completed Data +- $\approx$ 2230.35GB + +## Average Grad Norm +- 1.22 + +## Progress +- 92.66% + +## Comment + +Our model succeeds in giving an itinerary for a tour of Chongqingbut it is too abbreviated. diff --git a/logs/2023-01-19.md b/logs/2023-01-19.md new file mode 100644 index 0000000..7c9ff12 --- /dev/null +++ b/logs/2023-01-19.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January19) + +Time: January19 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.04 + +## Completed Data +- $\approx$ 2234.77GB + +## Average Grad Norm +- 1.24 + +## Progress +- 92.76% + +## Comment + +In today's testCPM-Bee is a very competent shopping guideexcept for recommending the lipstickas this contradicts the context. diff --git a/logs/2023-01-20.md b/logs/2023-01-20.md new file mode 100644 index 0000000..471335c --- /dev/null +++ b/logs/2023-01-20.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (January20) + +Time: January20 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.05 + +## Completed Data +- $\approx$ 2239.18GB + +## Average Grad Norm +- 1.25 + +## Progress +- 92.86% + +## Comment + +We have entered the 100th day of CPM-Live training. Glad to see that everything is running smoothly! + diff --git a/logs/2023-01-21.md b/logs/2023-01-21.md new file mode 100644 index 0000000..f6473cf --- /dev/null +++ b/logs/2023-01-21.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January21) + +Time: January21 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.02 + +## Completed Data +- $\approx$ 2243.59GB + +## Average Grad Norm +- 1.25 + +## Progress +- 92.96% + +## Comment + +We wish everyone a Happy Chinese New Year! diff --git a/logs/2023-01-22.md b/logs/2023-01-22.md new file mode 100644 index 0000000..b716433 --- /dev/null +++ b/logs/2023-01-22.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January22) + +Time: January22 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.02 +- End: 2.05 + +## Completed Data +- $\approx$ 2248.01GB + +## Average Grad Norm +- 1.25 + +## Progress +- 93.06% + +## Comment + +The first day of the Chinese New Yearpeople are celebratingthe model is still learning. diff --git a/logs/2023-01-23.md b/logs/2023-01-23.md new file mode 100644 index 0000000..4264f4a --- /dev/null +++ b/logs/2023-01-23.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January23) + +Time: January23 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.05 + +## Completed Data +- $\approx$ 2252.42GB + +## Average Grad Norm +- 1.25 + +## Progress +- 93.16% + +## Comment + +Today I consulted CPM-Bee when organizing a family entertainment activity and it recommended playing mahjong. I adopted it and had a great time. diff --git a/logs/2023-01-24.md b/logs/2023-01-24.md new file mode 100644 index 0000000..f04131a --- /dev/null +++ b/logs/2023-01-24.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January24) + +Time: January24 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.08 + +## Completed Data +- $\approx$ 2256.85GB + +## Average Grad Norm +- 1.25 + +## Progress +- 93.26% + +## Comment + +When I asked CPM-Bee for its opinion on exploding fireworksit seriously gave me a hilarious answer🤣. diff --git a/logs/2023-01-25.md b/logs/2023-01-25.md new file mode 100644 index 0000000..a76f4b1 --- /dev/null +++ b/logs/2023-01-25.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January25) + +Time: January25 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.08 +- End: 2.04 + +## Completed Data +- $\approx$ 2261.27GB + +## Average Grad Norm +- 1.25 + +## Progress +- 93.36% + +## Comment + +If you want to know the content of *"Man Jiang Hong"*ask CPM-Bee😉. diff --git a/logs/2023-01-26.md b/logs/2023-01-26.md new file mode 100644 index 0000000..61e6f77 --- /dev/null +++ b/logs/2023-01-26.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January26) + +Time: January26 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.03 + +## Completed Data +- $\approx$ 2265.70GB + +## Average Grad Norm +- 1.26 + +## Progress +- 93.46% + +## Comment + +What's the *Wondering Earth* project? CPM-Bee can tell you a version. diff --git a/logs/2023-01-27.md b/logs/2023-01-27.md new file mode 100644 index 0000000..cf6b587 --- /dev/null +++ b/logs/2023-01-27.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January27) + +Time: January27 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.03 +- End: 2.05 + +## Completed Data +- $\approx$ 2270.12GB + +## Average Grad Norm +- 1.27 + +## Progress +- 93.56% + +## Comment + +This phase of the training has cost half a million Yuan. diff --git a/logs/2023-01-28.md b/logs/2023-01-28.md new file mode 100644 index 0000000..0ceaec0 --- /dev/null +++ b/logs/2023-01-28.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January28) + +Time: January28 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.06 + +## Completed Data +- $\approx$ 2274.54GB + +## Average Grad Norm +- 1.25 + +## Progress +- 93.66% + +## Comment + +The CPM-Bee training is in its final stretch. Keep it up! diff --git a/logs/2023-01-29.md b/logs/2023-01-29.md new file mode 100644 index 0000000..6837b26 --- /dev/null +++ b/logs/2023-01-29.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January29) + +Time: January29 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.06 + +## Completed Data +- $\approx$ 2278.95GB + +## Average Grad Norm +- 1.26 + +## Progress +- 93.76% + +## Comment + +My nephew asks me what the difference is between a laptop and a desktop computer. I ask CPM-Bee to tell him. diff --git a/logs/2023-01-30.md b/logs/2023-01-30.md new file mode 100644 index 0000000..f230cfa --- /dev/null +++ b/logs/2023-01-30.md @@ -0,0 +1,27 @@ + +# CPM-Live Training Log (January30) + +Time: January30 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.05 + +## Completed Data +- $\approx$ 2283.37GB + +## Average Grad Norm +- 1.26 + +## Progress +- 93.86% + +## Comment + +How to translate Classical Chinese to English? Let CPM-Bee try it out! + +Chinese: 围师必阙，穷寇勿迫 + +Translation: surround the enemy without closing in on himand do not press the defeated enemy diff --git a/logs/2023-01-31.md b/logs/2023-01-31.md new file mode 100644 index 0000000..ae9b967 --- /dev/null +++ b/logs/2023-01-31.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (January31) + +Time: January31 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.05 + +## Completed Data +- $\approx$ 2287.79GB + +## Average Grad Norm +- 1.26 + +## Progress +- 93.96% + +## Comment + +Today we let our model appreciate ancient poetry. diff --git a/logs/2023-02-01.md b/logs/2023-02-01.md new file mode 100644 index 0000000..4795840 --- /dev/null +++ b/logs/2023-02-01.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February1) + +Time: February1 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.07 + +## Completed Data +- $\approx$ 2292.21GB + +## Average Grad Norm +- 1.26 + +## Progress +- 94.06% + +## Comment + +How about the answer given by cpm-bee today? I think it is excellent! diff --git a/logs/2023-02-02.md b/logs/2023-02-02.md new file mode 100644 index 0000000..852d347 --- /dev/null +++ b/logs/2023-02-02.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February2) + +Time: February2 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.07 + +## Completed Data +- $\approx$ 2296.55GB + +## Average Grad Norm +- 1.26 + +## Progress +- 94.16% + +## Comment + +Want to know how to get to the airport? CPM-Bee can tell you. diff --git a/logs/2023-02-03.md b/logs/2023-02-03.md new file mode 100644 index 0000000..52f89a8 --- /dev/null +++ b/logs/2023-02-03.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February3) + +Time: February3 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.07 +- End: 2.04 + +## Completed Data +- $\approx$ 2300.82GB + +## Average Grad Norm +- 1.26 + +## Progress +- 94.26% + +## Comment + +Let our model show you some investment advice in the big model sector! diff --git a/logs/2023-02-04.md b/logs/2023-02-04.md new file mode 100644 index 0000000..783b64c --- /dev/null +++ b/logs/2023-02-04.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February4) + +Time: February4 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.02 + +## Completed Data +- $\approx$ 2305.11GB + +## Average Grad Norm +- 1.26 + +## Progress +- 94.36% + +## Comment + +When a big model has common senseit becomes even more fascinating. diff --git a/logs/2023-02-05.md b/logs/2023-02-05.md new file mode 100644 index 0000000..c5ddb2b --- /dev/null +++ b/logs/2023-02-05.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February5) + +Time: February5 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.02 +- End: 2.06 + +## Completed Data +- $\approx$ 2309.52GB + +## Average Grad Norm +- 1.26 + +## Progress +- 94.46% + +## Comment + +SurprisinglyCPM-Bee has the ability to generate several rounds of dialogue for you. diff --git a/logs/2023-02-06.md b/logs/2023-02-06.md new file mode 100644 index 0000000..67fd9da --- /dev/null +++ b/logs/2023-02-06.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February6) + +Time: February6 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.06 +- End: 2.05 + +## Completed Data +- $\approx$ 2313.91GB + +## Average Grad Norm +- 1.27 + +## Progress +- 94.56% + +## Comment + +CPM-Bee can generate a letter template for you. Check it! diff --git a/logs/2023-02-07.md b/logs/2023-02-07.md new file mode 100644 index 0000000..d6eb586 --- /dev/null +++ b/logs/2023-02-07.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February7) + +Time: February7 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.05 +- End: 2.04 + +## Completed Data +- $\approx$ 2318.29GB + +## Average Grad Norm +- 1.27 + +## Progress +- 94.66% + +## Comment + +Our model generates a few more rounds of conversation todayand I think it's quite insightful. What do you think? diff --git a/logs/2023-02-08.md b/logs/2023-02-08.md new file mode 100644 index 0000000..fa5f173 --- /dev/null +++ b/logs/2023-02-08.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February8) + +Time: February8 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.04 +- End: 2.38 + +## Completed Data +- $\approx$ 2323.23GB + +## Average Grad Norm +- 1.33 + +## Progress +- 94.76% + +## Comment + +The dialogue generated today is very vivid. Quite good😁. diff --git a/logs/2023-02-09.md b/logs/2023-02-09.md new file mode 100644 index 0000000..6f601f5 --- /dev/null +++ b/logs/2023-02-09.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February9) + +Time: February9 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.38 +- End: 2.33 + +## Completed Data +- $\approx$ 2334.51GB + +## Average Grad Norm +- 2.44 + +## Progress +- 94.86% + +## Comment + +As you can seethe training loss is higher than it wasbecause we are upgrading the model! Stay tuned! diff --git a/logs/2023-02-10.md b/logs/2023-02-10.md new file mode 100644 index 0000000..1cb541c --- /dev/null +++ b/logs/2023-02-10.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February10) + +Time: February10 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.33 +- End: 2.33 + +## Completed Data +- $\approx$ 2349.66GB + +## Average Grad Norm +- 2.30 + +## Progress +- 94.96% + +## Comment + +If you are looking for some material to help you write an articleCPM-Bee can help you. diff --git a/logs/2023-02-11.md b/logs/2023-02-11.md new file mode 100644 index 0000000..44375bc --- /dev/null +++ b/logs/2023-02-11.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February11) + +Time: February11 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.33 +- End: 2.33 + +## Completed Data +- $\approx$ 2364.83GB + +## Average Grad Norm +- 2.30 + +## Progress +- 95.06% + +## Comment + +I asked CPM-Bee to give me some advice today. The text it wrote is very well structuredand I hope it's right😄. diff --git a/logs/2023-02-12.md b/logs/2023-02-12.md new file mode 100644 index 0000000..eb6890a --- /dev/null +++ b/logs/2023-02-12.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February12) + +Time: February12 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.33 +- End: 2.20 + +## Completed Data +- $\approx$ 2376.99GB + +## Average Grad Norm +- 2.33 + +## Progress +- 95.16% + +## Comment + +There is a large drop in the training loss as we have added some new data in addition to the plain text😉. diff --git a/logs/2023-02-13.md b/logs/2023-02-13.md new file mode 100644 index 0000000..45cf4fc --- /dev/null +++ b/logs/2023-02-13.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February13) + +Time: February13 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.20 +- End: 2.17 + +## Completed Data +- $\approx$ 2392.13GB + +## Average Grad Norm +- 2.32 + +## Progress +- 95.26% + +## Comment + +It's fun to watch a model learn a skill from scratch like a child. If it eventually becomes an elite in that fieldwe will have a great sense of achievement. diff --git a/logs/2023-02-14.md b/logs/2023-02-14.md new file mode 100644 index 0000000..454cbc1 --- /dev/null +++ b/logs/2023-02-14.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February14) + +Time: February14 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.17 +- End: 2.17 + +## Completed Data +- $\approx$ 2407.23GB + +## Average Grad Norm +- 2.32 + +## Progress +- 95.36% + +## Comment + +Whether or not to have a Valentine's Day is a question worth ponderingand CPM-Bee gives a realistic answer. diff --git a/logs/2023-02-15.md b/logs/2023-02-15.md new file mode 100644 index 0000000..0410e74 --- /dev/null +++ b/logs/2023-02-15.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February15) + +Time: February15 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.17 +- End: 2.18 + +## Completed Data +- $\approx$ 2422.30GB + +## Average Grad Norm +- 2.33 + +## Progress +- 95.46% + +## Comment + +If you read CPM-Bee's answer today without the relevant background knowledgeyou will probably choose to trust it. diff --git a/logs/2023-02-16.md b/logs/2023-02-16.md new file mode 100644 index 0000000..09790ca --- /dev/null +++ b/logs/2023-02-16.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (February16) + +Time: February16 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.18 +- End: 2.16 + +## Completed Data +- $\approx$ 2437.43GB + +## Average Grad Norm +- 2.35 + +## Progress +- 95.56% + +## Comment + +Can you find any flaws in the text generated by the model today without searching for anything? + diff --git a/logs/2023-02-17.md b/logs/2023-02-17.md new file mode 100644 index 0000000..10ae8a1 --- /dev/null +++ b/logs/2023-02-17.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February17) + +Time: February17 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.16 +- End: 2.15 + +## Completed Data +- $\approx$ 2452.54GB + +## Average Grad Norm +- 2.38 + +## Progress +- 95.66% + +## Comment + +The training loss was reduced from 2.4 to 2.1and the model performance gained some improvement. diff --git a/logs/2023-02-18.md b/logs/2023-02-18.md new file mode 100644 index 0000000..fb8d98c --- /dev/null +++ b/logs/2023-02-18.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February18) + +Time: February18 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.15 +- End: 2.15 + +## Completed Data +- $\approx$ 2467.66GB + +## Average Grad Norm +- 2.41 + +## Progress +- 95.76% + +## Comment + +You could try to get CPM-Bee to give you some adviceand they seem to be reliable overall. diff --git a/logs/2023-02-19.md b/logs/2023-02-19.md new file mode 100644 index 0000000..d514669 --- /dev/null +++ b/logs/2023-02-19.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February19) + +Time: February19 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.15 +- End: 2.14 + +## Completed Data +- $\approx$ 2482.74GB + +## Average Grad Norm +- 2.41 + +## Progress +- 95.86% + +## Comment + +I asked the model a question about the three-body problemand its answer was about civilization and the universe🤔. It seems the model is very fond of that novel. diff --git a/logs/2023-02-20.md b/logs/2023-02-20.md new file mode 100644 index 0000000..b3b8340 --- /dev/null +++ b/logs/2023-02-20.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February20) + +Time: February20 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.14 +- End: 2.15 + +## Completed Data +- $\approx$ 2497.91GB + +## Average Grad Norm +- 2.42 + +## Progress +- 95.96% + +## Comment + +CPM-Bee is quite good at memorizing ancient poetries. diff --git a/logs/2023-02-21.md b/logs/2023-02-21.md new file mode 100644 index 0000000..d372223 --- /dev/null +++ b/logs/2023-02-21.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February21) + +Time: February21 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.15 +- End: 2.14 + +## Completed Data +- $\approx$ 2513.00GB + +## Average Grad Norm +- 2.44 + +## Progress +- 96.06% + +## Comment + +Today's answer: "I'll take them all"😆. diff --git a/logs/2023-02-22.md b/logs/2023-02-22.md new file mode 100644 index 0000000..e638499 --- /dev/null +++ b/logs/2023-02-22.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February22) + +Time: February22 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.14 +- End: 2.15 + +## Completed Data +- $\approx$ 2528.11GB + +## Average Grad Norm +- 2.45 + +## Progress +- 96.16% + +## Comment + +What do you think is the correct answer to today's quiz🤔? diff --git a/logs/2023-02-23.md b/logs/2023-02-23.md new file mode 100644 index 0000000..c3c9cd9 --- /dev/null +++ b/logs/2023-02-23.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February23) + +Time: February23 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.15 +- End: 2.12 + +## Completed Data +- $\approx$ 2543.21GB + +## Average Grad Norm +- 2.47 + +## Progress +- 96.26% + +## Comment + +Today our model wrote a qualified essay for primary school students. diff --git a/logs/2023-02-24.md b/logs/2023-02-24.md new file mode 100644 index 0000000..ec871a4 --- /dev/null +++ b/logs/2023-02-24.md @@ -0,0 +1,24 @@ + +# CPM-Live Training Log (February24) + +Time: February24 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.12 + +## Completed Data +- $\approx$ 2558.32GB + +## Average Grad Norm +- 2.55 + +## Progress +- 96.36% + +## Comment + +After reading today's answerhave you seen the progress of the model? + diff --git a/logs/2023-02-25.md b/logs/2023-02-25.md new file mode 100644 index 0000000..cb74ecc --- /dev/null +++ b/logs/2023-02-25.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February25) + +Time: February25 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.12 + +## Completed Data +- $\approx$ 2573.39GB + +## Average Grad Norm +- 2.54 + +## Progress +- 96.46% + +## Comment + +In today's testthe model recommends several reasonable ways to travelbut there are a few problems with minor details. diff --git a/logs/2023-02-26.md b/logs/2023-02-26.md new file mode 100644 index 0000000..3e9c7e1 --- /dev/null +++ b/logs/2023-02-26.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February26) + +Time: February26 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.12 + +## Completed Data +- $\approx$ 2588.50GB + +## Average Grad Norm +- 2.55 + +## Progress +- 96.56% + +## Comment + +Want CPM-Bee to answer two questions for you at once? No problem! diff --git a/logs/2023-02-27.md b/logs/2023-02-27.md new file mode 100644 index 0000000..6b7cb5a --- /dev/null +++ b/logs/2023-02-27.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February27) + +Time: February27 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.12 + +## Completed Data +- $\approx$ 2603.63GB + +## Average Grad Norm +- 2.58 + +## Progress +- 96.66% + +## Comment + +CPM-Bee is making rapid progress towards a goal of becoming a competent AI writing assistant. diff --git a/logs/2023-02-28.md b/logs/2023-02-28.md new file mode 100644 index 0000000..cb50aaa --- /dev/null +++ b/logs/2023-02-28.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (February28) + +Time: February28 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.12 + +## Completed Data +- $\approx$ 2618.78GB + +## Average Grad Norm +- 2.59 + +## Progress +- 96.76% + +## Comment + +Goodbye February! The training of CPM-Bee is in its final stretch. diff --git a/logs/2023-03-01.md b/logs/2023-03-01.md new file mode 100644 index 0000000..de022e7 --- /dev/null +++ b/logs/2023-03-01.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March1) + +Time: March1 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.12 +- End: 2.09 + +## Completed Data +- $\approx$ 2633.83GB + +## Average Grad Norm +- 2.60 + +## Progress +- 96.86% + +## Comment + +It seems that CPM-Bee has some knowledge of the "Dream of the Red Chamber". diff --git a/logs/2023-03-02.md b/logs/2023-03-02.md new file mode 100644 index 0000000..0d7eb79 --- /dev/null +++ b/logs/2023-03-02.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March2) + +Time: March2 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.09 +- End: 2.11 + +## Completed Data +- $\approx$ 2648.91GB + +## Average Grad Norm +- 2.61 + +## Progress +- 96.96% + +## Comment + +March is a vibrant time of yearwith endless possibilities for everything😉. diff --git a/logs/2023-03-03.md b/logs/2023-03-03.md new file mode 100644 index 0000000..d69966c --- /dev/null +++ b/logs/2023-03-03.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March3) + +Time: March3 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 2.11 +- End: 1.86 + +## Completed Data +- $\approx$ 2664.01GB + +## Average Grad Norm +- 2.63 + +## Progress +- 97.06% + +## Comment + +The loss is substantially reduced! Probably because the newly added training data is easier to learn. diff --git a/logs/2023-03-04.md b/logs/2023-03-04.md new file mode 100644 index 0000000..64f3684 --- /dev/null +++ b/logs/2023-03-04.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March4) + +Time: March4 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.86 +- End: 1.84 + +## Completed Data +- $\approx$ 2678.92GB + +## Average Grad Norm +- 2.60 + +## Progress +- 97.16% + +## Comment + +It always brings a smile to my face when I see the model acquiring another new ability. diff --git a/logs/2023-03-05.md b/logs/2023-03-05.md new file mode 100644 index 0000000..8bd9080 --- /dev/null +++ b/logs/2023-03-05.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March5) + +Time: March5 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.84 +- End: 1.82 + +## Completed Data +- $\approx$ 2693.74GB + +## Average Grad Norm +- 2.50 + +## Progress +- 97.26% + +## Comment + +The first week of March ended on a busy note. diff --git a/logs/2023-03-06.md b/logs/2023-03-06.md new file mode 100644 index 0000000..5f439e8 --- /dev/null +++ b/logs/2023-03-06.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March6) + +Time: March6 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.82 +- End: 1.79 + +## Completed Data +- $\approx$ 2708.53GB + +## Average Grad Norm +- 2.50 + +## Progress +- 97.36% + +## Comment + +Our model can tell you what can help you fall asleep. diff --git a/logs/2023-03-07.md b/logs/2023-03-07.md new file mode 100644 index 0000000..c88c8a3 --- /dev/null +++ b/logs/2023-03-07.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March7) + +Time: March7 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.79 +- End: 1.79 + +## Completed Data +- $\approx$ 2723.41GB + +## Average Grad Norm +- 2.49 + +## Progress +- 97.46% + +## Comment + +Give the model some specific requirements and it can help you write your article. diff --git a/logs/2023-03-08.md b/logs/2023-03-08.md new file mode 100644 index 0000000..cd0f545 --- /dev/null +++ b/logs/2023-03-08.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March8) + +Time: March8 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.79 +- End: 1.79 + +## Completed Data +- $\approx$ 2738.29GB + +## Average Grad Norm +- 2.51 + +## Progress +- 97.56% + +## Comment + +Happy IWD to every woman! diff --git a/logs/2023-03-09.md b/logs/2023-03-09.md new file mode 100644 index 0000000..3279526 --- /dev/null +++ b/logs/2023-03-09.md @@ -0,0 +1,23 @@ + +# CPM-Live Training Log (March9) + +Time: March9 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.79 +- End: 1.78 + +## Completed Data +- $\approx$ 2753.17GB + +## Average Grad Norm +- 2.46 + +## Progress +- 97.66% + +## Comment + +It would be very cool if a model could generate more than just plain text. diff --git a/logs/2023-03-10.md b/logs/2023-03-10.md new file mode 100644 index 0000000..03a2d1f --- /dev/null +++ b/logs/2023-03-10.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March10) + +Time: March10 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.78 +- End: 1.78 + +## Completed Data +- $\approx$ 2768.05GB + +## Average Grad Norm +- 2.48 + +## Progress +- 97.76% + +## Comment + +The dressing advice given by the model today is very well thought out. diff --git a/logs/2023-03-11.md b/logs/2023-03-11.md new file mode 100644 index 0000000..4baceda --- /dev/null +++ b/logs/2023-03-11.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March11) + +Time: March11 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.78 +- End: 1.79 + +## Completed Data +- $\approx$ 2782.78GB + +## Average Grad Norm +- 2.56 + +## Progress +- 97.86% + +## Comment + +It looks like the loss has stabilized again over the last few days. diff --git a/logs/2023-03-12.md b/logs/2023-03-12.md new file mode 100644 index 0000000..afed2a6 --- /dev/null +++ b/logs/2023-03-12.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March12) + +Time: March12 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.79 +- End: 1.78 + +## Completed Data +- $\approx$ 2797.49GB + +## Average Grad Norm +- 2.58 + +## Progress +- 97.96% + +## Comment + +The weekend is over. Let's see what happens next week! diff --git a/logs/2023-03-13.md b/logs/2023-03-13.md new file mode 100644 index 0000000..20592f9 --- /dev/null +++ b/logs/2023-03-13.md @@ -0,0 +1,24 @@ +# CPM-Live Training Log (March13) + +Time: March13 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.78 +- End: 1.77 + +## Completed Data +- $\approx$ 2812.35GB + +## Average Grad Norm +- 2.60 + +## Progress +- 98.06% + +## Comment + +Our model has a certain understanding of the parameters of TVand this advertisement is written reasonably. + + diff --git a/logs/2023-03-14.md b/logs/2023-03-14.md new file mode 100644 index 0000000..164aecd --- /dev/null +++ b/logs/2023-03-14.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March14) + +Time: March14 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.77 +- End: 1.77 + +## Completed Data +- $\approx$ 2827.24GB + +## Average Grad Norm +- 2.61 + +## Progress +- 98.16% + +## Comment + +The evolution of models seems never-ending... diff --git a/logs/2023-03-15.md b/logs/2023-03-15.md new file mode 100644 index 0000000..c580536 --- /dev/null +++ b/logs/2023-03-15.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March15) + +Time: March15 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.77 +- End: 1.76 + +## Completed Data +- $\approx$ 2842.09GB + +## Average Grad Norm +- 2.64 + +## Progress +- 98.26% + +## Comment + +Sometimes I wonder how many days a day in the model world equals in the human world. diff --git a/logs/2023-03-16.md b/logs/2023-03-16.md new file mode 100644 index 0000000..0c057de --- /dev/null +++ b/logs/2023-03-16.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March16) + +Time: March16 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.76 +- End: 1.77 + +## Completed Data +- $\approx$ 2856.94GB + +## Average Grad Norm +- 2.66 + +## Progress +- 98.36% + +## Comment + +Who do you think is betterNewton or Einstein? Check the model's answer. diff --git a/logs/2023-03-17.md b/logs/2023-03-17.md new file mode 100644 index 0000000..d84165c --- /dev/null +++ b/logs/2023-03-17.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March17) + +Time: March17 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.76 +- End: 1.73 + +## Completed Data +- $\approx$ 2871.81GB + +## Average Grad Norm +- 2.68 + +## Progress +- 98.46% + +## Comment + +What a lively week! diff --git a/logs/2023-03-18.md b/logs/2023-03-18.md new file mode 100644 index 0000000..6d3f9d8 --- /dev/null +++ b/logs/2023-03-18.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March18) + +Time: March18 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.73 +- End: 1.75 + +## Completed Data +- $\approx$ 2886.72GB + +## Average Grad Norm +- 2.70 + +## Progress +- 98.56% + +## Comment + +Todaywe test the performance of our model with a what if question. diff --git a/logs/2023-03-19.md b/logs/2023-03-19.md new file mode 100644 index 0000000..ff5bd05 --- /dev/null +++ b/logs/2023-03-19.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March19) + +Time: March19 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.75 +- End: 1.76 + +## Completed Data +- $\approx$ 2901.50GB + +## Average Grad Norm +- 2.72 + +## Progress +- 98.66% + +## Comment + +The weekend is over and next week is another one to look forward to. diff --git a/logs/2023-03-20.md b/logs/2023-03-20.md new file mode 100644 index 0000000..8f87fcc --- /dev/null +++ b/logs/2023-03-20.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March20) + +Time: March20 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.76 +- End: 1.74 + +## Completed Data +- $\approx$ 2916.33GB + +## Average Grad Norm +- 2.74 + +## Progress +- 98.76% + +## Comment + +We are lucky because we are witnessing the impossible becoming a reality. diff --git a/logs/2023-03-21.md b/logs/2023-03-21.md new file mode 100644 index 0000000..1a27395 --- /dev/null +++ b/logs/2023-03-21.md @@ -0,0 +1,23 @@ +# CPM-Live Training Log (March21) + +Time: March21 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.74 +- End: 1.73 + +## Completed Data +- $\approx$ 2925.37GB + +## Average Grad Norm +- 2.20 + +## Progress +- 98.86% + +## Comment + +Maybe the next step we have to do is to make the purchase information recommended by the model to us more accurate and convincing. + diff --git a/logs/2023-03-22.md b/logs/2023-03-22.md new file mode 100644 index 0000000..aef4edd --- /dev/null +++ b/logs/2023-03-22.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March22) + +Time: March22 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.73 +- End: 1.74 + +## Completed Data +- $\approx$ 2932.65GB + +## Average Grad Norm +- 2.06 + +## Progress +- 98.96% + +## Comment + +In the pastwe asked what models can dobut now we ask what models cannot do. diff --git a/logs/2023-03-23.md b/logs/2023-03-23.md new file mode 100644 index 0000000..38ab7e2 --- /dev/null +++ b/logs/2023-03-23.md @@ -0,0 +1,23 @@ +# CPM-Live Training Log (March23) + +Time: March23 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.74 +- End: 1.80 + +## Completed Data +- $\approx$ 2939.92GB + +## Average Grad Norm +- 2.07 + +## Progress +- 99.06% + +## Comment + +Sometimes we often sigh at the rapid development of technology. Perhaps AGI is coming soon? + diff --git a/logs/2023-03-24.md b/logs/2023-03-24.md new file mode 100644 index 0000000..59669e3 --- /dev/null +++ b/logs/2023-03-24.md @@ -0,0 +1,22 @@ +# CPM-Live Training Log (March24) + +Time: March24 2023 19:00 + +Recorder: @zh-zheng + +## Loss +- Begin: 1.80 +- End: 1.77 + +## Completed Data +- $\approx$ 2946.88GB + +## Average Grad Norm +- 2.09 + +## Progress +- 99.16% + +## Comment + +In the pastit might have been hard to imagine that we would one day judge a paragraph to have been written by a model because it was so well written. diff --git "a/plans/CPM-Ant\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" "b/plans/CPM-Ant\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" new file mode 100644 index 0000000..4579841 --- /dev/null +++ "b/plans/CPM-Ant\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" @@ -0,0 +1,102 @@ +# CPM-Ant训练计划书 + +综合考虑数据和算力规模，CPM-Live将以10B模型训练为起点，我们将其命名为CPM-Ant。CPM-Ant 10B大模型训练将于2022年5月29日启动，预计整个训练周期为5个月。 + +## 一、模型架构 + +CPM-Ant的模型架构与主要设置如下： + +

+ +

+ +- **基于提示模板的多段式框架设计（prompt-based multi-segment framework）**：提示模板（prompt）用以实现模型在理解、生成、摘要等功能之间的快速切换，也易于添加新的功能（学习新的prompt）。 + + 文本段（segment）提供基础的文本编码能力，通过段表示（segment embedding）来影响模型编码模式，复杂的文本编码模式可以拆解成若干基础段的组合，例如编码-解码框架可以拆解成一个编码段+一个解码段的组合。对于每一个基础段，段内采用相对位置编码。 + + 基于提示模板和文本段的组合、拼接，结构简单且易于实现增加、修改模块，进行持续学习和功能更新。 + +- **共享embedding**：CPM-Ant输入embedding及输出embedding会共享参数，这点与BERT、GPT、T5一致，与T5-1.1、mT5不一致。我们的实验表明共享输入输出的embedding参数会极大增强训练稳定程度，而不共享embedding参数易于导致训练过程出现NaN。 +- **无bias**：我们的模型中，各类线性变换及layer norm均不设置bias。一方面源于不设置bias的模型训练稳定性会更强，另一方面也是不设置bias的模型在计算速度及显存消耗上要更占优。 +- **动态词表**：对于词表，初始阶段我们将提供大小为30000的中文词表，在后续训练过程中会结合新数据情况进行动态变动。 + +## 二、模型训练 + +模型训练过程中重点进行性能监控与问题处理，收集功能倡议与社区讨论意见并进行模型修改与反馈。主要计划开展相关工作如下： +- 实时：展示模型训练指标曲线 +- 每日：发布单日模型训练日志 +- 每周：集中处理反馈来自社区的讨论与意见 +- 不定期：发布模型训练的中间checkpoint并提供用户下载 + +模型训练完成后，CPM-Ant将对社区模型提议进行整理与集成，并将模型进行相应尺度的压缩，准备总结报告并陆续公布相关模型。此外，对于未能采纳的部分社区提议，我们将考虑引入到下一代模型训练中，并开始进行下一代模型的训练筹备。 + +## 三、数据分析 + +CPM-Ant采用1TB原始数据，清洗后得到200GB高质量数据，数据详情如下表所示： + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

数据来源	占比	文档平均长度（字）	句子平均长度（字）	平均PPL（mGPT）*
书籍	33.02%	248495.71	32.93	273.777
网页	21.52%	665.83	28.304	141.53
小说	20.76%	62317.79	30.839	69.98
杂志	11.95%	2534.16	39.06	83.22
学术	4.77%	173.8	58.044	39.04
百科	2.25%	1081.33	32.466	2072.53
新闻	1.79%	717.87	43.717	56.85
其它	3.95%	852.36	37.68	395.26
* 使用mGPT计算得到句子平均PPL

diff --git "a/plans/CPM-Bee\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" "b/plans/CPM-Bee\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" new file mode 100644 index 0000000..b393a53 --- /dev/null +++ "b/plans/CPM-Bee\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" @@ -0,0 +1,70 @@ +# CPM-Bee训练计划书 + +## 一、模型架构 + +CPM-Bee的结构整体与CPM-Ant保持一致，可参考[模型细节](https://www.openbmb.org/community/blogs/blogpage?id=98afef2ce45f4fe9a4bc15a66d7ccb92)。 + +相较于CPM-Ant，CPM-Bee增加了如下特性： + +- **增加结构化数据处理（Structured Data Processing）**：CPM-Bee将支持结构化数据（例如Json文件）的处理，以便更好地支持下游应用； +- **增加预训练硬提示（Pre-trained Hard Prompts）**：为了让CPM-Bee更好地掌握不同任务所对应的结构化数据形式，我们通过硬提示来组织结构化数据，并在预训练中充分学习这些硬提示； +- **增加词表扩展功能（Vocabulary Expansion）**：CPM-Bee是多语言模型，因此我们加入了词表扩展功能，以支持更多语言的持续学习，我们也引入了对不同语言同时进行处理的分词器； +- **增加数量自适应[UNK]和[MASK]**：传统的预训练模型采用统一的[UNK]和[MASK]标签（或者类似T5预训练固定数量的[MASK]标签），难以对[UNK]及[MASK]进行精细处理。这里我们将引入数量自适应的[UNK]和[MASK]机制，即模型能自动识别输入数据中的[UNK]和[MASK]，编码为[MASK#1]、[MASK#2]…以及[UNK#1]、[UNK#2]…，并对每个[UNK]和[MASK]单独进行特征处理。该机制十分有利于处理一些具有复杂上下文的网络文本，例如带有emoji的文本。 + +出于性能影响以及使用便利性的角度，CPM-Bee在CPM-Ant的基础上也引入了一些简化： + +- **删除段向量（Segment Embeddings）**：基于CPM-Ant的大量实验表明，在CPM-Ant的多段机制及相对位置编码（Multi-segment Mechanism & Relative Position Bias）中，结合段信息进行相对位置编码已经可以有效区分输入序列的分段特性，无需再采用类似BERT中的段向量来指导模型。此外，段向量往往面临泛化性的问题，即大量的预训练数据只存在1~2个分段，训练出的模型难以处理拥有2个分段以上的输入数据。因此，我们在CPM-Bee中去除了段向量的设计。 + + +## 二、模型特性 + +CPM-Bee具有如下模型特性： + +### （一）多语言融合 + +在CPM-Ant中，我们着重构建以中文为核心的大模型。在CPM-Bee中，我们将逐渐加入英文以及其他语种的数据（包括各语言独立数据及跨语言平行数据），最终形成以中文为核心，多语种兼顾的大规模预训练语言模型。 + +### （二）复杂结构处理 + +已有的预训练语言模型主要立足于利用非结构化文本进行训练，因而对于半结构化及结构化数据的处理能力较弱。在CPM-Bee中，我们会加入各类半结构化及结构化数据的处理功能，以更好地支持网页、代码等结构化复杂文本的处理能力。下面展示了一条结构化训练数据： + +```on +{ + "document": "今天天气是真的，我们去了，玩得非常。", + "": { + "": "好", + "": "颐和园", + "": "开心" + } +} +``` + +### （三）任务模式增强 + +在CPM-Bee中，我们会在预训练过程中引入各类常见任务模式的数据增强，包括生成、问答、摘要、翻译等，支持CPM-Bee在各类文本处理任务上开箱即用，提升结合少量样本的参数高效微调的性能。 + + +## 三、训练细节 + +### （一）训练数据 +CPM-Ant采用1TB原始数据，清洗后得到200GB高质量中文数据，这部分数据会被继续在CPM-Bee中使用。此外，CPM-Bee引入了400GB的多语言数据来进行预训练。基于上述600GB数据，我们设计了多样化的数据增广算法，使CPM-Bee具备多种语言能力，这些数据增广算法我们会在后续的技术报告中详细展开。 + +### （二）训练工具 +在CPM-Ant中，我们已经验证了OpenBMB的大模型全流程计算框架的可靠性，因此CPM-Bee也将基于下列工具进行训练、微调、压缩与推理： +- 训练工具 [[BMTrain](https://github.com/OpenBMB/BMTrain)] +- 微调工具 [[OpenDelta](https://github.com/thunlp/OpenDelta)] +- 压缩工具 [[BMCook](https://github.com/OpenBMB/BMCook)] +- 推理工具 [[BMInf](https://github.com/OpenBMB/BMInf)] + + +### （三）训练计划 + +CPM-Bee 10B大模型训练将于2022年10月13日启动，并将每月发布一版模型。 + +模型训练过程中重点进行性能监控与问题处理，收集功能倡议与社区讨论意见并进行模型修改与反馈。主要计划开展相关工作如下： +- 实时：展示模型训练指标曲线 +- 每日：发布单日模型训练日志 +- 每周：集中处理反馈来自社区的讨论与意见 +- 定期：发布模型训练的中间checkpoint并提供用户下载 + +模型训练完成后，CPM-Bee将对社区模型提议进行整理与集成，并将模型进行相应尺度的压缩，准备总结报告并陆续公布相关模型。此外，对于未能采纳的部分社区提议，我们将考虑引入到下一代模型训练中，并开始进行下一代模型的训练筹备。 diff --git "a/plans/CPM-Live\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" "b/plans/CPM-Live\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" index e663734..6b70568 100644 --- "a/plans/CPM-Live\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" +++ "b/plans/CPM-Live\350\256\255\347\273\203\350\256\241\345\210\222\344\271\246.md" @@ -1,5 +1,17 @@ # CPM-Live训练计划书 +## 目录 + +- [一、计划总览](#一计划总览) +- [二、CPM-Live模型](#二cpm-live模型) +- [三、开源共建](#三开源共建) +- [四、使用协议](#四使用协议) +- [五、风险提示](#五风险提示) +- [六、支持方案](#六支持方案) +- [七、CPM-Ant训练计划](#七cpm-ant训练计划) +- [八、CPM-Bee训练计划](#八cpm-bee训练计划) + + ## 一、计划总览 CPM-Live直播训练是以CPM系列模型为架构基础的开源大模型直播训练，训练过程将采用持续学习方式并实时直播训练进展。和现有的开源大模型以及相关直播相比，CPM-Live具有以下特点： @@ -10,26 +22,10 @@ CPM-Live直播训练是以CPM系列模型为架构基础的开源大模型直播 - **高效计算**。CPM-Live训练、压缩、推理过程将基于OpenBMB开源社区系列工具包。通过大模型训练“发动机”**BMTrain**，我们可以在小规模的集群中训练百亿以上规模的超大模型，这大幅降低了模型训练的成本，也使我们的训练更加低碳和高效。使用大模型“瘦身”工具库**BMCook**和高效推理工具包**BMInf**，普通用户可以在消费级显卡上运行大模型，从而更加便捷地体验大模型的魅力。 ## 二、CPM-Live模型 -### （一）模型架构 -CPM-Live的模型架构与主要设置如下： - -

- -

- -- **基于提示模板的多段式框架设计（prompt-based multi-segment framework）**：提示模板（prompt）用以实现模型在理解、生成、摘要等功能之间的快速切换，也易于添加新的功能（学习新的prompt）。 - - 文本段（segment）提供基础的文本编码能力，通过段表示（segment embedding）来影响模型编码模式，复杂的文本编码模式可以拆解成若干基础段的组合，例如编码-解码框架可以拆解成一个编码段+一个解码段的组合。对于每一个基础段，段内采用相对位置编码。 - - 基于提示模板和文本段的组合、拼接，结构简单且易于实现增加、修改模块，进行持续学习和功能更新。 - -- **共享embedding**：CPM-Live输入embedding及输出embedding会共享参数，这点与BERT、GPT、T5一致，与T5-1.1、mT5不一致。我们的实验表明共享输入输出的embedding参数会极大增强训练稳定程度，而不共享embedding参数易于导致训练过程出现NaN。 -- **无bias**：我们的模型中，各类线性变换及layer norm均不设置bias。一方面源于不设置bias的模型训练稳定性会更强，另一方面也是不设置bias的模型在计算速度及显存消耗上要更占优。 -- **动态词表**：对于词表，初始阶段我们将提供大小为30000的中文词表，在后续训练过程中会结合新数据情况进行动态变动。 -### （二）参数规模 +### （一）参数规模 -在训练中我们将通过多阶段的持续学习来实现参数不断增长。为了兼顾实际使用过程中的计算效率，我们会提供各种常用尺寸的大模型压缩版本，具体训练规模参数和开源参数规模如下表所示。 +在CPM-Live的训练中我们将通过多阶段的持续学习来实现参数不断增长。为了兼顾实际使用过程中的计算效率，我们会提供各种常用尺寸的大模型压缩版本，具体训练规模参数和开源参数规模如下表所示。 @@ -74,7 +70,7 @@ CPM-Live的模型架构与主要设置如下： - + @@ -111,7 +107,6 @@ CPM-Live的模型架构与主要设置如下： - :bug: : 发布模型尺寸 - :ant: : 目标模型尺寸 -我们将以训练10B参数的CPM-Ant模型为起点，提供中英文支持和结构化输入输出功能，后续训练过程中将陆续增大模型规模和数据量，并提供对更多语言、更多功能的支持。 训练工具详见： - [BMTrain](https://github.com/OpenBMB/BMTrain) @@ -120,17 +115,17 @@ CPM-Live的模型架构与主要设置如下： 压缩工具详见： - [BMCook](https://github.com/OpenBMB/BMCook) -### （三）模型特性 +### （二）模型特性 CPM-Live具有如下模型特性： #### 规模持续增大 -对于CPM-Live，我们不谋求一次性塑成一个拥有巨大参数量的模型，而是从百亿参数规模开始，立足模型知识继承、参数动态增长等技术，持续推动预训练模型能力提升与参数增长，同时有效避免灾难性遗忘问题对持续学习过程带来的负作用。模型参数是否增长取决于模型是否达到了学习能力瓶颈。相关内容可参考论文 [Knowledge inheritance for pre-trained language models](https://scholar.google.co.jp/citations?view_op=view_citation&hl=zh-CN&user=rz4rOSMAAAAJ&sortby=pubdate&citation_for_view=rz4rOSMAAAAJ:e5wmG9Sq2KIC)。 +对于CPM-Live，我们不谋求一次性塑成一个拥有巨大参数量的模型，而是从百亿参数规模开始，立足模型知识继承、参数动态增长等技术，持续推动预训练模型能力提升与参数增长，同时有效避免灾难性遗忘问题对持续学习过程带来的负作用。模型参数是否增长取决于模型是否达到了学习能力瓶颈。相关内容可参考论文 [Knowledge inheritance for pre-trained language models](https://arxiv.org/pdf/2105.13880.pdf)。 #### 能力持续扩展 -我们会在CPM-Live的预训练过程中引入通用性的任务提示，并利用自监督的方式同步预训练这些任务提示，促进大模型能力到下游任务的适配。在最初的阶段中，我们将预训练文本续写、改写、摘要、理解等基础的通用提示。后续会结合具体需求在预训练中添加新功能的任务提示。有关预训练提示微调方法可参考论文 [Ppt: Pre-trained prompt tuning for few-shot learning](https://scholar.google.co.jp/citations?view_op=view_citation&hl=zh-CN&user=rz4rOSMAAAAJ&cstart=20&pagesize=80&citation_for_view=rz4rOSMAAAAJ:4JMBOYKVnBMC)。 +我们会在CPM-Live的预训练过程中引入通用性的任务提示，并利用自监督的方式同步预训练这些任务提示，促进大模型能力到下游任务的适配。在最初的阶段中，我们将预训练文本续写、改写、摘要、理解等基础的通用提示。后续会结合具体需求在预训练中添加新功能的任务提示。有关预训练提示微调方法可参考论文 [PPT: Pre-trained prompt tuning for few-shot learning](https://arxiv.org/pdf/2109.04332.pdf)。 #### 数据持续增多 @@ -149,7 +144,7 @@ CPM-Live具有如下模型特性： 已有的预训练语言模型主要立足于利用非结构化文本进行训练，因而对于半结构化及结构化数据的处理能力较弱。CPM-Live在持续学习过程中，会逐步加入各类半结构化及结构化数据的处理功能，以更好地支持网页、代码等结构化复杂文本的处理能力。 -### （四）数据处理 +### （三）数据处理 CPM-Live对于原始数据进行了一系列的处理，用于提升数据质量、减少信息冗余、增强模型安全性，主要包含了如下三个阶段： - **数据过滤**：数据过滤阶段使用了模板、规则、PPL等多种过滤方案，筛除了原始语料中的低质量数据和敏感信息，用于提升模型安全性和训练阶段的稳定性。 @@ -211,85 +206,8 @@ CPM-Live系列发布模型适用协议为“[通用模型许可协议-来源说 ## 七、CPM-Ant训练计划 -综合考虑数据和算力规模，CPM-Live将以10B模型训练为起点，我们将其命名为CPM-Ant。CPM-Ant 10B大模型训练将于2022年5月29日启动，预计整个训练周期为5个月。 +详见[这里](CPM-Ant%E8%AE%AD%E7%BB%83%E8%AE%A1%E5%88%92%E4%B9%A6.md)。 -### （一）模型训练 +## 八、CPM-Bee训练计划 -模型训练过程中重点进行性能监控与问题处理，收集功能倡议与社区讨论意见并进行模型修改与反馈。主要计划开展相关工作如下： -- 实时：展示模型训练指标曲线 -- 每日：发布单日模型训练日志 -- 每周：集中处理反馈来自社区的讨论与意见 -- 不定期：发布模型训练的中间checkpoint并提供用户下载 - -模型训练完成后，CPM-Ant将对社区模型提议进行整理与集成，并将模型进行相应尺度的压缩，准备总结报告并陆续公布相关模型。此外，对于未能采纳的部分社区提议，我们将考虑引入到下一代模型训练中，并开始进行下一代模型的训练筹备。 - -### （二）数据分析 - -CPM-Ant采用1TB原始数据，清洗后得到200GB高质量数据，数据详情如下表所示： - -

4096

:bug: 7B

:ant: 10B CPM-Ant

:ant: 10B CPM-Ant/Bee

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

数据来源	占比	文档平均长度（字）	句子平均长度（字）	平均PPL（mGPT）*
书籍	33.02%	248495.71	32.93	273.777
网页	21.52%	665.83	28.304	141.53
小说	20.76%	62317.79	30.839	69.98
杂志	11.95%	2534.16	39.06	83.22
学术	4.77%	173.8	58.044	39.04
百科	2.25%	1081.33	32.466	2072.53
新闻	1.79%	717.87	43.717	56.85
其它	3.95%	852.36	37.68	395.26
* 使用mGPT计算得到句子平均PPL

\ No newline at end of file +详见[这里](./CPM-Bee%E8%AE%AD%E7%BB%83%E8%AE%A1%E5%88%92%E4%B9%A6.md)