metrological magic

めも - memo

きらびやかに、美しく、痛烈に。

Keras/Tensorflow : CIFAR-10のMobileNetV2-likeなアーキテクチャを作った.

0. 前提

MobileNetV2は、2018/04にGoogleのMark Sandlerらによって発表された論文:"MobileNetV2: Inverted Residuals and Linear Bottlenecks"にて導入された、ニューラルネットワークモデルです。その名の通り、モバイル機器向けの高速なフォワード処理に適しており、精度を保持した上でパラメータの削減に成功しています。これは、同じくGoogleから発表された論文:"Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications"で提唱されたMobileNetアーキテクチャのアップグレード版という位置付けで、出力チャネル数を明に増加させる「Expand-layer」やResNet系で馴染み深い「skip-connection」の導入などが特徴的です。

github.com


1. 動作環境

OS: Ubuntu 16.04

Package             Version  
------------------- -------
python              3.5.0
tensorboard         1.9.0    
tensorflow          1.9.0    
h5py                2.8.0    
Keras               2.2.2    
Keras-Applications  1.0.4    
Keras-Preprocessing 1.0.2  

2. プログラム

import_CIFAR-10.py

CIFAR-10データセットをnumpy配列(ndarray)形式で保存しておく実行ファイルです. CIFAR-10の画像は一枚あたり「32w(pixel) × 32h(pixel) × 3ch(RGB)」個のpixelからできています. 今回は, 画像データに対する前処理としてRGB-channelに応じて標準化(平均0, 標準偏差1)を行います.

データセット用のディレクトリと実行ファイルを用意しておくとバージョン管理が楽です.

import numpy as np
from keras.datasets import cifar10
from keras.utils import np_utils

nb_classes = 10
argmax_ch  = 255.0

if __name__=='__main__':
    # load CIFAR-10 data
    (X_train, y_train), (X_test, y_test) = cifar10.load_data()

    # set data type as 'float32'
    X_train = X_train.astype('float32') #argmax_ch
    X_test  = X_test.astype('float32')  #argmax_ch

    def ch_wise_normalization(X_type, ch):
        mean_ch = X_type[:, :, :, ch].mean()
        std_ch = X_type[:, :, :, ch].std()
        X_type[:, :, :, ch] = (X_type[:, :, :, ch] - mean_ch) / std_ch
        return X_type[:, :, :, ch]

    # normalize data for each R-G-B(0, 1, 2) channel 
    X_train[:, :, :, 0] = ch_wise_normalization(X_train, 0)
    X_train[:, :, :, 1] = ch_wise_normalization(X_train, 1)
    X_train[:, :, :, 2] = ch_wise_normalization(X_train, 2)

    X_test[:, :, :, 0]  = ch_wise_normalization(X_test, 0)
    X_test[:, :, :, 1]  = ch_wise_normalization(X_test, 1)
    X_test[:, :, :, 2]  = ch_wise_normalization(X_test, 2)

    # convert class label (0-9) to one-hot encoding format
    y_train = np_utils.to_categorical(y_train, nb_classes)
    y_test  = np_utils.to_categorical(y_test, nb_classes)

    # save datasets as np.ndarray class format files
    np.save('X_train', X_train)
    np.save('y_train', y_train)
    np.save('X_test' , X_test)
    np.save('y_test' , y_test)

mobilenetv2.py

MobileNetV2のアーキテクチャを定義してそのインスタンスを返すためのファイルです.
Kerasではkeras.applications.mobilenetv2.MobileNetV2で、定義ずみアーキテクチャの利用が可能なのですが, CIFAR-10, CIFAR-100の画像データは一片が32 pixelと非常に小さく、一辺が224 pixelで構成されるImageNet用に書かれている原論文のモデルでは, うまく学習ができません. そのため, Githubの実装を参考にして, アーキテクチャを作りました.

import os
import warnings
import numpy as np
from keras.layers import Input, Activation, Conv2D, Dense, Dropout, BatchNormalization, ReLU, DepthwiseConv2D, GlobalAveragePooling2D, GlobalMaxPooling2D, Add
from keras.models import Model
from keras import regularizers

# define the filter size
def _make_divisible(v, divisor, min_value=None):
    if min_value is None:
        min_value = divisor
    new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)
    # Make sure that round down does not go down by more than 10%.
    if new_v < 0.9 * v:
        new_v += divisor
    return new_v


# define the calcuration of each 'Res_Block'
def _inverted_res_block(inputs, expansion, stride, alpha, filters, block_id):
    prefix = 'block_{}_'.format(block_id)

    in_channels = inputs._keras_shape[-1]
    pointwise_conv_filters = int(filters * alpha)
    pointwise_filters = _make_divisible(pointwise_conv_filters, 8)
    x = inputs

    # Expand
    if block_id:
        x = Conv2D(expansion * in_channels, kernel_size=1, strides=1, padding='same', use_bias=False, activation=None, kernel_initializer="he_normal", kernel_regularizer=regularizers.l2(4e-5), name=prefix + 'expand')(x)
        x = BatchNormalization(epsilon=1e-3, momentum=0.999, name=prefix + 'expand_BN')(x)
        x = ReLU(6., name=prefix + 'expand_relu')(x)
    else:
        prefix = 'expanded_conv_'

    # Depthwise
    x = DepthwiseConv2D(kernel_size=3, strides=stride, activation=None, use_bias=False, padding='same', kernel_initializer="he_normal", depthwise_regularizer=regularizers.l2(4e-5), name=prefix + 'depthwise')(x)
    x = BatchNormalization(epsilon=1e-3, momentum=0.999, name=prefix + 'depthwise_BN')(x)
    x = ReLU(6., name=prefix + 'depthwise_relu')(x)

    # Project
    x = Conv2D(pointwise_filters, kernel_size=1, strides=1, padding='same', use_bias=False, activation=None, kernel_initializer="he_normal", kernel_regularizer=regularizers.l2(4e-5), name=prefix + 'project')(x)
    x = BatchNormalization(epsilon=1e-3, momentum=0.999, name=prefix + 'project_BN')(x)


    if in_channels == pointwise_filters and stride == 1:
        return Add(name=prefix + 'add')([inputs, x])
    return x

# build MobileNetV2 models
def MobileNetV2(input_shape=(32, 32, 3),
                alpha=1.0,
                depth_multiplier=1,
                include_top=True,
                pooling=None,
                classes=10):

    # fileter size (first block)
    first_block_filters = _make_divisible(32 * alpha, 8)
    # input shape  (first block)
    img_input = Input(shape=input_shape)

    # model architechture
    x = Conv2D(first_block_filters, kernel_size=3, strides=1, padding='same', use_bias=False, kernel_initializer="he_normal", kernel_regularizer=regularizers.l2(4e-5), name='Conv1')(img_input)
    #x = BatchNormalization(epsilon=1e-3, momentum=0.999, name='bn_Conv1')(x)
    #x = ReLU(6., name='Conv1_relu')(x)

    x = _inverted_res_block(x, filters=16,  alpha=alpha, stride=1, expansion=1, block_id=0 )

    x = _inverted_res_block(x, filters=24,  alpha=alpha, stride=1, expansion=6, block_id=1 )
    x = _inverted_res_block(x, filters=24,  alpha=alpha, stride=1, expansion=6, block_id=2 )

    x = _inverted_res_block(x, filters=32,  alpha=alpha, stride=2, expansion=6, block_id=3 )
    x = _inverted_res_block(x, filters=32,  alpha=alpha, stride=1, expansion=6, block_id=4 )
    x = _inverted_res_block(x, filters=32,  alpha=alpha, stride=1, expansion=6, block_id=5 )

    x = _inverted_res_block(x, filters=64,  alpha=alpha, stride=2, expansion=6, block_id=6 )
    x = _inverted_res_block(x, filters=64,  alpha=alpha, stride=1, expansion=6, block_id=7 )
    x = _inverted_res_block(x, filters=64,  alpha=alpha, stride=1, expansion=6, block_id=8 )
    x = _inverted_res_block(x, filters=64,  alpha=alpha, stride=1, expansion=6, block_id=9 )

    x = _inverted_res_block(x, filters=96,  alpha=alpha, stride=1, expansion=6, block_id=10)
    x = _inverted_res_block(x, filters=96,  alpha=alpha, stride=1, expansion=6, block_id=11)
    x = _inverted_res_block(x, filters=96,  alpha=alpha, stride=1, expansion=6, block_id=12)

    x = _inverted_res_block(x, filters=160, alpha=alpha, stride=2, expansion=6, block_id=13)
    x = _inverted_res_block(x, filters=160, alpha=alpha, stride=1, expansion=6, block_id=14)
    x = _inverted_res_block(x, filters=160, alpha=alpha, stride=1, expansion=6, block_id=15)
    x = Dropout(rate=0.25)(x)

    x = _inverted_res_block(x, filters=320, alpha=alpha, stride=1, expansion=6, block_id=16)
    x = Dropout(rate=0.25)(x)

    # define fileter size (last block)
    if alpha > 1.0:
        last_block_filters = _make_divisible(1280 * alpha, 8)
    else:
        last_block_filters = 1280


    x = Conv2D(last_block_filters, kernel_size=1, use_bias=False, kernel_initializer="he_normal", kernel_regularizer=regularizers.l2(4e-5), name='Conv_1')(x)
    x = BatchNormalization(epsilon=1e-3, momentum=0.999, name='Conv_1_bn')(x)
    x = ReLU(6., name='out_relu')(x)
    
    # top layer ("use" or "not use" FC)
    if include_top:
        x = GlobalAveragePooling2D(name='global_average_pool')(x)
        x = Dense(classes, activation='softmax', use_bias=True, name='Logits')(x)
    else:
        if pooling == 'avg':
            x = GlobalAveragePooling2D()(x)
        elif pooling == 'max':
            x = GlobalMaxPooling2D()(x)

    # create model of MobileNetV2 (for CIFAR-10)
    model = Model(inputs=img_input, outputs=x, name='mobilenetv2_cifar10')
    return model

main.py

各ファイルを統合して, 学習を実行するメインファイルです. このファイルによる実行がトリガーとなって, 各処理が進みます.

import os
import h5py
import numpy as np
import keras.backend.tensorflow_backend as KTF
import tensorflow as tf
from keras.models import Sequential, Model
from keras.applications.mobilenetv2 import MobileNetV2
from keras.engine.topology import Input
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.normalization import BatchNormalization
from keras.layers import Conv2D, SeparableConv2D, MaxPooling2D
from keras.optimizers import SGD, Adam, RMSprop
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard, CSVLogger, LearningRateScheduler
from datetime import datetime

from datafeeders.datafeeder import DataFeeder
from app.mobilenet_v2 import MobileNetV2

# set meta params
batch_size = 128
nb_classes = 10
nb_epoch   = 300
nb_data    = 32*32

# set meta info
log_dir         = '../train_log/mobilenet_v2-like5_log'
dataset_dir     = '../../CIFAR-10/datasets/dataset_norm'
model_name      = 'mobilenet_v2-like5__' + datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
model_arch_path = os.path.join(log_dir, (model_name + '_arch.png'))
model_cp_path   = os.path.join(log_dir, (model_name + '_checkpoint.h5'))
model_csv_path  = os.path.join(log_dir, (model_name + '_csv.csv'))


# load data
DF = DataFeeder(dataset_dir)
X_train, y_train, X_test, y_test = DF.X_train, DF.y_train, DF.X_test, DF.y_test

# data augumatation
datagen = ImageDataGenerator(
        width_shift_range=0.2, 
        height_shift_range=0.2,
        vertical_flip=False,
        horizontal_flip=True
        )
datagen.fit(X_train)


# build model
model = MobileNetV2(input_shape=X_train.shape[1:], include_top=True, alpha=1.0)
model.summary()
print('Model Name: ', model_name)

# save model architechture plot (.png)
from keras.utils import plot_model
plot_model(model, to_file=model_arch_path, show_shapes=True)

# set learning rate
learning_rates=[]
for i in range(5):
    learning_rates.append(2e-2)
for i in range(50-5):
    learning_rates.append(1e-2)
for i in range(100-50):
    learning_rates.append(8e-3)
for i in range(150-100):
    learning_rates.append(4e-3)
for i in range(200-150):
    learning_rates.append(2e-3)
for i in range(300-200):
    learning_rates.append(1e-3)

# set callbacks
callbacks = []
callbacks.append(TensorBoard(log_dir=log_dir, histogram_freq=1))
callbacks.append(ModelCheckpoint(model_cp_path, monitor='val_loss', save_best_only=True))
callbacks.append(LearningRateScheduler(lambda epoch: float(learning_rates[epoch])))
callbacks.append(CSVLogger(model_csv_path)) 

# compile & learning model
model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=2e-2, momentum=0.9, decay=0.0, nesterov=False), metrics=['accuracy'])
history = model.fit_generator(
              datagen.flow(X_train, y_train, batch_size=batch_size),
              steps_per_epoch=len(X_train) / batch_size,
              epochs=nb_epoch,
              verbose=1,
              callbacks=callbacks,
              validation_data=(X_test, y_test))
   
# validation
val_loss, val_acc = model.evaluate(X_test, y_test, verbose=0)
print('Model Name: ', model_name)
print('Test loss     : {:.5f}'.format(val_loss))
print('Test accuracy : {:.5f}'.format(val_acc))

# save model "INSTANCE"
f1_name = model_name + '_instance'
f1_path = os.path.join(log_dir, f1_name) + '.h5'
model.save(f1_path)

# save model "WEIGHTs"
f2_name = model_name + '_weights'
f2_path = os.path.join(log_dir, f2_name) + '.h5'
model.save_weights(f2_path)

# save model "ARCHITECHTURE"
f3_name = model_name + '_architechture'
f3_path = os.path.join(log_dir, f3_name) + '.json'
json_string = model.to_json()
with open(f3_path, 'w') as f:
    f.write(json_string)

# save plot figure
from utils.plot_log import save_plot_log
save_plot_log(log_dir, model_csv_path, index='acc')
save_plot_log(log_dir, model_csv_path, index='loss')


3. 学習結果

最終的な Validation Accuracyは 91% くらいでした。パラメータ数が250万弱なのでなかなかの性能。
学習率のスケジューリングによって精度が±5%動くので、難しいです。
※lossグラフの縦軸に「crossentropy」とありますが、多変量なので正しくは「categorical crossentropy」です。さーせん。

f:id:yumaloop:20180910124348p:plain
acc

f:id:yumaloop:20180910124359p:plain
loss

確率変数に関する4つの「収束概念」と「大数の法則&中心極限定理」

確率変数の収束 : Convergence of random variables についてまとめてみました.
数理統計学の核となる漸近理論や極値理論で活躍します.

https://upload.wikimedia.org/wikipedia/commons/4/49/Notions-of-convergence-in-probability-theory.jpg


1.確率収束 : convergence in probability

確率変数列  \{U_n\}_{n=1, 2, ...}に対して,任意の(どんなに小さな)  {\varepsilon}>0についても.以下の式が成り立つとき,
「確率変数列 \{U_n\}は確率変数 Uに確率収束する.」といい,これを「 U_n \to {}_p~U」と記す.

 \lim_{n \to \infty} P(~|U_n-U|{\geq}{\varepsilon}~)~=~0


2.p次平均収束 : convergence in mean

確率変数列  \{U_n\}_{n=1, 2, ...}に対して,以下の式が成り立つとき,
「確率変数列 \{U_n\}は確率変数 Uにp次平均収束する.」という.

 \lim_{n \to \infty} E(~{|U_n-U|}^{~p}~)~=~0


3.分布収束(法則収束): convergence in distribution (law)

確率変数列  \{U_n\}_{n=1, 2, ...}に対して,確率変数 Uの分布関数 F_U(x)の連続点 {x}で,以下の式が成り立つとき,
「確率変数列 \{U_n\}は確率変数 Uに分布収束(法則収束)する.」といい,これを「 U_n \to {}_d~U」と記す.

 \lim_{n \to \infty} P(~{U_n}{\leq}{x}~) ~=~ P(U{\leq}x)~=~F_U(x)


4.概収束 : almost sure convergence

確率変数列  \{U_n\}_{n=1, 2, ...}に対して,以下の式が成り立つとき,
「確率変数列 \{U_n\}は確率変数 Uに概収束する.」といい,これを「 U_n \to U_{a.s.}」と記す.

 \lim_{n \to \infty} P(~{\omega~|~{\lim_{n \to \infty} |U_n(\omega)-U(\omega)|=0}}~) ~=~1




参考


http://kristen-ressurs.no/images/Fysikk/Hilbert-space.jpg
http://radhakrishna.typepad.com/.a/6a00d83453b94569e2014e87c56ff9970d-pi


coming soon ...

Keras/Tensorflow : CIFAR-10のVGG-likeなアーキテクチャを作った.

1. 動作環境

OS: Ubuntu 16.04

Package             Version  
------------------- -------
python              3.5.0
tensorboard         1.9.0    
tensorflow          1.9.0    
h5py                2.8.0    
Keras               2.2.2    
Keras-Applications  1.0.4    
Keras-Preprocessing 1.0.2  

2. プログラム

import os
import numpy as np
import keras.backend.tensorflow_backend as KTF
import tensorflow as tf
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.normalization import BatchNormalization
from keras.layers import Conv2D, MaxPooling2D
from keras.optimizers import SGD, Adam
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard, CSVLogger
from datetime import datetime

# Set Meta Parameters & Information
img_row = 32.0
img_col = 32.0
img_ch  = 255.0

batch_size = 128
nb_classes = 10
nb_epoch   = 200
nb_data    = 32*32

log_dir        = '../train_log/vgg-like_log'
dataset_dir    = '../../CIFAR-10/datasets'
model_name     = 'vgg-like__' + datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
model_cp_path  = os.path.join(log_dir, (model_name + '_checkpoint.h5'))
model_csv_path = os.path.join(log_dir, (model_name + '_csv.csv'))

# Load CIFAR-10 Dataset
argmax_ch  = 255.0
(X_train, y_train), (X_test, y_test) = cifar10.load_data()

# convert pixel value range 0.0-255.0 to 0.0-1.0
X_train = X_train.astype('float32') / img_ch
X_test  = X_test.astype('float32')  / img_ch

# convert class label (0-9) to one-hot encoding format
y_train = np_utils.to_categorical(y_train, nb_classes)
y_test  = np_utils.to_categorical(y_test, nb_classes)

# save datasets as "np.ndarray" format files
np.save('X_train', X_train)
np.save('y_train', y_train)
np.save('X_test' , X_test)
np.save('y_test' , y_test)

# Data Augumatation
datagen = ImageDataGenerator(
        featurewise_center=True,
        featurewise_std_normalization=True,
        rotation_range=20,
        width_shift_range=0.2,
        height_shift_range=0.2,
        horizontal_flip=True
        vertical_flip=False)
datagen.fit(X_train)



old_session = KTF.get_session()

with tf.Graph().as_default():
    session = tf.Session('')
    KTF.set_session(session)
    KTF.set_learning_phase(1)

    # build model
    model = Sequential()
    with tf.name_scope('inference') as scope:
        model.add(Conv2D(64, (3, 3), padding='same', input_shape=X_train.shape[1:]))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(Conv2D(64, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))

        model.add(Conv2D(128, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(Conv2D(128, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))

        model.add(Conv2D(256, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(Conv2D(256, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(Conv2D(256, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(Conv2D(256, (3, 3), padding='same'))
        model.add(BatchNormalization())
        model.add(Activation('relu'))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))

        model.add(Flatten())
        model.add(Dense(1024))
        model.add(Activation('relu'))
        model.add(Dropout(0.5))
        model.add(Dense(1024))
        model.add(Activation('relu'))
        model.add(Dropout(0.5))
        model.add(Dense(nb_classes))
        model.add(Activation('softmax'))
    model.summary()

    # set callbacks
    cp_cb  = ModelCheckpoint(model_cp_path, monitor='val_loss', save_best_only=True)
    tb_cb  = TensorBoard(log_dir=log_dir, histogram_freq=1)
    csv_cb = CSVLogger(model_csv_path) 
    callbacks = [cp_cb, tb_cb, csv_cb]

    # compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    history = model.fit_generator(
        datagen.flow(X_train, y_train, batch_size=batch_size),
        steps_per_epoch=len(X_train) / batch_size,
        epochs=nb_epoch,
        verbose=1,
        callbacks=callbacks,
        validation_data=(X_test, y_test))
    
    # validation
    score = model.evaluate(X_test, y_test, verbose=0)
    print('val score    : ', score[0])
    print('val accuracy : ', score[1])


# save model "INSTANCE"
f1_name = model_name + '_instance'
f1_path = os.path.join(log_dir, f1_name) + '.h5'
model.save(f1_path)

# save model "WEIGHTs"
f2_name = model_name + '_weights'
f2_path = os.path.join(log_dir, f2_name) + '.h5'
model.save_weights(f2_path)

# save model "ARCHITECHTURE"
f3_name = model_name + '_architechture'
f3_path = os.path.join(log_dir, f3_name) + '.json'
json_string = model.to_json()
with open(f3_path, 'w') as f:
    f.write(json_string)


# end of session
KTF.set_session(old_session)

3.アーキテクチャ

Kerasのkeras.models.Model()クラスもつ属性(attribute)である"summary()"を使う.
プログラム上のmodel.summary()で、標準出力にモデルの構造(architechture)の要約情報が表示される.

Modelクラス (functional API) - Keras Documentation

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 32, 32, 64)        1792      
_________________________________________________________________
batch_normalization_1 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
activation_1 (Activation)    (None, 32, 32, 64)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 32, 64)        36928     
_________________________________________________________________
batch_normalization_2 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
activation_2 (Activation)    (None, 32, 32, 64)        0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 16, 16, 64)        0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 16, 16, 64)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 16, 16, 128)       73856     
_________________________________________________________________
batch_normalization_3 (Batch (None, 16, 16, 128)       512       
_________________________________________________________________
activation_3 (Activation)    (None, 16, 16, 128)       0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 16, 16, 128)       147584    
_________________________________________________________________
batch_normalization_4 (Batch (None, 16, 16, 128)       512       
_________________________________________________________________
activation_4 (Activation)    (None, 16, 16, 128)       0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 128)         0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 8, 8, 128)         0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 8, 8, 256)         295168    
_________________________________________________________________
batch_normalization_5 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
activation_5 (Activation)    (None, 8, 8, 256)         0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 8, 8, 256)         590080    
_________________________________________________________________
batch_normalization_6 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
activation_6 (Activation)    (None, 8, 8, 256)         0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 8, 8, 256)         590080    
_________________________________________________________________
batch_normalization_7 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
activation_7 (Activation)    (None, 8, 8, 256)         0         
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 8, 8, 256)         590080    
_________________________________________________________________
batch_normalization_8 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
activation_8 (Activation)    (None, 8, 8, 256)         0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 4, 4, 256)         0         
_________________________________________________________________
dropout_3 (Dropout)          (None, 4, 4, 256)         0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 4096)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 1024)              4195328   
_________________________________________________________________
activation_9 (Activation)    (None, 1024)              0         
_________________________________________________________________
dropout_4 (Dropout)          (None, 1024)              0         
_________________________________________________________________
dense_2 (Dense)              (None, 1024)              1049600   
_________________________________________________________________
activation_10 (Activation)   (None, 1024)              0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 1024)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 10)                10250     
_________________________________________________________________
activation_11 (Activation)   (None, 10)                0         
=================================================================
Total params: 7,586,378
Trainable params: 7,583,562
Non-trainable params: 2,816
_________________________________________________________________


4.学習結果

validation accuracy は 90% くらい

f:id:yumaloop:20180822170332p:plain

f:id:yumaloop:20180822170409p:plain

Optimal Brain Damage : 局所二次近似と極値判定(凸関数, 勾配, ヘッセ行列)

ニューラルネットワークモデルにおける「パラメータを訓練(学習)させるためのアルゴリズム」に注目したいと思います.ニューラルネットワークモデルについては,

「特徴量(=パラメータ)を人間が与えなくとも,機械(=コンピュータ)が勝手にその "重要度" を評価し,学習する」

という紹介で,その前進性を紹介されることが多いですが.ここで主張されている要点は,

  • 特徴量(params)の"重要さ"を機械(model)が自動的に評価すること
  • 特徴量(params)の"重要さ"を機械(model)が評価した後に,この情報を機械(model)に反映させること

の2つに集約できます.しかし,前者に関しては,適応信号処理や情報検索,機械学習などで既に多くの一般的な研究成果が与えられているため,ニューラルネットワーク特有の性質とは言えません,その意味で,個人的には,後者:「特徴量(params)の"重要さ"を機械(model)が評価した後に,この情報を機械(model)に自動的に反映させること」こそがニューラルネットワークモデルのクリティカルな性質であると言って良いでしょう.そしてこれは,具体的には「誤差逆伝播法」というアイデアと,これを解くための「最急降下法」という最適化アルゴリズムに還元されます.

以上を踏まえ,今回はニューラルネットワークモデルの核である「誤差逆伝播法」「最急降下法」に注目した上で,その原理となる「局所二次近似」や「凸最適化」についてまとめてみたいと思います.


1. 背景

https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Monkey_saddle_surface.svg/300px-Monkey_saddle_surface.svg.png

「Optimal Brain Damage (: OBD)」とは,ニューラルネットワークを用いた目的変数の予測問題において、パラメータ数を縮減するための方法であり,「局所二次近似 (Approximation of Second-order Derivatives)」とは,OBDを実装する際に必要となる計算アルゴリズムのことです。具体的には,目的関数(=誤差関数, 損失関数)を2次の項までのテイラー展開によって近似することで,あるパラメータの組み合わせに対する「勾配」を求めるという手法を意味しています.

なお.OBDは、Yann Le Canらによって書かれた初期のニューラルネットワーク研究における有名な論文:Optimal Brain Damageによって提案されています.

  • OBD: Optimal Brain Damage
  • OBS: Optimal Brain Surgeon
  • OCD: Optimal Cell Damage


ここからは、この局所二次近似について考えてみましょう.


2. 準備:凸関数とテイラー展開

まず,凸関数(Convex function),勾配(Gradient),ヘッセ行列(Hessian)の定義を与えます.

https://www.me.utexas.edu/~jensen/ORMM/computation/unit/functions/formulas/h_z_deriv.gif

2. 1. 凸関数

n個のスカラー変量  \{x_n\} からなる多変数関数: f(x_1, x_2, ... x_n)~=f({\bf x}) を考える.

定義域上の任意の2点  {\bf x_a}, ~{\bf x_b} (n次元ベクトル)において,次の不等式を満たすとき, fを「凸関数」(: Convex function)という.

 {\lambda}{\cdot}f({\bf x_a})+(1-{\lambda}){\cdot}f({\bf x_b})~~{\geq}~~f({\lambda}{\cdot}{\bf x_a}+(1-{\lambda}){\cdot}{\bf x_b})


なお,等号成立が {\bf x_a}~=~{\bf x_b}の場合に限るとき, f を「狭義の凸関数」という.


2. 2. 勾配 - Gradient

n個のスカラー変量  \{x_n\} からなる多変数関数: f(x_1, x_2, ... x_n)~=f({\bf x}) を考える.

 f(x_1, x_2, ... x_n) が,任意の変数 x_1, x_2, ... x_n について1階偏導関数が定義されているとき, fに対する1階微分の情報を包括的に含む以下のn次元ベクトル  \bf\nabla が定められる.これを「 fの勾配(gradient,  grad f)」という.

 {\bf\nabla}f({\bf x})=\left(
    \begin{array}{cccc}
      \frac{\partial f({\bf x})}{\partial x_1} \\
      \frac{\partial f({\bf x})}{\partial x_2} \\
      \vdots \\
      \frac{\partial f({\bf x})}{\partial x_n}
    \end{array}
  \right)


 {\bf\nabla}f({\bf x})={\large\frac{\partial f}{\partial x_1}}\mathbf{e_1}+{\large\frac{\partial f}{\partial x_2}}\mathbf{e_2}+\ldots+{\large\frac{\partial f}{\partial x_n}}\mathbf{e_n}



2. 3. ヘッセ行列 - Hessian Matrix

n個のスカラー変量  \{x_n\} からなる多変数関数: f(x_1, x_2, ... x_n)~=f({\bf x}) を考える.

 f(x_1, x_2, ... x_n) が,任意の変数 x_1, x_2, ... x_n について2階偏導関数が定義されているとき, fに対する2階微分の情報を包括的に含む以下のn×n正方行列:  {\bf H}~={\nabla^2}f({\bf x}) が定められる.これをヘッセ行列という.

 {\bf H}={{\bf\nabla}^2}f({\bf x})=\left(
    \begin{array}{cccc}
      {\frac{{\partial}^2 f({\bf x})}{{\partial x_1}{\partial x_1}}} & {\frac{{\partial}^2 f({\bf x})}{{\partial x_1}{\partial x_2}}} & \ldots & {\frac{{\partial}^2 f({\bf x})}{{\partial x_1}{\partial x_n}}} \\
      {\frac{{\partial}^2 f({\bf x})}{{\partial x_2}{\partial x_1}}} & {\frac{{\partial}^2 f({\bf x})}{{\partial x_2}{\partial x_2}}} & \ldots & {\frac{{\partial}^2 f({\bf x})}{{\partial x_2}{\partial x_n}}} \\
      \vdots & \vdots & \ddots & \vdots \\
      {\frac{{\partial}^2 f({\bf x})}{{\partial x_n}{\partial x_1}}} & {\frac{{\partial}^2 f({\bf x})}{{\partial x_n}{\partial x_2}}} & \ldots & {\frac{{\partial}^2 f({\bf x})}{{\partial x_n}{\partial x_n}}}
    \end{array}
  \right)

ただし,ヘッセ行列 {\bf H}の各要素は次のように定義される.

 h_{ij}~=~\frac{{\partial}^2 f}{{\partial x_i}{\partial x_j}}



2. 4. テイラー展開による2次の近似

n個のスカラー変量  \{x_n\} からなる多変数関数: f(x_1, x_2, ... x_n)~=f({\bf x}) を考える.


テイラー展開より, f を任意の点  {\bf \hat{x}~(=a)} によって近似すると,次式を得る.

 f({\bf x})= f({\bf a})+\sum_{n=1}^{\infty}{\large\frac{f^{(n)}({\bf a})}{n!}}({\bf x}-{\bf a})^n

 f({\bf x}) = f({\bf a})+{\large\frac{f^{(1)}({\bf a})}{1!}}({\bf x}-{\bf a})+{\large\frac{f^{(2)}({\bf a})}{2!}}({\bf x}-{\bf a})^2+{\large\frac{f^{(3)}({\bf a})}{3!}}({\bf x}-{\bf a})^3+~\cdots



ここで、 f(x_1, x_2, ... x_n)~=f({\bf x})を、2次までのテイラー展開によって近似すると、

 f({\bf x})~{\fallingdotseq}~f({\bf a})+{\large\frac{f^{(1)}({\bf a})}{1!}}({\bf x}-{\bf a})+{\large\frac{f^{(2)}({\bf a})}{2!}}({\bf x}-{\bf a})^2

となり、既知の値  {\bf \hat{x}~(=a)}によって、 f を推定することができる。



3. 局所二次近似

さて,ここまでの議論で共通して登場してきた

「n個のスカラー変量  \{x_n\} からなる多変数関数: f(x_1, x_2, ... x_n)~=f({\bf x})

は,予測問題の文脈では

「n次元のベクトル {\bf w} に関する誤差関数(=目的関数):  E({(w_1, w_2, ... w_n)}^{\mathrm{T}})~=E({\bf w})

とみなすことができます.


誤差関数  E({\bf x}) に対するパラメータベクトル  {\bf w} に,既知の値  \hat{w} を与えて,2次のテイラー展開によって近似してみましょう.

 E({\bf w})~{\fallingdotseq}~E({\bf \hat{w}})+{\large\frac{E^{(1)}({\bf \hat{w}})}{1!}}({\bf w}-{\bf \hat{w}})+{\large\frac{E^{(2)}({\bf \hat{w}})}{2!}}({\bf w}-{\bf \hat{w}})^2

  • 誤差関数を最小にするようなパラメータベクトル  {\bf w} を求めたい.
  • 誤差関数はパラメータベクトル  {\bf w} に対して凸関数となる.

誤差関数の

coming soon ..



4. 極値判定

x において正定値対称行列であるとき、f は x において極小である。
x において負定値対称行列であるとき、f は x において極大である。
x において正負両方の固有値を持つとき、x は f の鞍点である(これは x が退化する場合にも正しい)。

coming soon ..

参考

Javaで文字列として使える物理フォント一覧を表示させる方法

java.awt.GraphicsEnvironmentクラスにある
getLocalGraphicsEnvironment().getAllFonts();メソッドを使う。

GraphicsEnvironment (Java Platform SE 8)


Code sample.

import java.awt.Font;
import java.awt.GraphicsEnvironment;

public class Main {
	public static void main(String[] args) throws Exception {
	    Font[] fonts  = GraphicsEnvironment.getLocalGraphicsEnvironment().getAllFonts();
	    for (int i = 0; i < fonts.length; i++) {
	      System.out.print(fonts[i].getFontName() + ", ");
	      System.out.print(fonts[i].getFamily() + ", ");
	      System.out.print(fonts[i].getName());
	      System.out.println();
	    }
	}	
}


Result here.

Serif, Serif, Serif, 
SansSerif, SansSerif, SansSerif, 
Monospaced, Monospaced, Monospaced, 
Dialog, Dialog, Dialog, 
DialogInput, DialogInput, DialogInput, 
.SFNSText, .SF NS Text, .SFNSText, 
.SFNSText-Bold, .SF NS Text, .SFNSText-Bold, 
AlBayan, Al Bayan, AlBayan, 
AlBayan-Bold, Al Bayan, AlBayan-Bold, 
AlNile, Al Nile, AlNile, 
AlNile-Bold, Al Nile, AlNile-Bold, 
AlTarikh, Al Tarikh, AlTarikh, 
AmericanTypewriter, American Typewriter, AmericanTypewriter, 
AmericanTypewriter-Bold, American Typewriter, AmericanTypewriter-Bold, 
AmericanTypewriter-Condensed, American Typewriter, AmericanTypewriter-Condensed, 
AmericanTypewriter-CondensedBold, American Typewriter, AmericanTypewriter-CondensedBold, 
AmericanTypewriter-CondensedLight, American Typewriter, AmericanTypewriter-CondensedLight, 
AmericanTypewriter-Light, American Typewriter, AmericanTypewriter-Light, 
AmericanTypewriter-Semibold, American Typewriter, AmericanTypewriter-Semibold, 
AndaleMono, Andale Mono, AndaleMono, 
Apple-Chancery, Apple Chancery, Apple-Chancery, 
AppleBraille, Apple Braille, AppleBraille, 
AppleBraille-Outline6Dot, Apple Braille, AppleBraille-Outline6Dot, 
AppleBraille-Outline8Dot, Apple Braille, AppleBraille-Outline8Dot, 
AppleBraille-Pinpoint6Dot, Apple Braille, AppleBraille-Pinpoint6Dot, 
AppleBraille-Pinpoint8Dot, Apple Braille, AppleBraille-Pinpoint8Dot, 
AppleColorEmoji, Apple Color Emoji, AppleColorEmoji, 
AppleGothic, AppleGothic, AppleGothic, 
AppleMyungjo, AppleMyungjo, AppleMyungjo, 
AppleSDGothicNeo-Bold, Apple SD Gothic Neo, AppleSDGothicNeo-Bold, 
AppleSDGothicNeo-ExtraBold, Apple SD Gothic Neo, AppleSDGothicNeo-ExtraBold, 
AppleSDGothicNeo-Heavy, Apple SD Gothic Neo, AppleSDGothicNeo-Heavy, 
AppleSDGothicNeo-Light, Apple SD Gothic Neo, AppleSDGothicNeo-Light, 
AppleSDGothicNeo-Medium, Apple SD Gothic Neo, AppleSDGothicNeo-Medium, 
AppleSDGothicNeo-Regular, Apple SD Gothic Neo, AppleSDGothicNeo-Regular, 
AppleSDGothicNeo-SemiBold, Apple SD Gothic Neo, AppleSDGothicNeo-SemiBold, 
AppleSDGothicNeo-Thin, Apple SD Gothic Neo, AppleSDGothicNeo-Thin, 
AppleSDGothicNeo-UltraLight, Apple SD Gothic Neo, AppleSDGothicNeo-UltraLight, 
AppleSymbols, Apple Symbols, AppleSymbols, 
Arial-Black, Arial Black, Arial-Black, 
Arial-BoldItalicMT, Arial, Arial-BoldItalicMT, 
Arial-BoldMT, Arial, Arial-BoldMT, 
Arial-ItalicMT, Arial, Arial-ItalicMT, 
ArialHebrew, Arial Hebrew, ArialHebrew, 
ArialHebrew-Bold, Arial Hebrew, ArialHebrew-Bold, 
ArialHebrew-Light, Arial Hebrew, ArialHebrew-Light, 
ArialHebrewScholar, Arial Hebrew Scholar, ArialHebrewScholar, 
ArialHebrewScholar-Bold, Arial Hebrew Scholar, ArialHebrewScholar-Bold, 
ArialHebrewScholar-Light, Arial Hebrew Scholar, ArialHebrewScholar-Light, 
ArialMT, Arial, ArialMT, 
ArialNarrow, Arial Narrow, ArialNarrow, 
ArialNarrow-Bold, Arial Narrow, ArialNarrow-Bold, 
ArialNarrow-BoldItalic, Arial Narrow, ArialNarrow-BoldItalic, 
ArialNarrow-Italic, Arial Narrow, ArialNarrow-Italic, 
ArialRoundedMTBold, Arial Rounded MT Bold, ArialRoundedMTBold, 
ArialUnicodeMS, Arial Unicode MS, ArialUnicodeMS, 
Athelas-Bold, Athelas, Athelas-Bold, 
Athelas-BoldItalic, Athelas, Athelas-BoldItalic, 
Athelas-Italic, Athelas, Athelas-Italic, 
Athelas-Regular, Athelas, Athelas-Regular, 
Avenir-Black, Avenir, Avenir-Black, 
Avenir-BlackOblique, Avenir, Avenir-BlackOblique, 
Avenir-Book, Avenir, Avenir-Book, 
Avenir-BookOblique, Avenir, Avenir-BookOblique, 
Avenir-Heavy, Avenir, Avenir-Heavy, 
Avenir-HeavyOblique, Avenir, Avenir-HeavyOblique, 
Avenir-Light, Avenir, Avenir-Light, 
Avenir-LightOblique, Avenir, Avenir-LightOblique, 
Avenir-Medium, Avenir, Avenir-Medium, 
Avenir-MediumOblique, Avenir, Avenir-MediumOblique, 
Avenir-Oblique, Avenir, Avenir-Oblique, 
Avenir-Roman, Avenir, Avenir-Roman, 
AvenirNext-Bold, Avenir Next, AvenirNext-Bold, 
AvenirNext-BoldItalic, Avenir Next, AvenirNext-BoldItalic, 
AvenirNext-DemiBold, Avenir Next, AvenirNext-DemiBold, 
AvenirNext-DemiBoldItalic, Avenir Next, AvenirNext-DemiBoldItalic, 
AvenirNext-Heavy, Avenir Next, AvenirNext-Heavy, 
AvenirNext-HeavyItalic, Avenir Next, AvenirNext-HeavyItalic, 
AvenirNext-Italic, Avenir Next, AvenirNext-Italic, 
AvenirNext-Medium, Avenir Next, AvenirNext-Medium, 
AvenirNext-MediumItalic, Avenir Next, AvenirNext-MediumItalic, 
AvenirNext-Regular, Avenir Next, AvenirNext-Regular, 
AvenirNext-UltraLight, Avenir Next, AvenirNext-UltraLight, 
AvenirNext-UltraLightItalic, Avenir Next, AvenirNext-UltraLightItalic, 
AvenirNextCondensed-Bold, Avenir Next Condensed, AvenirNextCondensed-Bold, 
AvenirNextCondensed-BoldItalic, Avenir Next Condensed, AvenirNextCondensed-BoldItalic, 
AvenirNextCondensed-DemiBold, Avenir Next Condensed, AvenirNextCondensed-DemiBold, 
AvenirNextCondensed-DemiBoldItalic, Avenir Next Condensed, AvenirNextCondensed-DemiBoldItalic, 
AvenirNextCondensed-Heavy, Avenir Next Condensed, AvenirNextCondensed-Heavy, 
AvenirNextCondensed-HeavyItalic, Avenir Next Condensed, AvenirNextCondensed-HeavyItalic, 
AvenirNextCondensed-Italic, Avenir Next Condensed, AvenirNextCondensed-Italic, 
AvenirNextCondensed-Medium, Avenir Next Condensed, AvenirNextCondensed-Medium, 
AvenirNextCondensed-MediumItalic, Avenir Next Condensed, AvenirNextCondensed-MediumItalic, 
AvenirNextCondensed-Regular, Avenir Next Condensed, AvenirNextCondensed-Regular, 
AvenirNextCondensed-UltraLight, Avenir Next Condensed, AvenirNextCondensed-UltraLight, 
AvenirNextCondensed-UltraLightItalic, Avenir Next Condensed, AvenirNextCondensed-UltraLightItalic, 
Ayuthaya, Ayuthaya, Ayuthaya, 
Baghdad, Baghdad, Baghdad, 
BanglaMN, Bangla MN, BanglaMN, 
BanglaMN-Bold, Bangla MN, BanglaMN-Bold, 
BanglaSangamMN, Bangla Sangam MN, BanglaSangamMN, 
BanglaSangamMN-Bold, Bangla Sangam MN, BanglaSangamMN-Bold, 
Baskerville, Baskerville, Baskerville, 
Baskerville-Bold, Baskerville, Baskerville-Bold, 
Baskerville-BoldItalic, Baskerville, Baskerville-BoldItalic, 
Baskerville-Italic, Baskerville, Baskerville-Italic, 
Baskerville-SemiBold, Baskerville, Baskerville-SemiBold, 
Baskerville-SemiBoldItalic, Baskerville, Baskerville-SemiBoldItalic, 
Beirut, Beirut, Beirut, 
BigCaslon-Medium, Big Caslon, BigCaslon-Medium, 
BodoniOrnamentsITCTT, Bodoni Ornaments, BodoniOrnamentsITCTT, 
BodoniSvtyTwoITCTT-Bold, Bodoni 72, BodoniSvtyTwoITCTT-Bold, 
BodoniSvtyTwoITCTT-Book, Bodoni 72, BodoniSvtyTwoITCTT-Book, 
BodoniSvtyTwoITCTT-BookIta, Bodoni 72, BodoniSvtyTwoITCTT-BookIta, 
BodoniSvtyTwoOSITCTT-Bold, Bodoni 72 Oldstyle, BodoniSvtyTwoOSITCTT-Bold, 
BodoniSvtyTwoOSITCTT-Book, Bodoni 72 Oldstyle, BodoniSvtyTwoOSITCTT-Book, 
BodoniSvtyTwoOSITCTT-BookIt, Bodoni 72 Oldstyle, BodoniSvtyTwoOSITCTT-BookIt, 
BodoniSvtyTwoSCITCTT-Book, Bodoni 72 Smallcaps, BodoniSvtyTwoSCITCTT-Book, 
BradleyHandITCTT-Bold, Bradley Hand, BradleyHandITCTT-Bold, 
BrushScriptMT, Brush Script MT, BrushScriptMT, 
Chalkboard, Chalkboard, Chalkboard, 
Chalkboard-Bold, Chalkboard, Chalkboard-Bold, 
ChalkboardSE-Bold, Chalkboard SE, ChalkboardSE-Bold, 
ChalkboardSE-Light, Chalkboard SE, ChalkboardSE-Light, 
ChalkboardSE-Regular, Chalkboard SE, ChalkboardSE-Regular, 
Chalkduster, Chalkduster, Chalkduster, 
Charter-Black, Charter, Charter-Black, 
Charter-BlackItalic, Charter, Charter-BlackItalic, 
Charter-Bold, Charter, Charter-Bold, 
Charter-BoldItalic, Charter, Charter-BoldItalic, 
Charter-Italic, Charter, Charter-Italic, 
Charter-Roman, Charter, Charter-Roman, 
Cochin, Cochin, Cochin, 
Cochin-Bold, Cochin, Cochin-Bold, 
Cochin-BoldItalic, Cochin, Cochin-BoldItalic, 
Cochin-Italic, Cochin, Cochin-Italic, 
ComicSansMS, Comic Sans MS, ComicSansMS, 
ComicSansMS-Bold, Comic Sans MS, ComicSansMS-Bold, 
Copperplate, Copperplate, Copperplate, 
Copperplate-Bold, Copperplate, Copperplate-Bold, 
Copperplate-Light, Copperplate, Copperplate-Light, 
CorsivaHebrew, Corsiva Hebrew, CorsivaHebrew, 
CorsivaHebrew-Bold, Corsiva Hebrew, CorsivaHebrew-Bold, 
Courier, Courier, Courier, 
Courier-Bold, Courier, Courier-Bold, 
Courier-BoldOblique, Courier, Courier-BoldOblique, 
Courier-Oblique, Courier, Courier-Oblique, 
CourierNewPS-BoldItalicMT, Courier New, CourierNewPS-BoldItalicMT, 
CourierNewPS-BoldMT, Courier New, CourierNewPS-BoldMT, 
CourierNewPS-ItalicMT, Courier New, CourierNewPS-ItalicMT, 
CourierNewPSMT, Courier New, CourierNewPSMT, 
DFKaiShu-SB-Estd-BF, BiauKai, DFKaiShu-SB-Estd-BF, 
DFWaWaSC-W5, Wawati SC, DFWaWaSC-W5, 
DFWaWaTC-W5, Wawati TC, DFWaWaTC-W5, 
DINAlternate-Bold, DIN Alternate, DINAlternate-Bold, 
DINCondensed-Bold, DIN Condensed, DINCondensed-Bold, 
Damascus, Damascus, Damascus, 
DamascusBold, Damascus, DamascusBold, 
DamascusLight, Damascus, DamascusLight, 
DamascusMedium, Damascus, DamascusMedium, 
DamascusSemiBold, Damascus, DamascusSemiBold, 
DecoTypeNaskh, DecoType Naskh, DecoTypeNaskh, 
DevanagariMT, Devanagari MT, DevanagariMT, 
DevanagariMT-Bold, Devanagari MT, DevanagariMT-Bold, 
DevanagariSangamMN, Devanagari Sangam MN, DevanagariSangamMN, 
DevanagariSangamMN-Bold, Devanagari Sangam MN, DevanagariSangamMN-Bold, 
Didot, Didot, Didot, 
Didot-Bold, Didot, Didot-Bold, 
Didot-Italic, Didot, Didot-Italic, 
DiwanKufi, Diwan Kufi, DiwanKufi, 
DiwanMishafi, Mishafi, DiwanMishafi, 
DiwanMishafiGold, Mishafi Gold, DiwanMishafiGold, 
DiwanThuluth, Diwan Thuluth, DiwanThuluth, 
EuphemiaUCAS, Euphemia UCAS, EuphemiaUCAS, 
EuphemiaUCAS-Bold, Euphemia UCAS, EuphemiaUCAS-Bold, 
EuphemiaUCAS-Italic, Euphemia UCAS, EuphemiaUCAS-Italic, 
FZLTTHB--B51-0, Lantinghei TC, FZLTTHB--B51-0, 
FZLTTHK--GBK1-0, Lantinghei SC, FZLTTHK--GBK1-0, 
FZLTXHB--B51-0, Lantinghei TC, FZLTXHB--B51-0, 
FZLTXHK--GBK1-0, Lantinghei SC, FZLTXHK--GBK1-0, 
FZLTZHB--B51-0, Lantinghei TC, FZLTZHB--B51-0, 
FZLTZHK--GBK1-0, Lantinghei SC, FZLTZHK--GBK1-0, 
Farah, Farah, Farah, 
Farisi, Farisi, Farisi, 
Futura-Bold, Futura, Futura-Bold, 
Futura-CondensedExtraBold, Futura, Futura-CondensedExtraBold, 
Futura-CondensedMedium, Futura, Futura-CondensedMedium, 
Futura-Medium, Futura, Futura-Medium, 
Futura-MediumItalic, Futura, Futura-MediumItalic, 
GB18030Bitmap, GB18030 Bitmap, GB18030Bitmap, 
GeezaPro, Geeza Pro, GeezaPro, 
GeezaPro-Bold, Geeza Pro, GeezaPro-Bold, 
Geneva, Geneva, Geneva, 
Georgia, Georgia, Georgia, 
Georgia-Bold, Georgia, Georgia-Bold, 
Georgia-BoldItalic, Georgia, Georgia-BoldItalic, 
Georgia-Italic, Georgia, Georgia-Italic, 
GillSans, Gill Sans, GillSans, 
GillSans-Bold, Gill Sans, GillSans-Bold, 
GillSans-BoldItalic, Gill Sans, GillSans-BoldItalic, 
GillSans-Italic, Gill Sans, GillSans-Italic, 
GillSans-Light, Gill Sans, GillSans-Light, 
GillSans-LightItalic, Gill Sans, GillSans-LightItalic, 
GillSans-SemiBold, Gill Sans, GillSans-SemiBold, 
GillSans-SemiBoldItalic, Gill Sans, GillSans-SemiBoldItalic, 
GillSans-UltraBold, Gill Sans, GillSans-UltraBold, 
GujaratiMT, Gujarati MT, GujaratiMT, 
GujaratiMT-Bold, Gujarati MT, GujaratiMT-Bold, 
GujaratiSangamMN, Gujarati Sangam MN, GujaratiSangamMN, 
GujaratiSangamMN-Bold, Gujarati Sangam MN, GujaratiSangamMN-Bold, 
GurmukhiMN, Gurmukhi MN, GurmukhiMN, 
GurmukhiMN-Bold, Gurmukhi MN, GurmukhiMN-Bold, 
GurmukhiSangamMN, Gurmukhi Sangam MN, GurmukhiSangamMN, 
GurmukhiSangamMN-Bold, Gurmukhi Sangam MN, GurmukhiSangamMN-Bold, 
HannotateSC-W5, Hannotate SC, HannotateSC-W5, 
HannotateSC-W7, Hannotate SC, HannotateSC-W7, 
HannotateTC-W5, Hannotate TC, HannotateTC-W5, 
HannotateTC-W7, Hannotate TC, HannotateTC-W7, 
HanziPenSC-W3, HanziPen SC, HanziPenSC-W3, 
HanziPenSC-W5, HanziPen SC, HanziPenSC-W5, 
HanziPenTC-W3, HanziPen TC, HanziPenTC-W3, 
HanziPenTC-W5, HanziPen TC, HanziPenTC-W5, 
Helvetica, Helvetica, Helvetica, 
Helvetica-Bold, Helvetica, Helvetica-Bold, 
Helvetica-BoldOblique, Helvetica, Helvetica-BoldOblique, 
Helvetica-Light, Helvetica, Helvetica-Light, 
Helvetica-LightOblique, Helvetica, Helvetica-LightOblique, 
Helvetica-Oblique, Helvetica, Helvetica-Oblique, 
HelveticaNeue, Helvetica Neue, HelveticaNeue, 
HelveticaNeue-Bold, Helvetica Neue, HelveticaNeue-Bold, 
HelveticaNeue-BoldItalic, Helvetica Neue, HelveticaNeue-BoldItalic, 
HelveticaNeue-CondensedBlack, Helvetica Neue, HelveticaNeue-CondensedBlack, 
HelveticaNeue-CondensedBold, Helvetica Neue, HelveticaNeue-CondensedBold, 
HelveticaNeue-Italic, Helvetica Neue, HelveticaNeue-Italic, 
HelveticaNeue-Light, Helvetica Neue, HelveticaNeue-Light, 
HelveticaNeue-LightItalic, Helvetica Neue, HelveticaNeue-LightItalic, 
HelveticaNeue-Medium, Helvetica Neue, HelveticaNeue-Medium, 
HelveticaNeue-MediumItalic, Helvetica Neue, HelveticaNeue-MediumItalic, 
HelveticaNeue-Thin, Helvetica Neue, HelveticaNeue-Thin, 
HelveticaNeue-ThinItalic, Helvetica Neue, HelveticaNeue-ThinItalic, 
HelveticaNeue-UltraLight, Helvetica Neue, HelveticaNeue-UltraLight, 
HelveticaNeue-UltraLightItalic, Helvetica Neue, HelveticaNeue-UltraLightItalic, 
Herculanum, Herculanum, Herculanum, 
HiraKakuPro-W3, Hiragino Kaku Gothic Pro, HiraKakuPro-W3, 
HiraKakuPro-W6, Hiragino Kaku Gothic Pro, HiraKakuPro-W6, 
HiraKakuProN-W3, Hiragino Kaku Gothic ProN, HiraKakuProN-W3, 
HiraKakuProN-W6, Hiragino Kaku Gothic ProN, HiraKakuProN-W6, 
HiraKakuStd-W8, Hiragino Kaku Gothic Std, HiraKakuStd-W8, 
HiraKakuStdN-W8, Hiragino Kaku Gothic StdN, HiraKakuStdN-W8, 
HiraMaruPro-W4, Hiragino Maru Gothic Pro, HiraMaruPro-W4, 
HiraMaruProN-W4, Hiragino Maru Gothic ProN, HiraMaruProN-W4, 
HiraMinPro-W3, Hiragino Mincho Pro, HiraMinPro-W3, 
HiraMinPro-W6, Hiragino Mincho Pro, HiraMinPro-W6, 
HiraMinProN-W3, Hiragino Mincho ProN, HiraMinProN-W3, 
HiraMinProN-W6, Hiragino Mincho ProN, HiraMinProN-W6, 
HiraginoSans-W0, Hiragino Sans, HiraginoSans-W0, 
HiraginoSans-W1, Hiragino Sans, HiraginoSans-W1, 
HiraginoSans-W2, Hiragino Sans, HiraginoSans-W2, 
HiraginoSans-W3, Hiragino Sans, HiraginoSans-W3, 
HiraginoSans-W4, Hiragino Sans, HiraginoSans-W4, 
HiraginoSans-W5, Hiragino Sans, HiraginoSans-W5, 
HiraginoSans-W6, Hiragino Sans, HiraginoSans-W6, 
HiraginoSans-W7, Hiragino Sans, HiraginoSans-W7, 
HiraginoSans-W8, Hiragino Sans, HiraginoSans-W8, 
HiraginoSans-W9, Hiragino Sans, HiraginoSans-W9, 
HiraginoSansCNS-W3, Hiragino Sans CNS, HiraginoSansCNS-W3, 
HiraginoSansCNS-W6, Hiragino Sans CNS, HiraginoSansCNS-W6, 
HiraginoSansGB-W3, Hiragino Sans GB, HiraginoSansGB-W3, 
HiraginoSansGB-W6, Hiragino Sans GB, HiraginoSansGB-W6, 
HoeflerText-Black, Hoefler Text, HoeflerText-Black, 
HoeflerText-BlackItalic, Hoefler Text, HoeflerText-BlackItalic, 
HoeflerText-Italic, Hoefler Text, HoeflerText-Italic, 
HoeflerText-Ornaments, Hoefler Text, HoeflerText-Ornaments, 
HoeflerText-Regular, Hoefler Text, HoeflerText-Regular, 
ITFDevanagari-Bold, ITF Devanagari, ITFDevanagari-Bold, 
ITFDevanagari-Book, ITF Devanagari, ITFDevanagari-Book, 
ITFDevanagari-Demi, ITF Devanagari, ITFDevanagari-Demi, 
ITFDevanagari-Light, ITF Devanagari, ITFDevanagari-Light, 
ITFDevanagari-Medium, ITF Devanagari, ITFDevanagari-Medium, 
ITFDevanagariMarathi-Bold, ITF Devanagari Marathi, ITFDevanagariMarathi-Bold, 
ITFDevanagariMarathi-Book, ITF Devanagari Marathi, ITFDevanagariMarathi-Book, 
ITFDevanagariMarathi-Demi, ITF Devanagari Marathi, ITFDevanagariMarathi-Demi, 
ITFDevanagariMarathi-Light, ITF Devanagari Marathi, ITFDevanagariMarathi-Light, 
ITFDevanagariMarathi-Medium, ITF Devanagari Marathi, ITFDevanagariMarathi-Medium, 
Impact, Impact, Impact, 
InaiMathi, InaiMathi, InaiMathi, 
InaiMathi-Bold, InaiMathi, InaiMathi-Bold, 
IowanOldStyle-Black, Iowan Old Style, IowanOldStyle-Black, 
IowanOldStyle-BlackItalic, Iowan Old Style, IowanOldStyle-BlackItalic, 
IowanOldStyle-Bold, Iowan Old Style, IowanOldStyle-Bold, 
IowanOldStyle-BoldItalic, Iowan Old Style, IowanOldStyle-BoldItalic, 
IowanOldStyle-Italic, Iowan Old Style, IowanOldStyle-Italic, 
IowanOldStyle-Roman, Iowan Old Style, IowanOldStyle-Roman, 
IowanOldStyle-Titling, Iowan Old Style, IowanOldStyle-Titling, 
JCHEadA, HeadLineA, JCHEadA, 
JCfg, PilGi, JCfg, 
JCkg, GungSeo, JCkg, 
JCsmPC, PCMyungjo, JCsmPC, 
Kailasa, Kailasa, Kailasa, 
Kailasa-Bold, Kailasa, Kailasa-Bold, 
KannadaMN, Kannada MN, KannadaMN, 
KannadaMN-Bold, Kannada MN, KannadaMN-Bold, 
KannadaSangamMN, Kannada Sangam MN, KannadaSangamMN, 
KannadaSangamMN-Bold, Kannada Sangam MN, KannadaSangamMN-Bold, 
Kefa-Bold, Kefa, Kefa-Bold, 
Kefa-Regular, Kefa, Kefa-Regular, 
KhmerMN, Khmer MN, KhmerMN, 
KhmerMN-Bold, Khmer MN, KhmerMN-Bold, 
KhmerSangamMN, Khmer Sangam MN, KhmerSangamMN, 
Klee-Demibold, Klee, Klee-Demibold, 
Klee-Medium, Klee, Klee-Medium, 
KohinoorBangla-Bold, Kohinoor Bangla, KohinoorBangla-Bold, 
KohinoorBangla-Light, Kohinoor Bangla, KohinoorBangla-Light, 
KohinoorBangla-Medium, Kohinoor Bangla, KohinoorBangla-Medium, 
KohinoorBangla-Regular, Kohinoor Bangla, KohinoorBangla-Regular, 
KohinoorBangla-Semibold, Kohinoor Bangla, KohinoorBangla-Semibold, 
KohinoorDevanagari-Bold, Kohinoor Devanagari, KohinoorDevanagari-Bold, 
KohinoorDevanagari-Light, Kohinoor Devanagari, KohinoorDevanagari-Light, 
KohinoorDevanagari-Medium, Kohinoor Devanagari, KohinoorDevanagari-Medium, 
KohinoorDevanagari-Regular, Kohinoor Devanagari, KohinoorDevanagari-Regular, 
KohinoorDevanagari-Semibold, Kohinoor Devanagari, KohinoorDevanagari-Semibold, 
KohinoorTelugu-Bold, Kohinoor Telugu, KohinoorTelugu-Bold, 
KohinoorTelugu-Light, Kohinoor Telugu, KohinoorTelugu-Light, 
KohinoorTelugu-Medium, Kohinoor Telugu, KohinoorTelugu-Medium, 
KohinoorTelugu-Regular, Kohinoor Telugu, KohinoorTelugu-Regular, 
KohinoorTelugu-Semibold, Kohinoor Telugu, KohinoorTelugu-Semibold, 
Kokonor, Kokonor, Kokonor, 
Krungthep, Krungthep, Krungthep, 
KufiStandardGK, KufiStandardGK, KufiStandardGK, 
LaoMN, Lao MN, LaoMN, 
LaoMN-Bold, Lao MN, LaoMN-Bold, 
LaoSangamMN, Lao Sangam MN, LaoSangamMN, 
LiGothicMed, Apple LiGothic, LiGothicMed, 
LiHeiPro, LiHei Pro, LiHeiPro, 
LiSongPro, LiSong Pro, LiSongPro, 
LiSungLight, Apple LiSung, LiSungLight, 
Lucida Bright Demibold, Lucida Bright, Lucida Bright Demibold, 
Lucida Bright Demibold Italic, Lucida Bright, Lucida Bright Demibold Italic, 
Lucida Bright Italic, Lucida Bright, Lucida Bright Italic, 
Lucida Bright Regular, Lucida Bright, Lucida Bright Regular, 
Lucida Sans Demibold, Lucida Sans, Lucida Sans Demibold, 
Lucida Sans Regular, Lucida Sans, Lucida Sans Regular, 
Lucida Sans Typewriter Bold, Lucida Sans Typewriter, Lucida Sans Typewriter Bold, 
Lucida Sans Typewriter Regular, Lucida Sans Typewriter, Lucida Sans Typewriter Regular, 
LucidaBright, Lucida Bright, LucidaBright, 
LucidaBright-Demi, Lucida Bright, LucidaBright-Demi, 
LucidaBright-DemiItalic, Lucida Bright, LucidaBright-DemiItalic, 
LucidaBright-Italic, Lucida Bright, LucidaBright-Italic, 
LucidaGrande, Lucida Grande, LucidaGrande, 
LucidaGrande-Bold, Lucida Grande, LucidaGrande-Bold, 
LucidaSans, Lucida Sans, LucidaSans, 
LucidaSans-Demi, Lucida Sans, LucidaSans-Demi, 
LucidaSans-Typewriter, Lucida Sans Typewriter, LucidaSans-Typewriter, 
LucidaSans-TypewriterBold, Lucida Sans Typewriter, LucidaSans-TypewriterBold, 
Luminari-Regular, Luminari, Luminari-Regular, 
MLingWaiMedium-SC, LingWai SC, MLingWaiMedium-SC, 
MLingWaiMedium-TC, LingWai TC, MLingWaiMedium-TC, 
MalayalamMN, Malayalam MN, MalayalamMN, 
MalayalamMN-Bold, Malayalam MN, MalayalamMN-Bold, 
MalayalamSangamMN, Malayalam Sangam MN, MalayalamSangamMN, 
MalayalamSangamMN-Bold, Malayalam Sangam MN, MalayalamSangamMN-Bold, 
Marion-Bold, Marion, Marion-Bold, 
Marion-Italic, Marion, Marion-Italic, 
Marion-Regular, Marion, Marion-Regular, 
MarkerFelt-Thin, Marker Felt, MarkerFelt-Thin, 
MarkerFelt-Wide, Marker Felt, MarkerFelt-Wide, 
Menlo-Bold, Menlo, Menlo-Bold, 
Menlo-BoldItalic, Menlo, Menlo-BoldItalic, 
Menlo-Italic, Menlo, Menlo-Italic, 
Menlo-Regular, Menlo, Menlo-Regular, 
MicrosoftSansSerif, Microsoft Sans Serif, MicrosoftSansSerif, 
Monaco, Monaco, Monaco, 
MonotypeGurmukhi, Gurmukhi MT, MonotypeGurmukhi, 
Mshtakan, Mshtakan, Mshtakan, 
MshtakanBold, Mshtakan, MshtakanBold, 
MshtakanBoldOblique, Mshtakan, MshtakanBoldOblique, 
MshtakanOblique, Mshtakan, MshtakanOblique, 
Muna, Muna, Muna, 
MunaBlack, Muna, MunaBlack, 
MunaBold, Muna, MunaBold, 
MyanmarMN, Myanmar MN, MyanmarMN, 
MyanmarMN-Bold, Myanmar MN, MyanmarMN-Bold, 
MyanmarSangamMN, Myanmar Sangam MN, MyanmarSangamMN, 
MyanmarSangamMN-Bold, Myanmar Sangam MN, MyanmarSangamMN-Bold, 
Nadeem, Nadeem, Nadeem, 
NanumBrush, Nanum Brush Script, NanumBrush, 
NanumGothic, Nanum Gothic, NanumGothic, 
NanumGothicBold, Nanum Gothic, NanumGothicBold, 
NanumGothicExtraBold, Nanum Gothic, NanumGothicExtraBold, 
NanumMyeongjo, Nanum Myeongjo, NanumMyeongjo, 
NanumMyeongjoBold, Nanum Myeongjo, NanumMyeongjoBold, 
NanumMyeongjoExtraBold, Nanum Myeongjo, NanumMyeongjoExtraBold, 
NanumPen, Nanum Pen Script, NanumPen, 
NewPeninimMT, New Peninim MT, NewPeninimMT, 
NewPeninimMT-Bold, New Peninim MT, NewPeninimMT-Bold, 
NewPeninimMT-BoldInclined, New Peninim MT, NewPeninimMT-BoldInclined, 
NewPeninimMT-Inclined, New Peninim MT, NewPeninimMT-Inclined, 
Noteworthy-Bold, Noteworthy, Noteworthy-Bold, 
Noteworthy-Light, Noteworthy, Noteworthy-Light, 
NotoNastaliqUrdu, Noto Nastaliq Urdu, NotoNastaliqUrdu, 
Optima-Bold, Optima, Optima-Bold, 
Optima-BoldItalic, Optima, Optima-BoldItalic, 
Optima-ExtraBlack, Optima, Optima-ExtraBlack, 
Optima-Italic, Optima, Optima-Italic, 
Optima-Regular, Optima, Optima-Regular, 
OriyaMN, Oriya MN, OriyaMN, 
OriyaMN-Bold, Oriya MN, OriyaMN-Bold, 
OriyaSangamMN, Oriya Sangam MN, OriyaSangamMN, 
OriyaSangamMN-Bold, Oriya Sangam MN, OriyaSangamMN-Bold, 
Osaka, Osaka, Osaka, 
Osaka-Mono, Osaka, Osaka-Mono, 
PTMono-Bold, PT Mono, PTMono-Bold, 
PTMono-Regular, PT Mono, PTMono-Regular, 
PTSans-Bold, PT Sans, PTSans-Bold, 
PTSans-BoldItalic, PT Sans, PTSans-BoldItalic, 
PTSans-Caption, PT Sans Caption, PTSans-Caption, 
PTSans-CaptionBold, PT Sans Caption, PTSans-CaptionBold, 
PTSans-Italic, PT Sans, PTSans-Italic, 
PTSans-Narrow, PT Sans Narrow, PTSans-Narrow, 
PTSans-NarrowBold, PT Sans Narrow, PTSans-NarrowBold, 
PTSans-Regular, PT Sans, PTSans-Regular, 
PTSerif-Bold, PT Serif, PTSerif-Bold, 
PTSerif-BoldItalic, PT Serif, PTSerif-BoldItalic, 
PTSerif-Caption, PT Serif Caption, PTSerif-Caption, 
PTSerif-CaptionItalic, PT Serif Caption, PTSerif-CaptionItalic, 
PTSerif-Italic, PT Serif, PTSerif-Italic, 
PTSerif-Regular, PT Serif, PTSerif-Regular, 
Palatino-Bold, Palatino, Palatino-Bold, 
Palatino-BoldItalic, Palatino, Palatino-BoldItalic, 
Palatino-Italic, Palatino, Palatino-Italic, 
Palatino-Roman, Palatino, Palatino-Roman, 
Papyrus, Papyrus, Papyrus, 
Papyrus-Condensed, Papyrus, Papyrus-Condensed, 
Phosphate-Inline, Phosphate, Phosphate-Inline, 
Phosphate-Solid, Phosphate, Phosphate-Solid, 
PingFangHK-Light, PingFang HK, PingFangHK-Light, 
PingFangHK-Medium, PingFang HK, PingFangHK-Medium, 
PingFangHK-Regular, PingFang HK, PingFangHK-Regular, 
PingFangHK-Semibold, PingFang HK, PingFangHK-Semibold, 
PingFangHK-Thin, PingFang HK, PingFangHK-Thin, 
PingFangHK-Ultralight, PingFang HK, PingFangHK-Ultralight, 
PingFangSC-Light, PingFang SC, PingFangSC-Light, 
PingFangSC-Medium, PingFang SC, PingFangSC-Medium, 
PingFangSC-Regular, PingFang SC, PingFangSC-Regular, 
PingFangSC-Semibold, PingFang SC, PingFangSC-Semibold, 
PingFangSC-Thin, PingFang SC, PingFangSC-Thin, 
PingFangSC-Ultralight, PingFang SC, PingFangSC-Ultralight, 
PingFangTC-Light, PingFang TC, PingFangTC-Light, 
PingFangTC-Medium, PingFang TC, PingFangTC-Medium, 
PingFangTC-Regular, PingFang TC, PingFangTC-Regular, 
PingFangTC-Semibold, PingFang TC, PingFangTC-Semibold, 
PingFangTC-Thin, PingFang TC, PingFangTC-Thin, 
PingFangTC-Ultralight, PingFang TC, PingFangTC-Ultralight, 
PlantagenetCherokee, Plantagenet Cherokee, PlantagenetCherokee, 
Raanana, Raanana, Raanana, 
RaananaBold, Raanana, RaananaBold, 
SIL-Hei-Med-Jian, Hei, SIL-Hei-Med-Jian, 
SIL-Kai-Reg-Jian, Kai, SIL-Kai-Reg-Jian, 
STBaoliSC-Regular, Baoli SC, STBaoliSC-Regular, 
STBaoliTC-Regular, Baoli TC, STBaoliTC-Regular, 
STFangsong, STFangsong, STFangsong, 
STHeiti, STHeiti, STHeiti, 
STHeitiSC-Light, Heiti SC, STHeitiSC-Light, 
STHeitiSC-Medium, Heiti SC, STHeitiSC-Medium, 
STHeitiTC-Light, Heiti TC, STHeitiTC-Light, 
STHeitiTC-Medium, Heiti TC, STHeitiTC-Medium, 
STIXGeneral-Bold, STIXGeneral, STIXGeneral-Bold, 
STIXGeneral-BoldItalic, STIXGeneral, STIXGeneral-BoldItalic, 
STIXGeneral-Italic, STIXGeneral, STIXGeneral-Italic, 
STIXGeneral-Regular, STIXGeneral, STIXGeneral-Regular, 
STIXIntegralsD-Bold, STIXIntegralsD, STIXIntegralsD-Bold, 
STIXIntegralsD-Regular, STIXIntegralsD, STIXIntegralsD-Regular, 
STIXIntegralsSm-Bold, STIXIntegralsSm, STIXIntegralsSm-Bold, 
STIXIntegralsSm-Regular, STIXIntegralsSm, STIXIntegralsSm-Regular, 
STIXIntegralsUp-Bold, STIXIntegralsUp, STIXIntegralsUp-Bold, 
STIXIntegralsUp-Regular, STIXIntegralsUp, STIXIntegralsUp-Regular, 
STIXIntegralsUpD-Bold, STIXIntegralsUpD, STIXIntegralsUpD-Bold, 
STIXIntegralsUpD-Regular, STIXIntegralsUpD, STIXIntegralsUpD-Regular, 
STIXIntegralsUpSm-Bold, STIXIntegralsUpSm, STIXIntegralsUpSm-Bold, 
STIXIntegralsUpSm-Regular, STIXIntegralsUpSm, STIXIntegralsUpSm-Regular, 
STIXNonUnicode-Bold, STIXNonUnicode, STIXNonUnicode-Bold, 
STIXNonUnicode-BoldItalic, STIXNonUnicode, STIXNonUnicode-BoldItalic, 
STIXNonUnicode-Italic, STIXNonUnicode, STIXNonUnicode-Italic, 
STIXNonUnicode-Regular, STIXNonUnicode, STIXNonUnicode-Regular, 
STIXSizeFiveSym-Regular, STIXSizeFiveSym, STIXSizeFiveSym-Regular, 
STIXSizeFourSym-Bold, STIXSizeFourSym, STIXSizeFourSym-Bold, 
STIXSizeFourSym-Regular, STIXSizeFourSym, STIXSizeFourSym-Regular, 
STIXSizeOneSym-Bold, STIXSizeOneSym, STIXSizeOneSym-Bold, 
STIXSizeOneSym-Regular, STIXSizeOneSym, STIXSizeOneSym-Regular, 
STIXSizeThreeSym-Bold, STIXSizeThreeSym, STIXSizeThreeSym-Bold, 
STIXSizeThreeSym-Regular, STIXSizeThreeSym, STIXSizeThreeSym-Regular, 
STIXSizeTwoSym-Bold, STIXSizeTwoSym, STIXSizeTwoSym-Bold, 
STIXSizeTwoSym-Regular, STIXSizeTwoSym, STIXSizeTwoSym-Regular, 
STIXVariants-Bold, STIXVariants, STIXVariants-Bold, 
STIXVariants-Regular, STIXVariants, STIXVariants-Regular, 
STKaiti, STKaiti, STKaiti, 
STKaitiSC-Black, Kaiti SC, STKaitiSC-Black, 
STKaitiSC-Bold, Kaiti SC, STKaitiSC-Bold, 
STKaitiSC-Regular, Kaiti SC, STKaitiSC-Regular, 
STKaitiTC-Black, Kaiti TC, STKaitiTC-Black, 
STKaitiTC-Bold, Kaiti TC, STKaitiTC-Bold, 
STKaitiTC-Regular, Kaiti TC, STKaitiTC-Regular, 
STLibianSC-Regular, Libian SC, STLibianSC-Regular, 
STLibianTC-Regular, Libian TC, STLibianTC-Regular, 
STSong, STSong, STSong, 
STSongti-SC-Black, Songti SC, STSongti-SC-Black, 
STSongti-SC-Bold, Songti SC, STSongti-SC-Bold, 
STSongti-SC-Light, Songti SC, STSongti-SC-Light, 
STSongti-SC-Regular, Songti SC, STSongti-SC-Regular, 
STSongti-TC-Bold, Songti TC, STSongti-TC-Bold, 
STSongti-TC-Light, Songti TC, STSongti-TC-Light, 
STSongti-TC-Regular, Songti TC, STSongti-TC-Regular, 
STXihei, STHeiti, STXihei, 
STXingkaiSC-Bold, Xingkai SC, STXingkaiSC-Bold, 
STXingkaiSC-Light, Xingkai SC, STXingkaiSC-Light, 
STXingkaiTC-Bold, Xingkai TC, STXingkaiTC-Bold, 
STXingkaiTC-Light, Xingkai TC, STXingkaiTC-Light, 
STYuanti-SC-Bold, Yuanti SC, STYuanti-SC-Bold, 
STYuanti-SC-Light, Yuanti SC, STYuanti-SC-Light, 
STYuanti-SC-Regular, Yuanti SC, STYuanti-SC-Regular, 
STYuanti-TC-Bold, Yuanti TC, STYuanti-TC-Bold, 
STYuanti-TC-Light, Yuanti TC, STYuanti-TC-Light, 
STYuanti-TC-Regular, Yuanti TC, STYuanti-TC-Regular, 
Sana, Sana, Sana, 
Sathu, Sathu, Sathu, 
SavoyeLetPlain, Savoye LET, SavoyeLetPlain, 
Seravek, Seravek, Seravek, 
Seravek-Bold, Seravek, Seravek-Bold, 
Seravek-BoldItalic, Seravek, Seravek-BoldItalic, 
Seravek-ExtraLight, Seravek, Seravek-ExtraLight, 
Seravek-ExtraLightItalic, Seravek, Seravek-ExtraLightItalic, 
Seravek-Italic, Seravek, Seravek-Italic, 
Seravek-Light, Seravek, Seravek-Light, 
Seravek-LightItalic, Seravek, Seravek-LightItalic, 
Seravek-Medium, Seravek, Seravek-Medium, 
Seravek-MediumItalic, Seravek, Seravek-MediumItalic, 
ShreeDev0714, Shree Devanagari 714, ShreeDev0714, 
ShreeDev0714-Bold, Shree Devanagari 714, ShreeDev0714-Bold, 
ShreeDev0714-BoldItalic, Shree Devanagari 714, ShreeDev0714-BoldItalic, 
ShreeDev0714-Italic, Shree Devanagari 714, ShreeDev0714-Italic, 
SignPainter-HouseScript, SignPainter, SignPainter-HouseScript, 
SignPainter-HouseScriptSemibold, SignPainter, SignPainter-HouseScriptSemibold, 
Silom, Silom, Silom, 
SinhalaMN, Sinhala MN, SinhalaMN, 
SinhalaMN-Bold, Sinhala MN, SinhalaMN-Bold, 
SinhalaSangamMN, Sinhala Sangam MN, SinhalaSangamMN, 
SinhalaSangamMN-Bold, Sinhala Sangam MN, SinhalaSangamMN-Bold, 
Skia-Regular, Skia, Skia-Regular, 
Skia-Regular_Black, Skia, Skia-Regular_Black, 
Skia-Regular_Black-Condensed, Skia, Skia-Regular_Black-Condensed, 
Skia-Regular_Black-Extended, Skia, Skia-Regular_Black-Extended, 
Skia-Regular_Bold, Skia, Skia-Regular_Bold, 
Skia-Regular_Condensed, Skia, Skia-Regular_Condensed, 
Skia-Regular_Extended, Skia, Skia-Regular_Extended, 
Skia-Regular_Light, Skia, Skia-Regular_Light, 
Skia-Regular_Light-Condensed, Skia, Skia-Regular_Light-Condensed, 
Skia-Regular_Light-Extended, Skia, Skia-Regular_Light-Extended, 
SnellRoundhand, Snell Roundhand, SnellRoundhand, 
SnellRoundhand-Black, Snell Roundhand, SnellRoundhand-Black, 
SnellRoundhand-Bold, Snell Roundhand, SnellRoundhand-Bold, 
SukhumvitSet-Bold, Sukhumvit Set, SukhumvitSet-Bold, 
SukhumvitSet-Light, Sukhumvit Set, SukhumvitSet-Light, 
SukhumvitSet-Medium, Sukhumvit Set, SukhumvitSet-Medium, 
SukhumvitSet-SemiBold, Sukhumvit Set, SukhumvitSet-SemiBold, 
SukhumvitSet-Text, Sukhumvit Set, SukhumvitSet-Text, 
SukhumvitSet-Thin, Sukhumvit Set, SukhumvitSet-Thin, 
Superclarendon-Black, Superclarendon, Superclarendon-Black, 
Superclarendon-BlackItalic, Superclarendon, Superclarendon-BlackItalic, 
Superclarendon-Bold, Superclarendon, Superclarendon-Bold, 
Superclarendon-BoldItalic, Superclarendon, Superclarendon-BoldItalic, 
Superclarendon-Italic, Superclarendon, Superclarendon-Italic, 
Superclarendon-Light, Superclarendon, Superclarendon-Light, 
Superclarendon-LightItalic, Superclarendon, Superclarendon-LightItalic, 
Superclarendon-Regular, Superclarendon, Superclarendon-Regular, 
Symbol, Symbol, Symbol, 
Tahoma, Tahoma, Tahoma, 
Tahoma-Bold, Tahoma, Tahoma-Bold, 
TamilMN, Tamil MN, TamilMN, 
TamilMN-Bold, Tamil MN, TamilMN-Bold, 
TamilSangamMN, Tamil Sangam MN, TamilSangamMN, 
TamilSangamMN-Bold, Tamil Sangam MN, TamilSangamMN-Bold, 
TeluguMN, Telugu MN, TeluguMN, 
TeluguMN-Bold, Telugu MN, TeluguMN-Bold, 
TeluguSangamMN, Telugu Sangam MN, TeluguSangamMN, 
TeluguSangamMN-Bold, Telugu Sangam MN, TeluguSangamMN-Bold, 
Thonburi, Thonburi, Thonburi, 
Thonburi-Bold, Thonburi, Thonburi-Bold, 
Thonburi-Light, Thonburi, Thonburi-Light, 
Times-Bold, Times, Times-Bold, 
Times-BoldItalic, Times, Times-BoldItalic, 
Times-Italic, Times, Times-Italic, 
Times-Roman, Times, Times-Roman, 
TimesNewRomanPS-BoldItalicMT, Times New Roman, TimesNewRomanPS-BoldItalicMT, 
TimesNewRomanPS-BoldMT, Times New Roman, TimesNewRomanPS-BoldMT, 
TimesNewRomanPS-ItalicMT, Times New Roman, TimesNewRomanPS-ItalicMT, 
TimesNewRomanPSMT, Times New Roman, TimesNewRomanPSMT, 
ToppanBunkyuGothicPr6N-DB, Toppan Bunkyu Gothic, ToppanBunkyuGothicPr6N-DB, 
ToppanBunkyuGothicPr6N-Regular, Toppan Bunkyu Gothic, ToppanBunkyuGothicPr6N-Regular, 
ToppanBunkyuMidashiGothicStdN-ExtraBold, Toppan Bunkyu Midashi Gothic, ToppanBunkyuMidashiGothicStdN-ExtraBold, 
ToppanBunkyuMidashiMinchoStdN-ExtraBold, Toppan Bunkyu Midashi Mincho, ToppanBunkyuMidashiMinchoStdN-ExtraBold, 
ToppanBunkyuMinchoPr6N-Regular, Toppan Bunkyu Mincho, ToppanBunkyuMinchoPr6N-Regular, 
Trattatello, Trattatello, Trattatello, 
Trebuchet-BoldItalic, Trebuchet MS, Trebuchet-BoldItalic, 
TrebuchetMS, Trebuchet MS, TrebuchetMS, 
TrebuchetMS-Bold, Trebuchet MS, TrebuchetMS-Bold, 
TrebuchetMS-Italic, Trebuchet MS, TrebuchetMS-Italic, 
TsukuARdGothic-Bold, Tsukushi A Round Gothic, TsukuARdGothic-Bold, 
TsukuARdGothic-Regular, Tsukushi A Round Gothic, TsukuARdGothic-Regular, 
TsukuBRdGothic-Bold, Tsukushi B Round Gothic, TsukuBRdGothic-Bold, 
TsukuBRdGothic-Regular, Tsukushi B Round Gothic, TsukuBRdGothic-Regular, 
Verdana, Verdana, Verdana, 
Verdana-Bold, Verdana, Verdana-Bold, 
Verdana-BoldItalic, Verdana, Verdana-BoldItalic, 
Verdana-Italic, Verdana, Verdana-Italic, 
Waseem, Waseem, Waseem, 
WaseemLight, Waseem, WaseemLight, 
Webdings, Webdings, Webdings, 
WeibeiSC-Bold, Weibei SC, WeibeiSC-Bold, 
WeibeiTC-Bold, Weibei TC, WeibeiTC-Bold, 
Wingdings-Regular, Wingdings, Wingdings-Regular, 
Wingdings2, Wingdings 2, Wingdings2, 
Wingdings3, Wingdings 3, Wingdings3, 
YuGo-Bold, YuGothic, YuGo-Bold, 
YuGo-Medium, YuGothic, YuGo-Medium, 
YuKyo-Bold, YuKyokasho, YuKyo-Bold, 
YuKyo-Medium, YuKyokasho, YuKyo-Medium, 
YuKyo_Yoko-Bold, YuKyokasho Yoko, YuKyo_Yoko-Bold, 
YuKyo_Yoko-Medium, YuKyokasho Yoko, YuKyo_Yoko-Medium, 
YuMin-Demibold, YuMincho, YuMin-Demibold, 
YuMin-Extrabold, YuMincho, YuMin-Extrabold, 
YuMin-Medium, YuMincho, YuMin-Medium, 
YuMin_36pKn-Demibold, YuMincho +36p Kana, YuMin_36pKn-Demibold, 
YuMin_36pKn-Extrabold, YuMincho +36p Kana, YuMin_36pKn-Extrabold, 
YuMin_36pKn-Medium, YuMincho +36p Kana, YuMin_36pKn-Medium, 
YuppySC-Regular, Yuppy SC, YuppySC-Regular, 
YuppyTC-Regular, Yuppy TC, YuppyTC-Regular, 
ZapfDingbatsITC, Zapf Dingbats, ZapfDingbatsITC, 
Zapfino, Zapfino, Zapfino, 

GoogleObjectDetectionAPI : SSD(Single Shot MultiBox Detector)をローカル環境で動かす方法

1. 前提

   ・原論文(英語)  :https://arxiv.org/pdf/1512.02325.pdf
   ・日本語訳    :https://qiita.com/de0ta/items/1ae60878c0e177fc7a3a

  • ローカル環境で、SSDを動かす方法(Google TensorFlow - Github

github.com


  • git clone でTensorFlowの公式ブランチをローカルにコピーしておくと楽かも。
$ git clone https://github.com/tensorflow/models/


2. Githubから必要なファイルをダウンロード


① Configファイル(JSON形式)
以下のファイルをローカル環境にDownloadする。

download link:https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/ssd_mobilenet_v1_coco.config


② モデルの学習済み特徴量
以下のファイルをローカル環境にDownloadする。

download link:https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

Downloadしたら、ssd_moblienet_v1_coco.configを開き、ファイルの該当箇所にローカルのPATHを指定する。

fine_tune_checkpoint: "PATH_TO_BE_CONFIGURED/model.ckpt"


③ 元のデータセット
以下のファイルをローカル環境にDownloadする。
download link:https://github.com/tensorflow/models/blob/master/research/object_detection/dataset_tools/download_and_preprocess_mscoco.sh

Downloadしたら、ssd_moblienet_v1_coco.configを開き、ファイルの該当箇所にローカルのPATH(.shファイルがある場所)を指定する。

input_path: "PATH_TO_BE_CONFIGURED/mscoco_train.record-?????-of-00100"


download_and_preprocess_mscoco.shは、元データをTensorFlowが読み込めるデータセットに変換してくれるshellスクリプト。これを実行しておく。(10h~15hかかる)

$ /hogehoge/hoge/download_and_preprocess_mscoco.sh

④ ベクトルのラベルと名前の紐付け(ex. 0 → "cat")
以下のファイルをローカル環境にDownloadする。
download link:https://github.com/tensorflow/models/blob/master/research/object_detection/data/mscoco_label_map.pbtxt

Downloadしたら、ssd_moblienet_v1_coco.configを開き、ファイルの該当箇所にローカルのPATH(.pbtxtファイルがある場所)を指定する。

label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"


⑤ 実行ファイル(.py)を動かす。
・ローカル環境で、SSDを動かす方法は以下のGithubリンクに載っている。
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_locally.md

Tensorflow : MNISTを小規模なCNNで解いてみる

TensorFlowの練習がてら。
分類精度(accuracy)は98.9%

The simple implementation in python 3.6.6 with Tensorflow 1.9.0.


1.プログラム

# coding: utf-8

import tensorflow as tf
from tensorflow.contrib.learn.python.learn.datasets import mnist
from tensorflow.contrib.learn.python.learn.metric_spec import MetricSpec

learn = tf.contrib.learn
slim  = tf.contrib.slim


# モデル定義(Convolutional Neural Network)
def cnn(x, y):
    x = tf.reshape(x, [-1, 28, 28, 1])
    y = slim.one_hot_encoding(y, 10)

    #data→conv→pool→conv→pool→full→full→(softmax)→cls
    net = slim.conv2d(x,   48, [5, 5], scope = 'conv1')
    net = slim.max_pool2d(net, [2, 2], scope = 'pool1')
    net = slim.conv2d(net, 96, [5, 5], scope = 'conv2')
    net = slim.max_pool2d(net, [2, 2], scope = 'pool2')
    net = slim.flatten(net, scope = 'flatten')
    net = slim.fully_connected(net, 512, scope = 'fully_connected1')
    logits = slim.fully_connected(net, 10, activation_fn = None, scope = 'fully_connected2')
    prob = slim.softmax(logits)
    loss = slim.losses.softmax_cross_entropy(logits, y)
    train_op = slim.optimize_loss(loss, slim.get_global_step(), learning_rate = 0.001, optimizer = 'Adam')
    return {'class': tf.argmax(prob, 1), 'prob': prob}, loss, train_op


# データの読み込み
data_sets = mnist.read_data_sets('/tmp/mnist', one_hot = False)

# 変数のセット
X_train = data_sets.train.images
Y_train = data_sets.train.labels
X_test = data_sets.validation.images
Y_test = data_sets.validation.labels

# 学習ログ(validation)をコンソールに表示させる
tf.logging.set_verbosity(tf.logging.INFO)
validation_metrics = {"accuracy" : MetricSpec(metric_fn = tf.contrib.metrics.streaming_accuracy, prediction_key = "class")}
validation_monitor = learn.monitors.ValidationMonitor(X_test, Y_test, metrics = validation_metrics, every_n_steps = 100)

# 学習実行
classifier = learn.Estimator(model_fn = cnn, model_dir = '/tmp/cnn_log', config = learn.RunConfig(save_checkpoints_secs = 10))
classifier.fit(x = X_train, y = Y_train, steps = 3200, batch_size = 64, monitors = [validation_monitor])


2.ログ(コンソール画面)

ログはこんな感じ↓

...

INFO:tensorflow:global_step/sec: 2.20219
INFO:tensorflow:Starting evaluation at 2018-07-18-07:07:56
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/cnn_log/model.ckpt-3250
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2018-07-18-07:08:06
INFO:tensorflow:Saving dict for global step 3250: accuracy = 0.9898, global_step = 3250, loss = 0.03569727
INFO:tensorflow:Validation (step 3280): accuracy = 0.9898, loss = 0.03569727, global_step = 3250
INFO:tensorflow:loss = 0.0060795164, step = 3280 (40.518 sec)
INFO:tensorflow:Saving checkpoints for 3281 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Saving checkpoints for 3316 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Saving checkpoints for 3346 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Saving checkpoints for 3376 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:global_step/sec: 2.33117
INFO:tensorflow:Starting evaluation at 2018-07-18-07:08:39
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/cnn_log/model.ckpt-3376
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2018-07-18-07:08:51
INFO:tensorflow:Saving dict for global step 3376: accuracy = 0.9894, global_step = 3376, loss = 0.03725804
INFO:tensorflow:Validation (step 3380): accuracy = 0.9894, loss = 0.03725804, global_step = 3376
INFO:tensorflow:loss = 0.005827286, step = 3380 (45.278 sec)
INFO:tensorflow:Saving checkpoints for 3381 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Saving checkpoints for 3415 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Saving checkpoints for 3449 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Saving checkpoints for 3479 into /tmp/cnn_log/model.ckpt.
INFO:tensorflow:Loss for final step: 0.0017738211.

参考:
qiita.com

deepage.net

Stanford University CS231n: Convolutional Neural Networks for Visual Recognition