BH13.13/DataMapping

提供:TogoWiki

移動: 案内, 検索

目次

データマッピング自動化

  • 森(東工大), 山本(DBCLS), 岡本(DBCLS)

背景とスコープ

  • 菌株メタデータへのオントロジー/統制語彙の付与を自動化
  • 曖昧検索+語彙推薦
  • マニュアルマッピングのコスト低減のためにすぐにでも必要
  • とりあえず、既存NLPプログラムを組み合わせて処理系作る

処理系の概略

  • 個別の処理ステップの明確化(分割して使える、戻れる)
  • 多様なデータセット・辞書への対応
  • 完全一致は自動で置き換え
  • 曖昧一致はfalse positiveを許して語彙を推薦(スコアリング?)
  • アルゴリズム:text scan, n-grams and cosine similarity
  • 辞書:オントロジー(辞書)、マッピング正解セット

今後の課題

  • 辞書にGAZ, NCBI Taxnomyを加える
  • text scan時のパース粒度を調整する
  • マッピング正解セットの適応

パイロットワークの評価

  • データ:JCMのhabitatデータ、森さんによるマニュアルマッピング
  • 手法:text scan, bigram+cos similarityで各オントロジー(辞書)に一致させた
  • 結果ファイル:meoetc_match_result20140129.txt


  • text scanは有効
  • 文字列類似度が使えない概念的にはマッピングへは対応出来ない
TR     Creosoted wooden pole     manual     meo     wood
TR     Creosoted wooden pole     csso     no_hit
TR     Creosoted wooden pole     pdo     no_hit
TR     Creosoted wooden pole     fma     no_hit
TR     Creosoted wooden pole     meo     no_hit
  • bigram+cosが有効な例
TR     Buccal cavity     manual     fma     Oral cavity
TR     Buccal cavity     csso     no_hit
TR     Buccal cavity     pdo     no_hit
TR     Buccal cavity     fma     cs     Oral cavity [-1:cavity] @@ Cell cavity [-1:cavity] @@ Nasal cavity [-1:cavity] @@ Antral cavity [-1:cavity] @@ Orbital cavity [-1:cavity]
TR     Buccal cavity     meo     cs     Oral cavity [-1:cavity] @@ Nasal cavity [-1:cavity]
TR     Dried Japanese persimmon     manual     meo     dried persimmon
TR     Dried Japanese persimmon     csso     no_hit
TR     Dried Japanese persimmon     pdo     no_hit
TR     Dried Japanese persimmon     fma     no_hit
TR     Dried Japanese persimmon     meo     cs     dried persimmon [-1:persimmon]
TR     Fermented food     manual     meo     fermented food product
TR     Fermented food     all     ts     meo:food
TR     Fermented food     csso     no_hit
TR     Fermented food     pdo     no_hit
TR     Fermented food     fma     no_hit
TR     Fermented food     meo     cs     fermented food product [-1:food] @@ non-fermented food product [-1:food] @@ fermented meat [15:fermented] @@ fermented must [15:fermented] @@ fermented juice [15:fermented]
TR     Fermenting molasses     manual     meo     fermented molasses
TR     Fermenting molasses     all     ts     meo:molasses
TR     Fermenting molasses     csso     no_hit
TR     Fermenting molasses     pdo     no_hit
TR     Fermenting molasses     fma     no_hit
TR     Fermenting molasses     meo     cs     fermented molasses [-1:molasses] @@ fermented cane molasses [-1:molasses]
TR     Fingernail     manual     fma     Nail of finger
TR     Fingernail     csso     no_hit
TR     Fingernail     pdo     no_hit
TR     Fingernail     fma     cs     Nail of finger [10625:nail] @@ Finger [12838:finger]
TR     Fingernail     meo     cs     Nail of finger [10002:nail] @@ Finger [10003:finger]
  • bigram+cosでfalse positiveな例
TR     Dairy products     manual     meo     dairy product
TR     Dairy products     csso     no_hit
TR     Dairy products     pdo     no_hit
TR     Dairy products     fma     no_hit
TR     Dairy products     meo     cs     dairy product [-1:dairy] @@ honey product [10001:honey] @@ poultry product [10001:poultry]
TR     Dust     manual     meo     waste
TR     Dust     csso     no_hit
TR     Dust     pdo     no_hit
TR     Dust     fma     no_hit
TR     Dust     meo     cs     sawdust [10001:sawdust] #単語内の文字列類似度で違う意味にマッピングされてしまった例
TR     "Ear discharge, Japan"     manual     meo     discharge
TR     "Ear discharge, Japan"     manual     fma     Ear
TR     "Ear discharge, Japan"     all     ts     meo:Ear @ meo:discharge
TR     "Ear discharge, Japan"     csso     cs     Discharge [-1:discharge]
TR     "Ear discharge, Japan"     pdo     no_hit
TR     "Ear discharge, Japan"     fma     no_hit    #短い文字列が拾えない
TR     "Ear discharge, Japan"     meo     cs     discharge [-1:discharge]
TR     Fermenter     manual     meo     man-made structure
TR     Fermenter     csso     no_hit
TR     Fermenter     pdo     no_hit
TR     Fermenter     fma     no_hit
TR     Fermenter     meo     cs     fermented shrimp [10001:shrimp] @@ fermented starch [10001:starch] @@ fermented meat [10003:meat] @@ fermented must [10004:must] @@ fermented juice [10005:juice] #文字類似度は高いが意味が違う
  • tsとbi+cosの結果が矛盾する場合
TR     "Decayed leaf of deciduous tree, Saitama Pref., Japan"     manual     meo     leaf
TR     "Decayed leaf of deciduous tree, Saitama Pref., Japan"     all     ts     meo:leaf @ fma:Tree #どっちの粒度でつける?
TR     "Decayed leaf of deciduous tree, Saitama Pref., Japan"     csso     no_hit
TR     "Decayed leaf of deciduous tree, Saitama Pref., Japan"     pdo     no_hit
TR     "Decayed leaf of deciduous tree, Saitama Pref., Japan"     fma     no_hit
TR     "Decayed leaf of deciduous tree, Saitama Pref., Japan"     meo     no_hit
TR     "Decayed leaf of weed, Japan"     manual     meo     leaf
TR     "Decayed leaf of weed, Japan"     all     ts     meo:leaf
TR     "Decayed leaf of weed, Japan"     csso     no_hit
TR     "Decayed leaf of weed, Japan"     pdo     no_hit
TR     "Decayed leaf of weed, Japan"     fma     no_hit
TR      "Decayed leaf of weed, Japan"     meo     no_hit
TR     "Decayed leaves, Saitama Pref., Japan"     manual     meo     leaf #単複綴り違い問題
TR     "Decayed leaves, Saitama Pref., Japan"     csso     no_hit
TR     "Decayed leaves, Saitama Pref., Japan"     pdo     no_hit
TR     "Decayed leaves, Saitama Pref., Japan"     fma     no_hit
TR     "Decayed leaves, Saitama Pref., Japan"     meo     no_hit
個人用ツール