Modeling documents with Generative Adversarial Networks

Goal

GAN performence with nlp tasks, especially discovering methods for distributed representation documents using GAN.

Model

facts:

1. Energy based model perform better than probabilstic model
2. Denoising AutoEncoder within the discriminator to learn the document representation

Experiments

1. comparision against DocNade

2. t-sne doc representation

Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Goal

a framework for general question answering tasks trained using (input, question, answer).

Tasks

sequence tagging, classification, sequence-to-sequence(seq2seq), question answering(QA) that requires transiitve reasoning.

Model

Input Module

encode raw inputs(images, texts(sentences, story, tweets…etc) ) to distibuted vector representation.

GRU is used instead of LSTM for computation efficiency with similar performence.

Question Module

encode raw question of tasks. Then fed into espsodic memory module(EMM) as initial state/basis for it(EMM) to iterates.

Share word embedding matrix accross Input module and Question Module.

Episodic Memory Module

Iteratively attend different part of the input** given the current state.

(share some similarity to hops between controller module and memory module from End-To-End Memory Networks(MemN2N))

Compromised of an attention mechanism within a recurrent network.

Score function G of the answer is an 2-layer feed forward Neural Network as well.

Criteria for stopping includes a end-of-passes representation for the GRU to choose as well as a maximum number of iteration just in case.

Answer Module

Generate answer from final memory vector from EMM.

Training

Experiments

###QA

TODO: Experiments is not complete

Key-Value Memory Networks for Directly Reading Documents

Goal

Model goal: towards working on knowledge source from either unstructured documents or structured knowledge base such as FreeBase.

Structure goal: Storing key-value structured memory before reasoning, keys to address relevant memories to the question, value as facts supporting reasoning. In other words, key feature vectors should be design to match the question and value feature vectors should be design to match (partial) answers.

Model

Memory

key

key hashing is used to pre-select a subset of questions sharing same words excluding stop words. key addressing is used to assign probabilites to each element(key) in the set of the key hashing results.

value

value reading is then the weighted sum of the addressing probabilities on values features.

controller

as the query for each hops, query is updated for each hops back from memory network. key hashing is not performed during those hops.

Like previous MemNN and MemN2N, controller computed a final prediction.

query, anser,keys and values are free to define to suit the best performence of the dataset/task.

Train

key-value transforms while using standard backpropagation via SGD. key-value transforms is

Experiments

dataset: WIKIMOVIES, WIKIQA

WIKIQA:

##My questions

1. why number of hops can not be learned by a network?