Permutation invariant neural networks

Sinir ağı verildi $f$ girdi olarak alan $n$ Veri noktaları: $x_1, \dots, x_n$ . Diyoruz $f$ bir permütasyon değişmez halinde

$f (x_{1} . . . x_{n}) = f (p i (x_{1} . . . x_{n}))$ $f(x_1 ... x_n) = f(pi(x_1 ... x_n))$

for any permutation $pi$ .

Could someone recommend a starting point (article, example, or other paper) for permutation invariant neural networks?

neural-networks machine-learning reference-request

— Josef Ondrej
kaynak

Interesting question. You want to treat you inputs as a set rather than a vector. Assuming your inputs are scalar or otherwise comparable, did you consider sorting your inputs to create a permutation-invariant canonical representation and feeding that to a regular network?

— mjul

@mjul My inputs are series of bets of clients (each bet is represented by a vector of some cathegorical and continuous variables). I could order them chronologically for example, but as the time spacing between bets for each client is very different, this wouldn't make much sense. This could probably be solved using some function (fixed or learned) of time that would decay the coefficients for each bet. But I think the ordering really doesn't matter in this case so I wanted to try unordered data first, which obviously requires to treat the bets for each client symmetrically.

— Josef Ondrej

What is the network output? If the network is permutation invariant, the output will be the same for any order of inputs. Is this what you want?

— BlueMoon93

@BlueMoon93 Yes, that is exactly, what I want. The output can be anything (number, vector) as long as it does not depend on order of inputs.

— Josef Ondrej

why would you want a permutation invariant neural network?

— k.c. sayz 'k.c sayz'

Yanıtlar:

As far as I know, no one has tried this, due to the way the network is structured. Each input has a set of weights, that are connected to more inputs. If the inputs switch, the output will too.

However, you can build a network that approaches this behaviour. In your training set, use batch learning and for each training sample, give all possible permutations to the network such that it learns to be permutation invariant. This will never be exactly invariant, it just might be close.

Another way to do this is to have the weights replicated for all inputs. For example, lets assume you have 3 inputs (i0, i1, i2), and the next hidden layer has 2 nodes (hl0, hl1) and activation function F. Assuming a fully connected layer, you have 2 weights w0 and w1. The hidden layer's nodes hl0 and hl1 are given, respectively, by

hl0 = F(i0w0+i1w0+i2w0)
hl1 = F(i0w1+i1w1+i2w1)

Thus giving you a hidden layer whose values are permutation invariant from the input. From now on, you can learn and build the rest of the network as you see fit. This is an approach derived from convolutional layers.

Off-topic, this seems like a cool project. If you want to cooperate on some research project, contact me (check my profile)

— BlueMoon93
kaynak

The first approach suggested would be infeasible in my case due to the computational complexity. The second on the other hand seems maybe too restrictive. But it is certainly a good start. What I have come up with so far is approach similar to the one I found in this paper: arxiv.org/pdf/1612.04530.pdf. First I consider all pairs (generally all k-tuples) of inputs x_i, x_j, i,j in 0 ... n and apply some neural network on all of them (the same nn on each pair). This gives me n**2 outputs f(x_i, x_j) and then I average them (or take maximum) and apply anoter nn over the result.

— Josef Ondrej

This is what I have come up with so far: github.com/josefondrej/Symmetric-Layers

— Josef Ondrej

Here is a few that might be what you are looking for:

Deep Sets, https://papers.nips.cc/paper/6931-deep-sets.pdf
BRUNO: A Deep Recurrent Model for Exchangeable Data, https://arxiv.org/pdf/1802.07535.pdf
Deep Learning with Sets and Point Clouds, https://openreview.net/pdf?id=HJF3iD9xe
Permutation-equivariant neural networks applied to dynamics prediction, https://arxiv.org/pdf/1612.04530.pdf

— elgehelge
kaynak

I have implemented Permutational Layer here using Keras: https://github.com/off99555/superkeras/blob/master/permutational_layer.py

You can call the PermutationalModule function to use it.

Implemented following this paper: https://arxiv.org/pdf/1612.04530.pdf

The idea is to compare all pairs of N^2 pairs from N inputs, use the model with shared weights, then use pooling function N times on N inputs. The output you can use pooling again but in the paper, they don't mention another pooling.

— off99555
kaynak