Permutation invariant neural networks


Sinir ağı verildi f girdi olarak alan n Veri noktaları: x1,...,xn. Diyoruzfbir permütasyon değişmez halinde


for any permutation pi.

Could someone recommend a starting point (article, example, or other paper) for permutation invariant neural networks?

Interesting question. You want to treat you inputs as a set rather than a vector. Assuming your inputs are scalar or otherwise comparable, did you consider sorting your inputs to create a permutation-invariant canonical representation and feeding that to a regular network?

@mjul My inputs are series of bets of clients (each bet is represented by a vector of some cathegorical and continuous variables). I could order them chronologically for example, but as the time spacing between bets for each client is very different, this wouldn't make much sense. This could probably be solved using some function (fixed or learned) of time that would decay the coefficients for each bet. But I think the ordering really doesn't matter in this case so I wanted to try unordered data first, which obviously requires to treat the bets for each client symmetrically.
Josef Ondrej

What is the network output? If the network is permutation invariant, the output will be the same for any order of inputs. Is this what you want?

@BlueMoon93 Yes, that is exactly, what I want. The output can be anything (number, vector) as long as it does not depend on order of inputs.
Josef Ondrej

why would you want a permutation invariant neural network?
k.c. sayz 'k.c sayz'



As far as I know, no one has tried this, due to the way the network is structured. Each input has a set of weights, that are connected to more inputs. If the inputs switch, the output will too.

However, you can build a network that approaches this behaviour. In your training set, use batch learning and for each training sample, give all possible permutations to the network such that it learns to be permutation invariant. This will never be exactly invariant, it just might be close.

Another way to do this is to have the weights replicated for all inputs. For example, lets assume you have 3 inputs (i0, i1, i2), and the next hidden layer has 2 nodes (hl0, hl1) and activation function F. Assuming a fully connected layer, you have 2 weights w0 and w1. The hidden layer's nodes hl0 and hl1 are given, respectively, by

  • hl0 = F(i0w0+i1w0+i2w0)

  • hl1 = F(i0w1+i1w1+i2w1)

Thus giving you a hidden layer whose values are permutation invariant from the input. From now on, you can learn and build the rest of the network as you see fit. This is an approach derived from convolutional layers.

Off-topic, this seems like a cool project. If you want to cooperate on some research project, contact me (check my profile)

The first approach suggested would be infeasible in my case due to the computational complexity. The second on the other hand seems maybe too restrictive. But it is certainly a good start. What I have come up with so far is approach similar to the one I found in this paper: First I consider all pairs (generally all k-tuples) of inputs x_i, x_j, i,j in 0 ... n and apply some neural network on all of them (the same nn on each pair). This gives me n**2 outputs f(x_i, x_j) and then I average them (or take maximum) and apply anoter nn over the result.
Josef Ondrej

This is what I have come up with so far:
Josef Ondrej


I have implemented Permutational Layer here using Keras:

You can call the PermutationalModule function to use it.

Implemented following this paper:

The idea is to compare all pairs of N^2 pairs from N inputs, use the model with shared weights, then use pooling function N times on N inputs. The output you can use pooling again but in the paper, they don't mention another pooling.

Sitemizi kullandığınızda şunları okuyup anladığınızı kabul etmiş olursunuz: Çerez Politikası ve Gizlilik Politikası.
Licensed under cc by-sa 3.0 with attribution required.