For a p2sh multisig transaction, the serialized script looks like this:

m [pubkey] ... [pubkey] n OP_CHECKMULTISIG

The p2sh address is the hash of this script. The public keys can come in
any order, but the hash depends on the order. If you have a list of
public keys, to which address do you send your money? We need a standard
way of sorting the public keys so that the address generated is always
the same for the same public keys and m.

There are two kinds of public keys: compressed and uncompressed.
Uncompressed are longer than compressed.

There are a few obvious ways we could sort the public keys: as strings,
as big endian numbers, as little endian numbers.

The difference is this. Suppose one public key is 59234 (uncompressed),
and the other is 6903 (compressed). If we sort these as strings, then
6903 > 59234. But if we sort them as big endian numbers, then 6903 is
really 06903, and then 06903 < 59234. So it makes a critical difference.
Sorting as little endian is yet another option that is not the same as
the other two.

I noticed Alan Reiner's comment in an earlier message:

"Just like Jean-Pierre mentioned, we'll be using parallel
trees to generate P2SH addresses after sorting the keys
lexicographically."

It sounds like "lexicographically" probably means sorting as strings. I
have made an implementation of public key sorting in javascript where I
sort them as big endian numbers and fill in the 0s. IMO, the simpler
method is to sort them as strings, which has a simpler implementation
since it doesn't require filling in 0s first. However, I don't actually
care what method we use so long as everyone in the bitcoin world uses
the same standard. Which is the best way to sort public keys?

-- 
Ryan X. Charles
Software Engineer, BitPay