Implement side-by-side diffs

[ChrisPenner]

Overview

This represents a full overhaul of our diffing algorithm. The goal is to better align diffs in spite of change.

The old algorithm was entirely token-oriented, it didn't do anything to special-case lines, so if a bunch of lines were deleted on one side, the other side's diff would get out of alignment.

The new algorithm inserts spacers where necessary to keep similar chunks in line.

Here's an example diff using the new algorithm,
Each line is marked with -/+ when changed, and changed tokens within changed lines are highlighted by wrapping with {} (Don't worry, Simon will make the rendered version prettier)

=====left====
unchangedDefinition = 1
myDefinition : Nat -> Text -> Text
myDefinition n txt =
  -- Check for non-positive n
  if n == 0 then txt
  else Text.drop n txt

====right====
unchangedDefinition = 1
myNewDefinition : Int -> Text -> Text
myNewDefinition k input =
  -- Check for non-positive n
  if n <= 0
    then txt
  else
    Text.take n input

====diff====
  unchangedDefinition = 1                |   unchangedDefinition = 1
- {myDefinition} : {Nat} -> Text -> Text | + {myNewDefinition} : {Int} -> Text -> Text
- {myDefinition} {n} {txt} =             | + {myNewDefinition} {k} {input} =
    -- Check for non-positive n          |     -- Check for non-positive n
-   if n {==} 0 then txt                 | +   if n {<=} 0
-   else {Text.drop} n {txt}             | +  { }{ }{ }then txt
////                                     | +   else
////                                     | +   { }{ }{Text.take}{ }n {input}

Implementation notes

At a high level, the algorithm:

• First runs the diff algorithm using entire lines as tokens, this gives us a list of equal lines, with chunks of changed lines in between
• Now we can map over the changed trunks and run a token-diff on each of those. This behaves similar to the old algorithm
• We then compute the difference between the number of lines in the lhs and rhs diff sides; and add padding to fill out the smaller.
• This gives us two lists of lines, one for lhs, one for rhs. Each line is tagged with whether it's been changed or not. Lines with changes include annotations on each token indicating whether that particular token exists on the other side. This includes annotations for cases where the name is the same but the hash changed and vice versa.

This replaces the other diff algorithm entirely, which is currently unused in UCM, but will need matching updates from Simon when deploying to Share.

Test coverage

I added property tests to generate random diffs and assert that the diffs always have matching line-counts (asserting that spacers are being added)

I also added explicit test-cases using a text-rending of diffs.

There are also transcript tests to show the output json, but those aren't good for evaluating the diff effectiveness.

Loose ends

There's still another pass we can do on this to be smarter about where within a change block we put spacers, e.g. look at this VS code diff where spacers are interleaved within the change block to line up similar words.

Pretty sure I can iterate on this to get the same, but this is already a significant enough change that it's worth it to ship.

[ChrisPenner]

@hojberg note changes to the diff response.

Now the diff has {left: <lines>, right: <lines>}
where each side is an array of lines;

Each line is one of:

{kind: changed
, value: [<difftokens>]
}

{kind: unchanged
, value: [<difftokens>]
}

{kind: spacer}

The tokens themselves are similar to before, but for now each token is split up into its own object rather than having a both with a list of tokens.

We may actually want to look at more efficient serializations since I know Mitchell had noticed the diff sizes were getting really big.

I'm also looking at IDs which associate tokens on the lhs with their rhs counter-part when they match; that way you could hover the lhs and have it light up the matching token on the other side if we want :)

[hojberg]

@ChrisPenner how might I try this out with Share locally?

[ChrisPenner]

@ChrisPenner how might I try this out with Share locally?

@hojberg Here's a branch for ya!
unisoncomputing/share-api#155

[ChrisPenner]

@aryairani this is already up on Share and seems to be working well :)

Go ahead and merge unless you've got any concerns 👍🏼

[aryairani]

still want this?

[aryairani]

and these

[aryairani]

i have a draft to have CI use the latest 'stack' but I haven't had a chance to test it. fingers crossed though