Incremental Compiler:

taming Scalac

blog.typosafe.pl/scalar-2016

Is Scalac incremental compiler?

Scala compilation is

SLOOOW*

*compared to Java

What incremental compiler is?

Incremental compiler is a driver for scalac that for given change recompile only subset of sources that will produce the same output as full compilation.

Incremental compilers for Scala?

SBT (Zinc, ScalaIDE)

Intellij

SBT Incremental Compiler

explained in scala code

All scala examples below are pseudocode.

Actual sbt code is too complicated and hairy to be shown on such presentation.

github.com/sbt/sbt/tree/0.13/compile

def compile(compilation: Compilation): Unit = {
    var changed = changedSources(compilation)

    while(changed.nonEmpty){
        val oldAPIs = changed.map(currentAPI)
        val newAPIs = doCompile(changed)

        val changedAPIs = diff(oldAPIs, newAPIs)

        changed = changedAPIs.flatMap(allDependencies)
    }
}

Do we need recursion?

// Team.scala
class Team {
    def room = "1" // change to 1
}

// User.scala
class User(team: Team) {
    val room: String = team.room // this must fail
}

def compile(compilation: Compilation): Unit = {
    var changed = changedSources(compilation)

    while(changed.nonEmpty){
        val oldAPIs = changed.map(currentAPI)
        val newAPIs = doCompile(changed)

        val changedAPIs = diff(oldAPIs, newAPIs)

        changed = changedAPIs.flatMap(allDependencies)
    }
}

def changedSources(compilation: Compilation): Seq[File] = {
    if(outputChanged || optionsChanged) // Then compile all
        compilation.allSources
    else // Compile only changed files
        compilation.allSources.filter(changedSource)
}

private def doCompile(source: Seq[File]): Seq[API] = {
    val listener = new SourceListener

    runScalac(listener)

    updateDependencies(listener.dependencies)

    listener.apis
}

class SourceListener{
    def process(unit: CompilationUnit): Unit = {

    // Compute hashes
    val unitTraverser = new HashingTraverser.run(unit.tree)

    _apis +=  API(unit.source, unitTraverser.unitHash)

    // Compute dependencies
    val depsTraverser = new DepsTraverser.run(unit.tree)

    _dependencies += (unit.source -> depsTraverser.dependencies)
    }
}

trait TreeTraverser {
    def traverse(tree: Tree): Unit

    def run(tree: Tree): this.type
}

class HashingTraverser extends TreeTraverser {
    var unitHash = EmptyHash

    override def traverse(tree: Tree): Unit = tree match {

        case ClassTree(sig, members) if !sig.isPrivate =>
            unitHash.hash(sig)
            members.foreach(sig)

        case Method(sig, body) if !sig.isPrivate =>
            unitHash.hash(sig)

        //  ...

class DepsTraverser extends TreeTraverser {
    var dependecies: Set[File] = Set.empty

    override def traverse(tree: Tree): Unit = tree match {

        case ClassTree(signature, members) =>
            dependecies += signature.declaredIn
            members.foreach(traverse)

        case  Method(signature, body) =>
            dependecies += signature.declaredIn
            traverse(body)

        case Operation(tpe, _) =>
            dependecies += tpe.declaredIn
        // ...

def compile(compilation: Compilation): Unit = {
    var changed = changedSources(compilation)

    while(changed.nonEmpty){
        val oldAPIs = changed.map(currentAPI)
        val newAPIs = doCompile(changed)

        val changedAPIs = diff(oldAPIs, newAPIs)

        changed = changedAPIs.flatMap(allDependencies)
    }
}

SBT - name hashing

Explained as diff on scala code

case class API(source: File, hash: Hash)

// becomes

case class NamedAPI(source: File, hashes: Map[String,Hash])

var dependecies: Set[File] = Set.empty

becomes

var fileDependencies: Set[File] = Set.empty
var nameDependencies: Set[String] = Set.empty

def isAffected(changedAPI: Seq[File, Set[String],
               source: File): Boolean = {
    changedAPI.find {
        case (changedSource, changedNames) =>
            def useChangedNames =
                (usedNames(source) & changedNames).isDefined

            isDependency(source, changedSource) && useChangedNames
    }
}

Can we do better?

Smaller granulation

https://github.com/sbt/zinc/pull/86

Speed vs. Precision

Speed-up your incremental compilation

Provide return types

val it = Seq("Adam", "Marcin")
val hr = Seq("Ola", "Ala")

class TeamMember(name: String, team: String)
def allTeamMembers: Seq[TeamMember] = ???...

//This is Seq[Seq[String]]
def teams = Seq(it, hr)

//This is Iterable[Seq[String]]
def teams = allTeamMembers.groupBy(_.team).map {
    case (_, usersInTeam) =>
        usersInTeam.map(_.name)
}

One file - few classes

Krzysztof Romanowski

romanowski.kr@gmail.com

What was changed?

Class (file) is changed when

How to detect API changes?

signatures from .class files?

trait Dep

class Generic[E[V] <: Seq[V with Dep]]

// yields same javap output as

class Generic[E[V] <: Seq[V]]

public class romanowski.changes.Generic
    <E extends scala.collection.Seq<java.lang.Object>> {

    public romanowski.changes.Generic();

    Code:
    0: aload_0
    1: invokespecial #13  // Method java/lang/Object."<init>":()V
    4: return
}

scalap?

any other ideas?

What is affected?

// Implicits.scala
class Implicits(i: Int) {
    implicit class FooBarOps(from: FooImpl[_]){
        def fooBar = 1
    }
}

// FooImpl.scala
class FooImpl[A] extends Foo[A]

// Baz.scala
class Baz {
    val foo = new FooImpl[Bar]

    foo.fooBar
}

// Foo.scala
class Foo[A]

object Foo // extends Implicits(1) // Uncomment this


// Bar.scala
class Bar

object Bar extends Implicits(1)

*.class files?

There is no direct path that leads from changes in Foo$ (companion object from Foo) to Baz.

scalap?

There is no direct path that leads from changes in Foo$ (companion object from Foo) to Baz.

Concerete usages vs. potential usage

Do we want to reimplement compiler?

Krzysztof Romanowski

romanowski.kr@gmail.com