Use with Schema Stitching

[idibidiart]

Hi,

I'd like to get your opinion on using GrAMPS such that I limit each resulting GQL server to one data source (in our case each REST API is maintained by a separate team/contractor) and use Schema Stitching to create the unified schema/server. This way separate teams can expose their own GQL end points (for use by customers) while allowing us to integrate all REST APIs under one GQL server for another use case. This way things are more modular.

What issues do you foresee if any?

Thanks & great to see momentum building up for the GrAMPS approach

[jlengstorf]

The goal of GrAMPS is to allow each team/contractor to create a data source that wraps their REST API. They have full control over that code and publish the built files to a registry (or Artifactory, or an internal store, etc.).

A valid GrAMPS data source can be consumed by any GraphQL server, so the typical use case is to bring all the data sources into a single endpoint to lower the complexity of combining data sources.

Something like this:

import bodyParser from 'body-parser';
import gramps from '@gramps/gramps';
import { graphqlExpress } from 'apollo-server-express';
import playground from 'graphql-playground-middleware-express';

import T1DataSource1 from '@teamone/data-source-one';
import T1DataSource2 from '@teamone/data-source-two';
import T2DataSource3 from '@teamtwo/data-source-three';
import T3DataSource4 from '@teamthree/data-source-four';

const GraphQLOptions = gramps({
  dataSources: [
    T1DataSource1,
    T1DataSource2,
    T2DataSource3,
    T3DataSource4,
  ],
});

app.use(bodyParser.json());
app.use('/graphql', graphqlExpress(GraphQLOptions));
app.use('/playground', playground({ endpoint: '/graphql' }));

app.listen(8080);

This way, you don't have to worry about combining multiple remote schemas.

However, there's no issue with consuming data sources one at a time for other endpoints. In fact, that's a primary goal of GrAMPS. 😄

Long-term, we hope to see open source GrAMPS data sources that will allow any developer to use a company's data source to build apps (e.g. if you’re building a music app, you could install the Genius API data source and have lyrics ready to go).

Does that all make sense?

[kbrandwijk]

I see a few alternatives:

1. Define GrAMPS datasources (that connect to any backend datasource), expose them individually as GraphQL endpoint (basically the same as above, but with a single datasource each time), stitch those GraphQL endpoints together in a separate central server (using other ecosystem tools like graphql-binding)

2. Define GrAMPS datasources and stitch them together directly in a central server (like Jason's example above). This would mean defining the stitching behavior in GrAMPS, but not using individual GraphQL endpoints for each datasource.

Looking at these two options, I think the best approach depends on whether you need/want to expose individual datasources as 'stand-alone' GraphQL endpoints too.

[idibidiart]

Thank you both for following up.

I have the same conclusion as the one given by @kbrandwijk

The beauty of it is that both #1 and #2 are perfectly valid, and both are fully modular. I had misunderstood (for some reason) that in #2 one would have to manually create/edit the combined schema. But it seems that you're implying that each team would create the data source and the schema that goes along with it, and a unified schema is created automatically from those individual schema (not sure if it's actually created as a file that humans can consume or just an in-memory object, and I'm hoping the former)

Am I making any sense here?
:)

[kbrandwijk]

In option 2, the merged schema is created by GrAMPS. In memory, at runtime, but it is fairly easy to save the SDL to a file if you want to (would just be a fs.writeFileSync(print(gramps.schema)) call after the gramps() call in the example above), although you generally wouldn't need to.

In option 1, you would have full control over the resulting schema, and the way it delegates to the GrAMPS datasources, but it would require more manual work, as you would explicitly define the schema that you want to expose.

From personal experience, even though option 2 seems easiest, the fact that you don't control the resulting schema, as it is directly dictated by the individual data sources, has led me to the best practice of always explicitly defining my schema. When the current GraphQL ecosystem was emerging, I used to think that would cause a lot of extra maintenance, but in reality, the full control argument wins for me.

[idibidiart]

the fact that you don't control the resulting schema, as it is directly dictated by the individual data sources, has led me to the best practice of always explicitly defining my schema.

I agree. That was another subtle thing that I did not articulate.

Help me clarify what you stated here:

In option 1, you would have full control over the resulting schema, ... , but it would require more manual work, as you would explicitly define the schema that you want to expose.

You're saying that exposing a unified schema is an extra manual cost in 1 (not the individual schema for each data source since that's has to be done in 1 and 2)

Correct?

[idibidiart]

Clarifying and using same terminology: unified schema = merged schema

for each data source in both 1 and 2, we have to manually map the REST API response to GraphQL types, right?

In 1, we would also have to manually create the merged schema (I've not done schema stitching before but my assumption stemming from the use of the word 'stitch' is that it's a manual task)

Are these assumptions correct?

thank you again for explaining! really appreciate the insight

[kbrandwijk]

In both cases, you need to set up the individual GrAMPS datasource, so define schema and map REST API to it.

So with that out of the way, what's left is the way to combine them.

1. Like you said, that would mean manually creating the merged schema. It's basically a new schema from scratch, and in the resolvers for that schema, you would call the appropriate GrAMPS datasource. There's tooling available to statically (at build-time) generate Typescript definitions and binding classes for that, so it's very easy to implement these resolvers. A basic example (based on the example above) would be:

const typeDefs = `
type Query {
    myQuery: [ResultType]
    myOtherQuery: [OtherType]
}
`

const resolvers = {
  Query: {
    myQuery: forwardTo('binding1'), // simply forwards to a query with the same name and args
    myOtherQuery: (parent, args, context, info) => {
      return context.binding2.query.anotherQuery(args, info) // freedom to delegate to a differently named query
    }
  }
}

const server = new GraphQLServer({
  typeDefs,
  resolvers,
  context: req => ({
    ...req,
    binding1: new Binding(prepare({ datasources: [ T1DataSource1 ] }).schema),
    binding2: new Binding(prepare({ datasources: [ T1DataSource2 ] }).schema)
  })
})

server.start(() => console.log('Server is running on http://localhost:4000'))

Alternatively (depending on what you find easier), you can also still use gramps to merge all of your datasources, and expose them all through one single binding:

const typeDefs = `
type Query {
    myQuery: [ResultType]
    myOtherQuery: [OtherType]
}
`

const resolvers = {
  Query: {
    myQuery: forwardTo('gramps'), // simply forwards to a query with the same name and args
    myOtherQuery: (parent, args, context, info) => {
      return context.gramps.query.anotherQuery(args, info) // freedom to delegate to a differently named query
    }
  }
}

const gramps = prepare({
  dataSources: [
    T1DataSource1,
    T1DataSource2,
    T2DataSource3,
    T3DataSource4,
  ],
});

const server = new GraphQLServer({
  typeDefs,
  resolvers,
  context: req => ({
    ...req,
    gramps: new Binding(gramps.schema)
  })
})

server.start(() => console.log('Server is running on http://localhost:4000'))

2. There's GrAMPS itself, that can generate a merged schema from the set of GrAMPS datasources directly. This requires no manual work on the unified/merged schema, but also gives you no control over it.

@jlengstorf We should really take care of supergraphql/gramps-boilerplate#1...

[idibidiart]

That’s great info. Thank you.

What I was thinking, and please correct me if it’s not a good idea, is to use GrAMPS to create a GraphQL end point for each REST API and then use the techniques described here to build a merged schema in a robust manner:

https://www.apollographql.com/docs/graphql-tools/schema-stitching.html

Type conflict resolution is covered in the above link. I wonder if the GrAMPS way of auto-merging allows us to specify that behavior?

Looking forward to start playing with GrAMPS and scheme stitching starting tomorrow.

[kbrandwijk]

The GrAMPS merging does not allow you to specify these options. As all types are namespaced, you should not have any conflicts anyways.

Regarding graphql-tools. The current version of their schema stitching is not so great, they have an alpha version with a new merging API (ardatan/graphql-tools#527) which is a lot better. However, the issue remains that if you want control over which parts of the underlying endpoints you want to expose, graphql-tools merging is not going to give you the opportunity to do so (the new alpha version has better support, but the way to declare it is way less clear than by just defining your own schema explicitly).

[idibidiart]

Oh. I get it now. Sorry, I thought GraphQL-tools also allowed for manual creation of merged schema.

So then GrAMPS does make a lot of sense not only for Option 2 but also Option 1.

Control is key! I urge you guys to include what you just explained to me here in your boilerplate or at least in examples.

[kbrandwijk]

Yep, that was why I linked to the open issue regarding that :)

[idibidiart]

Great. Meanwhile, where can I ask questions if I get stuck. Do you guys have a Gitter or Slack channel? I’d hate to bug Jason on Twitter. And I don’t want to raise an issue for every question. I guess chat would be the best option.

[jlengstorf]

Yep! We have a Slack channel over on the Graphcool instance: https://graphcool.slack.com/messages/C8GG5TL0M

[idibidiart]

@kbrandwijk @jlengstorf

I need to be able to combine the data sources you provide in the examples (XKCD and IMDB) into one schema and (as a POC) I want to create a merged type in the merged schema (using SDL, since it's important for others to be able to edit the merged schema using SDL) that does not only return IMDB movie data but injects a random XKCD image.

This would provide a POC of "multiplexing REST APIs" that I could then take up the chain and overhaul how a $100B company build applications. GrAMPS is such an awesome life saver (and a massive time saver) if I can do this with it. I'm sure it can be done but not sure how to change the behavior so that instead of auto generating a merged schema I get to this:

const typeDefs = `

type someMergedType {
 someFieldFromSource1: someTypeFromSource1,
 someFieldFromSource2: someTypeFromSource2!
}

type Query {
    myQuery: [ResultType],
    myOtherQuery: [OtherType]
    myMergedQuery: [someMergedType]
}
`

const resolvers = {
  Query: {
    myQuery: forwardTo('gramps'), // simply forwards to a query with the same name and args
    myOtherQuery: (parent, args, context, info) => {
      return context.gramps.query.anotherQuery(args, info) // freedom to delegate to a differently named query
    },
    myMergedQuery: (parent, args, context, info) => { 
      // somehow return a list of someMergedType
    }
  },
  Types? 
// don't we also need a bunch of type resolvers to resolve the fields in Query/Mutation results from above e.g. resolve fields in someMergedType
// e.g.:
 someMergedType:  {
   someFieldFromSource1: (parent, args, context, info) => { 
        // resolve someFieldFromSource1 
     },
     someFieldFromSource2: (parent, args, context, info) => { 
        // resolve someFieldFromSource2
     },
  },
 someOtherMaybeNestedType: {
   // etc
 } ,
 someOtherType: {
  // etc 
}
   
 }
}

const gramps = prepare({
  dataSources: [
    T1DataSource1,
    T1DataSource2
  ],
});

const server = new GraphQLServer({
  typeDefs,
  resolvers,
  context: req => ({
    ...req,
    gramps: new Binding(gramps.schema)
  })
})

server.start(() => console.log('Server is running on http://localhost:4000'))

My confusion is very basic. I lack the step by step how-to, and it could take me days to sort out how to take the existing examples of single sources, make it multi-source, eliminate auto-generation of schema, and define the schema by hand as per the above.

A how-to example would be tremendously helpful and it could then be added to the docs for others to follow.

Thank you so much.

[Updated]

[jlengstorf]

@idibidiart The IMDB database that was originally used is pretty much terrible, so I'd strongly recommend not doing that.

What will probably make your life easier is doing something much smaller: create a data source using the quickstart, then stitch that with the XKCD data source.

[kbrandwijk]

It would really help to have 2 sample datasources @jlengstorf. Any chance you can come up with something similar to XKCD? I can then write up an example of how to create the 'front-end' server using our tooling.

[jlengstorf]

@kbrandwijk Do you know of any unauthenticated, stable APIs with a really small footprint (similar to XKCD)? I'm currently putting together an example of two POC data sources to show how stitching works, but I've been avoiding wrapping a large API and/or requiring that people taking the tutorial have to register for API keys.

[ecwyne]

https://github.com/toddmotto/public-apis

[kbrandwijk]

@jlengstorf When prepare is available, making that example would be a lot easier.

[jlengstorf]

This one might work. Anyone have time to wrap this? http://numbersapi.com/

[kbrandwijk]

Will do.

[jlengstorf]

@kbrandwijk Thank you! I just invited you to the GrAMPS org and created a repo for it: https://github.com/gramps-graphql/data-source-numbers