escritorio

Integrating Powergate: Intro to the Powergate JavaScript Client

Jun 15, 2020

Textile built Powergate as a way to bridge the gap between IPFS and Filecoin storage. You get flexible data storage configurations optimized for long term, cryptographically verified, affordable storage on Filecoin, and high availability, fast, distributed storage on IPFS, or both simultaneously. According to the Powergate docs:

Powergate is a multi-tiered file storage API built on Filecoin and IPFS, and and index builder for Filecoin data. It's designed to be modular and extensible.

Powergate is meant to be integrated into other systems and applications. It's the easiest way to leverage IPFS and Filecoin, and by using Powergate to orchestrate how they work together, you get the best of both worlds plus the added smarts of Powergate's powerful data storage configuration mechanism.

If you'd like more information about Powergate in general, please see the repo's README and FFS design documentation.

Options for Integration

Today, there are a few tools we provide for integrating Powergate into your system or application. If you're building in Go, there's the Go Powergate client. If you're writing shell scripts or want to quickly experiment with Powergate APIs, the CLI would be a great tool (Installable executables coming soon -- for now, follow instructions in the README about how to install from source). The newest addition is our TypeScript/JavaScript Powergate client, and that's what we'll focus on in this post.As a side note, all of the clients I just mentioned are built using gRPC.proto services exposed by Powergate, so in theory any type of client could be built in any language supported by gRPC. Browse around the repo to see the various files defining these services. For example, the service definition for the Deals module can be found here

.

The Example Project

Our simple web app showing some raw data about the Powergate instance it's connected to

The project I'll work through in this post is a simple Node.js web app built with Express as a server, router, and middleware and it uses Pug templates for rendering HTML. You can find the finished product in the examples folder of the Powergate JavaScript client repo. I'll use TypeScript because I like it, but you don't have to. The app will display some basic information about the Powergate instance it's connected to and allow users to authenticate using GitHub OAuth. Once authenticated, the web app will display additional information about the user's own FFS instance.

Initial Project Setup@textile/powergate-clientIf you have an existing project you'd like to use as we walk through this example, you'll just need to install

npm install @textile/powergate-client

and then adapt further code examples I provide to whatever tools your project uses.

git clone https://github.com/textileio/node-starter.git
cd node-starter
npm install

For everyone else, I created a minimal application you can clone or fork that includes all the dependencies we'll need, plus some basic plumbing for user authentication, and UI. To get started with it, run the following:

Setup a GitHub OAuth AppTo get GitHub authentication working in the example app, you'll need to log into your GitHub account and create a new OAuth App. You'll find this under Settings > Developer Settingshttp://127.0.0.1:3000 > OAuth Apps. Create a new one, you can use as the Homepage URLhttp://127.0.0.1:3000/auth/github/callback > OAuth Apps. Create a new one, you can use y Authorization callback URL. Take note of the generated Homepage URLclient idsecret

provided after you submit the form. .env.exampleBack in our project, copy .env to GITHUB_CLIENTHomepage URLGITHUB_CLIENT_SECRET and open it for editing. Update

with the values from your new OAuth app.Important.env: Don't check your .gitignore file into any public repository since it contains application secrets. The

in our starter project is already set up to do the right thing here.

Initial Run

npm run debug

You should now be able to run the app in debug mode:

And you should see the default application home page at http://localhost:3000.

The default home screen of the app

Application Anatomy

src/server.tsOur starter app already has some basics that we'll take advantage of. Here's an overview of the important pieces and how we'll use them when integrating Powergate.

src/util/env.ts is the main entry point to our application..env gives us access to variables defined in our

src/config/passport.ts file. We'll use those to specify where to connect to our Powergate instance as well as access other application secrets. configures our Passport

src/models/user.ts GitHub authentication strategy. We'll allocate Powergate resources for authenticated users.User contains our

src/app.ts data model and provides SQLite-based persistence of user data. We'll extend this model to associate Powergate information with users.

views/ is where our Express application lives. All routes and route-handling middleware are defined here. We'll use middleware to interact with the Powergate client, querying for data to display and creating Powergate resources for authenticated users.

contains Pug templates that we'll use to render information about Powergate and authenticated user Powergate resources.

Powergate currently uses two categories of API access; Unauthenticated requests to access general information about the Powergate and Lotus node health, network, miners and more, and authenticated requests to access to individual FFS instances. A single FFS instance provides partitioned access to and management of data storage on IPFS and Filecoin.

Our goal with this example app is to display some general information about Powergate using the unauthenticated APIs, and then create a FFS instance for  each authenticated user. The same FFS instance should be used by a user that logs out and then returns to the app at a later time. We'll display a bit of information about the authenticated user's FFS instance.

Running Powergate

  1. Of course, in order to integrate Powergate into our app, we'll need an instance of Powergate running. Textile is considering offering hosted Powergate services, so if that's something you'd be interested in, please get in touch. For now, there are a few options to get Powergate running yourself.BYO - Powergate depends on connecting to IPFS and Lotus nodes. You are free to run your own and then powdrun the Powergate server
  2. yourself, configuring it with the addresses of your IPFS and Lotus nodes.
  3. Docker + testnet - We provide a Docker compose configuration that will spin up Powergate, IPFS, and Lotus, plus some extra metrics tracking and visualization tools. The resulting setup runs on the Filecoin testnet (and mainnet once it launches). This is good for a production setup, but will feel slow for development purposes.

Docker + localnet - We created a special configuration of Lotus that runs a filecoin network locally at high speed, and it works great for testing and development. It's easy to use via a localnet Docker compose configuration. This is how we'll be running Powergate and its dependencies for this example app.El Powergate releases pagepackage-lock.json includes bundled up Docker Compose files that use appropriate Docker image versions of Powergate, Louts, and IPFS. Let's check our @textile/grpc-powergate-client to see the appropriate release to download and run. Look for

"@textile/grpc-powergate-client": {
    "version": "0.0.1-beta.13",
    ...
},

. Its version number corresponds to the version of Powergate the underlying gRPC bindings were created from.powergate-docker-<version>.zipDownload the 0.0.1-beta.13 from the appropriate release, in this case, version . Assuming you have Docker DesktopMakefile installed, start up Powergate using the provided

unzip powergate-docker-v0.0.1-beta.13.zip
cd powergate-docker-v0.0.1-beta.13
BIGSECTORS=true make localnet

:Important note on BIGSECTORSBIGSECTORS=false: When running the localnet setup, the Lotus node is configured with a mocked sector builder, using either "small" or "big" sector sizes. The practical effects of this configuration are on the size of files you can store in the localnet and how quickly the storage deals will complete. Using BIGSECTORS=true will limit you to storing files of around 700 bytes and deals will complete in 30-60 seconds. Using

will allow you to store files anywhere from 1Mb to 400Mb, but deals will complete in 3-4 minutes. Be sure to choose the value that makes sense for your development scenario.

Connecting to Powergate.envFirst, we'll configure the project to connect to our Powergate instance and then create the client we use to make calls to Powergate. Open

POW_HOST=http://0.0.0.0:6002

for editing and add a new variable for the Powergate API host. The below value is correct if you're running our Docker compose setup, but adjust it as needed if you're running Powergate elsewhere:src/util/env.tsThen, let's add an the exported value of this variable to

export const POW_HOST = process.env["POW_HOST"]

so we can easily read it's value from elsewhere in the application:src/app.tsNow we can create our instance of the Powergate client, connecting to the configured host. In POW_HOST we'll import our createPow variable and the Powergate client factory function pow installed, start up Powergate using the provided

import { createPow } from "@textile/powergate-client"
import { EXPRESS_PORT, POW_HOST, SESSION_SECRET } from "./util/env"

const pow = createPow({ host: POW_HOST })

, and finally create our Powergate client we'll call

Unauthenticated Requests + Displaysrc/app.tsNow that our Powergate client is ready to go, we can use it to query some data from Powergate, and then we'll display that data on the home page of the web app. In /, we'll update our render route handler to call some Powergate APIs and pass the resulting data to the Promise function so it's available in our HTML template. Most Powergate functions return asyncs since they are making async calls over a network. For that reason, we also need to label our route handler function as try/catch, use Promise as usual with nexts, and call the handler's

app.get("/", async (_, res, next) => {
  try {
    const [respPeers, respAddr, respHealth, respMiners] = await Promise.all([
      pow.net.peers(),
      pow.net.listenAddr(),
      pow.health.check(),
      pow.miners.get(),
    ])
    res.render("home", {
      title: "Home",
      peers: respPeers.peersList,
      listenAddr: respAddr.addrInfo,
      health: respHealth,
      miners: respMiners.index,
    })
  } catch (e) {
    next(e)
  }
})

parameter in case of an error. Here's the updated version:views/home.pugWe've now passed the results from our calls for peers, listen address, health status, and Filecoin miners into our render function, so we can update stringify to render that information. I updated the header text, and to avoid needing to make more UI, I'm just <pre>ing the data and printing it inside

extends layout

block content
  h1 Node Info
  p.lead A sample of generally available data, no auth required.
  hr
  .row
    - const jsonPeers = JSON.stringify(peers, null, 4)
    - const jsonAddr = JSON.stringify(listenAddr, null, 4)
    - const jsonHealth = JSON.stringify(health, null, 4)
    - const jsonMiners = JSON.stringify(miners, null, 4)
    .col-sm-6
      h2 Node Health
      pre= jsonHealth
    .col-sm-6
      h2 Listen Address
      pre= jsonAddr
    .col-sm-6
      h2 Peers
      pre= jsonPeers
    .col-sm-6
      h2 Miners
      pre= jsonMiners

tags:

Refresh, and you should see:

User Model UpdateUserWe need to update our ffsToken?: string model to represent the FFS instance associated with each user. We'll add a User property to the src/models/user.ts type in

export type User = {
  gitHubId: string
  email: string
  ffsToken?: string
}

. It will hold the token returned when we create a FFS instance for a user:UserNext, update the src/models/user.ts references and SQL statements in User for the ffsToken CRUD operations, taking into account the new findByGitHubId property. For example, becomes (notice we update the SQL statement with a new field y

export const findByGithubId = async function (gitHubId: string): Promise<User | undefined> {
  const q = "SELECT gitHubId, email, ffsToken FROM users WHERE gitHubId = ?"
  const row = await db.get(q, gitHubId)
  if (!row) {
    return undefined
  } else {
    const user: User = {
      gitHubId: row.gitHubId,
      email: row.email,
      ffsToken: row.ffsToken,
    }
    return user
  }
}

the response object with a new field):

This is a little tedious, so you may want to just copy the final version from the example app repo.

FFS Creation Middlewaresrc/app.tsIn /auth/github/callback, the route /user is called when GitHub redirects the user back to our application as part of the OAuth process. It currently finishes the authorization process and then redirects the user to User. We'll add another handler to that flow, after completing the authorization process, but before redirect, to create a FFS instance if needed for the now-authenticated user then update and save the

import { save, User } from "./models/user"

app.get(
  "/auth/github/callback",
  passport.authenticate("github", { failureRedirect: "/" }),
  async (req, _, next) => {
    if (req.user) {
      const user = req.user as User
      if (user.ffsToken) {
        pow.setToken(user.ffsToken)
        return next()
      } else {
        try {
          const createResp = await pow.ffs.create()
          user.ffsToken = createResp.token
          await save(user)
          pow.setToken(user.ffsToken)
          next()
        } catch (e) {
          next(e)
        }
      }
    } else {
      next(new Error("no user found in session"))
    }
  },
  (_, res) => {
    res.redirect("/user")
  },
)

instance:asyncOur handler is Promise since it calls the Powergate client's ffs.create()-based User save() and the

functions. UserWe first make sure we can access our Express user object (which we know is our

type). If not, this is an error.ffsTokenThen we check if the user already has a next(). If so, this is a returning user and we're done. We call

so the request handler chain simply continues. ffsTokenIf the user doesn't have a ffsToken, we create a new FFS instance, set the user's Userhere

property with the returned token and save the updated pow.setToken(user.ffsToken)The last step is calling

. This passes the user's FFS token into our Powergate client, and the client will now send that token along with each request to Powergate so it knows which FFS instance the user is interacting with.

Important note: The communication with Powergate is currently unencrypted http so the FFS token is sent in plain text. We plan on providing tooling to support encrypted https in an upcoming release.

FFS Creation Middlewaresrc/app.tsAuthenticated Request + Display/user, we already have a route, passportConfig.isAuthenticated, thar requires authentication (enforced by the pow middleware). Having an authenticated user at this point implies that we've already created a FFS instance for that user, and that our /user client is configured to use that FFS instance's token. So let's now make an authenticated call to Powergate. We'll do this by updating the handler for the

app.get("/user", passportConfig.isAuthenticated, async (_, res, next) => {
  try {
    const info = await pow.ffs.info()
    res.render("user", {
      title: "User",
      info: info.info,
    })
  } catch (e) {
    next(e)
  }
})

route:ffs.info()Here, we call user.pug to which returns some information about the user's FFS instance. We pass the result into our render function so we can display it in the updated

extends layout

block content
  h1 FFS (#{info.id})
  p.lead Data for your FFS instance.
  hr
  .row
    - const jsonInfo = JSON.stringify(info, null, 4)
    .col-sm-6
      h2 Instance Info
      pre= jsonInfo
template:

views/user.pug updated to display FFS info

If you refresh and make sure you're logged in, you should see:

`

If you log out and log back in, you should see that the FFS instance id remains the same, showing that the same user will always be interacting with their unique FFS instance.

Wrap Up

In this example, we've retrofitted a very simple, but typical, Node.js web app with Powergate capabilities. With this simple change, we can view information about our Powergate, IPFS, and Lotus nodes, and users now have access to powerful data storage on IPFS and Filecoin through their own FFS instance. I hope that seeing one way to map Powergate concepts onto an existing application gives you ideas about how you might integrate Powergate into your own systems.

Next Steps

The Powergate JavaScript client is brand new and improving every day. The most commonly used Powergate APIs are available already, but we'll be adding more and keeping the client up to date with the core Powergate APIs. Be sure to star or subscribe to the repo to stay updated.

Información de la licencia de JavaScript
¡Grandioso! Se ha suscrito con éxito.
¡Grandioso! A continuación, complete la comprobación para un acceso completo.
¡Bienvenido de nuevo! Has firmado con éxito.