New REST API RFC

From Koha Wiki
Jump to navigation Jump to search

This was discussed on IRC on 2015-01-22: http://irc.koha-community.org/koha/2015-01-22#i_1625133

Background

Related bug reports

Suggested way forward

  • Agree on some guidelines/rules for the new API
  • Start strictly RESTful api using /rest and version it ( e.g. /rest/v1/route' )
  • Some time in the future: deprecate and ultimately remove /svc

During the Koha Hackfest 2015 we did tech sprints for Swagger (to autodocument the API), Raisin and Mojolicious (web frameworks), to evaluate their viability as the engine driving the new REST API.
We strongly suggest using Mojolicious as the framework to build the Koha API on. Perl supports many frameworks and we could deploy any of them succesfully, however we chose Mojolicious due to the following reasons:

  • It runs on many FastCGI-implementations, like Plack and several other Plack-like servers.
  • It also supports CGI for backwards compatibility
  • It supports Swagger 2.0 for automated API documentation, routing, and client side code scaffolding. This makes sure that the API documentation stays fresh and accurate. This is one of the key reasons we want a powerful web framework, so we don't have to worry about maintaining documentation so much.
  • It is very easy to use and shouldn't add anything to the core Koha-libraries.
  • It can be configured to run from the default Koha installation, or as it should be, under a FastCGI-implementation.

Signed Off: Olli-Antti Kivilahti

Signed Off: Julian Maurice

Signed Off: Kyle M Hall

Signed Off: Claire Hernandez

Progress report 2015.06.24 @ Koha developer IRC meeting

Since the Marseilles Hackfest 2015, BibLibre (Julian Maurice [jajm]) has been working to implement the new Koha RESTful API using the agreed-on Mojolicious framework with Swagger2 as a critical key component in documenting the API. API is based on the Ebsco API requirements for Koha. Development was completed and code submitted to bugzilla.

Now Olli-Antti Kivilahti [kivilahtio] has been extending and improving some parts of the API implemetation. Some key issues:

Logging, see Bug 13799 - 2. Set up logging and configuration file reading for Mojolicious. Mojolicious kills STDOUT and STDERR, thus we get no warnings and notifications from the Koha-portion regarding the myriad of issues arising there. The bug has a workaround for that but it is very hacky. This doesn't prevent usage, but makes debugging very difficult.

  • Some clean/better way of redirecting STDERR and STDOUT to log files needs to be devised.

Authentication, there was no Swagger-driven way of doing both authentication and permission checking. Such feature is now implemented in Bug 13920 - 9. API authentication system - Swagtenticator authentication. Now the authentication framework is pretty decent, but the exact methods of authentication are still under discussion. -User must be able to authenticate to the REST API with the Koha CGISESSID-cookie based authentication, so we can use the API to augment existing Koha's features.

  • This will be supplied by kivilahtio and Vaara-kirjastot ASAP

-There must be a passwordless authentication mechanism, like the proposed X-Koha-Signature -based version, or some other more established way of doing API key authentication (ashimema knows about this)

  • jajm has a version in Bug 13920 - API authentication system - proposal, but it should be reviewed if it can be standardized and streamlined some more.

-Other authentication mechanisms are optional and not important at this stage.

Enabling optional parameters Currently there is no way of setting optional parameters in the Mojolicious::Plugins::Swagger2 and that is not cool.

  • This can be supplied by kivilahtio and Vaara-kirjastot as part of the Serials improvements in some point in time.

Existing API Services Many API services have been now implemented by BibLibre but contain a lot of bugs. See "Bug 13799 - Add base for building RESTful API" blocking dependencies.

  • These bugs need squashing.

Swagger UI After a heated debate in the IRC meeting, we decided to let go of Swagger UI as a part of the Koha git history, and instead move it to a new bug which helps people deploy API inspectors, other than just Swagger UI.

  • We decided to create good instructions on how to set up API inspectors for QA people.

see http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14455

Conclusion: sign-offs and QA are needed. To help with that the API inspector tutorials need to go online. However they will be tested at latest when Vaara-kirjastot pushes these features to production and signs them off themselves :) The current code base already facilitates creating further REST services, since the authentication system doesn't touch the service implementation except by defining the permissions required in the Swagger2 definitions. Koha REST API is not ready to be deployed to production, but we could start pushing the core dependency (Bug 13799) to master to make further developments more easier. When the authentication system has been tested and verified (if only for the CGISESSID), we can start using the API in production for the services that we feel confident in.

Propositions

We have some choices to make, lets clarify them here.. vote on them and then turn this page into a definitive documentation of the chosen path.

Should we use a 'Mojolicious + Swagger2' stack?

Pro's
  • Mojo is a lightweight, fast perl web framework with out of the box support for persistent, non-blocking apps
  • Swagger 2 is a lightweight restful api specification/documentation language
  • Mojo + Swagger encourages/enforces a 'documentation/specification first' style of development.. You write your documentation as a 'Swagger specification file', this can then be used to automagically build tests from (thus giving you test driven development).
  • The Swagger 2 mojo plugin gives you automagical scaffolding built from the swagger specification file (which should include json specifications for all acceptable api requests and responses and thus includes validation out of the box)
Con's
  • Ties us to the mojolicious web framework as a dependancy
  • Ties us to the swagger specification language (unless we or someone else out there build a similar mojolicious plugin to talk 'RAML' or whatever future spec language is created)
  • Works best as a self contained persistent application... could mean further work in making sure existing koha routines can work in that way.
  • Non-blocking should be strived for in Mojo apps, but can lead to a steep learning curve.
  • Swagger is not fully json-schema compatible :(

Should we use the RAML API Specification language + custom code/mojo/whatever without a plugin?

Pro's
  • More capable specification/documentation language compared to Swagger 2.0
  • Fully json-schema compatible
  • Good array of tools for working with the specification files
Con's
  • No pre-built plugin for Mojolicious
  • No pre-built perl support
  • Due to the above, all code would have to be fully crafted, nothing would get auto-generated for us from the specifications

Should we use Koha::Service + 'Specifications' to base work upon?

Pro's
  • Uses existing Koha routines and standards
  • Less 'heavy and invasive' than relying upon an new framework
  • Doesn't enforce a 'spec/test' first approach
Con's
  • Is re-inventing the wheel somewhat as there are great frameworks out there (mojolicious being only one of them) that already achieve the goals of this interface
  • Constrains us to live within the confines of how things already work in Koha.
  • Doesn't enforce a 'spec/test' first approach
  • Everything would have to start out hand crafted.

API Protocol Specification

Ack... the entire point of RAML/Swagger is that it's a specification language.. thus the could we convert the below freeform specification into a swagger (probably best in swagger as that seems to be where the majority of people here are headed) or raml document please MRenvoize 09:29, 11 May 2015 (EDT) 

        /rest/
        /rest/v1/
GET     /rest/v1/borrower?barcode=1234&limit=20&start=1                 Fetches all borrower
GET     /rest/v1/borrower/<borrowernumber>                              Gets data about borrower
?GET    /rest/v1/borrower/cardnumber:<cardnumber>                       Gets data about borrower
?GET    /rest/v1/borrower/userid:<userid>                               Gets data about borrower
POST    /rest/v1/borrower/                                              Create a new borrower
PUT     /rest/v1/borrower/<borrowernumber>                              Updates data about borrower
DEL     /rest/v1/borrower/<borrowernumber>                              Updates data about borrower

GET     /rest/v1/borrower/<borrowernumber>/hold                                         Gets data about borrower holds
POST    /rest/v1/borrower/<borrowernumber>/hold                                         Places a new hold for this patron
PUT     /rest/v1/borrower/<borrowernumber>/hold/<id>                                    Updates the given hold for this patron
DEL     /rest/v1/borrower/<borrowernumber>/hold/<id>                                    Cancels a hold for this patron

GET     /rest/v1/borrower/<borrowernumber>/hold/biblionumber:<biblionumber>/pickup      For a given bib/item and patron, return array of tuples
GET     /rest/v1/borrower/<borrowernumber>/hold/itemnumber:<itemnumber>/pickup          of available pickup locations
                                                                                        { BR1: "Branch One", BR2: "Branch Two", BR3: "Branch Three" }

GET     /rest/v1/borrower/<borrowernumber>/checkout                     Gets list of currently checked out items for this patron
POST    /rest/v1/borrower/<borrowernumber>/checkout/<barcode>           Check out an item to this patron
PUT     /rest/v1/borrower/<borrowernumber>/checkout/<barcode>           Renew an item already checked out to this patron
DEL     /rest/v1/borrower/<borrowernumber>/checkout/<barcode>           Return an item checked out to this patron

                                                                        [biblionumber, itemnumber, pickup location, start date, end date]
PUT     /rest/v1/borrower/<borrowernumber>/hold/<id>                    Updates an existing hold for this patron
DEL     /rest/v1/borrower/<borrowernumber>/hold/<id>                    Cancels an existing hold for this patron


Authentication:
POST    /rest/v1/borrower/userid:<userid>/authenticate                  Authenticates a Koha user
                                                                        [password]



####  SERIALS  #####
Exposing API-calls to get a hierarchy of serial Items, instead of just displaying them as one huge list.
Idea is to navigate Serials from volume to item, or from branch to item. Eg:

Serial Biblio -> Volume -> Number -> Branch -> Item
Serial Biblio -> Branch -> Volume -> Number -> Item

##Make a RESTful level3 search for serials, getting a resultset which contains links
## to the found resources.
GET     /rest/v1/serial?ti=Times&volume=2014&issue=12&number=2&limit=20&page=2
GET     /rest/v1/serial?ti=Times&enumeriation=volume,2014&enumeriation=Issue,5&issue=12&number=2&limit=20&page=2 ( alternative )
RESPONSE [{id: 10332412,
           link: { href: "/rest/v1/serial/10332412",
                   rel:  "serial detail"}
                 },
          {id: 10332414,
           link: { href: "/rest/v1/serial/10332414",
                   rel:  "serial detail"}
                 },
          ...
         ]

##Make a RESTful level3 search for serial Items, getting a linked resultset
## Enable link for placing a number level hold for any Item inside a serial number.
GET     /rest/v1/serial/<biblionumber>/item
GET     /rest/v1/serial/<biblionumber>/item?volume=2014&issue=12&number=2
GET /rest/v1/serial/<biblionumber>/item?enumeration=volume,2014&enumeration=issue,12&enumeration=number,2 ( alternative )
RESPONSE [{ id: 4123,
            link: { href: "/rest/v1/serial/<biblionumber>/item/4123",
                    rel:  "item detail"}
                  },
          { id: 4124,
            link: { href: "/rest/v1/serial/<biblionumber>/item/4124",
                    rel:  "item detail"}
                  },
           ...,
          link: { href: "/rest/v1/serial/<biblionumber>/vol/2014/num/12-2/hold",
                  rel:  "number hold"}
         ]

GET     /rest/v1/serial/<biblionumber>/item/<itemnumber>
RESPONSE { id: <itemnumber>,
           <stuff interesting for serialItems>,
           link: { href: "/rest/v1/borrowers/<borrowernumber>/hold",
                   rel:  "place hold",
                   verb: "PUT"},
           link: { "Useful serial item related actions" },
           link: { ... },
         }

##Get all volumes and their numbers.
GET     /rest/v1/serial/<biblionumber>/enumerations
RESPONSE { ALL VOLUMES AND THEIR NUMBERS, + links for navigating. }

##Get all branches and item barcodes for enumeration
GET     /rest/v1/serial/<biblionumber>/vol/2015/num/4/availability
RESPONSE (hash/collection) { ALL BRANCHES AND ITEM BARCODES FOR ENUMERATION /^2015 : 4$/, + links for navigating. }

##Place a hold on a Serial Number.
PUT     /rest/v1/serial/<biblionumber>/vol/2014/num/4/hold?borrowernumber=123411&pickup=CPL&startdate=YYYYMMDDTHHMMSSZZZZ&enddate=YYYYMMDDTHHMMSSZZZZ
RESPONSE 200 OK

Guidelines/rules for the new API

pianohacker's Koha::Service is a good start at supporting a number of the below:

  • Use HTTP request methods appropriately
    • POST = create a resource
    • GET = read a resource
    • PUT = update a resource
    • DELETE = delete a resource
  • Use HTTP status codes appropriately
    • 200 OK
    • 400 Bad Request
    • 404 Not Found
    • 500 Internal Server Error
    • etc
  • Use HTTP query strings appropriately
    • Query strings should be used for identifying a resource.. i.e. filtering, expanding/contracting scope
  • Use HTTP message bodies appropriately
    • Message bodies should be used to transfer objects.. i.e. the bulk of the data transfer should go in a body.. that's true for GET, PUT and POST requests
  • Use HTTP content negotion to return in the format requested
    • JSON bodies should be supported as minimum
    • XML bodies should be supported for legacy compatability if required ( the i/o handling of xml/json should be baked into Koha::Service if possible such that developers need not consider it - Kyle )
    • JSONP bodies should be supported if the request may be coming from a different origin domain
  • Use a feature documenting and acceptance test driven test framework (kivilahtio thinks)
 Like the Perl Cucumber integration to use as a wrapper for feature documentation and acceptance/integration testing framework.
 It looks very promising and could be a good candidate as a mandatory testing framework for the new REST-API.
 
 https://metacpan.org/pod/Test::BDD::Cucumber::Manual::Integration
 https://metacpan.org/pod/App::pherkin
 
 Oslo is using Ruby, so we can consider falling back on to it if the Perl implementation doesn't work well.
 Or improve on the Perl implementation of Cucumber.
 
 However we should consider some test driven mechanism as a mandatory part of getting REST API patches pushed to master.
 
 Cucumber works rather well with Selenium2.0/WebDriver from Jonathan http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13691 
 (However - to test a REST-api using Selenium is somewhat overkill - using a rest client from cucumber is easier, and sufficient. --akafred)
 (If we strictly document our API with somthing like RAML, then many off the shelf testing frameworks will be able to read that documentation to automagically build and run tests MRenvoize 07:44, 25 March 2015 (EDT))

We should consider a json-first approach, writing json schema files for requests and responses.. this aids in documentation, testing and would also lead to DRY code for validating requests before processing them.. json first also means we can easily add xml along the line, xml first is harder to convert to json. MRenvoize 07:44, 25 March 2015 (EDT)

Thoughts for the future

  • Authentication not mandatory, so we can use it also for anom on Opac (public reports for example, now they are outside C4::Service)
  • APIs should be written in such as way as to conform to a very standard RESTful convention ( success means ngResource should work with the API with little to no configuration [1] )
  • Should we support compression, and should it be default or optionally on?
  • Should we support HEAD requests and OPTIONS requests to aid in discoverability? ( I'd say recommended but not required at this point in time )
  • How do we test? ( There's no reason we can't unit test the individual sub's of each module, right? - Kyle )
  • Idea from Thomas: Add api key system for simple api access authorization that can be revoked ( bonus points: allow api key to be limited to certain parts of rest api. This may not be needed as the user permission already give this type of control ).
  • A couple of resources worth looking at:
    - timelessrepo.com/haters-gonna-hateoas
    - roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven 
    - martinfowler.com/articles/richardsonMaturityModel.html
Retrieved from "https://wiki.koha-community.org/w/index.php?title=New_REST_API_RFC&oldid=15583"