ILL Module
Interlibrary Loans Module
| Status: | unknown | |
| Sponsored by: | no-one | |
| Developed by: | Alex Sassmannshausen, Martin Renvoize, Colin Campbell | |
| Expected for: | 2014-03-15 | |
| Bug number: | Bug 7317 | |
| Work in progress repository: | https://github.com/PTFS-Europe/koha/tree/ill_master | |
| Description: | = Introduction =
Following on from the original RFC and significant work carried out for PTFS Europe customers, we now have a working ILLModule deployed for a small number of customers. It currently uses the British Library's BLDSS API, and also provides a "generic email" plugin for "peer-to-peer" Interlibrary Loans. Next concerns will be consolidating and cleaning up of the module so we can move towards submitting it to the Koha community. The next section of this RFC will detail the proposed method by which we will submit the module, and also the status of this stage of the project. In general, the design of the module does/will reflect the earlier draft of the RFC. For this reason the prior version exists below the "Submission To Community" section. However, information detailed in that section is subject to change. Submission To CommunityWe will use a 2 stage submission process:
The idea behind this two stage submission is that it will on the one hand ensure proper separation of concerns of the module, and on the other will clarify the working of the module to the community. This approach should not get in the way of unit testing, but it may get in the way of user testing, as no "backend" will exist when the ILLModule by itself is submitted. I would imagine the ILLModule submission will contain a "stub backend" that illustrates the user journey, if not much else. The next section will discuss the ILLModule submission in depth. ILLModule SubmissionThe ILLModule submission will comprise of the following elements:
Config.pm notesCurrently ILLModule configuration takes two forms: we have some high level sysprefs, and then we have configuration options in koha-conf.xml and ill/spec.yaml. The former are safe to be flicked on and off, and to be changed on the fly. The latter should normally only be touched once, when the ILLModule is configured on installation. The latter should also be implemented as yaml files outside of koha-conf.xml, with koha-conf merely pointing to the appropriate subdirectory. Finally, whilst ILLModule configuration is currently loaded as a Config object, this config object is normally loaded by the Abstract.pm object. In the expanded model that we are now implementing this would lead to a problem: the config object will contain global and backend configuration, and the global options need to be consulted before the backend is created. So we must load the gobal and backend options before creating Abstract and/or Backend. The config object should perhaps become an autonomous object, which is consulted prior to loading the backend, in ILLRequests. When ILLRequests returns either a Backend or an ILLRequest it should pass the config object along. When creating a Backend however, it can do so by simply looking at the "global configuration" part of the Config object. Current StatusAt present, as described in the introduction, we have a working ILLModule/BLDSS interface. The code base is currently too monolithic, and whilst there is an overall outline of the above modular design, at present that is more of a gesture than a usable feature. The following tasks are ranked roughly by priority:
These items should be checked off the list as they are implemented. High level architecture
Koha ILL moduleThe Koha ILL module provides the skeleton and infrastructure for ILL functionality. It provides routes, an object model for the concept of ILLRequests and a framework for 'ILL plugins' and 'API backends'. The primary interface to the API backends is through an "Abstract" object interface. This interface provides a limited set of methods for core ILL operations, but defers their evaluation to the active & registered API backend. DSLThe methods provided by abstract all return a hashref defined in terms of a mini ILL specific DSL. The DSL covers three parts:
input/output templatingThis DSL includes a meta templating language to specify the rendering of:
Thus, the templating aspect of the DSL, provides an alternative interface to a subset of HTML as rendered by Template Toolkit. action type specificationThe DSL also includes a vocabulary for describing set response strategies. Thes strategies are:
continuation identificationcontinuation identification provides a mechanism to either provide a procedure to perform the next step immediately, or provides an identifier which can be used, with Abstract's "continuation-index" introspective procedure to retrieve the correct procedure at a later stage. Abstract methodsThe following abstract methods are defined and should be implemented by every API backend:
Reasoning & ExplicationThe above design should make it feasible to implement a great number of ILL workflows: whilst we have a very limited number of simple core methods, each API can use continuation specifications to embed an arbitrary number of additional steps in the evaluation of one of these core methods. Another benefit to having a reduced set of core methods is that it encourages iterative development for API backends: instead of having to worry about implementing, even as stubs, 20 or 30 methods covering all bases, you can implement 5 stubs, then slowly add continuations to each stub until you reach full functionality. API backendsAPI backends are the meat of Koha's interaction with external API sources. Each API backend essentially promises to implement Abstract's interface, using the DSL described above. Internal logic, and configuration is entirely separate from core Koha logic, and can thus be an entirely separate module. A downside to this approach is that there is very limited interaction between Koha itself and the backends. Any information the backend relies upon would probably need to be injected at construction time. API backends would probably be loaded using a configuration switch in koha-conf.xml. For now it is thought that only one backend be active at a time. ILL pluginsILL plugins are additional mechanisms that can be enabled or disabled in sysprefs. Criteria for it are that they have to be API backend agnostic, and relatively stateless (they could use Koha state, and use those structures to implement its state). By and large, these plugins provide additional functionality that falls outside of the scope of the core ILL module. An example of a plugin would be a generic ILL email system which uses ILL information stored using the core API backend, to pass specific ILL request information to partner libraries. Additional ResourcesTraining DocumentationFile:ILL v2TrainingNotesKoha current.odt Presentation and discussion at Koha Hackfest 15ILL Module
Required Functionality
Implementation
Demo of what we havePlugin framework
Workflows (contd.)tracking items once issued from ILL patron to actual patron:
NorwayUK (BL)NCIPGermany SLNP (OCLC)
Possible Approach
=> The interface provides all possible steps, individual API implementations return stubs, and form values. Steps (API)
in unmediated, skip these ----
end -----
more decentralised responses (i.e. not BL :-) ) ----
Is there a decentralised API that has additional steps:
Steps (on receipt, return, local to library).Sometthing to think about. provide Routing SlipsAdditional potential features
Previous Noteshttp://wiki.koha-community.org/wiki/Inter_Library_Loan Previous RFC | |
Chat Log (30 Oct 2015)
<atheia> So I spent about two week working on an abstract interface to ILL. <magnuse> cool <atheia> The idea is, as described in the RFC, etc., to have a unified
frontend / koha-module [13:18]
<atheia> And then a number of backends that implement the interaction to the
specific ILL system in use.
<atheia> So for instance, I saw that you had written some custom .pls for the
user interface.
<atheia> Instead the idea would be that even with such diverging APIs as NNCIP
and BLDSS, we could still use the same front-end scripts. [13:19]
<magnuse> yeah, mostly because i wanted to experiment in my own little sandbox
:-)
<atheia> Originally the idea was to achieve this using a well-defined
interface, and then some way to pass "form snippets" back from the
API backend to the template. [13:20]
<atheia> But this breaks down for several reasons — one of which is
internationalization — another being complexity.
<magnuse> yeah... <atheia> So what I'm settling on now is to have: a) a clean facade of ILL
methods, used from the Koha side [13:21]
<atheia> b) each method in the facade can take an additional "template"
parameter, by which the API backend can perform additional steps for
that part of the API logic [13:22]
<atheia> c) the front-end templates use additional includes (one set of
includes for each API backend), which also get passed the "template"
parameter, so those include snippets can generate forms as necessary
for that particular part of the API backend logic. [13:23]
<atheia> I don't know if this is now utterly confusing — I could try to
illustrate? alternatively if you have question?
<atheia> I'll point you to the relevant code repo though. sec. <magnuse> hehe, i can't say i have a clear mental image of it yet... [13:24] <magnuse> but it does sound flexible <atheia> Right :-) [13:25] <atheia> The branch I've been committing to for this is:
https://github.com/PTFS-Europe/koha/tree/ill_community
<atheia> But as an example: <magnuse> the way i am looking at it now, nncipp is mainly about changing
statuses and having those status changes fire off messages to the
other library. and maybe have the responses from those messages
change the statuses again [13:26]
<atheia> OK. <magnuse> ah yes, lots of stuff going on in that branch i see [13:27] <atheia> And when you say change statuses — are these the ILL statuses that
are now an authorized list?
<magnuse> yup <atheia> And how do you create a "new ill request"? <magnuse> but that might not be relevant to what you were about to say :-)
[13:28]
<atheia> Do you manually fill in a form or, search against the stock of
partner libraries?
<atheia> No, it kind of is, I think :-) <magnuse> for now, i fill out a form <magnuse> but i can totally see a z39.50/sru search being the starting point <atheia> Right; we currently have two approaches too: form and API websearch.
[13:29]
<magnuse> also, we have this thing where any library can log into the opac of
any other library and place a request there, which is then sent back
to the home ils
<atheia> oh <atheia> wow. so that's a totally different workflow. <atheia> Would that still use nncipp or would that be something different
altogether?
<atheia> Like, would that create an ill record in the requesting library's ILL
module? [13:30]
<magnuse> well, the way it will be implemented is: the opac sends a message to
the home ils, then the home ils sends a message to the owner ils
(which is the same as the message sent when you fill out a form)
<magnuse> sorry, phone <atheia> np <magnuse> gah [13:31] <magnuse> i need to pick up my wife <magnuse> might be gone about 20 minutes <atheia> :-) np — we can pick it up later again. Already food for thought. <atheia> Just ping me in the private chate. <atheia> s/chate/chat/ <magnuse> ok, great! <atheia> later! <magnuse> back! [14:11] <atheia> Ah sorry — didn't see your message. [14:34] <atheia> you still about? <magnuse> yup <magnuse> i should have shouted louder ;-) [14:35] <atheia> :-) <atheia> k <atheia> So — if I understand correctly, the workflow you are using is:
[14:36]
<atheia> - create request (currently form, could be z39 search, or OPAC
message)
<atheia> - update request status to trigger further messages [14:37] <magnuse> yup [14:38] <magnuse> oh and there are also incoming messages that result in a changed
status. i use the NCIPServer rangi has been working on for that
[14:39]
<atheia> And would these incoming messages, which cause a status change, then
in return cause further "business logic" to be triggered? [14:40]
<magnuse> so i order something and it gets status ORDERED. then the owner
library sends it, and send a message at the same time, so my request
is upgraded to SHIPPING. then i receive it and change the status to
RECEIVED, and send a message to the owner library that it has been
recieved
<magnuse> hm, i think they are mostly informative [14:41] <atheia> Aha. <magnuse> there is renewal [14:42] <magnuse> i can send off a request for renewal and the answer can be yes or no <atheia> Right — but that is more of an "action" rather than a status update
right?
<magnuse> yeah, probably <atheia> BLDSS API also has a "status" action whereby we can request
information on what's going on with a request (e.g. see: "order
received" or "dispatched"). [14:43]
<atheia> So the feeling I'm getting here is that you could do something like
"status update hooks":
<atheia> whenever the "status update" action is performed, we also make a
method call to the API backend passing the STATUSCODE [14:44]
<atheia> The API can then perform actions (like send a message to a remote
server), and returns OK
<atheia> After which ILL updates the status. <atheia> That would handle all your status update logic no? <magnuse> yup, sounds like it [14:45] <atheia> OK — that is doable… I reckon we could use the same model as what I'm
doing now (he said optimistically) [14:46]
<atheia> Whereby the backend can return a "partial-success" with a specific
TEMPLATE, which in turn is passed back to the TT template, so it can
include an INCLUDE which can display a specific form for that
TEMPLATE.
<atheia> But — would the status updates require any further manual
intervention, or is it basically automatic after that? [14:47]
<atheia> i.e.: when you update to RECEIVED, do you need to pass additional
data to the backend for their call to a remote server (e.g. the
recipients comments).
<magnuse> as far as i can tell it's automatic [14:48] <atheia> OK <atheia> Then ignore my elaborations — let's KISS <atheia> I… mean… you know. <atheia> keep it simple. <atheia> *blush* <magnuse> no, i see changing the status to received as just reading the
barcode off the book, and then koha should shoot a "received"
message to the owner library
<magnuse> lol! [14:49] <magnuse> yeah, let's keep it simple :-) <atheia> :-) good <atheia> Ah. <atheia> But we need to pass to read the barcode and pass that to the backend
right?
<atheia> I imagine that would not necessarily be the same as the "request id"
— i.e. would the information already be contained in the request data
stored in the db? [14:50]
<atheia> So: workflow detail. <atheia> please. :-) <magnuse> the way i do it now, i read the barcode and use that to look up the
request data in the db [14:51]
<magnuse> over time there might be multiple requests for the same barcode, so
i use barcode and status to do the actual lookup [14:52]
<atheia> is th ebarcode the item barcode or borrower barcode? <magnuse> when i want to receive (and send off RECEIVED messages) there should
only be one record with the barcode and the status SHIPPED
<magnuse> item barcode [14:53] <atheia> of course. <atheia> ok. <atheia> So you barcode search an item, get to the ILL in the ILLrequest
module
<atheia> Then set status to the RECEIVED status <atheia> right? [14:54] <magnuse> yup <atheia> ok, in which case, I think simple works. <magnuse> or what i actually imagine is a form for reading barcodes and
setting status RECEIVED/sending RECEIVED messages [14:55]
<magnuse> so you'd open all the parcels you got that day <magnuse> pile up all the ill books and scan them <magnuse> into a text field [14:56] <atheia> ok, yeah, I think that works. <magnuse> and one thing i have not thought about so far: that should trigger a
message to the borrower that his/her book has arrived [14:57]
<magnuse> and probably also a slip to put in the book before it is placed on a
shelf to be picked up
<atheia> right — for now, I would suggest we kick that in the long grass. <atheia> Just to keep the whole thing manageable. <magnuse> yup [14:58] <magnuse> that is what i have been doing :-) <atheia> I could see us adding these features either to the status change bit
or in the API backend…
<atheia> dunno. <atheia> good call :-) <magnuse> (might be able to piggyback on existing holds functionality) <atheia> Right — I think that's what Janet over here has been doing: [14:59] <atheia> She added a UI button that allows users to create a Koha item from an
ILLRequest, which is then added to the usual holds for the borrower.
<atheia> *I think* <atheia> So do you feel like you want to add more context? [15:00] <magnuse> yeah, sounds reasonable <atheia> Or can I try to frame this in terms of the work I've been doing? <magnuse> hm, not that i can think of <atheia> So we can see how feasible it is for you? <magnuse> sure <atheia> OK. <atheia> So I would propose 2 points of contact between the Koha ILL module
and a specific backend: [15:01]
<atheia> 1) actions (these include things like "creating a new ILLRequest", or
"Renewing" or "Fetch Status";
<atheia> 2) status update hooks (these would be things like, set status to
received, send message to remote host; or set status to COMPLETE, add
data to the COMPLETION_DATE field) [15:02]
<atheia> For 1): the UI would expose specific buttons for each available
action in the "manage_ill" interface. [15:03]
<magnuse> sounds good to me <atheia> When the user clicks on the appropriate button, the relevant method
is called in the API backend, which responds either by performing
action automatically (e.g. firing off a renewal request and reporting
success) or by responding with the name of a TEMPLATE.
<atheia> In the latter case, the ILL UI then includes that Backend's Method
INCLUDE, passing the TEMPLATE name, and that include can present a
form to the end user where they can provide additional data. [15:04]
<atheia> For instance, for BL API: when placing the order, the API backend
passes back for the 'confirm' TEMPLATE, which causes the UI to
generate a form with a selection of delivery services/prices. [15:05]
<atheia> The user selects the price and clicks submit. <atheia> At this point the UI, knowing that it invoked a TEMPLATE, calls the
same method in the backend again, with the new data from the user,
and the backend then performs the operation [15:06]
<atheia> In our example — the backend then uses the price information to
finally place the order with BL.
<atheia> So we have something like 4 different actions (create order, confirm
order, get status, renew loan); but each Backend is able to perform
as many intermediary steps as required to complete each of those
actions. [15:07]
<atheia> Conversely, in your case, your backend might simply not implement
"confirm", or "status"
<atheia> as they don't make sense in your context. [15:08] <magnuse> yeah <atheia> Does this all make sense so far? <magnuse> i think so, yes :-) <atheia> OK. <atheia> So for 2) <atheia> The UI offers, in the ill-manage part, a chance for an end-user to
select a new status for an ILL. [15:09]
<atheia> Upon submission, the UI invokes, always the same method:
UPDATE_STATUS (or something), and passes the new status code.
<atheia> The backend upon which that method is invoked must handle each
possible status code passed to it. [15:10]
<atheia> (but it can choose to not perform any action for a specific code. <atheia> ) <magnuse> (and there should be some limit on "status paths", so from status A
you can only move to status B or C, not D)
<atheia> At this stage, the Backend can perform a specific action (like
sending a message to a remote server).
<atheia> OK. Interesting! [15:11] <atheia> Do you feel like that limitation would have to be present on the UI
(e.g. only show those statuses that are allowed in this context)
<magnuse> well, it would be nice <atheia> or would it be ok to always show all statuses, but let the backend
say "tihs makes no sense"
<atheia> Indeed :-) I also think it would be more complex for now. [15:12] <magnuse> yeah <atheia> We'd need to track status dependencies somehow. <atheia> If possible, I'd prefer the second (where the backend returns "this
makes no sense")
<atheia> WDYT? <magnuse> as a first pass i think it would be ok to rely on librarians to know
which changes makes sense
<atheia> awesome. <atheia> :-) <atheia> So in order to implement this, I think the "UPDATE_STATUS method
should take two arguments: new_status_code, old_status_code. [15:13]
<atheia> Then the backend can evaluate whether this is a valid transition, and
perform it's actions if it is.
<atheia> WDYT? <magnuse> works for me :-) <atheia> cool. <atheia> So the backend performs its action and returns "success" or "fail".
[15:14]
<atheia> If "success" the Koha ILL Module finally updates the status in the
ILL tables to the new status.
<atheia> If "fail" the Koha ILL Module displays an error message and does not
update the status in the ILL tables.
<atheia> And I think… that's all :-) <magnuse> hehe, yeah [15:15] <atheia> Now most importantly, do you think that this kind of infrastructure
would work for you as a backend implementor?
<atheia> (you know, with help and documentation :-) ) <magnuse> what i have done in a couple places is change the status to
RENEWAL_REQUESTED before the renewal request is ssent, and then
RENEWAL_OK after i get the confirmation. that way i can pick up
failed requests [15:16]
<magnuse> yeah, it sounds kind of logical :-) <atheia> OK, so that is a slightly more manual approach to the renewal
process.
<atheia> is renewal a realtime request for you in NNCIPP or is it
asynchroneous? [15:17]
<magnuse> with clear or it's just me being lazy and not taking all possible
outcomes into account...
<atheia> i.e. does the remote service provide an immediate response or do they
manually process requests?
<atheia> Sorry — I don't think I understand your last sentence? [15:18] <magnuse> s/with clear// <atheia> ah <atheia> ;-) <magnuse> immediate response <atheia> k, so you could implement that as a real-time action, as discussed
above and save yourself the hassle of the 2 additional ILL statuses.
<magnuse> yeah <atheia> But to clarify, as an implementor, you would basically implement:
[15:19]
<atheia> a) a Backend, which implements the 4 actions as methods as mentioned
above, and
<atheia> b) a method for the backend called UPDATE_STATUS <magnuse> yeah, that makes sense [15:20] <atheia> c) implement includes for the ILL templates for your specific
Backend, which would be either stubs (in case your backend has no
multi stage ACTIONS), or which implement additional forms.
<atheia> And that's all there should be to it for an implmentor. <atheia> cool <atheia> Unfortunately I am not quite there yet — but I am implementing this
system now on ill_community. [15:21]
<magnuse> shpuld be doable in an afternoon ;-) <atheia> optimistic ;-) <magnuse> hehe <magnuse> do you have an idea about a timeframe for that [15:22] <magnuse> ? <magnuse> i have my work based on the ill_master branch, i think. i need to
polish that to where it cooperates with the other norwegian
implementations (so i can get some of my money from the project :-)
[15:23]
<atheia> Yeah, I reckon I need another 2 weeks to get the infrastructure ready
and to implement the ACTIONS and UPDATE_STATUS, and template includes
for BLDSS API.
<atheia> OK. <magnuse> the next stage of the project is to get things into production <atheia> What I can do is imlement the interface in the abstract (i.e. without
the BLDSS backend).
<atheia> and then let you know, and you could implement NNCIPP at the same
time that I implement the rest of BLDSSAPI. [15:24]
<magnuse> that sounds like a perfect fit, if i can take the ncip/backend code
i have now and start to bake it into your framework
<atheia> and that way we can chat and improve things in the abstract
interfaces whilst we go along?
<atheia> OK. <magnuse> sounds like a really awesome plan <atheia> I will ask Martin if I can get another day next week to do that
specific thing, and then can let you know when you can start using
ill_community. [15:25]
<atheia> sec. <magnuse> great <atheia> OK. [15:26] <atheia> That's good. <atheia> I'll make sure to be in IRC everyday I'm actually working (I'll be
off on Wednesday/Thursday). [15:27]
<magnuse> ok <atheia> And plan to ping you with the green light on Tuesday at the latest. <magnuse> awesome <atheia> So then we can continue chatting as we go along. <atheia> yeah deffo! <magnuse> don't rush it for my sake, though <atheia> It's really helpful to have you work on this whilst we work on it
too. [15:28]
<magnuse> i need to fix an encoding thing in the code i have <magnuse> then get a "signoff" from another vendor, that our systems can talk
together
<magnuse> then i'll probably have some other stuff to attend to <atheia> OK, well, I'll get this part done anyway, and then we can revisit it
whenever you have another moment to look at this.
<magnuse> so it will be a week or two before i can start looking at the
"remake"
<atheia> :-) <magnuse> great <atheia> np <atheia> Hey, another question, [15:29] <atheia> Would you mind if I update the RFC/ Bug with a log of this chat? <atheia> So we can kind of share progress etc. without me having to distill
everything immediately?
<atheia> (np if you'd rather not) <magnuse> please do [15:30] <magnuse> i don't mind [15:31] <atheia> awesome :-)