Project: Implement Spring Data to LibreHealth Toolkit

toolkit
gsoc2018-project
gsoc2018

(Yash D. Saraf) #54

@sunbiz The solution I mentioned above is giving me null pointer errors, perhaps your way will work. Could you please point out your solution? Here’s the repo link again if you missed it in my previous post.


(Yash D. Saraf) #55

@sunbiz For filtering results based on parameters passed in queries, I have looked at
Reactive Cassandra Template and QueryDsl.
Of these,

  • QueryDsl is not supported in reactive apps
  • Reactive Cassandra Template will render the app database dependent (which I believe is against the database independency goal).

LibreHealth toolkit originally uses Hibernate criteria for filteration/search purposes, which again cannot be used in reactive apps. Any thoughts on this?


(Prashadi) #56

@yashdsaraf @sunbiz initally we discuss storing the entire content of the FHIR resource. Since FHIR resources contains attributes and hierarchical attributes, we might need to check how we can represent in cassandra bases on key value pairs.


(Saptarshi Purkayastha) #57

I suggest storing each resource JSON in it’s own datastore. And references should be reference only, with the referenced resource in it’s own datastore. I don’t think we should call these tables because it’s no longer in a relational database. @prashadi does that answer your question??

@yashdsaraf we don’t want to use criteria or filtering. It should be MapReduce or CQL. This is a useful example that I’ve based a project on recently - https://github.com/spring-projects/spring-data-cassandra/blob/master/src/main/asciidoc/reference/reactive-cassandra.adoc


(Yash D. Saraf) #58

@sunbiz ReactiveCassandraTemplate is the preferred way to go for using CQL in spring data cassandra. My question is wouldn’t using CQL render the project to be Cassandra dependent?
Since one of the goals is to “Document examples that show how spring data can generate a non relational and relational database”.


(Saptarshi Purkayastha) #59

That’s fine dependence for the GSoC project


(Prashadi) #60

@sunbiz thank you for the clarification.


(Yash D. Saraf) #61

@sunbiz I’ve made a very primitive filtering structure, here’s a blog post about it.
Although when trying to implement pattern based matching or partial search, I get indexing errors i.e LIKE cannot be used on non-indexed columns. I’m looking deeper into indexing in cassandra. Any ideas?

Update: Here’s the repo link.


(Yash D. Saraf) #62

The indexing errors were solved by creating a custom SASI Index for certain columns. I made an annotation to do this, here’s how a sample Patient looks like now.
@sunbiz Could you please elaborate on your solution for annotating the HAPI FHIR structures issue,


(Yash D. Saraf) #63

@sunbiz I have finally figured out the null pointer errors, Cassandra was unable to map java.lang.Object of the UserData property of HAPI FHIR structures. I solved it by annotating the getter of userdata with @CassandraType(type = DataType.Name.MAP, typeArguments = {DataType.Name.VARCHAR, DataType.Name.VARCHAR}) in spring 2.1.0 SNAPSHOT (I’ll try this in the RELEASE version today).
This solved the mapping errors as well since marking the getters with @Id annotations and subclasses with @Table got the models working with Cassandra.
I’ll push this today after configuring all the converters and indexed fields.


(Yash D. Saraf) #64

@sunbiz I’ve pushed the changes after configuring all the converters for the Patient model and disabled saving unwanted properties in the database as well. This is how a Patient model looks like now,

CREATE TABLE cassandra.patient (
    patient_id text PRIMARY KEY,
    active boolean,
    address text,
    animal text,
    birthdate timestamp,
    communication text,
    contact text,
    contained text,
    deceased text,
    empty boolean,
    extension text,
    formatcommentspost text,
    formatcommentspre text,
    gender text,
    generalpractitioner text,
    id text,
    identifier text,
    implicitrules text,
    language text,
    link text,
    managingorganization text,
    managingorganizationtarget text,
    maritalstatus text,
    meta text,
    modifierextension text,
    multiplebirth text,
    name text,
    photo text,
    primitive boolean,
    resource boolean,
    telecom text,
    text text,
    userdata map<text, text>
) 


(Saptarshi Purkayastha) #65

So you are extending the HAPI DSTU3 structures, but since you are extending and annotation, it is not serving the purpose. The purpose of using the HAPI structures was that they usually keep in sync with the standard, and if you have to rewrite each property of their class, it basically doesn’t make much sense.

I havent been able to figure out a way to annotate without extending either. I like your approach and it is fine for the moment, but we still should continue to find other way, instead of replicating all their model code.


(Yash D. Saraf) #66

@sunbiz I’ve added a data loader similar to the pre-requisite task project which retrieves sample resources from a FHIR server and the data is saved in the database just fine (except for StackOverFlow errors with Jackson which were solved using Gson instead), but retrieving data back from the database gets stuck for some reason without throwing any errors whatsoever.
I’ve succesfully tried logging the queries being executed so the error is not on the cassandra part. It seems the error must be somewhere in data conversion, I’ll keep you posted on this.

PS: Data retrieval worked fine when I was using a sample Patient model.


(Saptarshi Purkayastha) #67

@yashdsaraf by when do you think you will be able to support the CRUD on the full Patient resource at least? I want to start thinking about the way to distribute this such that it can be an EHR platform. i.e. support Authentication, module assembly etc?


(Yash D. Saraf) #68

@sunbiz As I said in my previous post I’m able to save the Patient which means implementing the Update and Delete functionalities should be quite straight forward as well. Once the READ operation is entirely sorted out, others shouldn’t take much longer. I’m working on fixing the no output issue I mentioned earlier, once that’s done I’ll be able to give you a much better answer as to when the CRUD support will be complete.

That sounds exciting! :grinning:


(Yash D. Saraf) #69

@sunbiz I’ve tried debugging the app to find where the app gets hung up, all I know this far is that CQL queries are executed successfully and the ResourceConverter is the last one to get executed before the app gets stuck. I also tried to make a custom model using just a single data type from HAPI FHIR structures e.g IdType where the app didn’t get stuck but Jackson threw some mapping errors which I’m afraid will come later in our app as well. I’m currently trying to get a non reactive version of the app working, I’ll get back to you with the progress in some time.


(Yash D. Saraf) #70

@sunbiz The non reactive version didn’t work either. It seems I’m back to square one regarding this error, it would be great if you could give the code a try https://gitlab.com/yashdsaraf/reactive-lh-toolkit.

Here’s what I know so far,

  • Saving works but the app freezes on select operation without any error or output
  • Select queries are successfully executed (you can enable logging from application.properties) so the error is not in the CQL queries
  • I used the debugger to track all the converters and they were successfully executed as well
  • Execution doesn’t reach the stage where the CustomPatient object is converted to JSON to be returned to the browser. (I know this because Jackson errors out on UriType fields and there are no such errors in this case.)

Any ideas on how to solve or debug this?


(Yash D. Saraf) #71

@sunbiz @namratanehete I’m working on the CREATE, UPDATE and DELETE operations for Patient.
I’m using https://www.hl7.org/fhir/http.html for reference.
It says that

servers MAY choose to allow clients to PUT a resource to a location that does not yet exist on the server - effectively, allowing the client to define the id of the resource.

Do we want to allow creating a resource with unknown id using PUT?


(Saptarshi Purkayastha) #72

Yes, I think we should allow it.


(Yash D. Saraf) #73

@sunbiz I’ve implemented CRUD on sample patient model, referring to https://www.hl7.org/fhir/http.html. Here’s the gist of the code.
I also figured out a way out of Jackson mapping errors, since we can’t directly annotate the HAPI models, we can use Jackson Mixins to overcome stackoverflow errors and we can choose which properties are displayed and which ones aren’t.
For the READ error, I created a sample spring project with spring boot snapshot version and it is throwing some mapping errors (unlike our original app which just keeps on loading without any output or error whatsoever). I’d like to go down this rabbit hole for now and see where it leads. I’ll report back the progress in some time.