Project: Improving FHIR Analytics Using Apache Spark and Cassandra

prashadi · March 12, 2019, 5:04am

@mshubhankar can you redeploy the entire application by cleaning the webapps directory? Do you see any backend error logs?

mshubhankar · March 12, 2019, 3:42pm

@prashadi I found the problem. I have both Java 8 and Java 11 on my laptop and the war file was built on Java 8 whereas tomcat was running on Java 11. Anyway, things are good and up running now. I tested the functionality after the PR and it’s working just fine

prashadi · March 17, 2019, 10:00am

Great I’ll merge the pull request after I tested locally as well. Good work.

mshubhankar · March 25, 2019, 6:20am

Hi @prashadi @sunbiz @namratanehete @r0bby,

I’m starting to write my proposal and wanted to get my objectives sorted out. I have some questions related to some objectives in mind and would be glad if someone can help me out with these :

Upgrade FHIR analytics to work with latest cerner bunsen - The latest cerner bunsen depends on Spark 2.3 and Hapi 0.4.6 . I have made these changes and requested a PR with the updates. Another objective of the project is also to get the latest functionality of bunsen but I’m not sure which functionalities we can add to the project. Do we want some added functionalities from here ?
Complete the TODO Implementations in utils/LibrehealthAnalyticsUtil.java file
Integrate with Apache Drill - Apache drill has a good documentation to be built with JDBC drivers and also in the Spark environment but there seems to be some problem with it being integrated with cassandra. The documentation mentions that a custom plugin can be made for cassandra but I don’t find any prior work to refer to. I have also found a post which is old and deprecated but can be used as reference. It would be great to have some information about this.
Enhance query interface - Are there any particular interface enhancements ?

Have I missed out on something except of writing a research paper which I hope can be done after being done with the objectives ?

mshubhankar · March 28, 2019, 12:13am

Hey @prashadi @sunbiz @namratanehete @r0bby,

I’ve written the first draft of my proposal here. Please have a look and revert me back with your feedback and valuable comments to make it better.

r0bby · March 28, 2019, 1:45am

I would like to remind you and everyone reading this that without doing the starter tasks, we will not consider your application at all, no matter how strong it is;

mshubhankar · March 31, 2019, 8:24pm

I am sorry I missed the Intro task. I have now completed the Intro task 1 and my code can be found here. I hope you can now review my proposal.

r0bby · March 31, 2019, 11:14pm

Link to it in your proposal.

mshubhankar · March 31, 2019, 11:29pm

I have added it under the “Contributions and work done” section of my proposal

prashadi · April 1, 2019, 2:40am

@mshubhankar Sorry for the delay. I’ll have a look at the proposal and get back to you. Was away just over a week.

judywawira · April 2, 2019, 3:56am

@mshubhankar great start on your proposal

I don’t see you recognizing the work @prashadi worked on last year in GSOC , and how your work will improve that

This year the outcome of this project is to make it usable - esp to non technical users , so expect a UI improvement too

mshubhankar · April 2, 2019, 4:16pm

Thank you for your feedback @judywawira. I will add another section called “Previous work done” to my proposal citing the work by @prashadi.

Ans yes, enhancing the UI is one of my goals for the project.

r0bby · April 2, 2019, 5:13pm

@mshubhankar – this will require wireframes in your proposal to show the UI changes you are proposing.

prashadi · April 4, 2019, 11:49am

@mshubhankar Proposal is looking good. Please include the suggestions made by @judywawira and @r0bby. We also need to evaluate the possibilities of integrating automatic query builder for the UI which will omit the requirement of writing the plain Spark SQL queries.

mshubhankar · April 7, 2019, 5:10am

Hello everyone,

It makes me very sad to say that I won’t be able to send a proposal for GSoC this year. One of my supervisors didn’t allow me to take up a GSoC project because of the time constraints. I was really excited by the idea of the project and hoped that I would’ve also learnt a lot from it. I’m sorry I won’t be able to continue as a GSoC participant but I wish to keep contributing to the community in future. I would also like to thank you for all your responses throughout !

prashadi · April 8, 2019, 5:29am

@mshubhankar Thank you for letting us know about the situation. It’s sad that you couldn’t do GSoC this year. I wish you all the very best with your work.

eunice18 · April 9, 2019, 5:25am

Hello All,

I was trying the project past few days. But I was busy with my midterm exams. I’ll try to submit a proposal before the deadline. Since I have to do a contribution, I will try my best to do a contribution.

I’m getting following error many times. Any idea on this?

Caused by: org.springframework.data.cassandra.CassandraInternalException: Query; CQL [CREATE CUSTOM INDEX ON patient (animal) USING ‘org.apache.cassandra.index.sasi.SASIIndex’ WITH OPTIONS = {‘mode’: ‘CONTAINS’, ‘analyzed’: ‘true’, ‘analyzer_class’: ‘org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer’, ‘case_sensitive’: ‘false’};]; An unexpected error occurred server side on localhost/127.0.0.1:9042: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.cassandra.index.sasi.SASIIndex; nested exception is com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on localhost/127.0.0.1:9042: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.cassandra.index.sasi.SASIIndex at org.springframework.data.cassandra.core.cql.CassandraExceptionTranslator.translate(CassandraExceptionTranslator.java:93) at org.springframework.data.cassandra.core.cql.CassandraAccessor.translate(CassandraAccessor.java:334) at org.springframework.data.cassandra.core.cql.CqlTemplate.translateException(CqlTemplate.java:725) at org.springframework.data.cassandra.core.cql.CqlTemplate.query(CqlTemplate.java:175) at org.springframework.data.cassandra.core.cql.CqlTemplate.queryForResultSet(CqlTemplate.java:252) at org.springframework.data.cassandra.core.cql.CqlTemplate.execute(CqlTemplate.java:150) at org.springframework.data.cassandra.core.CassandraPersistentEntitySchemaCreator.lambda$createIndexes$2(CassandraPersistentEntitySchemaCreator.java:110) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.springframework.data.cassandra.core.CassandraPersistentEntitySchemaCreator.createIndexes(CassandraPersistentEntitySchemaCreator.java:110) at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.performSchemaActions(CassandraSessionFactoryBean.java:175) at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.createTables(CassandraSessionFactoryBean.java:156) at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.performSchemaAction(CassandraSessionFactoryBean.java:91) at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.afterPropertiesSet(CassandraSessionFactoryBean.java:63) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1765) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1702) … 73 common frames omitted Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on localhost/127.0.0.1:9042: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.cassandra.index.sasi.SASIIndex at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:63) at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:25) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68) at org.springframework.data.cassandra.core.cql.CqlTemplate.query(CqlTemplate.java:171) … 92 common frames omitted

Thank you.

prashadi · April 9, 2019, 5:35am

@eunice18 without completing starter tasks before the deadline will lead to ignore your application considering for the GSoC. @r0bby I believe we have no other options.

judywawira · April 13, 2019, 4:03pm

that’s a bummer @mshubhankar

sunbiz · April 25, 2019, 1:29am

It is disappointing, but I am hoping to see you around